Implementation of a Comprehensive Testing Protocol for a Rapidly Manufactured Mechanical Ventilator

Abstract

Introduction

The COVID-19 pandemic highlighted that surges in critical care demand can overwhelm existing healthcare infrastructures, most evident in the acute shortage of mechanical ventilators across the globe. Despite efforts to encourage urgent authorization of newly developed emergency ventilators, the currently available testing protocols are not internationally accepted, standardized and none address testing in clinical settings. The aim of this study was to compile and perform a comprehensive clinical testing protocol for a newly developed emergency ventilator.

Methods

Using previously available guidance, we compiled a sequential testing protocol with a: 1. preclinical, 2. safety testing, 3. clinician usability test and 4. clinical stage involving respiratory failure patients. The protocol was then tested on the Luca ventilator, a mechanical ventilator capable of sophisticated ventilator settings rapidly developed specifically in response to the COVID-19 pandemic.

Results

During the pre-clinical/safety stages, the ventilator produced pressure and volume changes deemed acceptable by the Rapidly Manufactured Ventilator System guideline. Furthermore, our protocol allowed the identification of a number of issues that were easily resolved with minor software adjustments. Usability was excellent (overall System Usability Scale score = 90.5). Clinical testing revealed that a sampling frame of 15 critically ill patients was sufficiently powered to detect any significant, clinically relevant differences between the Luca ventilator and a standard ICU ventilator.

Conclusions

The ventilator was accurate, reliable, safe, and user-friendly. The implementation of a comprehensive, standardized pre-clinical/clinical testing protocol is feasible, potentially enabling the safe and timely emergency authorization of rapidly developed mechanical ventilators crucial in pandemic situations.

Resumen

Introducción

La pandemia de COVID-19 ha puesto de manifiesto que las sobrecargas en los cuidados intensivos pueden desbordar las infraestructuras sanitarias existentes, que fue evidente en la escasez inmediata de respiradores mecánicos observada en todo el mundo. A pesar de las tentativas de alentar la autorización urgente de respiradores de emergencia recién desarrollados, los protocolos de ensayos existentes no están aceptados en todo el mundo, no están normalizados y ninguno trata los ensayos en entornos clínicos. El objetivo de este estudio fue recopilar y ejecutar un protocolo de ensayos clínicos exhaustivo para un respirador de emergencia recién desarrollado.

Métodos

Aplicando las directrices previas recopilamos un protocolo secuencial de ensayos con: 1. estudio preclínico, 2. estudio de seguridad, 3. ensayo de operabilidad por los facultativos y 4. fase clínica con participación de pacientes con insuficiencia respiratoria. Se analizó el protocolo con el respirador Luca, un respirador mecánico con configuraciones sofisticadas desarrollado rápida y específicamente para responder a la pandemia de la COVID-19.

Resultados

Durante las fases preclínicas/de seguridad, los cambios de presión y volumen que produjo el respirador se consideraron aceptables, según la directriz Rapidly Manufactured Ventilator System. Asimismo, nuestro protocolo permitió identificar varios problemas que se resolvieron fácilmente con mínimos ajustes de software. La operabilidad resultó excelente (puntuación total de la escala de operabilidad del sistema = 90,5). Los ensayos clínicos revelaron que una muestra de 15 pacientes graves presentaba suficiente potencia estadística para detectar todas las diferencias significativas de interés clínico entre el respirador Luca y uno de Unidad de Cuidados Intensivos ordinario.

Conclusiones

El respirador resultó exacto, fiable, seguro y fácil de usar. La implantación de un protocolo de ensayos preclínicos y clínicos exhaustivo y normalizado es factible y puede habilitar la autorización de emergencia oportuna de respiradores mecánicos de desarrollo rápido, cruciales en situaciones pandémicas.

Introduction

The COVID-19 pandemic precipitated many challenges in the health care system relating to device allocation and supply-demand discrepancies, which was most evident in the acute shortage of mechanical ventilators across the globe.^1,2 A number of technical implementations and developments have been proposed in order to serve as alternatives to traditional commercially available mechanical ventilators,^3–5 however, the majority of these solutions do not permit sophisticated calibration and versatile treatment settings required when treating respiratory failure associated with COVID-19.^6–8

In order to facilitate the development of technically advanced, locally designed and manufactured ventilators that might contribute to covering the increased demand foreseen during subsequent waves of the pandemic,⁹ several countries offered urgent authorization pathways for emergency use of such devices.¹⁰ While there have been examples of previously unused or modified devices gaining emergency approval and national guidelines on requirements and testing for newly developed, rapidly manufactured ventilators have also been published,^9,11–13 so far there has been no report on newly developed devices receiving emergency authorization during the COVID-19 pandemic. However, given that subsequent pandemics and natural disasters might result in a similar overwhelm of the healthcare system,¹⁴ there is a rationale for an internationally accepted, comprehensive clinical testing protocol that may be used in such situations to ensure quick, thorough and local testing of newly developed emergency ventilators.

The most comprehensive testing guidance on the subject published to date is the Specification for Rapidly Manufactured Ventilator System (RMVS) guidance by the United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA),¹² but the guidance does not involve a clinical testing stage and there is little data on its usefulness, potential pitfalls, or limitations. The aim of this study was to compile and conduct a comprehensive, sequential pre-clinical and clinical testing protocol for a rapidly developed mechanical ventilator based on currently available guidance complemented by a clinical stage not currently described in any guideline. The testing process focused on accuracy, reliability, safety, and usability in both in vitro and in vivo settings and identified potential pitfalls in order to supply information for subsequent internationally accepted, comprehensive guidelines on emergency authorization of rapidly developed mechanical ventilators during a pandemic situation.

Materials and methods

The Luca ventilator

The Luca ventilator is a rapidly developed COVID-19 specific mechanical ventilator, designed and manufactured in a closed source process by Femtonics Kft. (Budapest, Hungary) in collaboration with the Department of Anaesthesiology and Intensive Therapy at Semmelweis University (Budapest, Hungary).

The electronically controlled pneumatic ventilator is equipped with an electronic display and manual interface (Fig. 1) and is able to operate with commonly used disposable, double limb ventilator circuits. Its software permits controlled, assist/controlled, and assisted pressure-controlled (e.g. pressure support) modes allowing spontaneously triggered breaths, a crucial factor in achieving patient-ventilator synchrony.¹⁵ During the design of the device, the main goal of development was to incorporate materials and parts that were readily available locally in order to ensure a continuous supply chain for possible increased production need. Main performance features corresponding to currently accepted requirements are presented in Supplementary Table 1. As can be seen, the design and features of the LUCA mechanical ventilator permits synchronization of mandatory breaths with the patient's respiratory effort as well as advanced patient tailored adjustments of ventilator parameters.

Fig. 1.

Luca mechanical ventilator with close-up of the user interface (online version: colour; print version: black and white).

Three prototypes of the Luca ventilator were used for the testing process with Intersurgical smoothbore breathing system 1.6 m (Intersurgical, Wakingham, Berkshire, UK) circuits, Virobac II viral/bacterial filters (King Systems Noblesville, Indiana, USA) on both the expiratory and inspiratory limbs additionally to a Humid-Vent Compact S (Teleflex Medical, Athlone, Ireland) heat and moisture exchange filter (HMEF) at the distal end.

Testing protocol

The testing process was designed with sequential stages, including an in vitro preclinical part, followed by a safety testing stage, a clinician usability test and concluding with in vivo, clinical testing on respiratory failure patients. All subsequent steps were dependent on the previous successful completion of the previous testing stage. Testing was performed in May 2020. All preclinical tests were performed during a 2-day period, except for endurance testing. Clinical tests were performed over a course of one week.

I. Stage: preclinical testing

Accuracy and reliability were tested according to the RMVS performance testing specifications¹² with additional endurance testing. Actual values of pressure and volume supplied by the Luca ventilators were measured with an ASL 5000 breathing simulator (IngMar Medical, Pittsburgh, PA) and compared to set/displayed parameters. The following settings were tested in a total of 72 combinations: compliance (10, 20 and 50 mL/cmH₂O), resistance (10, 20 and 50 cmH₂O), inspiratory pressure (15 and 30 cmH₂O), respiratory frequency (12 and 20/min), inspiratory and expiratory ratio (I:E) (1:2), FiO₂ (55 and 95%), and PEEP (5, 10 and 15 cmH₂O). As per the RMVS, accuracy was deemed adequate if the measured airway pressure was within ± (2 + (4% of set parameter)) cmH₂O; interface readings of expired volumes greater than 50 mL were within ± (4.0 + (15% of the actual measured volume expired through the patient-connection port)) mL; and measured oxygen concentrations were ±5% of the set value. Inspiratory time (time from the onset of the flow towards the ASL until flow reversal was detected), expiratory time (time from the onset of the flow away from the ASL until flow reversal was detected), respiratory rate and successful triggering rate were measured by the ASL and compared to displayed values and deemed adequate within a 5% deviation. FiO₂ readings were verified with a custom oxygen sensor.

Endurance was tested by continuous operation of a Luca prototype for 30 days in a hospital environment connected to an EasyLung (IMTMedical, Malaysia) test lung with a compliance of 25 mL/mbar and resistance of 20 mbar/L/s under continuous supervision.

II. Stage: safety testing

The safety testing stage involved a list of simulated alarm scenarios (high/low pressure, high/low respiratory volume, low/high minute volume, high respiratory rate, FiO₂ inaccuracy, gas supply inadequacy, circuit disconnect), based on previously published consensus on requirements and alarm profiles of commonly used ICU ventilators.^16,17 Emergency situations were simulated using the ASL 5000 breathing simulator in order to test the following alarm scenarios: high/low pressure, high/low respiratory volume, low/high minute volume, high respiratory rate, FiO₂ inaccuracy, gas supply inadequacy, circuit disconnect. Activation and timing of alarms were recorded. Closed Suctioning Test was performed according to RMVS guidance.¹²

III. Stage: usability testing

Usability was tested with the participation of 10 critical care doctors who each performed a total of 13 tasks (see Supplementary Table 2) on a ventilation model consisting of a Luca ventilator prototype connected to the ASL after a 5-min introduction to the ventilator. Ventilation model settings were as follows: 5 cmH₂O resistance, 70 mL/cH₂O compliance within an initially passive system [0/min respiratory rate], then one with a spontaneous breathing with 18/min respiratory rate and 8 cmH₂O breathing effort. Baseline ventilator settings for the LUCA ventilator prototype were 5 cmH₂O PEEP, 15 cmH₂O inspiratory pressure, 15/min respiratory rate, 1.0 s inspiratory time with the pressure trigger inactivated.

System Usability Scale (SUS), an evaluation comprising of 10 questions, was used to assess subjective usability.¹⁸ The SUS score has a range of 0–100, where higher scores indicate better usability,¹⁸ and scores correlate well with a standard adjective scale (worst imaginable, awful, poor, OK, good, excellent, best imaginable).¹⁹ SUS score for the LUCA mechanical ventilator was calculated based on the evaluation of the participating critical care specialists after the usability testing scenario.

IV. Stage: clinical testing

The clinical testing was a comparative, observational study, during which respiratory failure patients were ventilated by standard commercially available ventilators (Hamilton G5, Hamilton Medical, Bonaduz, Switzerland) for 60 min, followed by 60 min of ventilation by a Luca ventilator prototype with identical parameter settings. Assist/control pressure control ventilation mode as per the Chatburn classification (patient pressure or ventilator time triggered, pressure controlled, time cycled) mode was used in both the Luca and standard ICU ventilators. Vital parameters (heart rate, blood pressure, oxygen saturation) and ventilation parameters (respiratory rate, tidal volume, minute ventilation, pressure, FiO₂) were recorded every 10 min additionally to two blood gas analyses during both 60-min periods. Signs of dyspnoea (increased work of breathing) were assessed by the study physicians or if in case the patient was able to cooperate, by the Borg dyspnoea scale.²⁰ Patients were asked to rate the Luca ventilator as “similar or more comfortable compared to the standard ICU ventilator” or “less comfortable compared to the standard ICU ventilator”. Need for physician and nursing intervention as well as type and frequency of ventilator alarms were recorded. Rescue intervention (switching to the original ventilator with adequate clinical management of adverse events) was allowed in case testing physicians deemed it necessary.

Patients treated for respiratory failure were recruited for the observational study. Inclusion criteria were: >18 years of age, ongoing mechanical ventilation with pressure controlled, time-cycled mode. Exclusion criteria involved: pregnancy, unstable condition, lack of informed written consent and instability associated with rapidly changing respiratory patterns during the planed observation period. Primary endpoint was blood gas parameter values, secondary endpoints included clinical parameters recorded during the sessions.

Gas flowrate, volume, and leakage are expressed as standard temperature pressure dry (STPD) values for the preclinical stages and as body temperature pressure saturated (BTPS) values for the clinical stage of the study.

Statistical analysis

Data are presented as mean (±SD) and percentage (N). Groups were compared with Mann–Whitney U test, correlation was calculated with Spearman correlation. Generalized Linear Mixed Modelling (GLMM) with log normal distribution and a random effect by patient was used to assess the ventilator type controlling for ventilation time and the interaction between ventilator type and time on blood gas values and clinical parameters during clinical testing. Clinical relevance was evaluated by effect size according to Cohen.²¹ Cohen's D (the difference between two groups’ means divided by the pooled standard deviation) is considered small if the value is less than 0.2, medium if less than 0.5, and large if more than 0.8. A small Cohen's D values suggests that the difference is negligible, even if it is statistically significant. As Cohen's D might be effected when analysing variables of larger values, clinical relevance was confirmed with Cohen's f² for each variable. Cohen's f² (the proportion of variance accounted for by the variable of interest given the variance accounted for by all other variables in the model) is considered small if the value is less than 0.02, medium if less than 0.15, and large if more than 0.35. A small Cohen's f² value suggests that the difference is negligible, even if it is statistically significant. Given that the data contains repeated effects, Cohen's D is often considered to be an inappropriate effect size due to the correlation within the patient. However, Cohen's D is often easier to interpret. Therefore, both effect sizes are included.

The number of patients required to prove clinical equivalence in the clinical testing stage was evaluated with sample size calculation using the GLMM approach with RMVS criteria (4 + 15% and 2 + 4% respectively), 10% and 5% difference for inspiratory pressure and tidal volume and 10% and 5% difference for pO₂ and pCO₂ values respectively, based on previously published data focused on clinical variability of blood gas values in stable ICU patients.²²

Sample size calculation was conducted for difference in ventilator type with a Generalized Linear Mixed Model approach. The following assumptions were utilized. (1) There will be no clustering effect as all patients were treated at the same hospital. (2) Power was assumed to be 0.8 and Type 1 error was set to 0.05. (3) Unstructured correlation structure, for both time and ventilator Type. (4) Standard deviation was considered unchanged by time and ventilator type as found to be the case in the data. (5) Inspiratory pressure and tidal volume were used as the outcome variables of interest for the analysis.

Significance level was set at p < 0.05.

Statistical analysis was performed by IBM SPSS Statistics version 25.0 (IBM, Armonk, NY)) and SAS version 9.4 (SAS Institute, Cary, NC).

Ethical approval

This study was approved by the Hungarian Ethical Committee and the National Institute of Pharmacy and Nutrition (OGYÉI/25837/2020) and conducted in compliance with the Helsinki Declaration. All participating physicians and patients provided written consent before participating in the study. If a patient was unable to give written consent, consent was obtained from the next of kin and was reinforced once the patient was able to consent. All data were analyzed anonymously.

Results

I. Stage: preclinical testing

The 72 predefined settings were evaluated on all three prototypes of the Luca ventilator. As can be seen from Fig. 2, all measured parameter values were within the accepted range for all prototypes.

Fig. 2.

Set and measured pressure values and displayed and measured volume values of the Luca ventilator tested against lung models with different compliance and resistance specified in RMVS. Scatter plot of measured pressure readings for two set pressure values and measured and displayed volume readings charted using different lung models (as described in Methods). Different grey scale points represent the three different ventilator prototypes. Shaded zone represents accepted range as per RMVS specifications.

Measured and displayed inspiratory times differed by more than 5% when driving pressure (the difference between inspiratory pressure and PEEP levels) was within the range of 5 cmH₂O (data not shown). This issue was remedied by a more careful control of the rise time (the time during which inspiratory pressure is built up), which could be ensured with a minor software adjustment (using altered mathematical algorithm in the control system). Optimal trigger sensitivity was achieved by setting the manual inspiratory trigger to 1.5–2.5 cmH₂O below PEEP level.

II. Stage: safety testing

All prototypes displayed alarm activation in response to all tested alarm simulation scenarios within a 1 second timeframe, except for FiO₂ out of range alarms, which were activated within 1 min (which is the commercially accepted range for chemical sensors).

In one prototype, the circuit leakage alarm was activated at different times based on the location of the simulated disconnection of the circuit. Significant delay or activation of lower level alarms was observed in instances when the disconnection occurred at the junction of the Y-piece and the expiratory limb, a phenomenon not uncommon even in more advanced commercially available mechanical ventilators. A minor software adjustment rectified this discrepancy.

Closed Suctioning Test was passed according to RMVS guidance but it was noted that pressure readings were highly variable based on suction flow and timing of the procedure (data not shown).

III. Stage: usability testing

10 physicians (5 female and 5 male) participated in the study, with a mean critical care experience of 9.70 (SD = 8.11) years. All 10 physicians performed all 13 tasks correctly. SUS was 90.50 (SD = 8.40) for the cohort. SUS did not depend on physician sex (p = 0.17) nor correlated with years of experience (rs = 0.179, p = 0.62).

IV. Stage: clinical testing

Sample size calculation concluded that a minimum of 9 patients with 7 time points per patient would provide sufficient data to determine the difference between ventilators concerning volume, pressure and blood gas values. 15 patients, receiving invasive ventilation for a critical illness through an endotracheal tube on day 3–20 after admittance were recruited for the clinical testing stage. Characteristics of patients are listed in Table 1. Respiratory patterns were versatile (see Fig. 3).

Table 1.

Characteristics of patients participating in the clinical testing stage.

Characteristics	Mean ± SD or % (N)
Sex
Male	53.3% (8)
Female	46.7% (7)
Age (years)	62.4 ± 16.5
Diagnosis
Cardiac	13.3% (2)
Neuro	13.3% (2)
Respiratory	37.5% (6)
Surgical	26.7% (4)
Urogenital	6.7% (1)
COVID +	6.7% (1)
APACHE II (points)	24.5 ± 7.9
Predicted mortality (%)	45.7 ± 21.6
RASS score	−1.9 ± 1.9
PEEP (cmH2O)	8.0 ± 0.2
Inspiratory pressure (cmH2O)	23.9 ± 0.3
FiO2(%)	30 ± 0.8

APACHE II: acute physiology and chronic health evaluation II score, FiO₂: fraction of inspired oxygen, RASS: Richmond agitation sedation scale.

Fig. 3.

Mean respirator parameters during ventilator sessions for all patients. Mean respirator parameters calculated from all checkpoints during the ventilator sessions.

All 15 patients completed the stage. No adverse events or need for rescue intervention was recorded. No sudden inexplicable changes or variability greater than 20% were observed for any of the vital parameters monitored in the 15 patients during the use of the Luca ventilator (see Supplementary Fig. 1). All blood gas values were within the acceptable range for all patients during both the standard ICU ventilator and Luca ventilator session (see Supplementary Fig. 2). Ventilator parameters recorded during the ventilation sessions for all 15 patients are shown in Fig. 3.

We found no significant, clinically relevant differences between vital parameters, blood gas values or ventilator parameters recorded during ventilation sessions of the Luca ventilator and the commercially available ICU ventilator (see Table 2), except for inspiratory peak flow results, which showed a mathematically significant, but clinically irrelevant difference between the two ventilators.

Table 2.

Generalized Linear Mixed Models analysis of clinical parameters, ventilator type and time change during the clinical testing procedure.

	N	ICU ventilator	Luca ventilator	Ventilator type	Cohen's f^2	Cohen's D
Heart rate (/min)	210	87.39 ± 16.81	87.97 ± 16.70	p = 0.225	0.004	0.035
BP systolic (mmHg)	210	120.48 ± 19.35	123.73 ± 21.95	p = 0.009	0.038	0.157
BP diastolic (mmHg)	210	58.88 ± 11.36	60.4 ± 12.17	p = 0.010	0.007	0.129
SpO2 (%)	210	96.30 ± 1.57	96.46 ± 1.91	p = 0.272	0.030	0.093
Measured respiratory rate (/min)	210	17.02 ± 4.83	17.08 ± 4.10	p = 0.532	0.020	0.013
pH	58	7.43 ± 0.07	7.43 ± 0.06	p = 0.244	0.031	0.079
pCO2 (mmHg)	58	34.12 ± 6.07	34.57 ± 5.58	p = 0.278	0.028	0.078
pO2 (mmHg)	58	93.20 ± 17.92	97.30 ± 20.54	p = 0.106	0.017	0.213
HCO3 (mmol/L)	58	22.59 ± 3.73	22.64 ± 3.81	p = 0.824	0.054	0.015
Lactate (mmol/L)	58	1.43 ± 0.84	1.39 ± 0.82	p = 0.304	0.014	0.050
Measured PEEP (cmH2O)	210	9.46 ± 3.09	9.4 ± 3.09	p = 0.005	0.014	0.021
Measured inspiratory pressure (cmH2O)	210	23.38 ± 4.44	23.56 ± 4.31	p = 0.026	0.018	0.069
Inspiratory peak flow (L/s)	164	61.71 ± 9.42	67.65 ± 10.02	p = 0.000	0.036	0.610
Expiratory peak flow (L/s)	124	24.85 ± 47.13	28.12 ± 47.32	p = 0.086	0.073	0.069
Tidal volume (mL)	208	581.80 ± 107.19	567.63 ± 81.39	p = 0.184	0.046	0.149
Minute ventilation (L/min)	208	11.30 ± 12.38	11.04 ± 11.14	p = 0.767	0.009	0.021
FiO2 (%)	201	0.31 ± 0.05	0.31 ± 0.05	p = 0.103	0.058	0.012
Set PEEP (cmH2O)	208	9.60 ± 3.11	9.57 ± 3.15	p = 0.416	0.049	0.010
Set inspiratory pressure (cmH2O)	210	22.97 ± 4.21	23.26 ± 4.12	p = 0.000	0.008	0.042
Set respiratory rate (/min)	210	13.00 ± 2.46	13.00 ± 2.46	p = 1.00	0.086	0.000
Ti (s)	210	1.11 ± 0.18	1.11 ± 0.18	p = 0.061	0.013	0.016
Alarm (/h)	105	0.03 ± 0.17	0.22 ± 0.69	p = 0.07

BP: blood pressure, HCO₃: bicarbonate, FiO₂: fraction of inspired oxygen, pCO₂: partial pressure of carbon-dioxide, PEEP: positive end expiratory pressure, pO₂: partial pressure of oxygen, SpO₂: oxygen saturation, Ti: inspiratory time.

Measurement numbers (N) are a result of available data point values. Cohen's D (the difference between two groups’ means in standard deviation) is considered small if the value is less than 0.2, medium if less than 0.5 and large if more than 0.8. A small Cohen's D values suggests that the difference is negligible, even if it is statistically significant. Cohen's f² (the square of the difference between two groups’ means in standard deviation) is considered small if the value is less than 0.02, medium if less than 0.15 and large if more than 0.35. A small Cohen's f² value suggests that the difference is negligible, even if it is statistically significant.

There were no differences between the number of physician or nurse interventions (data not shown), and although there was a significant difference in alarm frequency, most of these alarms were self-limited and required no intervention.

None of the patients showed signs of dyspnoea as per Borg score. Only 3 of the 15 patients were able to supply a subjective assessment about the Luca ventilator, all of which were “similar or more comfortable compared to the standard ICU ventilator”. Borg scale values for these patients were all 0.

Discussion

The aim of this study was to compile a comprehensive clinical testing protocol for a rapidly developed mechanical ventilator to aid emergency authorization in the event of overwhelming critical care provision demand. We compiled a four-stage sequential testing protocol to limit the potential high risk associated with fast-track approval of newly developed devices.²³ Using the protocol, we found that the Luca ventilator was accurate, reliable, safe and user-friendly. During the pre-clinical testing stage, our protocol allowed the identification of a number of issues that were easily resolved with minor software adjustments. Furthermore, during the clinical testing stage, we found that a sampling frame of 15 critically ill patients was sufficient to reliably show no significant, clinically relevant differences between the vital parameters, blood gas values and ventilator parameters associated with the Luca ventilator compared to a standard ICU ventilator.

Preclinical testing was based on the RMVS guidance, which suggests testing the ventilator on a test lung with variable compliance and resistance.¹² The exact manner of pressure assessment is not detailed in the RMVS. As previously noted, it is important that pressure waveforms that do not resemble the ideal square-wave shape of commercially available ventilators could have vastly different values of pressure during the inspiratory as well as the expiratory phase, further influenced by the characteristics of rise time, which are seldom quantified even in advanced commercially available ventilators.²⁴ In our routine, we used the peak pressure recorded during the inspiratory phase, which is indicative in cases where square pressure waveform can be demonstrated. However, in other, less sophisticated designs, where a sawtooth or rising pressure waveform is more likely, an integration of pressure-over-time value might be more informative (see Fig. 4).

Fig. 4.

Possible pressurization patterns during pressure accuracy testing. Possible pressure profiles during testing of pressure controlled ventilation mode. (A) Slow profile with insufficient initial flow response, (B) unstable regulation with pressures beyond tolerance range, (C) optimal pressure regulation with fast onset, pressures close to target range. Grey line: target pressure profile, grey dashed lines: acceptable pressure range, black line: measured pressure profile, grey shading: area under the measured pressure curve.

The issue uncovered concerning the set and measured inspiratory time discrepancy with lower driving pressure settings is a well-known phenomenon associated with pressure-controlled modes and is a result of slower build-up of inspiratory flow.²⁵ Patient-tailored adjustment of rise time, especially in spontaneous modes and with lower driving pressure values, is crucial in optimizing inspiratory time and tidal volume and is an important comfort factor for patients in the weaning stage.²⁶ As this issue is less of a concern in the early stage of respiratory failure associated with diseases causing a critical care surge, where higher driving pressures might be needed in order to provide adequate ventilation, the altered software setting should be optimal for patients requiring ventilation during the initial period of their disease. Our experience highlights that ongoing communication and real-time cooperation between clinical testing and design teams is important throughout the testing process.

Concerning safety testing our experience highlights that both preclinical and clinical evaluation of alarm functions is important and that the safety testing stage should include several disconnection scenarios (e.g. disconnection at the distal end of the circuit, disconnection at the HMEF, disconnection at the Y-piece, disconnection at the proximal part of the expiratory or inspiratory limb). Rational tailoring of alarm functions during the testing stage is of great significance in order to avoid alarm fatigue.²⁷

Usability testing, especially important in situations where complex personal protective equipment might limit both sensory and operative abilities of clinicians, is not thoroughly detailed in existing emergency authorization guidances, and there is no standard usability scale designed for mechanical ventilators as previously published works have used different adapted scales.^28,29 For the current evaluation, a task list based on known requirements of ICU ventilators¹⁷ was created and usability was evaluated using the System Usability Scale (SUS)¹⁸ for its reported easy administration and valid, reliable results on small sample sizes.³⁰ As SUS scores are influenced by user experience, the testing cohort should include clinicians with different level of expertise to provide usability information in real-life critical care practice.

Previously published studies on testing of newly designed ventilators notably lack clinical testing scenarios.³ As opposed to ventilators developed according to internationally accepted standards that achieve CE certification rapidly developed and deployed mechanical ventilators cannot be tested in clinical studies with large patient frames, nevertheless it is crucial that these devices be tested with real-life critical care patients. The clinical testing algorithm presented here permits establishing equivalence compared to commercially available ICU ventilators in a timely fashion based on physiological, respiratory and subjective factors and provides guidance on how to calculate adequate sample size taking into account the specific reliability and accuracy of the ventilator in question.

Our study has several limitations. Although the preclinical stage of our protocol is based on the RMVS, we did not examine the efficacy of this stage in identifying parameter dependent inaccuracies. Although more extensive testing with additional frequency and I/E values might increase dependability, we deemed the parameters listed in the RMVS guidance sufficient for this stage. Furthermore, the testing protocol was designed based on the characteristics of the Luca ventilator, and other emergency devices might require adjustment of the preclinical stage according (i.e. if the device uses flow controlled modes or flow trigger). In the clinical stage, the observation period for the tested ventilator always followed the one for the standard ICU ventilator. Because this might lead to potential bias, we advise alternating the testing and control periods when performing the clinical stage. The strength of our current study is that it is the first extensive report of a nationally implemented testing protocol for a mechanical ventilator rapidly developed for use during the COVID-19 pandemic. The testing procedure is comprehensive, simple to perform during a short period of time, in compliance with current guidance, and includes preclinical and clinical stages. Further clinical tests which might be considered for inclusion in an internationally accepted guideline are simple and rapid testing of environmental effects (such as radiation and electromagnetic field) on device performance, software performance checks and long term endurance testing as well as testing on environmental aerosolized viral load in the case of ventilators designed for use in infectious diseases.

In conclusion, the protocol presented here allows the comprehensive and timely evaluation of rapidly developed mechanical ventilators. Based on our results, vital issues can be identified and addressed during the testing process, which might aid the safe and timely emergency authorization of rapidly developed mechanical ventilators playing possible crucial roles in future pandemic situations.

Informed consent

All participating physicians and patients provided written consent before participating in the study.

Authors’ contributions

SB took part in study design, acquisition of data and interpretation of data. LV participated in study design, analysis and interpretation of data. VAG took part in study design, analysis and interpretation of data. MLL participated in study design, analysis and interpretation of data. JG oversaw study design and interpretation of data. AL coordinated study design and acquisition and took part in the analysis and interpretation of data. All of the authors took part in drafting and revising the manuscript and approved the final version.

Funding

The study was performed at the Department of Anaesthesiology and Intensive Therapy, Semmelweis University, Budapest, Hungary. Luca ventilator prototypes were provided by Femtonics Inc. No additional funding was utilized.

Conflict of interest

Dr. Baglyas, Valko and Lorx are part of the Respiratory Research Team at the Department of Anaesthesiology and Intensive Care at Semmelweis University, which has been awarded a Hungarian National Research, Development and Innovation Office grant (2020-2.1.1-ED-2020-00016), but have no relevant competing interests. V. Anna Gyarmathy, Michelle LaPradd and Dr. Gal have no competing interests to declare.

Acknowledgements

The study was performed at the Department of Anaesthesiology and Intensive Therapy, Semmelweis University, Budapest, Hungary. Luca ventilator prototypes were provided by Femtonics Inc. No additional funding was utilized. The authors wish to thank the engineering staff at Femtonics Inc. and the staff of the Department of Anaesthesiology and Intensive Care at Semmelweis University for their assistance with data collection.