Effectiveness of a program for improving teamwork in Operanting Rooms

Abstract

Introduction

The purpose of the trial was to establish whether a training program could improve teamwork in the Operating Rooms and the Obstetric Suite at a healthcare institution.

Method

Quasi-experimental before and after study – A multiphase and multi-method training program was implemented for improving teamwork, measured with the OTAS-S instrument in 40 surgical procedures, pre and post intervention. Training was given to 80% of the population. The descriptive analysis was abased on the data collected from each stage and the effectiveness was determined based on the comparison of the scores obtained.

Results

Post-intervention measurements revealed differences in the scores obtained by the teamwork between the two stages (MW U-test; z = 48.879, P = .0000). This effectiveness was shown in the surgical and obstetrics area. Two or more points of improvement in the average OTAS-S scores was identified in every phase, behaviors and sub-teams.

Conclusion

Intervention was effective for improving teamwork in the operating rooms and in the obstetrics suites in the short term. Further research is suggested including a control group and a more extended follow-up to establish long-term effectiveness.

Resumen

Introducción

El objetivo fue determinar si un programa de entrenamiento podía mejorar el trabajo en equipo en las salas de cirugía y obstetricia de una institución de salud.

Método

Estudio cuasi experimental de antes y después. Se aplicó un programa de entrenamiento multi-fase y multi-método para mejorar el trabajo en equipo, el cual se midió aplicando el instrumento OTAS-S en 40 procedimientos quirúrgicos en las etapas pre y post intervención. La capacitación fue dada al 80% de la población. Se realizaron análisis descriptivos de los datos obtenidos en cada etapa y se determinó la efectividad a partir de la comparación de los puntajes obtenidos.

Resultados

La medición post-intervención reveló diferencias en los puntajes obtenidos en el trabajo en equipo entre ambas etapas (MW U-test; z = 48.879, P = .0000). Esta efectividad se dio en el área de cirugía y obstetricia. Todas las fases, comportamientos y sub-equipos presentaron mejora de dos o más puntos en los puntajes promedios del OTAS-S.

Conclusión

La intervención fue efectiva para mejorar el trabajo en equipo en las unidades de cirugía y obstetricia a corto plazo, se sugiere realizar investigación futura que incluya grupo de control y hacer seguimiento más prolongado en el tiempo para determinar efectividad a largo plazo.

Introduction

Poor communication and teamwork deficiencies have been identified as contributing factors to medical errors or adverse events¹; some trials indicate that poor communication among the team is responsible for about 20% of the mistakes in healthcare in our country.² Morey et al.³ found that the implementation of initiatives for improving teamwork reduced the number of events from 30% to 4.4%. However, quality evidence is not yet available to support these findings or to determine which of these intervention plans are the major contributors to the reduction of those mistakes.^4,5

One of the most popular initiatives to improve teamwork is Team Stepps based on some of the Crew Resource Management (CRM) resources developed for aviation after the 80s, acknowledging the importance of non-technical skills in the case of mistakes leading to accidents. The training includes the development of collective intelligence, understood as the active ability of the team members to learn, teach, communicate, reason and think together, regardless of their position in the hierarchy, focused on shared objectives and a mission.⁶

There are different instruments for measuring teamwork in the OR. Some of these are the Anesthetists Non-Technical Skills (ANTS),⁷ the Non-Technical Skills (NOTECHS),⁸ the Non-Technical Surgical Skills for Surgeons (NOTSS)⁹ and the Observational Teamwork Assessment for Surgery (OTAS).¹⁰ Only the latter has a validated Spanish version.

This paper intends to answer the question: What is the effectiveness of a training program for improving teamwork skills of staff members working in surgery and obstetrics at a healthcare institution in Colombia?

Method

Quasi experimental before and after study – A multiphase and multi-method training program was implemented based on the TeamSTEPPS^® model,^11,12 in addition to the CRM (Crew Resource Management)¹³ intervention models, and on the motivational theories of health psychology.¹⁴

The intervention included the following components (Fig. 1):

• (1)

  4-h workshop: Previously prepared audiovisual material was used (PowerPoint slides, awareness video and videos of demonstrative cases). The topics included: Patient Safety Systemic Model, teamwork, non-technical skills (communication, cooperation, coordination, leadership and situational awareness), Time-out and effective use of the WHO checklist.

• (2)

  Five virtual modules were uploaded to a Moodle^® platform on a weekly basis. The topics corresponded to an in depth analysis of the workshop discussions.

• (3)

  Time-Out training and correct use of the WHO surgical checklist. By consensus with the surgical, obstetrics, safety and quality leaders, an adaptation was made to the WHO checklist and then the professionals from both areas were trained in its application. Additionally, training was given on the use of Briefing and Debriefing meetings, Time-out and Sign-out procedures that are consistent with the surgical checklist. However, in our case there is a difference because the Time-Out was done prior to the induction of anesthesia and not prior to the incision and was not limited to the items on the list, but included the planning and evaluation of every aspect in the procedure.

• (4)

  Institutional Actions – Meetings aimed at getting the Institution's commitment to the change process and with the strategies needed to maintain the improvement processes in the long term.

Fig. 1.

General structure of the intervention program for improving teamwork in the OR and the obstetric suite.

Source: Authors.

Procedure

Stage 1. Baseline

An initial OTAS-S–based teamwork measurement was made.^15,16 A previously trained psychologist administered the instrument, following the procedure established in the instrument's handbook.

The sample size for paired data was estimated, resulting in a minimum sample size of 16 pairs, for a minimum detectable difference of 1.5 points in the pre and post intervention stages, a standard deviation of 1.2, a 95% confidence interval and 80% power.

Stage 2. Application of the intervention

The implementation of the intervention lasted two months. The physical workshops were repeated for three weeks, and each staff member attended only once. The on-line workshops began once the physical workshops were over. Training in time-out was done directly in the ORs.

Stage 3. Measurement of effectiveness

One week after the intervention was completed, a post-measurement was done using the OTAS-S instrument again.

Instruments

The OTAS-S measures five teamwork dimensions: communication, coordination, cooperation/support, leadership and supervision/awareness of the situation. It is rated with a 7-point Kinkert scale; it measures the performance of 4 sub-teams – anesthesia, surgery, nursing and instrumentation – in one of the three surgical phases (pre, intra and postoperative).^15,16

The participants evaluated the workshop responding to statements about three topics: contents, quality of delivery as a whole and overall satisfaction. This evaluation format was adapted from Hull et al.¹⁷ Each statement was rated on a 5-point Likert scale (1 = totally disagree, 5 = totally agree).

Results

Stage 1. Baseline

Teamwork was measured in 40 surgical procedures in total, distributed by units, as shown on Table 1.

Table 1.

Frequency and percentage of surgical procedures reviewed by unit (pre-phase).

Unit	Frequency	Percentage
Surgery	24	60%
Obstetrics	16	40%
Total	40	100%

Source: Authors.

The minimum number planed for each unit sample was met; in the surgical unit 8 procedures beyond the number planned were observed.

The average surgical time was 51 min (SD = 26 min), with a minimum of 18 and a maximum of 352 min. The teams were made up by an average of 7 people ±2; the smallest team size was 3 people and the maximum 11. With regard to the procedures observed, 94% of the obstetric surgeries were cesarean sections, while in the surgical unit there were general procedures and specialized surgeries such as orthopedics, plastic surgery and gynecological surgeries.

Table 2 shows the total and per unit results prior to the intervention.

Table 2.

OTAS-S scores for the total number of observations and per unit (measurement before the intervention).

	Mean	SD	Median	Percentile 25%	Percentile 75%
Total	3.3	1.3	4	3	4
Unit
Surgery	3.5	1.2	4	3	4
Obstetrics	3.1	1.2	3	3	4

Source: Authors.

The test scores range from 0 to 6, where 0 indicates a problem behavior and a serious hurdle for the team's performance and 6 indicates an exemplary behavior, extremely effective for improving teamwork performance, while 3 means that the performance of the team does not improve or deteriorate by the particular behavior. The results show that the teams had a baseline neutral behavior.

Furthermore, no significant difference was noted in the units (MW U-test; z = −1.781, P = .075).

Table 3 presents the results initially obtained based on behavior, sub-team and phase.

Table 3.

OTAS-S scores based on behavior, sub-team and surgical phase (measurement before the intervention).

Dimension	Mean	SD	Median	Percentile 25%	Percentile 75%
Surgical phase
Pre	3.7	1.0	4	4	4
Intra	3.6	0.98	4	4	4
Post	2.6	1.4	3	3	4
Behavior
Communication	3.5	1.2	4	4	4
Coordination	3.4	1.4	4	4	4
Cooperation	3.5	1.3	4	4	4
Leadership	3.1	1.1	3	3	4
Situational awareness	3.2	1.2	3	3	4
Sub-team
Surgery	3.2	1.4	3	2	4
Nursing	3.7	0.8	4	3	4
Instrumentation	3.4	0.96	3	3	4
Anesthesia	3.1	1.6	3	2	4

Source: Authors.

Significant statistical differences by phase were identified (KW-test; Chi-square = 338.138, P = .000), in terms of behavior (KW-test; Chi-square = 43.978, P = .000) and in terms of sub-team (KW-test; Chi-square = 63.702, P = .000). The behaviors with the highest opportunity for improvement were leadership and situational awareness; overall however, all of the behaviors could be improved with improved teamwork. As a whole, the nursing team showed the best teamwork scores, as well as the pre-operative phase.

Stage 2. Administration of the intervention

18 workshops were organized to train 186 staff members in person, from the areas of intervention (76% of the staff). The results of the evaluation made by the participants are illustrated on Table 4.

Table 4.

Overall assessment of the in-person workshop.

Area assessed	Item	Mean (SD)^a	Range
Contents	This workshop met my expectations	4.90 (0.4)	2–5
	I would recommend this workshop to my colleagues	4.91 (0.5)	1–5
	The contents of the workshop were consistent with the objectives set	4.92 (0.3)	2–5
Quality of Delivery	The style of the presenter was appropriate and she made it interesting	4.95 (0.2)	3–5
	The information was given in an easy to understand form	4.97 (0.2)	4–5
	The quality of the teaching materials was adequate	4.91 (0.3)	2–5
	The various activities of the workshop helped me to a better understanding of the topic	4.96 (0.2)	4–5
Satisfaction	Overall, I am pleased with this workshop	4.96 (0.2)	3–5
	The learning objectives were accomplished	4.93 (0.3)	4–5
	After the workshop I feel better prepared to understand the importance of non-technical skills and their impact on patient safety	4.96 (0.2)	3–5

a

SD: Standard deviation.

Source: Authors.

In all of the areas evaluated, the workshop was considered very positive (range 4.90–4.97 on a 5-point scale), so it can be considered a valuable tool to create awareness and educate the staff on the importance of teamwork.

The design included 5 modules that were uploaded to S.C.A.R.E.’s central platform “Siempre Educando” (Constantly Educating). There were 134 staff members participating (86% of the workshop participants); the groups with the lowest participation were surgeons and anesthesiologist. The major difficulty experienced was poor knowledge of some staff members about how to use these platforms and this required establishing fixed timetables to provide assistance and hands-on training.

Training on the correct use of the WHO checklist and the implementation of time-out extended over four weeks. 73 interventions were assessed with 47 different teams trained, including 95% of the nurses and scrub nurses, 80% of the anesthesiologists and approximately 70% of the specialized surgeons.

Stage 3. Effectiveness measurement

For the second measurement OTAS-S was used again in 40 procedures, following the same percentages as the preliminary stage (60% surgery, 40% obstetrics).

The surgical time in the second round was 65 min in average (SD = 32 min), range 29 through 182 min. In average, the teams consisted of 8 people, with a standard deviation of 1, ranging from 5 to 10.

Measurement of the post-intervention stage revealed differences in the teamwork scores, as illustrated on Tables 5 and 6.

Table 5.

Comparison of OTAS-S scores from the before and after phases of the intervention (total and by unit).

	Mean Pre	SD^a Pre	Median Pre	Mean Post	SD Post	Median Post	Z^b
Total	3.3	1.3	4	5.3	1.01	6	−48,879^c
Unit
Surgery	3.5	1.2	4	5.4	0.98	6	−38,188^c
Obstetrics	3.1	1.2	3	5.2	1.04	6	−30,674^c

a

SD: Standard deviation.

b

Mann–Whitney U-test, two-sided test.

c

P = .0000.

Source: Authors.

Table 6.

Comparison of OTAS-S scores for the before and after phases of the intervention (based on behavior, sub-team and surgical phase).

Dimension	Mean Pre	SD^a Pre	Median Pre	Mean Post	SD Post	Median Post	Z^b
Surgical phase
Pre	3.7	1.0	4	5.5	0.8	6	−28,903^c
Intra	3.6	0.98	4	5.4	0.9	6	−28,189^c
Post	2.6	1.4	3	5.1	1.2	6	−28,631^c
Behavior
Communication	3.5	1.2	4	5.3	1.0	6	−20,592^c
Coordination	3.4	1.4	4	5.5	0.95	6	−22,739^c
Cooperation	3.5	1.3	4	5.4	1.0	6	−21,728^c
Leadership	3.1	1.1	3	5.2	1.1	6	−21,795^c
Situational Awareness	3.2	1.2	3	5.4	1.0	6	−22,914^c
Sub-team
Surgery	3.2	1.4	3	5.3	0.96	6	−23,845^c
Nursing	3.7	0.8	4	5.2	0.93	6	−23,476^c
Scrub nurses	3.4	0.96	3	5.7	0.63	6	−29,004^c
Anesthesia	3.1	1.6	3	5.1	1.3	6	−21,976^c

a

SD: Standard deviation.

b

Mann–Whitney U-test, two-sided test.

c

P = .0000.

Source: Authors.

The final average score was 5.3, representing considerable improvement in the team's performance.

All stages, behaviors and sub-teams showed improvements of two or more points in the average scores obtained. Significant differences were identified in every comparison: total score, by unit, surgical phase, behavior, and sub-teams (surgery and obstetrics) all showed a considerable and similar improvement (post means: 5.4–5.2).

Discussion

This trial is considered a forerunner in the development of human factors-related research and patient safety in our country. As stated by Martin et al.¹⁸ “…following the publication “Errar es humano” (To err is human) from the Institute of Medicine in 1999, there have been numerous grants awarded for research, publications, editorials, letters and reviews on the topic of Human Factors in medicine”; however, no trials have yet been completed in Colombia showing the effectiveness of interventions to improve non-technical skills among healthcare surgical practitioners. The aim of the study was to establish the effectiveness of a training program for enhancing team skills of the surgical and obstetrics staff at a healthcare institution in Colombia.

The results show that the baseline team measurements indicated a mild trend toward teamwork, but if failed to impact patient safety. The behaviors with higher probability of improvement were leadership and situational awareness, but as a whole, every one of them could be improved toward enhanced teamwork. As a whole, the nursing team showed higher teamwork scores, as well as the preoperative phase. The post-operative period showed the lowest teamwork behaviors, with an average score of 2, indicating a mild disruption in the team's performance, whether due to omissions or inadequate behavior. This result could be associated with the lack of a debriefing of the surgical process since some of the team members often left the OR even before the patient had been transferred to the gurney and delivered to the recovery room.

One of the key surgical variables to reduce the number of medical errors is on-the-job training, in addition to strategies and team roles.¹⁹ Teamwork includes technical skills such as handling and use of surgical equipment and non-technical skills such as communication, decision making and shared situational awareness.²⁰ Survey analysis and observations, as well as near misses and adverse events, have evidenced the key role of miscommunication among the team members. Likewise, the analysis of communications and healthcare team work has enabled the identification of recurrent issues due to cultural barriers and a rigid hierarchical structure.²¹

Overall, the final scores measured in this trial in the various groups show teamwork-oriented behaviors after the implementation of the intervention program. The one to two points difference in the pre and post-intervention indicates that the intervention was effective to improve teamwork in the surgical and obstetric units at the institution in the short term.

Some of the limitations of this trial were the quasi-experimental method with just one institution involved and the lack of a control group. The pre and post-test designs limit the ability of the researchers to identify causal relationships, although the results may be statistically significant.²¹

Literature reviews have been made on the relationship between training in non-technical skills and technical performance and the clinical results. One review revealed that poor quality of the trials was due to masking issues, subjective measurements and potential Hawthorne effect. The results reported were improved attitude, improved teamwork, improved technical performance, efficiency or decline in the number of errors. None of the randomized clinical trials found any evidence of technical or clinical improvement.²¹

Another review considered 12 trials where significant changes in the team practices were reported. The training strategies were: Briefing–debriefing, WHO checklists, in-person workshops and simulation environments. These results suggest that these types of training help to improve communications and group unity. However, no obvious differences were found in terms of the number of errors or incidents.²⁰

The results of our investigation are based on the observations on teamwork and a survey on intervention satisfaction. Measurements to assess the clinical effect, as well as the drop in adverse events or improved performance of the technical team are lacking. A measurement was made one week after the intervention but longer follow-up is required to ensure the stability of the changes identified.

The strengths of the trial were meeting the requirement of the minimum sample size per unit, the use of observational instruments adapted and validated to our environment and the implementation of an evidence-based program.

The intervention lasted for two months and 186 staff members were trained in the areas of intervention (76% of the staff). To comply with the interdisciplinary nature of the team as suggested by the TeamSTEPPS program, the participants included physicians, nurses, scrub nurses and support staff of the institution.²²

Additionally, the CRM intervention models were taken into account.¹³ Besides urging the participants to acknowledge the importance of a safety culture and teamwork, tools were used to standardize procedures, ensure effective communication, and improve awareness of all the environmental variables.

One of these tools in particular was the use of the checklists introduced in aviation back in 1935. Its recent use in surgery has proven to reduce the number of complications by 36% and deaths by 47%. The implementation of the WHO checklist has shown to influence the perception about a safety culture. At a University Hospital in Norway, a limited impact on safety culture was identified, with a 77% checklist compliance and 85% positive changes in the frequency of reported events and improvement plans.²³ A research done in Colombia found a reduction in the number of reported surgical adverse events, following the implementation of the checklist (7.26% in 2009 vs. 3.29% in 2010) and a perception of enhanced quality and patient satisfaction.²⁴ These findings show the importance of adopting the checklist as a routine in healthcare centers in our country; however, as Gómez (2013) said “…the implementation of the checklist is not an easy task and requires leadership, teamwork, flexibility and adaptation to change, both by the institution itself and by the healthcare professionals”.²⁵ So the recommendation when adopting the checklist is to simultaneously introduce programs to enhance these types of non-technical skills such as the one herein discussed.

Some additional strategies that can be combined with the use of the checklist are the Time-Out and Briefings and Debriefings that were also used in this intervention. These are pre and post-surgery team discussions that usually last from 3 to 5 min following a checklist. Prior to the procedure, the list shall include: (1) introduction of team members, (2) goals of surgery, (3) clarification of roles, (4) review of the procedure, and (5) identification of potential hazards. At the end of the procedure, the list includes: (1) checking and counting gausses, needles and instruments, (2) confirmation and labeling of surgical specimens, (3) agreement on post-operative care, and (4) discussion around the team's performance: positive and negative aspects and improvement strategies.

Before the program the post-operative was the surgical phase with fewer teamwork behaviors, associated with not having a debriefing, including transferring the patient and hand-off to the recovery suite. The phase with the highest post-intervention score was the pre-operative phase, but there were significant differences between the before and after measurements for all the surgical phases, including the post-op.

The available evidence about the technical or clinical benefit of teamwork training interventions is weak. The studies, which are mainly descriptive, are of poor quality and exhibit flaws in the psychometric characteristics of the instruments. Many trials base their results on the change in perceptions and attitudes, but this does not guarantee an effective behavioral change.

Studies have focused on communication and teamwork using tools such as briefings–debriefings and checklists. However, there is no information in the literature about other factors such as shared situational awareness, leadership, management of workload and decision-making. There is also a lack of follow-up to ensure the stability of the changes identified, with post-intervention measures for up to 12–18 months.

Most research focuses on surgeons and anesthesiologists, but the other professionals involved in the team, including nurses, scrub nurses, etc., have not been taken into account, or their results have not been frequently reported. In contrast, the participation of surgeons and anesthesiologists was minimal in this trial. Probably as a result of a cultural bias, a strong hierarchy characterizes the medical environment. This is quite concerning because the performance of the surgical team depends on the successful interaction of many people from diverse disciplines that work together in the OR.²⁶

The suggestion is to use a more stringent methodology in future research projects, with the participation of a larger number of institutions, including a control group, the use of validated instruments with evidence in our environment, both for observation and for measurement of the impact on attitudes, team satisfaction and other indicators; finally, a longer follow-up would be appropriate.

Funding

The financing was done by Colombian Society of Anesthesiology (S.C.A.R.E.).

Conflict of interest

The authors Ana Carolina Amaya and María Luz Gómez are currently working in the Deputy Medical Advisory Specialized S.C.A.R.E.