Development and Validation of a Tool for the Evaluation of Laparoscopic Intestinal Anastomosis in Simulation

Abstract

Introduction

Laparoscopic side-to-side intestinal anastomosis is a common in clinic practice and training simulation. The aim of this study is to design and validate a reliable and reproducible tool for its evaluation.

Methods

A modified Delphi method was used to design a tool with elements that determine quality, including 5 items: separation between stiches, eversion, tension, leak and iatrogenesis. The study included 21 participants (10 first-year residents and 11 experts) who performed a 5 cm laparoscopic intestinal side-to-side anastomosis with porcine viscera. The evaluations were blinded and done independently by 2 evaluators.

Results

The means obtained by novice and expert participants were, respectively: separation between stiches 3.2 vs 5.7 (P<.001), eversion 3.3 vs 5.9 (P=.004), tension 2.9 vs 5.9 (P=.001), leak tightness 3.2 vs 5.7 (P=.005), iatrogenesis 6.9 vs 7 (P=.47). The iatrogenesis parameter was not discriminatory, so it was removed from the tool. The total results were 12.5 for novices and 23.2 for experts (P=.001).

The correlation between observers presented an intraclass correlation coefficient of 0.99 for the separation between stiches, 0.94 for eversion, 0.98 for tension and 0.99 for leak.

The correlation between the score and the leak without pressure presented a Rosenthal's R of −0.71 (P<.001); with pressure R=−0.55 (P=.01).

Conclusions

The designed tool is valid to discriminate between novice and expert participants, presents very high concordance between observers and correlates with the risk of leak.

Resumen

Introducción

La anastomosis intestinal laterolateral laparoscópica es una práctica habitual en la clínica y entrenada en simulación. El objetivo del estudio es el diseño y posterior validación de una herramienta fiable y reproducible para su evaluación.

Métodos

Se utilizó un método Delphi modificado para desarrollar los elementos de evaluación al que finalmente incluyeron 5 apartados (separación entre puntos, eversión, tensión, estanqueidad y iatrogenia). Se incluyeron 21 participantes, 10 residentes quirúrgicos de primer año y 11 expertos. Realizaron anastomosis enteroentéricas laterolateral laparoscópica en víscera ex vivo porcina de 5 cm. Las evaluaciones fueron ciegas y realizadas por 2 evaluadores de forma independiente.

Resultados

Las medias obtenidas por noveles y expertos fueron respectivamente: separación entre puntos 3,2 vs 5,7 (p<0,001), eversión 3,3 vs 5,9 (p=0,004), tensión 2,9 vs 5,9 (p=0,001), estanqueidad 3,2 vs 5,7 (p=0,005), iatrogenia 6,9 vs 7 (p=0,47). El parámetro iatrogenia no es discriminatorio, por lo que fue eliminado de la herramienta. Los resultados totales fueron 12,5 los noveles y 23,2 los expertos (p=0,001).

La correlación entre observadores presenta un coeficiente de correlación intraclase de 0,99 para la separación entre puntos, 0,94 la eversión, 0,98 la tensión y 0,99 la estanqueidad.

La relación entre la puntuación y la fuga anastomótica sin presión presenta una R de Rosenthal de −0,71 (p<0,001); con presión se obtiene una R=−0,55 (p=0,01).

Conclusiones

La herramienta diseñada es válida para discriminar entre participantes noveles y expertos, presenta muy alta concordancia entre observadores y se correlaciona con el riesgo de fuga.

Introduction

The use of laparoscopic simulation as a teaching method is a fundamental pillar in surgeon training. It is often more effective than traditional methods for integrating knowledge, acquiring complex clinical–surgical skills^1–4 and behaviors, as well as decision making.^5,6

Surgical simulation has evolved from carrying out initiation exercises and developing simple skills^7–9 to carrying out complex techniques^10,11 in order to subsequently transfer these capabilities to clinical practice.¹² This goal has generated the need for tools to verify the level of performance of participants throughout their training, establish the optimal time to initiate the technique in patients under supervision, and analyze teaching methods that most efficiently enable this transition.^13–15 To this end, the Objective Structured Assessment of Technical Skill (OSATS)¹⁶ has been developed to assess the performance of procedures, although it does not evaluate the final results.

As for the types of surgical procedure, one of the most commonly trained skills in our setting is manual side-to-side intestinal anastomosis because it is considered a frequent technique in laparoscopy, and patient safety depends on its quality.^17–19 Its use as a moderate-level practice has been widely studied and extended. However, the evaluation of its results has varied between and within groups, depending on monitors and evaluators.

Therefore, the objective of this study is the design and subsequent validation of a reliable and reproducible tool for the evaluation of laparoscopic manual side-to-side intestinal anastomoses in the context of simulation.

Methods

Development of the Tool

To establish an expert consensus on the quality elements of anastomoses, as well as their descriptors and measurement (content validity), a modified Delphi method was used.²⁰ An iterative process of anonymous surveys and voting was conducted, with the participation of 10 expert laparoscopic surgeons who were considered representative of the General Surgery specialty (at least 10 years of experience, having performed more than 100 laparoscopic anastomoses in clinical practice, from 5 different hospitals in Spain). Participation was voluntary, with no type of remuneration, and consent was obtained before the start of the study. The first round asked about the elements (which are high-level concepts that describe a characteristic of the anastomosis) and descriptors (which reflect the specific parameters that determine the quality of each element) that should be evaluated by the tool. In the second round, each element and descriptor were assessed by importance. An area for comments was also provided. In the third round, once the variables were determined, participants were asked about their evaluation criteria. With these data, a global assessment tool was designed based on 5 elements, each of which was assessed with scores from 1 to 7, including: separation between stitches, eversion, leak test and iatrogenesis. An objective data collection form (Annex 1), an evaluation tool (Table 1) and instructions for the evaluator (Annex 2) were standardized.

Table 1.

Tool for the Evaluation of Laparoscopic Intestinal Anastomosis Done Under Simulation (Szabo-Berci-Sackier Laparoscopic Trainer Storz).

Dimension	Score
	1	2	3	4	5	6	7	Points
	Definite Risk of Dehiscence	Very Probable Risk of Dehiscence	Probable Risk of Dehiscence	Minimal Risk of Dehiscence	No Risk of Dehiscence but Poor Quality	No Risk of Dehiscence and Good Quality	No Risk of Dehiscence and Excellent Quality
Element 1: Separation between stitches on the anterior side of the anastomosis: distance between each of the stitches and number of stitches that did not meet the proper distance.	Unacceptable distance between stitches and discrepancy			Acceptable distance between stitches and discrepancy			Precise distance between stitches and no discrepancy
Element 2: Eversion of mucosa: Visualization of the intestinal mucosa outside of the suture lines, anterior o posterior to the anastomosis	Eversion of the mucosa is unacceptable.			Eversion of the mucosa is acceptable.			There is no eversion of the mucosa.
Element 3: Suture tension: Assessment of the size of the anastomosis through the measurement of the length of the anastomosis and the traction of the suture.	The measurement of the anastomosis from angle to angle is unacceptable.The excess suture material after tensing the suture line is unacceptable.			The measurement of the anastomosis from angle to angle is acceptable.The excess suture after tensing the suture line is acceptable.			The measurement of the anastomosis from angle to angle is accurate.There is no excess suture material after tensing.
Element 4: Leak test: Evaluation of air leaks through the suture line when the intestines of the anastomosis are distended. It is considered ‘leak-tight’ when no air leaks out.	There is air leakage in several areas around the anastomosis, which does not allow for the distension of the anastomosis.			There is small air leakage through needle-holes, but not between the stitches when the intestinal loops are distended.			There is no air leakage through any point of the anastomosis and the intestinal loops are distended.
Element 5: Iatrogenesis: Assessment of the existence of tears or loss of serosa in the intestinal loops of the anastomosis.	Intestinal tear with perforation.Extensive loss of serosa in the anastomosed loop.			No tears or intestinal perforation.Small loss of serosa affecting only the superficial layer.			No tear or loss of serosa
Total=1+2+3+4+5 (minimum 5 and maximum 35)

* The element of iatrogenesis was eliminated from the final version of the tool because it was not discriminative.

Cohort Study

In order to study the construct validity of the tool and to analyze whether the measurements correlated with the learning trajectory of novice surgeons and experts when performing intestinal anastomoses, 21 participants were included in the study: 10 first-year surgical residents initiated in laparoscopic surgery, and 11 expert specialists (at least 5 years of experience and having performed more than 50 anastomoses), who gave their written informed consent to participate. The study was carried out at the Hospital Virtual Valdecilla (Santander, Cantabria) between November 2017 and August 2018. Data were collected for the characteristics and experience of the participants, as well as the time used, the length and the leakage of the anastomosis.

Two evaluators participated, who were general surgeons and instructors from the simulation center and had not participated in the Delphi survey. The evaluations were blind, and each was carried out independently by both, except for the leak test, which was done jointly. The evaluators’ training consisted of a 30-minute session to explain the tool and a practical anastomosis assessment workshop. The consistency or stability of the measurements obtained from the same surgeon (reliability) was evaluated by the two evaluators.

Procedure

Manual side-to-side intestinal anastomoses measuring 5 cm were performed with ex vivo porcine viscera in a laparoscopic endo trainer (Szabo-Berci-Sackier Laparoscopic Trainer Storz). The anastomosis technique involved performing interrupted tacking sutures and continuous double suture with Connell suture at the vertices (Fig. 1).

Fig. 1.

Manual side-to-side intestinal anastomosis with ex vivo porcine viscera.

The teaching methodology used consisted of providing the participant with a bibliography about the procedure, a video to model the technique, and later practice with the instructor (always the same, and not an evaluator).

Subsequently, the anastomoses were evaluated using the designed tool, and air-leak tests were performed without pressure and then measuring the pressure at which anastomotic leakage occurred (WIKA® DG-10 digital manometer).

Sample Size

The calculation of the sample size was based on previous experiences and carried out with the OSAT scores obtained among novice participants and experts in performing laparoscopic manual side-to-side intestinal anastomoses. It was calculated using the comparison of means, with 3.47 for novices and 4.63 for experts, and a standard deviation of 0.504. The need of 4.1 participants in each group was obtained for an α risk=0.05 and a statistical power of 95%.

Statistical Analysis

The statistical analysis was done with the IBM SPSS Statistics program for Windows, Version 24.0 (2016; IBM Corp, Armonk, NY). To evaluate the reproducibility of the tool, we calculated the interobserver agreement rate (intraclass correlation coefficient). The Mann–Whitney U test was used to compare the means of novice and expert participants as well as the relationship between anastomotic leak and its score. This correlation was analyzed using Rosenthal's R. Statistical significance was established at P<.05.

Results

Delphi

According to responses of the survey, the optimal manual lateral intestinal anastomosis should have a distance between stitches of 3–4 mm, without being able to visualize intestinal mucosa between them, which would be classified as eversion. The ideal size should be between 4 and 5 cm.

To assess suture tension, the suture was gently pulled with an instrument to determine whether it slid, and the length of the remaining thread was measured. Sutures were considered loose when the excess material measured 5 mm or more.

Iatrogenesis was defined as the presence of perforations or visceral tears due to inadequate manipulation.

Cohort

The study cohort included 21 general surgeons: 12 women and 9 men, all of whom were right-handed. Out of the total, 10 were surgery residents with a mean age of 26.4 years and 11 experts with a mean of 48.9 years, with an overall age of 38.2. Previous experience was nil for the first group, without having independently performed any laparoscopic procedure or anastomosis, either clinical or in simulation. The expert group had an average of 545.5 laparoscopic procedures, with an average of 220 clinical laparoscopic anastomoses (enteroenteric, gastroenteric and coloenteric) and 43.6 in simulation. The time used per anastomosis was 52.2 min.

Interobserver Reliability

A very high level of agreement was reached between the different evaluators with an intraclass correlation coefficient of 0.99 for the separation between stitches, 0.94 in eversion, 0.98 in tension and 0.99 in the leak-tightness test (Table 2). Iatrogenesis was similar between all observations.

Table 2.

Interobserver Agreement.

	Mean	Variance	Interclass Correlation
Separation between stitches	4.24	0.007	0.986
Eversion	4.52	0.27	0.944
Tension	3.43	0.061	0.981
Leakage	4.48	0.088	0.989

Internal Validity

The means obtained by the novice participants and the experts were respectively: separation between stitches 3.2 vs 5.7 (P<.001), eversion 3.3 vs 5.9 (P=.004), tension 2.9 vs 5.9 (P=.001), leak test 3.2 vs 5.7 (P=.005), iatrogenesis 6.9 vs 7 (P=.47). With a total score of 19.4 vs 30.2 (P<.001) (Table 3). The average time was 122 min vs 52.2 min (P<.001).

Table 3.

Means of Scores Obtained Between Novice and Expert Participants.

	Novice	Expert	P
Separation between stitches	3.149	5.757	.0003
Eversion	3.299	5.878	.004
Tension	2.865	5.848	.001
Leakage	3.231	5.682	.005
Iatrogenesis	6.850	7	.468
Total	19.394	30.166	.0003
Total without iatrogenesis	12.544	23.166	.0003

It can be seen how the iatrogenesis parameter is very similar between groups and not significant or discriminatory. Therefore, it can be eliminated from the tool without changes in significance, obtaining total results of 12.5 for novices and 23.2 for experts (P=.001) (Fig. 2).

Fig. 2.

Total score without iatrogenesis for novices and experts.

Likewise, the relationship between the total score and anastomotic leak test was calculated. In the case of the test without pressure, the Rosenthal R was −0.71 (P<.001) (Fig. 3), while with pressure it was R=−0.55 (P=.01).

Fig. 3.

Total score of anastomoses, divided according to the result of the leak test without pressure.

The group of novices obtained a correlation of R=−0.69 (P=.03) in the leak without pressure and R=−0.66 (P=.04) with pressure. Meanwhile, the results in the group of experts were R=−0.48 (P=.11) in the leak without pressure and R=−0.66 in the leak with pressure (P=.03).

Discussion

In the end, the tool developed for the evaluation of simulation laparoscopic intestinal anastomoses was comprised of 4 elements with their corresponding quality descriptors and was able to significantly discriminate between experienced and inexperienced participants. Likewise, the tool obtained an inversely proportional correlation with anastomotic leakage and very good agreement between observers. The iatrogenic parameter was initially included in the tool as a result of the Delphi study; however, as the results were very similar between groups and neither significant nor discriminatory, this variable was removed from the final version of the tool (Annex 3).

The evaluation of the results of an intestinal anastomosis is complex, due to the large number of variables involved and the difficulty of testing and correlating the technique with the final result. For this reason, it continues to be studied in current clinical practice.^21,22 To this, we must add the staticity of the tissues and the peculiarities of the simulation, as a greater number of verification and evaluation methods are available in other disciplines, such as microvascular surgery.^23–25 However, training and technological development hastens the need for valid, simple, reproducible tools with which to evaluate progress.²⁶ The need to develop skills in different techniques and their incorporation in comprehensive curricula has led to the appearance of studies that range from basic skills²⁷ to more complex techniques such as cholecystectomies,²⁸ fundoplications,²⁹ etc. In our experience, the OSATS, which has already been validated by Reznick³⁰ for intestinal anastomoses in live animals, seems to be a good estimator of procedural ability, although it does not adequately correlate with the final quality of the anastomosis.¹⁵ In addition, OSATS requires more time and significant human and financial resources. The tool designed in this study can alleviate these barriers, while objectivizing data collection^31,32 and correlating with the final technical result.

The implications for teaching in subsequent training courses include the individualization of the learning curve of participants and determining the elements of anastomosis in which proficiency has been achieved and in which progress needs to be made. This makes it possible to transform a training curriculum based on practice time into another based on performance evaluation according to standardized and validated parameters. In addition, it allows future lines of research to be opened in order to create a reference framework for evaluating the number of anastomoses or the score that would determine the transition to clinical practice.

This study has several limitations. On the one hand, the evaluators were familiar with the technique and simulation, which may limit the possibility of generalizing to evaluators who are not experts in or do not perform this technique. The inclusion of new evaluators at other medical centers may be of interest in order to verify whether agreement is maintained. Dissemination of the tool among new evaluators will require evaluation guidelines as well as a training workshop.

Possible future research would include expanding the sample size and testing whether the statistical power obtained is maintained by introducing trained and mid-level participants, who were not represented in the present study.

After these steps, its implementation and validation in anastomoses performed in live animals could be studied and subsequently considered in human clinical practice as a quality criterion.

Conclusions

In this study, we have designed and validated a tool for the evaluation of laparoscopic manual intestinal anastomoses in ex vivo viscera, confirming its ability to discriminate between novice participants and experts as well as very high agreement between observers. In addition, this tool obtains a good correlation with the risk for leakage without pressure.

Conflict of Interests

The authors have no conflict of interests to declare. Hospital Virtual Valdecilla is affiliated with the Center for Medical Simulation (Boston, USA), both of which are non-profit teaching institutions that offer clinical training programs and enrolled education programs.