Post-Ureteroscopic Lesion Scale to determine ureteral wall damage, not so easy to employ

Abstract

Objective

To analyze the level of agreement of the Post-Ureteroscopy Lesion Scale (PULS) and the consequences on its application in clinical practice with more reliable statistical data than the one used in the original work.

Methods

14 URS and 14 micro-URS were performed in 14 female porcine model. All the procedures were video recorded and an anatomopathological analysis was performed in each ureter. Sixteen urologists (9 endourologists and 7 general urologists) and 4 residents evaluated the ureteral lesions according to the PULS, with degrees 0, 1 and ≥2. The agreement was calculated with percentages, Kendall’s W coefficient and the indicators Fleiss’ Kappa and Krippendorff’s Alpha, while the inter-rater agreement was calculated with Spearman’s correlation and Cohen’s Kappa.

Results

The percent of agreement was 11.1%. The coefficients were likewise classified as low or very low, with the greatest agreement found among the inexperienced. Also, 50% of the raters did not agree with the rest.

Conclusions

The low inter-rater agreement, the specificity of the PULS and the clinical-pathological correlation suggests that this scale is not simple, and probably has a long learning curve.

Resumen

Objetivo

Analizar el nivel de concordancia de la Post-Ureteroscopic Lesion Scale (PULS), y examinar las consecuencias de su aplicación en la práctica clínica con datos estadísticos más fiables que los utilizados en el trabajo original.

Métodos

Se realizaron 14 ureteroscopias (URS) y 14 micro-ureteroscopias (micro-URS) en 14 cerdos hembra. Todos los procedimientos se grabaron en vídeo y se realizó un análisis anatomopatológico en cada uréter. Dieciséis urólogos (9 endourólogos y 7 urólogos generales) y 4 médicos internos evaluaron las lesiones ureterales según la escala PULS, con grados 0, 1 y ≥2. La concordancia se calculó mediante porcentajes, el coeficiente W de Kendall, el índice Kappa de Fleiss y el Alfa de Krippendorff. La concordancia entre evaluadores se calculó con la correlación de Spearman y el coeficiente Kappa de Cohen.

Resultados

El porcentaje de concordancia fue del 11,1%. Los coeficientes se clasificaron como bajos o muy bajos, y encontramos una mayor concordancia entre los evaluadores sin experiencia. Por otro lado, no hubo acuerdo/concordancia en/entre el 50% de los evaluadores.

Conclusiones

La baja concordancia entre evaluadores, la especificidad de la PULS y la correlación clínico-patológica sugieren las dificultades del uso de esta escala y una curva de aprendizaje probablemente larga.

Introduction

Ureteroscopy (URS) is an endourological technique that is fundamental for the diagnostic and treatment of ureteral pathologies. URS has advantages over extracorporeal shock wave lithotripsy (ESWL) in the management of lithiasis lodged in the distal ureter and in the need for a decreased number of sessions.¹ Nevertheless, URS is associated to the lengthening of the hospital stay and a greater number of complications.²

The classifications of the lesions have two fundamental objectives. On the one hand, they allow for homogenizing the treatment each lesion receives, and on the other hand, they facilitate the comparison of the results of the different groups when the results are shared.

Ureteral lesions, which may have resulted from the application of endourological techniques, are classified by the American Association for the Surgery of Trauma (AAST) according to the Organ Injury Severity Scale for the Ureter.³ The ureter lesions can also be classified according to the treatment provided to manage them, as done by the Dindo-modified Clavien Classification.^4,5 Just as the AAST scale, it is a valid scale for all the ureter lesions produced by external or iatrogenic trauma.

In 2012, Schoenthaler et al. developed the Post-Ureteroscopic Lesion Scale (PULS). This scale intended to classify the ureteral lesions that existed after an ureteroscopy, independently of their existence before it. It associated a degree of lesion with the need or not of placing a ureteral stent and also the duration of the stenting.^6–8 The PULS offers 6 degrees, from 0 to 5, as shown in Supplementary Fig. 1.

Since the date it was published, the use and presence of the PULS has gradually increased in clinical research of lesions after an ureteroscopy, as a tool to measure the lesions produced after this procedure. According to the Web of Science, the increase of the citations with respect to this scale has been constant, from 2 citations in 2015 to 61 on September 2020. In addition, with the development of flexible endoscopes and ureteral access sheaths (UAS), the PULS traditionally has been used in the assessment of injuries related to UAS placement.^6,9,10

In 2014, the authors showed a certain inter-rater agreement of the scale with Kendall’s W test working with 37 evaluators and 100 videos of procedures.⁷ In 2018, the authors of the PULS studied the agreement between the assistance personnel and surgeons. However, this overall agreement was put into question as each patient was evaluated by a unique assistant and a unique surgeon, and not by their entire set.¹¹

Nowadays, there are better indicators for analyzing agreement between raters other than Kendall’s W test and value.¹² Based on Cohen’s Kappa for evaluating the agreement between 2 raters, Fleiss’ Kappa¹³ or Light’s Kappa¹⁴ can be implemented, and could be applicable if there is a situation where all the raters evaluate all the cases. Another interesting indicator of agreement between raters is Krippendorff’s Alpha.¹⁵

Therefore, the objective of this study is to analyze the inter-rater agreement of the use of PULS in an experimental model for its application on the clinical setting.

Material and methods

Videos

The videos correspond to an experimental study on 14 Large White breed female pig animals, weighing between 30 and 35 kg. Males were avoided as they tend to have a particular anatomical conformation, which makes them inadequate for endourological studies. The pig’s kidneys are multipapillar, just as humans. The pig models that weigh between 30−35 kg are similar in size to a human who weighs 70 kg approximately. Also, the ureter has a length of approximately 22−26 cm and an internal diameter of 4–6 Fr, which is very similar to that of a human.¹⁶ Anomalies were discarded under general anesthesia. The experimental phase was conducted by two endourologists who had experience in both ureteroscopy and micro-ureteroscopy (m-URS). Each surgeon performed 7 ureteroscopies and 7 micro-ureteroscopies.¹⁷

The videos showed 14 URS and 14 m-URS. Each video lasted approximately 30 s, and included the final step of the ureteroscopy, which showed the 17–18 distal centimeters of the ureter. Of the 28 initial videos, the study of one of the procedures conducted was excluded due to problems in the viewing corresponding to a m-URS.

This study was approved by the Institutional Ethics Committee for Animal Research. During the experimental phase, the animal experiment care guidelines as well as the European Union guidelines regarding the protection of animals with scientific aims were complied with.

Evaluators

Sixteen urologists were selected (9 of them endourologists) and 4 residents. This is the sample size of the study since our aim was to analyze the agreement between their responses. All the participants received the same instructions for the classification of lesions (Supplementary Fig. 1). These instructions were extracted from the studies published by Schoenthaler et al.^6,7 Some endourologists were already familiarized with the scale.

Evaluation

The observers were given a questionnaire to evaluate the degree of ureteral lesions. In this questionnaire, a single response was consigned to each of the videos. There were 6 options available, given that the PULS scoring was between 0 and 5.⁷

To categorize the perforating lesions (degree > 2), an anterograde pyelogram needs to be performed with verification with fluoroscopy. However, in our study this could not be performed, so that the degree > 2 lesions were ultimately denominated “degree ≥ 2”.

Other variables. Pathological assessment

Besides the videos, an anatomopathological analysis was conducted. The experimental study was completed by removing the urinary tract en bloc for pathologic study. The pathologist, blinded to the animal’s group, performed all histopathological ureteral evaluations. A hematoxylin and eosin stain was used on the samples, and a validated healing score was used with 5 parameters were classified, scored from 0 to 3, where 0 was equivalent to not showing any changes, and 3 showing severe changes.¹⁷ These parameters were: inflammation, lamina propia fibrosis, muscular layer fibrosis, muscle integrity and serous membrane alterations. By the results from the anatomy of the pathology, each video was classified according to the PULS. Histology scores were used a reference to establish a correspondence between the PULS scores and the microscopic ureteral damage.

Data analysis

To analyze the agreement, the percentage of agreement, Kendall’s W and its significance, and the agreement indicators Fleiss’s and Light’s Kappa and Krippendorff’s Alpha were calculated. To evaluate the agreement with Fleiss’s and Light’s Kappa, values between 0.21 and 0.40 were considered low, between 0.41 and 0.60 normal, 0.61 and 0.80 good and more than 0.81 excellent.¹⁸ For Krippendorff’s Alpha, values of less than 0.67 were considered bad agreement, between 0.67 and 0.80 considerable agreement, and more than 0.80 positive agreement.¹⁵

As agreement between two raters, Spearman’s correlation coefficient and Cohen’s Kappa were calculated, and these were interpreted in the same scale as those from Fleiss’s and Light’s Kappa. Also, the percentage of peers who were significantly in agreement were calculated for both indicators.

Results

The indicators calculated can be observed in Table 1. The percentage of agreement was 11.1%, although this value is an overestimation of the agreement, as it does not consider agreement due to chance alone. Also, Fleiss’s Kappa, as well as the correction by Light for complete models, does not exceed 0.40 for all the raters, and only comes close to 0.40 among the newest raters, who showed more agreement amongst themselves. Krippendorff’s Alpha values were classified as “very low” for all the raters in the different groups, with greater agreement again shown for the most inexperienced. The classic Kendall’s W coefficient was significant as expected, although its values were not excessively high.

Table 1.

Agreement.

Indicators	Total	Beginners	Expert endoscopists	Others
Number of Raters	20	4	9	7
% Agreement	11.1%	51.9%	37%	22.2%
Fleiss’ Kappa	0.24**	0.37**	0.25**	0.17**
Light’s Kappa	0.26	0.4	0.26	0.20
Krippendorff’s Alpha	0.24	0.37	0.25	0.17
Kendall’s W Coefficient by pairs	0.40**	0.67**	0.43**	0.45**
% Significance Cohen’s Kappa	57.2%	50.0%	50.0%	52.4%
% Significance Spearman Correlation	54.4%	100%	58.3%	61.9%

**

Significance < 0.01.

As for the agreement by peers, it can be observed that for Cohen’s Kappa, as well as for Spearman’s correlation, the percentage of rater pairs with these non-zero indicators revolved around 57.2% and 54.4% for the set of evaluators (Table 2). However, in the case of Cohen’s Kappa, and following the classification proposed, only 8 pairs of raters exceed values of 0.60 considered to be good agreement (Table 2). Following this criterion, only raters 1, 2 and 16 would have a certain agreement between them, rater 20 with raters 4 and 12, and rater 11 with rater 17 and 1, and rater 6 with 19. Therefore, according to Cohen’s Kappa, 10 of the 20-rater polled were not notably in agreement with the rest of their peers.

Table 2.

Cohen’s Kappa and Spearman’s Correlation among raters.

Spearman’s correlation	Cohen’s Kappa
	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	R11	R12	R13	R14	R15	R16	R17	R18	R19	R20
R1		0.763*	0.252	0.287	0.329*	0.140*	0.232	0.330*	−0.024	0.428*	0.618*	0.301*	0.369*	0.028	0.338*	0.614*	0.563*	0.217*	0.140*	0.344*
R2	0.774*		0.327*	0.491*	0.329*	0.140*	0.232	0.330*	−0.024	0.428*	0.428*	0.301*	0.369*	0.028	0.338*	0.614*	0.563*	0.217*	0.140*	0.563*
R3	0.667*	0.594*		0.278	0.391*	0.121*	−0.104	0.359*	−0.062	0.361*	0.289	0.151	0.256*	0.136	0.359*	0.163	0.340*	0.204	0.121*	0.193
R4	0.423*	0.623*	0.397*		0.285	0.221*	0.032	−0.055	0.244	0.382*	0.206	0.314*	0.156	0.049	0.133	0.139	0.320	0.137	0.221*	0.709*
R5	0.455*	0.455*	0.403*	0.293		0.150*	0.184	0.304	0.137	0.536*	0.536*	0.201	0.241*	0.281*	0.345*	0.224	0.184	0.197	0.150*	0.184
R6	0.485*	0.485*	0.368	0.429*	0.384*		0.115	0.081	0.377*	0.192*	0.192*	0.481*	0.377*	0.258*	0.029	0.176*	0.258*	0.077	1.000*	0.258*
R7	0.268	0.268	−0.025	0.165	0.425*	0.452*		0.265	0.182	−0.007	−0.007	0.247	−0.034	−0.112	0.002	0.296	0.076	0.147*	0.115	0.076
R8	0.376	0.376	0.392*	0.072	0.374	0.411*	0.303		−0.096	0.408*	0.408*	0.166	0.318*	0.013	0.530*	0.400*	0.359*	0.252	0.081	−0.007
R9	0.143	0.143*	−0.043	0.295*	0.189	0.530*	0.338	0.048		0.200	0.000	0.524*	0.161	0.073	−0.182	0.010	0.061	0.225*	0.377*	0.530*
R10	0.568+	0.568*	0.453*	0.411	0.520*	0.411*	0.124	0.513*	0.258		0.491*	0.477*	0.513*	0.013	0.330*	0.303*	0.449*	0.314*	0.192*	0.449*
R11	0.759*	0.568*	0.515*	0.241*	0.520*	0.411*	0.124	0.513*	0.048	0.513*		0.267*	0.513*	0.372*	0.464*	0.303*	0.633*	0.314*	0.192*	0.265
R12	0.485*	0.485*	0.237	0.390	0.290	0.600*	0.432*	0.356	0.554*	0.593*	0.356		0.215	0.125	0.019	0.368*	0.369*	0.175*	0.481*	0.621*
R13	0.646*	0.646	0.395*	0.311	0.384*	0.530*	0.106	0.451*	0.187	0.677*	0.677*	0.247		0.314*	0.172	0.301*	0.540*	0.191*	0.377*	0.196
R14	0.299	0.299*	0.218	0.193	0.378	0.451*	0.015	0.117	0.106	0.151	0.494*	0.170	0.392*		0.164	0.189	−0.011	0.114	0.258*	0.090
R15	0.501*	0.501*	0.563*	0.278	0.403*	0.372	0.056	0.628*	−0.110	0.481*	0.628*	0.207	0.460*	0.385*		0.276*	0.337*	0.186	0.029	0.071
R16	0.638*	0.638*	0.303	0.295	0.377	0.530*	0.338	0.468*	0.188	0.468*	0.468*	0.554*	0.447*	0.328	0.460*		0.416*	0.106	0.176*	0.182
R17	0.690*	0.690*	0.487*	0.352	0.206	0.452*	0.212	0.482*	0.107	0.482*	0.661*	0.432*	0.550*	0.015	0.542*	0.569*		0.330*	0.258*	0.382*
R18	0.419*	0.419*	0.287	0.183	0.144	0.371	0.292	0.367	0.393*	0.367	0.367	0.371	0.393*	0.392*	0.353	0.510*	0.447*		0.077*	0.330*
R19	0.485*	0.485*	0.368	0.429*	0.384*	1.000*	0.452*	0.411*	0.530*	0.411*	0.411*	0.600*	0.530*	0.451*	0.372	0.530*	0.452*	0.371		0.258*
R20	0.479*	0.690*	0.339	0.725*	0.206	0.452*	0.212	0.124	0.569*	0.482*	0.303	0.693*	0.355	0.242	0.218	0.338	0.409*	0.447*	0.452*

R1-R4, Beginner Raters; R5-R13, Expert endoscopists; R14-R20, Other Urologists.

*

Significance < 0.05.

According to the pathological study (Table 3), 12 of the 27 videos (44.4%) had a degree 1 in the PULS, with the rest having a degree 0. Comparing with what was stated by the raters, substantial discrepancies were observed, especially when the pathological score was classified as degree 1. In this case, only 23.5% of the classifications made by the raters agreed, with most of the discrepancies due to under-estimation (67.5%), and only 8.7% due to over-estimation. This agreement increased to 80.7% when it dealt with degree 0 for the classification made through with the pathological score, resulting in an overestimation of 19.3% (Table 3).

Table 3.

Classification of the videos by the raters and by the Anatomy of the pathology. Percentage of agreement between evaluators and Anatomy of the pathology.

	0	I	II o + II	A.P.
Video	% (n)	% (n)	% (n)	Epithelial	Muscular	PULS (A.P)	% A.A.P
1	90 (18)	5 (1)	5 (1)	1	0	0	0.9
2	100 (20)			1	0	0	1
3	100 (20)	.	.	1	0	0	1
4	65 (13)	35 (7)	.	2	0	1	0.35
5	75 (15)	25 (5)	.	2	0	1	0.25
6	65 (13)	35 (7)	.	3	0	1	0.35
7	40 (8)	60 (12)	.	3	0	1	0.6
8	95 (19)	5 (1)	.	1	0	0	0.95
9	95 (19)	5 (1)	.	1	0	0	0.95
10	95 (19)	5 (1)	.	1	0	0	0.95
11	85 (17)	15 (3)	.	2	0	1	0.15
12	25 (5)	75 (15)	.	1	0	0	0.25
13	80 (16)	20 (4)	.	1	0	0	0.8
14	.	.	100 (20)	3	0	1	0
15	70 (14)	25 (5)	5 (1)	2	0	1	0.25
16	75 (15)	25 (5)	.	1	0	0	0.75
17	70 (14)	30 (6)	.	3	0	1	0.3
18	65 (13)	35 (7)	.	1	0	0	0.65
19	80 (16)	15 (3)	5 (1)	1	0	0	0.8
20	80 (16)	20 (4)	.	3	0	1	0.2
21	100 (20)	.	.	1	0	0	1
22	80 (16)	20 (4)	.	2	0	1	0.2
23	90 (18)	10 (2)	.	2	0	1	0.1
24	90 (18)	10 (2)	.	2	0	1	0.1
25	90 (18)	10 (2)	.	1	0	0	0.9
26	40 (8)	45 (9)	15 (3)	1	0	0	0.4
27	80 (16)	20 (4)	.	1	0	0	0.8
						Mean	0.55

A.P., Anatomy of the pathology; % APA Percent of Agreement Anatomy of the pathology. Degree = 0, I, II or +II.

With respect to the sensitivity and the specificity registered in the set of the raters for diagnosing a degree 0 with the PULS, 242 positive results were obtained (Classified as Degree 0) from 300 visualizations of Degree 0 videos according to the pathological score, and 162 positive results (classified as Degree 0) from 240 videos of Degree 1 according to the anatomy of the pathology. Therefore, a sensitivity of 0.807 and a specificity of 0.325 were obtained. The predictive values are shown in Fig. 1. A skewed behavior can be observed for both predictive values, almost without a curve.

Figure 1.

Positive and negative predictive values for prevalence of grade 0.

Discussion

The PULS intends to be a simple and understandable classification system not only for medical specialists but also for medical residents. Under the premise of being an intuitive and simple scale, a high level of agreement is sought to make it highly reproducible.

The authors of the scale limit its utility to three aspects: the need or not of a ureteral stent, recommended duration of the stenting or the need of immediate repair of the ureter due to a serious lesion. Thus, the PULS is considered a scale that contributes complementary to other existing scales such as the AAST, the modified Clavien-Dindo system or the Satava grading system.^19,20 The study of the usefulness and reliability of this scale is of great interest, as ureter stents decrease the quality of life of up to 80% of the patients.²¹

The results of the inter-rater agreement reported by Schoenhaler et al.⁶ are limited to Kendall’s W and its significance. Knowing that Kendall’s W test can provide a contrast if there is at least one rater who agrees with another from the group, an increase in the number of raters would increase the probability of obtaining significant results, so that the value of Kendall’s W would be more interesting than its significance. The values obtained in this study are also significant but slightly inferior. Nevertheless, a scale has not been developed to evaluate agreement as a function of the value of Kendall’s W. Therefore, other indicators were utilized, such as the ones presented in the study, Fleiss’ Kappa, Light’s Kappa o Krippendorff’s Alpha, with all of them showing a low or slight agreement, for the set of raters whole as well as the 3 groups classified.

Other indicators according to rater pairs such as Cohen’s Kappa and Spearman’s correlation also coincided in the low agreement between viewers. That 50% of the viewers did not obtain values for Cohen’s Kappa above 0.6 with the rest of the raters corroborated the lack of agreement in the PULS scale.

The first study in which the results from the PULS were correlated with histopathological findings was developed by Lildal et al.²² The results of this experimental study in a porcine model for retrograde intrarenal surgery (RIRS) with the introduction of a ureteral access sheath showed that the PULS underestimated the ureteral lesions in at least 1 degree with respect to the histopathological findings. Bozzini et al.⁹ revealed in their study that there is no increased number of ureteral injuries in RIRS using UAS in comparison with not using it. Injury assessment was based on the PULS. According to our study, the results of Bozzini et al. could have a different meaning according to the low inter-rate agreement.

When comparing the results of the raters with the histopathological ones, the underestimation that reduced the degree 1 to degree 0 would result in the lack of care for 67.5% of the patients who would truly need it. And the over-estimation of degree 0 to degree 1 that would be produced in 19.3% of the cases would result in the over-treatment, with possible secondary discomforts due to the ureteral stent (hematuria, lower urinary tract symptoms, infections, etc.).⁶ Considering this under-rating of ureteral lesions, the inefficiency of the PULS to differ between degrees 0 and 1 and that Schoenthaler et al. suggest that degrees from 0 to 2 should not be reported as post-surgical complication, PULS may represent a limited tool for reporting complications related to ureteroscopic procedures.⁷

One of the main drawbacks of the PULS is that it does not consider the state of the ureter at the beginning of the URS. Patients that have undergone previous endourological procedure or an ESWL on the same ureter, the categorization of the post-ureteroscopy lesion may be over-estimated when using PULS.

Finally, we would like to consider some points that could improve the concordance in evaluation of ureteral lesion using the PULS. First, it is necessary to develop a training and monitoring program to enable urologists to learn how to use the PULS correctly. It is important to generalize the use of pyelography for any suspected ureteral injury during ureteroscopy, and to know to evaluate the severity of the lesion in relation to the amount of extravasated contrast. In relation to this, it could be helpful the application of technology to automatize and reduce the error risk during the evaluation process. The development of a software tool that allows the comparison of pyelography images could guide the impartial assessment of the lesion.

Limitations

Despite the results regarding the concordance between histological and clinical features and their support to clinical findings, we must consider some limitations in this aspect.

First, we don’t have information about the previous histological status of the ureteral tissues, which would allow us to analyze what changes are generated by the intervention itself.

On the other hand, histopathological analysis was exhaustive but was focal too, and therefore it is a transverse or static technique, while the visualization of the ureter through videos is considered a longitudinal or dynamic technique. This is the reason why, in the cases where the PULS could overestimate the results or the sensitivity and specificity of the scale, we cannot be sure that they differ due to the lack of the analysis of the ureteral portion that could be damaged.

Finally, the histopathological analysis cannot distinguish between scores 3, 4 or 5. The difference between the grades are relevant in clinical practice because of the different therapeutic requirements.

Conclusion

In our study, the Post-Ureteroscopic Lesion Scale (PULS) had a low inter-rater agreement, a scarce correlation with the histopathological findings and a low specificity. Furthermore, its suitable use may imply a long learning curve, and may not be as easy to use as the authors advocated a priori.

Funding

This study has no funding.

Compliance with ethical standards

The authors declare we have no financial interests. No funding was received to assist with the preparation of this manuscript. The videos correspond to an experimental study of 14 white female pigs and had the approval of the ethics committee for studies with animal experimentation [15]. The evaluators participating in the study signed an informed consent form to use their judgments. Because of the characteristics of the study, no ethics committee approval is required.

Ethics approval

Data used comes from secondary sources, so it does not require the approval of an ethics committee.

Authors’ contribution

Rebeca Polo: Protocol/project development, Data collection.

Àngela Canós-Nebot: Project development, Manuscript writing.

Juan Pablo Caballero-Romeu: Project development, Manuscript writing, Management.

Juan Antonio Galán-Llopis: Project development, Manuscript editing.

Pablo Caballero: Data analysis, project development, Manuscript writing.

Federico Soria: Data collection, Manuscript editing.

Julia E. De La Cruz-Conty: Data collection.

Jose Tuells: Project development, Manuscript editing.

Conflicts of interests

On behalf of the authors, I declare no conflict of interest.