Inter-observer variability of the traffic rate system (law 35/2015) and utility of biomechanical tests in the assessment of sequels

Abstract

Introduction

Law 35/2015 assesses damages in traffic accidents and tries to guarantee an equal response to identical situations. The biomechanical functional assessment (BFA) is a complementary medical test that provides objectivity in the evaluation of post-traumatic neck pain. This study analyses the interobserver variability of the assessment system defined by Law 35/2015 and studies the effect of having BFA tests and the profile of the evaluator in determining sequelae.

Materials and methods

To do this, 49 professionals in the assessment of bodily injury evaluated 5 clinical cases of accident victims with post-traumatic neck pain; a month later, 35 of them assessed the same cases including a BFA report.

Results

The results show high variability in determining the days of personal injury (Kappa coefficients between 0.04 and 0.073) with or without BFA; high interobserver variability in the assessment of sequelae in cases without BFA (Kappa coefficients between 0.022 and 0.044), which slightly improves with BFA (Kappa coefficients between 0.128 and 0.26), even showing weak concordance. The BFA has an influence on the determination of sequelae (P < .01), but the profile of the evaluator does not. More than 79.4% of the evaluators found the BFA tests useful to reveal or to confirm symptoms, recovery, or simulation

Conclusions

There is variability in the application of Law 35/2015 for the assessment of traffic accidents among professionals of bodily injury. The BFA is useful for evaluators and influences the determination of sequelae.

Resumen

Introducción

La Ley 35/2015 valora daños y perjuicios en accidentes de circulación y trata de garantizar una respuesta igualitaria ante situaciones idénticas. La valoración funcional biomecánica (VFB) es una prueba médica complementaria que aporta objetividad en la evaluación de la cervicalgia postraumática. Este estudio analiza la variabilidad interobservador del sistema de valoración de la Ley 35/2015 y estudia el efecto de la VFB y del perfil del evaluador en la determinación de secuelas.

Material y Métodos

Para ello, 49 profesionales de la valoración del daño corporal evaluaron 5 casos clínicos de accidentados con cervicalgia postraumática; un mes más tarde, 35 de los anteriores valoraron los mismos casos incluyendo informe de VFB.

Resultados

Hay una elevada variabilidad en la determinación de los días de perjuicio personal (coeficientes Kappa entre 0,04 y 0,073) con o sin VFB; elevada variabilidad interobservador en la valoración de secuelas en los casos sin VFB (coeficientes Kappa entre 0,022 y 0,044), que mejora discretamente con VFB (coeficientes Kappa entre 0,128 y 0,26), aun mostrando concordancia débil. El resultado de VFB tiene influencia en la determinación de secuelas (p < 0,01), pero no el perfil del evaluador. Más del 79.4% de los evaluadores encontraron útiles las pruebas para poner de manifiesto o confirmar sintomatología, recuperación, o simulación

Conclusiones

Existe variabilidad en la aplicación de la Ley 35/2015 para la valoración de los accidentados de tráfico con cervicalgia postraumática entre los profesionales del daño corporal. La VFB resulta de utilidad para los evaluadores y tiene influencia en la determinación de secuelas

Introduction

The system for the evaluation of damages caused to persons in traffic accidents, established in Royal Legislative Decree 8/2004,¹ was replaced by Law 35/2015,² which came into force on 1 January 2016. The latter places emphasis on providing certainty to the injured party and the insurers, guaranteeing an equal response to identical situations.³

The capacity of a personal injury assessment system to offer similar results when assessing the same case can be studied by analysing its reproducibility, corresponding to the capacity of a test to be replicated or to offer an identical result under equivalent measurement conditions.⁴

However, agreement between medical decisions has been the subject of studies in different fields, with unequal results depending on the field.^5–7 The reasons for the low agreement found in some cases include factors such as training in the use of the tool (related to the professional profile of the assessor) or failures of the assessment system.⁸

Regarding the reproducibility of personal injury assessment systems, the study by Ordóñez-Mayán⁹ on the previous assessment system, established in RLD 8/2004, stands out. Contrary to the researchers' proposal that an injury assessment system should be clear, simple, and rigorous, the results of their study found weak reproducibility, i.e., weak concordance between the results obtained by different assessors for the same information. Therefore, although the existence of standard criteria aims to increase concordance, it does not seem to be a sufficient condition.

Although there are studies on variability in relation to the establishment of the causal link under the current legal framework,¹⁰ there are no known studies similar to that of Ordóñez-Mayán et al.⁹ that evaluate the reproducibility of the assessment system of Law 35/2015 for the determination of sequelae. Thus, despite the interest in ensuring equal treatment in this framework, no information is available.

In the 1990s, Biomechanical Functional Assessment (BFA) began to be used in Spain in occupational and forensic medicine to assist in decision-making. Through the use of instrumental techniques, the BFA makes it possible to evaluate functions and activities of the locomotor apparatus, such as ambulation, cervical mobility, etc.¹¹ In 2007, it was extended as a complementary medical test in the assessment of incapacity, and the Directorate for the Regulation of Social Security and the Association of Mutual Insurance Companies for Accidents at Work confirmed its recognition, including it in the catalogue of medical tests and explorations useful in the assessment, review, and qualification of incapacity for work.^12,13

The usefulness of BFA in the medico-legal assessment of neck pain after a traffic accident was demonstrated by Vivas Broseta et al.¹⁴ under the RLD 8/2004 assessment system. In this study, in which 3 forensic physicians from the Institute of Legal Medicine and Forensic Sciences of Valencia (IMLCCFF) assessed 51 injured subjects, the majority of participants found the BFA useful. In fact, they modified their criteria in the determination of sequelae, which was related to a reduction in the overall cost of compensation.

Given the interest in guaranteeing an equal response with the damage assessment system of Law 35/2015, the lack of studies on the reproducibility of this system and the scarcity of data on the effect of BFA tests in medico-legal assessment, we proposed the present research.

The main objectives of this study are to evaluate the inter-observer reproducibility in the application of Law 35/2015, of 22 September, on the reform of the system for the assessment of damages caused to persons in traffic accidents² in cases of post-traumatic cervicalgia, and to study the effect of the use of the BFA in the determination of the sequelae under this law. The secondary objectives are, on the one hand, to study the influence of the assessor's profile on the determination of after-effects under Law 35/2015, taking into account the length of experience in the assessment of the injury and the main area of practice, and on the other, to describe the usefulness of the BFA reports as perceived by the assessors.

In essence, the present project aims to overcome the limitations of the studies by Ordóñez-Mayan et al.⁹ and Vivas Broseta et al.¹⁴ by complementing the findings of these authors in the context of the new Law 35/2015.

Material and methods

For the identification of the participants, professionals from the Institutes of Legal Medicine and Forensic Sciences and from different Bodily Injury Valuation Societies were involved. Thanks to them, 108 doctors with more than 6 months' experience in the field of personal injury assessment and an altruistic interest in participating in the research were identified. Participants from the study by Vivas Broseta et al.¹⁴ were excluded to avoid bias. Participants were informed of the objectives and characteristics of the project, gave their consent and provided their data to be contacted and recruited.

In a manner equivalent to the Ordóñez-Mayán,⁹ participants had to evaluate a series of clinical cases, in our case, 5 in 2 phases (10 evaluations in total). In the first phase, cases were sent excluding BFA reports (cases 1 to 5, SIN_BFA). In the second phase, the same cases were sent including BFA reports (cases 6 to 10, CON_BFA). Since one of the objectives was to assess the influence of BFA on the determination of sequelae, the non-relevant data in the second phase (CON_BFA) were modified to avoid recognising that these were the same cases. Specifically, under the supervision of 2 professionals from the IMLCCFF in Valencia, the name, avatar (fictitious graphic representation of the patient), and profession were modified. In addition, there was at least one month of separation between the phases.

The clinical cases selected by the IMLCCFF in Valencia were real cases with cervical disease after a traffic accident with BFA. Two forensic doctors ensured that the information provided was comprehensible, synthetic, and sufficient to obtain the study variables in the framework of the current scale (Law 35/2015),² including data on the evolution of the process and the symptomatology at the time of the assessment (at the end of the process).

The BFA reports of the CON_BFA cases, carried out by medical experts in biomechanics, contained the results of the tests carried out at the time of being assessed by the IMLCCFF in Valencia. This BFA was performed using NedCervical/IBV,¹⁵ an application that analyses cervical movement, providing a Normality Index (NI), which summarises the degree of functionality, and a Collaboration Index (CI), which assesses the compatibility of the movement with a simulator pattern (Fig. 1). Both are based on comparison with databases of non-functionally impaired individuals, impaired individuals, and individuals simulating the impairment. The reports included final conclusions summarising functionality (normal function/mild functional impairment/mild functional impairment) and collaboration (patient has collaborated/patient has not collaborated), which relates to the compatibility of the effort made with the maximum of the assessed person's capabilities. To improve the comprehensibility of the report, basic information on the assessment methodology and the cut-off points of the final indices (NI and CI) were also included.

Fig. 1.

Image taken during the biomechanical functional assessment test of the cervical spine (above) and NI and CI results as shown in the report provided (below). The cut-off points established for NI and CI are 90% (between 90% and 100% is considered functional) and 50% (below 50% compatible with simulator pattern), respectively.

The BFA results were:

• •

  Case 6 (1 in first phase): NI 77% and CI 57%, compatible with functional impairment and cooperative subject.

• •

  Case 7 (2 in first phase): NI 93% and CI 90%, normal function (with NI close to the lower limit of normality of 90%) and cooperative subject.

• •

  Case 8 (3 in first phase): NI 70% and CI 19%, simulator pattern, i.e., the patient was uncooperative.

• •

  Case 9 (4 in first phase): NI 80% and CI 71%, mild functional impairment and cooperative subject.

• •

  Case 10 (5 in first phase): NI 98% and CI 80%, normal function and cooperative subject.

• •

  To compile the results, a battery of questions with closed response fields was designed. This was entered into the surveymonkey.com web platform, which allows the creation of surveys in a simple way and facilitates the collection and recording of responses through any device with mobile data. The study variables were:

Participant's reference

- Length of experience in damage assessment (6 months to 1 year / 1–5 years / more than 5 years).

- Scope of practice (public / private / both).

- Geographical origin (Autonomous Community of residence and practice).

Referring to the assessment of the case

- Days of personal injury (individual) for moderate, serious, or very serious loss of quality of life.

- Days of basic personal injury due to a temporary injury.

- Sequelae (yes / no).

- Points awarded for the sequel(s) (total sum of points).

Referring to the usefulness of the BFA

They were asked to tick one of the following options:

A. The BFA report has been useful in confirming or highlighting:

• •

  Active symptoms.

• •

  Recovery.

• •

  Simulation.

B. The BFA report was not useful.

For the study of inter-observer variability in the SIN_BFA and CON_BFA cases, the kappa coefficient¹⁶ was used (see Table 1 for the meaning of this coefficient).

Table 1.

Meaning of the kappa coefficient

Kappa coefficient scores	Concordance strength
0	Equivalent to random
0.1–0.20	Poor
0.21–0.40	Weak
0.41–0.60	Moderate
0.61–0.80	Good
0.81–0.99	Very good
1	Perfect

The non-parametric Chi-Square test was used to study the influence of the BFA and the profile of the assessor in the determination of the sequel/s points. For this purpose, the data were previously adjusted using the Cumulative Link Model¹⁷ methodology, which allows working with ordinal variables. For the BFA, a redesign of the model was also applied, considering that the NI and CI interact (cases with a CI below 50% have a NI below 90%). Statistical significance was established according to a 95% confidence interval (P < .05).

Results

Participant characteristics

Of the 108 doctors who agreed to be contacted, 49 took part in the first phase (the first 5 SIN_BFA cases) and 35 also took part in the second phase (CON_BFA cases).

Of the latter 35, 57.14% came from Institutes of Forensic Medicine and Forensic Sciences (public sector), and 42.86% were mainly in private practice. All had more than 5 years of experience in personal injury assessment except 2: 1 less than 5 years, and 1 between 6 months and 1 year.

In order of frequency, there were 7 participants from the Community of Valencia, 4 from Catalonia, 3 from the Community of Madrid, 3 from the Canary Islands, 3 from Andalusia, 2 from the Basque Country, 2 from Aragon, and 1 participant from each of the following: Murcia, Galicia, Ceuta, Cantabria, Castilla La Mancha, and the Balearic Islands.

Interobserver variability

The results related to variability are summarised in Table 2. Kappa coefficients less than 0.1 (concordance strength equivalent to chance), represent high variability in the "Days of (particular) personal injury for moderate, severe or very severe loss of quality of life" (DPP_CV) and in the "Days of basic personal injury for temporary injury" (DPP_LT), in the first (SIN_BFA cases) and in the second phase (CON_BFA cases). The kappa coefficient was also less than 0.1 for the proposed sequel (yes/no) and the number of points awarded for sequel/s in the first phase (SIN_BFA).

Table 2.

Variability in the proposed Days of personal injury (individual) for moderate, severe or very severe loss of quality of life (DPP,_CV), Days of basic personal injury for temporary injury (DPP,_LT), proposed existence or non-existence of sequelae and in the total sequelae points, calculated through the kappa coefficient.

	DPP_CV	DPP_LT	Existence of sequelae (YES/NO)	Sequelae points
kappa C.phase 1 (SIN_^aBFA)	0.073	0.060	0.044	0.022
kappa C.phase 2 (CON_^aBFA)	0.068	0.040	0.260	0.128

a

BFA = biomechanical functional assessment.

For the sequelae proposal (yes/no) and the number of points awarded for sequela/e in the second phase (CON_BFA), the kappa coefficient was 0.260 ("weak" correlation) and 0.128 ("poor" correlation), respectively. This increase in the kappa coefficient compared to the first phase is related to the increase in agreement in 3 of the 5 clinical cases (CON_BFA): cases 6, 8, and 10.

Influence of the BFA and valuer profile

There is a positive trend in the awarding of sequela points in cooperative subjects with limited functionality in the BFA (lower NI more sequela points if the CI is greater than 50%), and negative in cases of non-cooperation, regardless of NI (variable "NI < 90 & CI < 50", P = .00; Table 3). That is, if the CI is low (less than 50%: non-collaboration), fewer points are awarded regardless of the NI. Conversely, if the CI is higher than 50% and the NI lower than 90% (cooperative patient with impaired functionality), more points are awarded.

Table 3.

Relationship of the Normality Index, Collaboration Index and assessor profile to the sequelae points awarded. In the column "Estimate" sign (-) when the points in the second phase decrease with respect to the first phase and vice versa.

Variable	Estimate	Standard error	Level of significance
^aNI = o > 90	− 1.88	0.40	0.00
^bCI = o > 50	3.41	0.71	0.00
^cNI < 90 & IC < 50	− 1.53	0.71	0.03
^dNI < 90 & IC = o > 50	1.88	0.40	0.00
^ePub/Priv profile	− 0.01	0.37	0.98

a

Normality Index greater than or equal to 90 is equivalent to normal function and collaborating subject.

b

Collaboration Index greater than or equal to 50 is equivalent to a collaborating subject.

c

Normality Index less than 90 with Collaboration Index less than 50, equals functional impairment with simulator pattern.

d

Normality Index less than 90 with Collaboration Index greater than or equal to 50, equals functional impairment in a collaborating subject.

e

Pub/Priv profile = profile of the assessor, assesses whether there is a relationship between both profiles (public or private) and the points awarded for sequelae.

There is no relationship between the scope of the exercise (public or private profile) and the sequelae points (Table 3). It was not possible to study the influence of the exercise time of the assessor because all but 2 participants belonged to the same group.

Usefulness of the BFA

The majority of participants found the BFA report useful. The percentages for each case and the reasons can be seen in Table 4.

Table 4.

Perceived usefulness of the assessors in each of the clinical cases with biomechanical functional assessment.

	Case 6	Case 7	Case 8	Case 9	Case 10
Useful for confirming or revealing active symptomatology	77.14%	25.72%	3.03%	63.64%	29.41%
Useful for confirming or revealing recovery	2.86%	62.86%	3.03%	15.15%	50%
Useful for confirming or revealing simulation	8.57%	5.71%	90.91%	6.1%	0%
Not useful	11.43%	5.71%	3.03%	15.15%	20.59%

Discussion

Few research studies the concordance in the assessment of bodily injury in traffic accidents, and none assesses the influence of the BFA on such concordance under Law 35/2015. The design of this study has made it possible to evaluate both aspects. Therefore, despite the limitations, we consider the results found to be relevant. These limitations are mainly related to the possibility of having missed some relevant data in the process of selection and preparation of the clinical cases, and to the fact that the BFA was performed in the final phase of the sequelae, not influencing the variables related to the healing process.

In relation to the number of participants, we consider our sample to be adequate, exceeding that of the study by Ordóñez-Mayán et al.,⁹ which included 24 observers assessing a single case. Even so, we do not rule out the possibility that some of the professionals initially contacted did not participate due to disagreement with some of the objectives or with the proposed methodology, which may have biased our results.

There is great variability in the assessment of "Days of (particular) personal injury due to loss of quality of life" and "Days of basic personal injury due to temporary injury in both phases" (SIN_BFA and CON_BFA). The lack of influence of the BFA here could be related to the fact that it was done at the end of the process, not providing information on the previous functional status. The inter-observer variability in the determination of sequelae (yes/no) and the points awarded in the first phase (SIN_BFA) is also very high. It must be borne in mind that medico-legal decision-making is subject to multiple external sources of error and variability, dependent on the assessment system or the assessor himself. Even if the necessary examinations are carried out, decisions in medicine will always be made under conditions of uncertainty.¹⁸ This affects multiple areas of medicine, such as radiology, where high interobserver variability has been reported in tests such as MRI for the diagnosis of brachial plexus avulsions,⁷ assessment of the lumbar spine,⁶ or in the interpretation of abdominal X-rays.¹⁹ Other studies report high variability between professionals with different profiles for a complementary test^20,21 or clinical scale, e.g., in some sections of the Barthel index.⁵ In our study, we ruled out differences in the profile of the assessors as a source of variability, given the results of the statistical analysis and the fact that practically all participants had similar length of experience. On the other hand, we cannot rule out that some aspects related to our methodology may have influenced variability. Along these lines, several participants missed data related to injury stabilisation or complementary tests performed, among other aspects.

The kappa coefficients of our study are lower than those obtained by Órdoñez-Mayán et al.,⁹ which means that under Law 35/2015 the overall concordance was somewhat lower than under the RLD 8/2004¹ system. In any case, this comparison should be made with caution, as these are different assessment systems that have evaluated different clinical cases. Furthermore, neck pain after a road traffic accident is one of the most controversial injuries in the field of clinical assessment and personal injury,²² with the complexity that this entails in relation to the assessment of the injury.

It should be noted that the results of the study by Órdoñez-Mayán et al.⁹ were not good either (overall kappa coefficient of 0.37, weak agreement). As in our study, this could be related to differences in the criteria of the assessors or to the inaccuracy of the scales which, although they attempt to systematise the quantification of harm, are still imperfect due to the difficulty of assessing some constituent elements of harm.²³ So, although these systems have the advantage of unifying criteria, it is inevitable to find a certain margin of uncertainty.²⁴ This is even more evident in scales with a greater component of subjectivity, such as those for the valuation of aesthetic damage, which have been called into question on equivalent grounds.²⁵

A certain increase in agreement has been found when BFA reports are provided in the determination of the sequelae. In addition, there is a statistically significant relationship between the results (NI and CI) and the points awarded for sequela/s, in line with what has already been reported by Vivas-Broseta et al.¹⁴ However, on that occasion the correlation was not statistically significant, in the context of an insufficient sample size. In our study, with a larger number of participants, this has been corroborated in a statistically significant way. Therefore, the BFA could be useful in improving settlement, being postulated as one of the complementary examinations useful in the issuing of well-founded medico-legal conclusions, establishing injuries and sequelae, and determining the degree of impairment for usual occupations.²⁶

Most participants found the BFA useful for some reason. Again, this is in line with the findings of Vivas-Broseta et al.,¹⁴ where 98% of the tests were useful to the participants. Unlike that study, our participants were not trained in biomechanical assessment, although basic information about the methodology and its significance was included in the reports themselves. We hypothesise that this methodological change may explain the minimal difference in the perceived usefulness of the BFA (somewhat less in our case). It is striking that the greatest usefulness of BFA was found in the case of CI compatible with non-collaboration (CI < 50%). This results in less variability in the assessment of this particular case and highlights the importance of the simulation diagnosis. We relate this to the prevalence of symptom exaggeration in the context of neck pain, up to 60% in post-traumatic neck pain and around 50% in chronic neck pain.²⁷

In short, given the scarcity of reliable instrumental resources to objectify temporary incapacity and sequelae, biomechanical tests are a useful complementary tool to assess the residual situation of the subject, supporting the observer in making decisions on functionality and in assessing the degree of collaboration of the examinee.

The conclusions reached are as follows:

• 1.

  The inter-observer variability of the determination of (particular) personal injury days due to moderate loss of quality of life and of basic personal injury due to a temporary injury was very high in cases without and with BFA.

• 2.

  The inter-observer variability in the determination of the existence or not of -sequelae/s and in the awarding of points for them was very high in the cases without BFA, decreasing slightly with the use of BFA.

• 3.

  The points awarded for sequelae in cases with BFA were statistically significantly related to the results of the evaluation of the BFA, being reduced in cases of absence of functional alteration and non-cooperation.

• 4.

  There was no influence of the profile of the assessor (in this case the field of public or private practice) on the points awarded for sequelae.

• 5.

  Most of the assessors found the BFA useful for the evaluation of clinical cases, as it revealed or confirmed symptomatology, recovery, or simulation.

In future studies, we propose to evaluate the influence of the BFA performed at various evolutionary moments of the healing process, to include other types of BFA (balance or hand strength), and to ensure that there is no lack of data of interest for the evaluation of the case according to the scale. In addition, it would be advisable to extend the study with an economic cost analysis, calculating the savings in relation to the sequelae points in cases without and with BFA.

Funding

The activity was carried out within the framework of the IBERUS project, Biomedical Engineering Technology Network applied to degenerative pathologies of the neuromusculoskeletal system in clinical and out-of-hospital environments (CER-20211003), CERVERA Network funded by the Ministry of Science and Innovation through the Centre for Industrial Technological Development (CDTI), charged to the General State Budget 2021 and the Recovery, Transformation and Resilience Plan. Part of this research has been carried out within the framework of the Mov-Hum Project (Generation of criteria for the assessment and design of products based on the study of biomechanical models of the upper limb and spine), which has the support of the Valencian Institute for Business Competitiveness (IVACE) and 50% co-financing by the European Union through the European Regional Development Fund (ERDF), within the ERDF Operational Programme of the Valencian Community 2014–2020, with the file number IMDEEA/2018/89.

Conflict of interests

The authors have no conflict of interests to declare.

Acknowledgements

To the Institute of Legal Medicine and Forensic Sciences of Valencia, for the important work carried out during the project and for their efforts in the dissemination and recruitment of participants-professionals in this field-in the project. To the Basque Society for the Valuation of Bodily Injury and the Spanish Society for the Valuation of Bodily Injury for their collaboration, effort and work carried out, as well as for encouraging the participation of professionals in the valuation of injury in this study. To Dr. Emilio Félix Aranda Uriarte, member of the aforementioned societies, and Dr. Eugenio Laborda Calvo, president of the Spanish Society for the Valuation of Bodily Injury, for their contributions, which have been key to the development of the study and the interpretation of the results. To all the participants, doctors specialising in the assessment of bodily injury of different profiles, without whose effort and work it would not have been possible to carry out this study.