Development and validation of meta-measurement instruments: A methodological approach

Abstract

A valid and reliable nursing assessment is essential for identifying required care and ensuring patient safety. The convenience of conducting a comprehensive assessment of the patient has led to a significant increase in assessment tools that may slow down the process. Nevertheless, the possibility of consolidating various instruments that measure common or similar constructs into a meta-instrument is considered an alternative that could enhance assessment efficiency.

A meta-instrument can be defined as a measurement tool that consolidates other instruments based on measuring related constructs and sharing dimensions or items, aiming to achieve a more parsimonious measurement. Literature on such assessment tools is scarce, and there are numerous options for their construction and initial validation. Additionally, it is advisable to confirm their psychometric properties and ensure that they maintain, at the very least, the same diagnostic capacity as the original instruments.

This article presents a proposal for the phases to follow in constructing meta-instruments, along with various methodological alternatives that can be employed based on the characteristics of the original instruments and the purpose of creating the meta-instrument. Furthermore, special attention is given to the checklists that should be used to study the psychometric properties and diagnostic capacity of the meta-instruments. Finally, future lines of research and challenges in the development of nursing assessment meta-instruments are discussed.

Resumen

Una valoración enfermera válida y fiable resulta imprescindible para identificar los cuidados requeridos y garantizar la seguridad del paciente. La conveniencia de realizar una valoración integral al paciente ha propiciado un considerable aumento de instrumentos de valoración que restan agilidad al proceso. No obstante, la posibilidad de colapsar varios instrumentos que presentan constructos comunes o similares en un meta-instrumento que los integre se plantea como una alternativa que puede facilitar la eficiencia de la valoración.

Un meta-instrumento puede definirse como un instrumento de medida que colapsa otros instrumentos en base a que miden constructos relacionados y comparten dimensiones o ítems, con el objetivo de obtener una medición con un enfoque más parsimonioso. La literatura sobre este tipo de herramientas de valoración es escasa y las opciones para su construcción y validación inicial son amplias. Además, es conveniente confirmar sus propiedades psicométricas y que mantienen, al menos, la misma capacidad diagnóstica que los instrumentos originales.

En este artículo se presenta una propuesta de las fases a seguir para la construcción de meta-instrumentos, así como diferentes alternativas metodológicas que pueden utilizarse en función de las características de los instrumentos originales y el objetivo de la creación del meta-instrumento. Además, se hace especial mención los listados de verificación que será conveniente utilizar para estudiar las propiedades psicométricas y la capacidad de diagnóstica de los meta-instrumentos. Por último, se plantean futuras líneas de investigación y retos en el desarrollo de los meta-instrumentos de valoración enfermera.

Introduction

Undertaking a correct nursing assessment is essential to diagnose and plan the care required by patients, as well as evaluating the effectiveness of interventions. In addition, thanks to that information, nurses can provide conditions of optimal care on the premise of guaranteeing patient safety. To achieve these objectives, nursing assessments must integrate valid and reliable information¹ that can be obtained from various sources, highlighting a wide range of instruments to assess different situations such as, for example, the risk of falls or loss of skin integrity; dependence and frailty; malnutrition; impaired sleep quality; sensory alterations; mental health problems or quality of life issues. among other aspects.

These instruments of assessment have developed exponentially in recent decades.² This development has entailed an increase in the amount of information collected by nurses, which also often implies duplication of documentation,³ leading to redundant nursing assessment.⁴ This causes nurses to use these instruments with a certain level of skepticism, perceiving their use as an administrative burden and a waste of time.⁵ All this translates into inaccurate, routine completion with little participation from nurses, thus returning information that has little validity or reliability for health care.^6,7

It is therefore necessary to offer alternatives to improve nursing assessment through the prism of a comprehensive and structured patient assessment that is more valid, reliable and somewhat more sparing.⁸ To this end, the literature is beginning to offer proposals for new meta-instruments,^4,9 which could be defined as measuring instruments that collapse other instruments based on the fact that they measure related constructs and share dimensions or items, with the aim of obtaining an assessment with a more sparing approach, but at least with the same psychometric properties and diagnostic capability as the original instruments.

Some of these meta-instruments have been proposed as a theoretical exercise,⁴ while others were developed with the aim of increasing the completion rates of nursing records in an attempt to remedy the lack of validity of the information that has an impact on healthcare.⁹ However, there are few experiences presented in the literature and it would be advisable to try to lay down methodological foundations and alternatives that would bring progress in the future development of this type of measuring instruments. Therefore, the objective of this article is to present a methodological approach for the construction and validation of meta-instruments for measurement in the field of the health sciences, in general, and nursing in particular.

Methodological proposal: an overview

By way of synthesis, the first phase consists of establishing the set of constructs to be measured and identifying the profile of the recipients, level of care, or other factors that may influence the selection of the set of measuring instruments that is to be collapsed. The selection of these instruments may be based on their psychometric properties or by creating groups of experts that can be selected on the basis of academic qualifications, experience in design and validation of measurement instruments, or clinical expertise.

As a second phase, we propose analysing the framework of conceptual and semantic relationships between the different dimensions of the instruments and items to detect similarities, duplicities and redundancies that establish direct and indirect conceptual relationships. Different techniques of analysis are proposed and we recommend using broad sampling frames to ensure the representativeness of the population under study. At this point, a descriptive and bivariate analysis must be undertaken to identify and assess possible relationships that should be taken into account in the next phase.

The third phase consists of developing the meta-instrument. To do this, different analytical techniques can be explored, depending on the nature of the items that make up the initial instruments. The objective is to obtain a meta-instrument that is capable of measuring the desired constructs but with the fewest possible items.

Once the new meta-instrument has been built, in the fourth phase initial validation will be undertaken to confirm its accuracy and capacity for discrimination. The text mentions some alternatives, it being necessary to adjust to what is recommended by the analytical technique used in the development of the meta-instrument. Finally, we underline the need to study the psychometric properties of the instrument, as well as its diagnostic capability. Fig. 1 presents a flowchart with the process of development and validation of a meta-instrument.

Figure 1.

Development and validation process of a meta-instrument.

Construction of the meta-instrument

PHASE I: selection of constructs and instruments

Establishing the set of constructs that you want to measure is a good starting point to begin building a meta-instrument. A latent construct or variable can be defined as a specific attribute that cannot be measured directly but can be inferred through a set of observed items, reagents, or variables. This is probably one of the most controversial points in this whole process, since it can be tackled from different approaches. On the one hand, the literature recommends organising nursing assessment using a specific nursing model or framework.⁸ However, there is a long list to choose from and, in addition, the debate as to whether or not theories and models reflect the reality and complexity of care is still open.¹⁰ In addition, it is possible to find literature on meta-instruments⁴ or nursing assessment systems that do not use any nursing framework, model, or theory.¹¹ On the other hand, the selection of constructs can also be based on nursing indicators, quality of care indicators, or results sensitive to nursing practice. Without intending to enter into a discussion about the most appropriate nomenclature, this approach presents a certain complexity since there is no clear consensus as to which indicators really capture the quality of care¹² and the measurement instruments are useful for only some of these.

In both cases, techniques such as nominal groups, focus groups or the Delphi method can be used to reach a consensus on the constructs to be measured and that will be part of the meta-instrument.¹³ In addition, broader methodological frameworks such as the one recommended by the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) initiative can be used to establish sets of main results,¹⁴ including checklists of the methodological quality of validation studies to facilitate the selection of measuring instruments.¹⁵ Along with validity (degree to which a measurement instrument measures the construct it purports to measure); measurement reliability (degree to which the measurement is error-free); and diagnostic accuracy (ability of a test to discriminate between people affected and not affected by the condition of interest). Of the instruments chosen, specific aspects related to the profile of the users, the level of care, the availability of data, professional skills or the institutional context¹⁶ will require to be considered, which may influence the selection of the set of constructs and measurement instruments. For example, retrospective data may be used - or the toolkit may be protocolised - by the health ccentre.¹⁷ It is also possible that the instruments are only available in the original language, making it necessary to adapt them cross-culturally and undertake prior studies for validation and diagnostic accuracy.

PHASE II: analysis of the instruments, constructs and items

Once the constructs to be measured have been decided and the instruments that will be used to construct the meta-instrument have been selected, it is advisable to analyse the conceptual and semantic relationships between the dimensions of the instruments, along with the model, theory or conceptual framework that has been chosen. In addition, analysing the items of the instruments will make it possible to detect similarities, duplications, and redundancies in order to establish direct conceptual relationships (items from different instruments directly linked to the same care) or indirect relationships (relationships of the items with other care to which they are not directly linked).¹⁷ To do this, traditional methods of conceptual and semantic analysis can be used¹⁸ or even natural language processing algorithms can be applied.¹⁹ The results of these analyses of conceptual and semantic relationships will guide - and need to be confirmed by - subsequent statistical analyses.

Retrospective cross-sectional designs based on data from previous studies⁴ or registry data¹⁷ appear to be the most affordable type of study to quantitatively study the relationships between dimensions and items of different instruments, despite the limitation that the availability and quality of data in the medical record may entail.²⁰ We recommend that data be collected prospectively and with trained personnel, although this may require significant resources and time if the sample sizes used in previous studies are taken into account.^4,17

Beyond the classic recommendation of using between 5 and 10 participants per item or at least 100 participants in total,¹⁵ determining the sample size in validation studies is still an aspect to be resolved, with sample sizes ranging from 100 to more than 1000 participants^21,22 and, in addition, no specific recommendations have been made on the sample needed to collapse or unify several instruments into a single one. However, Palese et al.⁴ used a sample of 1464 assessments in their theoretical exercise for a total of 42 items, and Luna-Aleixos et al.¹⁷ used a sample of 1352 evaluations for a total of 21 items. Therefore, we recommend using broad sampling frames that, on the one hand, guarantee the representativeness of the population under study and, on the other hand, create sub-samples and develop analytical strategies to explore the relationships between dimensions and items, build the meta-instrument and analyse its psychometric properties.

When exploring the relationships between dimensions and items, a window of options opens up that ranges from classic analysis of association and correlation, depending on the nature of the variables,· to factor analysis⁴ or Gaussian graphical models.²³ In addition, the possibility of using techniques such as multiple correspondence analysis and cluster analysis to explore possible groupings of items remains open.^24–26 In the absence of a specific recommendation, the choice of these techniques will depend on the type of instruments and items that form part of the analysis, and the expertise and experience of the researchers.

Regardless of the techniques used, it should not be forgotten that a detailed descriptive and bivariate analysis must be run on the sociodemographic, clinical, or other variables included in the study and the instruments and their items, in order to understand the data in depth and identify possible associations that should be included in the next phase of construction of the meta-instrument.

PHASE III: development of the meta-instrument

All the information and prior learning about the operation of the instruments to be collapsed, as well as the interactions with the sociodemographic variables included in the study, will be essential to advance in this phase. It is advisable to remember that the objective is to achieve a meta-instrument that collapses the original instruments with a more sparing approach and this will imply that the number of items will be reduced but ensuring that the same number of constructs will be measured.

To this end, previous studies have used different analytical techniques. For example, Palese et al.⁴ used the classical theory of tests with a logical sequence of exploratory factor analysis and confirmatory factor analysis. However, item response theory offers alternatives such as the Rasch²⁷ analysis, which overcome some limitations of classical factor analyses, since this technique assumes that the relationship between the response to the item and the construct does not necessarily have to be linear: It is capable of detecting differential functioning in the item, based on sociodemographic variables, and can be used with any type of item. These techniques are commonly used in instrument validation studies, however, their objective here is not to analyse the structural validity of a questionnaire but to group the items that are related into dimensions with the intention of being able to select the most representative ones and reduce the total amount to achieve greater sparingity between the instruments.

In another study, the development of a meta-instrument relied on linear models to predict the scores of the original instruments, based on a combination of their own items.¹⁷ However, these models require certain conditions to be met, such as normality of the residuals, homogeneity of variance (homoscedasticity), linearity between the variables and independence of observations. The fulfilment of these conditions should be verified to avoid reaching spurious conclusions and, if these are not met, other extensions of the linear models can be explored according to Table 1. At this point, it should be noted that the TRIPOD statement (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) recommends the use of methods of analysis such as bootstrapping or cross-validation, thus increasing methodological rigour.²⁸

Table 1.

Compliance with properties of linear models.

Models	Normality	Homogeneity	Linearity	Independence
Linear	✓	✓	✓	✓
Generalised Linear	X	✓	✓	✓
Gen. least squares*	✓	X	✓	✓
Additive Models	✓	✓	X	✓
Mixed models	✓	✓	✓	X

*

Generalised least squares (GLS).

In addition, there are previous experiences with other predictive models that should be taken into account for future developments. For example, Chi-squared Automatic Interaction Detection (CHAID models) models are traditionally used in market research to identify customer profiles, based on categorical data and using the chi-square test result. In fact, some authors have already used this type of analysis to reduce the number of items in a questionnaire.²⁹ Similarly, predictive models based on neural networks have been used to predict the onset of Parkinson's disease using questionnaires that measure the intensity of symptoms.³⁰ In this case, no items were eliminated, but it was observed that not all symptoms had the same relevance when predicting the onset of the disease. Other authors are committed to the development of clinical prediction rules based on the analysis of risk factors³¹ or artificial intelligence models.³²

PHASE IV: initial validation of the meta-instrument

Once the model on which the new meta-instrument is to be based has been built, it is advisable to run the first tests to study the reliability of the predictive model.³³ To this end, Palese et al.⁴ used structural model equations, with the dependent variables being the outcome measures of the questionnaires and the results of the confirmatory factor analysis as independent variables. Again, this is another example of how techniques commonly used in validation studies can be applied for other purposes.

Luna-Aleixos et al.¹⁷ took a different approach and analysed the reliability between the values predicted by the model and the original values, as well as the agreement between the predicted and original classification categories. This first reliability analysis that contrasts initial results of the model does not replace the analysis of the reliability of psychometric tests that includes intra- and inter-observer reliability and internal consistency. At this point, it is again convenient to comment that the TRIPOD²⁸ statement recommends the use of techniques such as bootstrapping or cross-validation to study the reliability and agreement of linear models. However, it is best to ensure that the validation procedure is in line with what is recommended, based on the analytical technique used to develop the meta-instrument.

Psychometric properties and diagnostic accuracy

The meta-instrument constructed in the previous phase will include the measurement of several constructs and will do so in a more sparing way, but this is still a tool that will be used by different professionals to assess the care required by users of health services. Therefore, it will be necessary to study its psychometric properties and ensure that the new meta-instrument has adequate content validity, construct validity and criterion validity, as well as internal consistency, reliability and sensitivity to change.

Given that the methodology for analysing the psychometric properties of measurement instruments is widely described in the literature, and even assuming that there are controversies in some of the techniques and procedures used, we believe that the most appropriate course of action is to refer to the recommendations laid down by the COSMIN initiative and the sequence of steps taken in the study of the psychometric properties in the VALENF Instrument.²⁰ Although this proposal has room for improvement, this instrument establishes the need to study both the content validity and the construct validity of the new meta-instrument for each of the constructs that are initially measured. The hope would be that a reflection of each of these will be observed in the statistical results. To do this, the aforementioned techniques such as factor analysis, Rasch analysis or structural equation models can be used, although in this case the objective will be to study the structural validity of the meta-instrument. Regarding the sample for the analysis of psychometric properties, it will be advisable to do this with a different sample that can be obtained in different ways, for example, by running a new study or with cross-validations and obtaining random samples on the same database.

Finally, we recommend studying the diagnostic capacity of the new meta-instrument by running a study of diagnostic tests that establish the indicators of diagnostic validity (sensitivity and specificity), diagnostic safety (predictive values and curves of functional characteristics) and diagnostic reliability. To do this, it will be necessary to run a new study and obtain a new sample. We suggest following the statement for reporting diagnostic accuracy studies (STARD)³⁴ and also the statement on transparent reporting of a multivariate prediction model for individual prognosis or diagnosis (TRIPOD).²⁸ In addition, the only reference retrieved so far in meta-instruments is the diagnostic test study protocol in the VALENF Instrument.³⁵

The future of meta-instruments

This work deals with a new approach in the design and validation of instruments that can bring important benefits. However, the lack of a more or less accepted methodology when building the meta-instrument leaves most of the decision-making in the hands of the researchers. In addition, it implies an increase in complexity in the analytical procedures that nursing traditionally uses to design and validate instruments, further enriching this entire process.

These aspects can make it difficult to develop new proposals with a similar approach, and it is advisable for people interested in the development of meta-instruments to generate the necessary synergies to prevent this from happening. In addition, it is necessary to take advantage of the impact and momentum of disruptive technologies such as big data or artificial intelligence³⁶ to advance in the design of instruments that facilitate the assessment of the care required by our patients and users, as is also the case of the use of biomarkers³⁷ that can be routinely incorporated into the assessment of patients to improve the detection of risk situations.

In addition to facing the learning curve involved in incorporating new analytical techniques and technologies - and implementing meta-instruments in clinical practice, this will be an important challenge for those who decide to undertake it, since they will have to overcome resistance to the use of assessment instruments, given their loss of credibility for nurses, as well as resistance to change.

The development and implementation of meta-instruments can improve care by increasing the duration of direct care that nursing provides to the patient, as this is a more sparing and streamlined instrument. In addition, they can be tools that facilitate nurses' decision-making, enabling care to be prescribed through the completion of these meta-instruments and assisting in the evaluation of the real impact of the care provided.

Conclusion

A meta-instrument is a measurement instrument that collapses other instruments based on the fact that they measure related constructs and share dimensions or items, providing a measurement with a more sparing approach.

The development of meta-instruments aims to assist the important work involved in assessing the care required by users of the health system. In addition, this enables the detection of risk situations and, therefore, can facilitate the prescription of care and decision-making. To do this, it is necessary to develop this with rigorous methods and confirm that, at the very least, they maintain the same psychometric properties and diagnostic capability as the original instruments from which they were designed.

Funding

This work has been funded by the Universitat Jaume I, [reference UJI-A2020-08] and by the Conselleria de Innovación, Universidades, Ciencia y Sociedad Digital de la Generalitat Valenciana (reference CIGE 2022/150).

Conflicts of interest

The authors declare that they have no conflicts of interest.

Authors' contribution

The manuscript has been read and approved by all authors.