metricas
covid
Buscar en
Cirugía Española (English Edition)
Toda la web
Inicio Cirugía Española (English Edition) The size of the effect. Does magnitude matter?
Información de la revista
Vol. 100. Núm. 12.
Páginas 790-792 (diciembre 2022)
Visitas
374
Vol. 100. Núm. 12.
Páginas 790-792 (diciembre 2022)
Methodological letter
Acceso a texto completo
The size of the effect. Does magnitude matter?
El tamaño del efecto. ¿La magnitud importa?
Visitas
374
Antonio Ríosa,
Autor para correspondencia
arzrios@um.es
arzrios4@gmail.com

Corresponding author.
, Marina Iniesta-Sepúlvedab
a Servicio de Cirugía General y del Aparato Digestivo, IMIB - Hospital Clínico Universitario Virgen de la Arrixaca, Departamento de Cirugía, Pediatría, Obstetricia y Ginecología, Universidad de Murcia, Murcia, Spain
b Departamento de Psicología, Facultad de Ciencias de la Salud, Universidad Católica de Murcia (UCAM), Murcia, Spain
Este artículo ha recibido
Información del artículo
Texto completo
Bibliografía
Descargar PDF
Estadísticas
Texto completo

Statistical significance tests have generally been relied on to evaluate research outcomes1, where the researcher concludes that an effect must exist in the population if the probability of having obtained it by chance in the sample is very small (usually less than 1% or 5%). However, it is not only important to know about the existence of the effect in the population under study, but also to know the magnitude of the effect. Here we must remember that the statistical significance test (p-value) does not provide information about the magnitude of an effect and is not necessarily associated with the clinical or practical significance of outcomes. Clinical significance refers to the actual impact of the observed effect on the patient2. Unfortunately, it is common to find interpretations that confuse statistical significance with practical significance.

This aspect is particularly relevant today. Statistical significance is highly dependent on sample size, and current studies in surgery and other medical disciplines are primarily multi-centre studies with large samples of participants. This can lead to significant outcomes in most cases, which are not always relevant at a practical level3.

Effect size indices were developed to understand the practical significance of research outcomes; they report the extent to which the phenomenon under study occurs in the population4. Numerous effect size indices exist and are generally classified according to whether they refer to the magnitude of differences between groups or the degree of association between variables4 (Table 1).

Table 1.

Associated statistical test and interpretation of some of the main effect size indices.

Effect size indices for the evaluation of differences between groups
Index  Statistical test  Interpretation 
Standardised mean difference  Student’s t-test for independent samples  Magnitude of the difference in means between two groups. Cohen’s criterion (1988): 
    around .20 low 
    around .50 moderate 
    >.8 high 
Relative Risk (RR)  2 × 2 contingency table  Magnitude of the difference between hazard ratios in two groups. 
Odds Ratio (OR)  2 × 2 contingency table  Magnitude of the difference between the advantages of two groups. 
Effect size indices of the evaluation of the relationship between variables
Index  Statistical test  Interpretation 
Pearson’s correlation (r)  Linear correlation analysis  Degree of linear association between two variables. Cohen’s criterion (1988): 
    <.10 null association 
    .10−.29 low 
    .30–.49 moderate 
    >.50 high 
Coefficient of Determination (R2)  Simple linear regression  Proportion of variance of the dependent variable explained by the independent variable 
Adjusted Coefficient of Determination (adjR2)  Multiple linear regression  Proportion of variance of the dependent variable explained by the model 
Eta-squared (η2)  One-factor ANOVA  Proportion of variance of the dependent variable explained by the factor. 
Partial eta-squared (pη2Factorial ANOVA  Proportion of variance of the dependent variable explained by each factor 
Omega-squared (ω2)  One-factor ANOVA  Proportion of variance of the dependent variable explained by the model. 
  Factorial ANOVA   
Indices for evaluating the magnitude of differences between groups

The most commonly used are those in the d family, which are based on differences between means. Among them, the standardised mean difference is the most widely used and allows comparison of two groups in a single outcome measurement. Cohen's formula5 is indicated when both groups are of a similar size and variability. There are other indices within this family, for example, in the case of pretest posttest designs, it is more appropriate to use indices based on change scores6, such as the standardised mean change index (one group) or the difference between standardised mean changes (two groups).

When two groups are to be compared in a dichotomous variable, indices based on hazard ratios are used, i.e., the probability of an event of interest occurring as a function of the presence or absence of a factor. The most commonly used indices are the risk ratio or relative risk (RR) and the odds ratio (OR). The RR index indicates the extent to which the probability of the event occurring in one group is greater than the probability of the event occurring in the other, while the OR should be interpreted in terms of advantages rather than probability7. The advantage tells us how many times the probability of the event occurring is higher than the probability of it not occurring, or vice versa, therefore the OR would be equal to the ratio between the advantages of the two groups. The RR and OR values will be similar when the hazard ratios are low.

Indices to evaluate the degree of association between variables

The best known of the association indices is Pearson’s correlation coefficient, which measures the magnitude and direction of the linear relationship between two variables. This index varies between −1 and 1, the strength of the association between variables being lower as the correlation approaches 0. The non-parametric alternative to Pearson's correlation coefficient (also used when the variables are ordinal) is Spearman's correlation (Rho).

In the context of the analysis of association between variables, it is interesting to determine what proportion of the variance of the dependent variable is explained by the independent variable. In linear regression models, this proportion is given by the coefficient of determination (R2), which is equal to the square of the Pearson correlation coefficient. When several predictors are included, the adjusted coefficient of determination (R2a⁡d⁡j) is preferred as it presents better control of the error variance taking into account the sample size and the number of predictors of the model8. Logistic regression is a special case, where the dependent variable is dichotomous, whose estimated effect size is the natural logarithm of the OR7.

Another index of explained variance, derived in this case from analysis of variance (ANOVA), is the eta-squared (η2). It reports the proportion of variance explained of the dependent variable by a categorical independent variable. In the case of a factorial ANOVA, partial η2 indicates the percentage of variance explained by each of the predictors. There are other effect size indices for ANOVA, such as the omega-squared2), which corrects for the possible overestimation of the variance explained by η29.

Considerations for publication

The main current guidelines for the publication of medical research indicate that in addition to statistical significance, reporting of effect size should be considered, as it provides a measure of the clinical importance of outcomes2. The main reasons for reporting effect size include1,4,8: 1) to know the practical significance of the results; 2) to enable the calculation of statistical power. Thus, in the design of a new study, the effect size observed in previous scientific literature is used to calculate statistical power and estimate the necessary sample size; and 3) to allow comparison between studies and the integration of empirical evidence in meta-analysis.

A number of considerations should be taken into account when reporting the effect size10. First, it should be specified which of the indices was used to obtain the effect size and the effect size should be appropriate to the type of analysis performed. Secondly, the effect size describes the properties of a sample, being a potential estimator of its corresponding parameter in the population; therefore, the confidence interval, which gives information on the degree of precision of the estimate, must be provided. Thirdly, it is worth remembering that, despite having mathematical criteria for the interpretation of the effect size, there are other factors that determine the practical relevance of the outcomes. Thus, researchers who are familiar with the context and the phenomenon under study must provide an explanation of the meaning of the effect found in the real world5. A small effect (according to mathematical criteria) with large health or economic consequences may be relevant for society.

To conclude, it is important to note that the objective evaluation of scientific evidence requires the complementary assessment of statistical significance tests and measures of effect size, which, together with substantive and contextualised interpretation by researchers, will provide a more accurate idea of the meaning of findings for clinical reality.

References
[1]
D.N. Kyriacou.
The enduring evolution of the P value.
JAMA., 315 (2016), pp. 1113-1115
[2]
American Medical Association.
AMA manual of style: a guide for authors and editors.
Oxford University Press, (2020),
[3]
E.H. Livingston, A. Elliot, L. Hynan, J. Cao.
Effect size estimation: a necessary component of statistical analysis.
Arch Surg., 144 (2009), pp. 706-712
[4]
P.D. Ellis.
The essential guide to effect sizes: statistical power, meta-analysis, and the interpretation of research results.
Cambridge University Press, (2010),
[5]
J. Cohen.
Statistical power analysis for the behavioral sciences.
2nd ed, Lawrence Erlbaum Associates, (1988),
[6]
J.P. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M.J. Page, et al.
Cochrane handbook for systematic reviews of interventions.
Wiley, (2019),
[7]
E.C. Norton, B.E. Dowd, M.L. Maciejewski.
Odds ratios-current best practice and use.
JAMA., 320 (2018), pp. 84-85
[8]
C.O. Fritz, P.E. Morris, J.J. Richler.
Effect size estimates: current use, calculations, and interpretation.
J Exp Psychol Gen., 141 (2012), pp. 2-18
[9]
K. Okada.
Is omega squared less biased? A comparison of three major effect size indices in one-way ANOVA.
Behaviormetrika., 40 (2013), pp. 129-147
[10]
O.A. Karadaghy, H. Hong, N. Scott-Wittenborn, P. Sinha, J. Suko, S. Tait, et al.
Reporting of effect size and confidence intervals in JAMA Otolaryngol Head Neck Surg.
JAMA Otolaryngol Head Neck Surg., 143 (2017), pp. 1075-1080
Copyright © 2021. AEC
Descargar PDF
Opciones de artículo
es en pt

¿Es usted profesional sanitario apto para prescribir o dispensar medicamentos?

Are you a health professional able to prescribe or dispense drugs?

Você é um profissional de saúde habilitado a prescrever ou dispensar medicamentos