We included 78 individuals with a current manic episode without mixed symptoms recruited from the Hospital Benito Menni, Hospital General de Granollers, Hospital Sant Rafael, and Hospital del Sagrat Cor, Barcelona. We were interested in manic relapses in general, transdiagnostically. Thus we included individuals with either bipolar disorder type I or schizoaffective disorder. Inclusion criteria were (a) current manic episode according to DSM-IV diagnostic criteria; (b) age 18–65 years; and (c) right-handedness. Exclusion criteria were: (a) hypomanic episode; (b) Young Mania Rating Scale (YMRS)18<18 to ensure discarding hypomania; (c) Hamilton Depression Rating Scale (HDRS)19>8 to ensure discarding mixed episodes; (d) history of traumatic brain injury; (e) severe neurological disease; (f) premorbid intelligence quotient (IQ) <70; (g) contraindications to MRI (e.g., metallic implants); and (h) severe substance dependence during the previous six months.

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. All procedures involving human subjects/patients were approved by the Clinical Research Ethics Committee of Germanes Hospitalàries del Sagrat Cor de Jesús, Barcelona (approval number PR-2014-19). All participants gave written informed consent before participation. Healthy controls received a gift card as compensation for their participation in the study.

We detail the collection of baseline clinical and structural MRI data and the standard preprocessing of the latter with SPM12 (Wellcome Trust Center for Neuroimaging, London, UK) in the Supplement. Mainly, experienced psychiatrists confirmed the manic episode diagnosis and administered the YMRS, HDRS, and Positive and Negative Syndrome Scale (PANSS).20

Follow-up visits were at 2, 4, 6, 9, 12, 15, and 18 months after the MRI scanning session. In case of relapse between two visits, the psychiatrist wrote the approximate date of relapse. If there were no relapses before study termination (at 18 months or earlier in case of loss to follow-up or depressive episode), we saved the date of the last visit. For 30 patients, we could determine relapses (or their absence) during an extended period after 18 months up to 10.6 years without ambiguities. We discarded the follow-up information for that extended period whenever doubts existed.

We first conducted simple Cox proportional hazards regressions for descriptive purposes to find baseline clinical conditions associated with manic relapse. The dependent variable was the time from the remission of the current manic episode to the first manic relapse (or the last visit for patients with no registered relapse). The independent variable was the clinical variable or voxel volume. In the case of voxel volumes, we first removed the effects of age and sex from the gray and white matter volumes using linear models. We included the whole brain; we did not pre-plan selecting some brain regions or anatomical targets. For clinical variables, we used p<0.05, and for voxel volumes, p<0.005.21 We did not correct this analysis for multiple comparisons because of its sole descriptive purpose.

We created a user-friendly script for R for this analysis based on the “survival” package.22 We provide this script in the Supplement to allow other groups to conduct these analyses.

As we detail later, we created the risk estimation models exclusively using the data from the training set. The clinical data included age, sex, premorbid IQ, age at onset, age at first admission, illness duration, diagnosis of schizoaffective disorder (vs. bipolar disorder type I), lithium, antipsychotic dosage (chlorpromazine equivalents), and clinical symptoms based on YMRS/HDRS/PANSS items (plus total scores). The MRI data included the modulated and unmodulated gray and white matter images because we have found elsewhere that segments convey complementary volumetric information, and their combination increases the accuracy.23,24 Again, we included the whole brain; we did not pre-plan selecting some brain regions or anatomical targets.

First, we used “lasso” regressions to impute the missing values twenty times.25 Second, we removed the effects of the MRI device with ComBat.26 This tool has been reported to improve simpler methods27,28 and avoid most biases for simple machine-learning models such as lasso.29 Third, we removed the effects of age and sex from the MRI images using linear models. Finally, we fitted a multiple “lasso” Cox regression. The dependent variable was the time to the first manic relapse (i.e., one measure per participant), and the independent variables were the clinical and MRI data. We conducted 18 lasso regressions, each with the voxels of half brain (resulting from dividing the brain in two by planes at 0° or 45° of the anatomical planes). Afterward, we combined the regressions. We have found elsewhere that this ensemble approach substantially increases the accuracy.24

We conducted all analyses with our freely available graphical software MRI Predict (https://www.mripredict.com/).

We used a cross-validation approach to assess the accuracy of the estimations in new individuals. To guarantee that the cross-validation-derived performance was unbiased, we conducted all preprocessing and modeling operations exclusively using one subset of patients (the “training set”) and afterward estimated the hazard ratio (HR) of the other patients (the independent “test set”). HR>1 means increased, and HR<1 means decreased relapse risk.

Specifically, we divided the overall cohort into ten groups or “folds”. For the first fold, we used individuals from folds 2–10 for the training set and individuals from fold 1 for the test set. For the second fold, we used individuals from folds 1 and 3–10 for the training set and individuals from fold 2 for the test set. And so on. With this strategy, we could estimate all individuals’ HRs. Still, we never used the same individuals for training and validating a model, ensuring the unbiasedness of the scheme.

To assess the accuracy of the estimations, we tested whether the estimated manic relapse risk was associated with a higher relapse hazard. Specifically, we fitted a mixed-effects Cox proportional hazards regression model with the “multisite.accuracy” package for R, in turn, based on the “coxme” package.30,31 The dependent variable was the time to the first manic relapse, the independent variable was the logarithm of the estimated HR, and the MRI device was a random-effects factor. We used the multisite.accuracy version (instead of simpler Cox models) to avoid biases due to combining patients scanned with different MRI devices.29,30 We tested the Cox proportional hazard assumption with the “cox.zph” function.22

We also assessed the discrimination accuracy and calibration32 at different follow-up times (every six months until ten years). To assess discrimination, we calculated the multisite version of the area under the curve (AUC) at each follow-up time with the “multisite.accuracy” package30 for R (based on the “AROC” package33). The AUC provides an aggregate measure of sensitivity/specificity across all possible classification thresholds. For instance, we could define individuals with an estimated HR>1 as having a high risk of relapse. Using this classification threshold, we could calculate the sensitivity and specificity of the detection of relapses at a 3-year follow-up. However, we could also use other thresholds, such as HR>1.1, HR>1.2, and so long. The AUC represents the aggregation of the sensitivities and specificities at all thresholds. An AUC=50% means that the classification is as good as tossing a coin, while an AUC=100% means that the classification is perfect. We used the multisite.accuracy version (instead of simpler AUC) to avoid biases due to combining patients scanned with different devices.29,30 On the other hand, to assess calibration, we calculated the expected calibration error (ECE) using the “CalibratR” package for R34 (though we modified the function to conduct the calculations separately for each site). The ECE compares the probabilities of relapse derived from the model with the observed probabilities. To obtain the former, we first converted the logarithms of the HR of the (cross-validation) training set to probabilities using the normal function, we calibrated them using isotonic regression calibration with the “rfUtilities” package for R,35 and finally interpolated the calibration estimated with training test to the test set. To calculate the ECE, we divided the patients into bins (the first bin including the patients with lowest estimated probabilities and the last bin including those with the highest, separately for sites). Afterward, for each bin, we subtracted the average estimated probability and the observed probability (proportion of relapses). Finally, we averaged the absolute errors from the different bins. We tried to create ten equal-sized bins for each MRI device, as long as each bin had at least 5 patients. We conducted weighted averages to account for differences in bin sizes.

To assess whether the models’ risk estimation accuracy was mainly related to MRI or clinical data, we also created models exclusively based on these variables separately.

To assess the relevance of the model's variables, we calculated the relative absolute coefficients (rβ) based on the “lares” R package.36 Specifically, we calculated the absolute value of each coefficient and divided it by the sum of the absolute values of all model's coefficients. We repeated this operation separately for each model and then averaged across models.

We encourage independent groups to replicate our model's accuracy assessment. To help them, we provide a simplified website-based version of the tool (https://www.mripredict.com/mania/) that quickly estimates the manic relapse risk of an individual. For the website, we fitted a model using the whole cohort and selected the coefficients with an absolute value≥0.05 (see Supplement).

The risk estimation model used the diagnosis of schizoaffective disorder (vs. bipolar disorder type I) and the presence of several clinical symptoms, with the most relevant in the model being poor impulse control, unusual thought content, delusions, tension, grandiose or paranoid ideas, mannerisms and lack of illness insight.

It is worth noting that all clinical symptoms with rβ>0.5% model relevance come from PANSS, with the only exceptions of grandiose or paranoid ideas (YMRS 8) and lack of illness insight (HDRS 17), which did have indeed a relatively small relevance compared to the two most relevant PANSS items (rβ=0.6–0.7% vs. 1.5–2.8%). While PANSS may also measure affective symptoms, the relevance of PANSS items for estimating manic relapse risk invites speculation on whether a schizoaffective spectrum disorder may exist. In that spectrum, some patients would have schizophrenia-like symptoms without fulfilling the criteria of schizoaffective disorder (e.g., including periods with psychotic but not affective symptoms). Indeed, gray matter abnormalities in schizoaffective disorder resemble schizophrenia more than bipolar disorder.38 This resemblance suggests that schizoaffective disorder is a subtype of schizophrenia or that in a continuum spectrum model of psychosis, schizoaffective disorder is more skewed toward schizophrenia than bipolar disorder.

Interestingly, the model selected a diagnostic label (schizoaffective disorder) even if it had also selected its associated symptoms, supporting that diagnostic labels are more than a mere collection of symptoms. Even more, the coefficient of the diagnostic label was substantially higher than the coefficients of the symptoms, supporting that diagnostic labels are clinically helpful. These observations may be relevant in the current debate about DSM/ICD diagnoses.39

Also interestingly, some of the items selected by the model (e.g., delusions and lack of illness insight) have been previously identified as the main risk factors for medication nonadherence in bipolar disorder and schizophrenia.40 And nonadherence has been associated, in turn, with a higher number of manic episodes and hospitalizations.41 Thus, it may be that the mechanism by which these symptoms increase the risk of relapse is by decreasing treatment adherence. Such a mechanism would indicate the need to include interventions to increase adherence, especially in patients with a higher risk of relapse. Moreover, the importance of increasing adherence is higher when non-adherence has been associated with higher suicidal risk.42

An unexpected finding was that lithium usage was not a relevant variable in the model. Indeed, the percentage of patients receiving lithium and the mean dosage were similar between patients who did or did not relapse. Looking at the data, we could not find an apparent reason for this finding, i.e., lithium was not statistically associated with more severe symptoms (other than PANSS N1, blunted affect, p=0.001, and HDRS 6, early awakening, p=0.035). However, as noted above, it could be that even if the indication of lithium was not inferior in patients with higher relapse risk, the actual usage was lower due to non-adherence.

The brain abnormality showing the highest relevance in the model was the cerebellum gray matter decrease, followed at a considerable distance by a white matter increase in the angular gyrus, several gray matter increases (especially in middle temporal and inferior frontal gyri), and a white matter decrease in the superior occipital region.

These regions have little overlap with those reported in meta-analyses of gray and white matter abnormalities in bipolar disorder.13,43–45 However, our machine-learning algorithms did not look for abnormalities. Instead, they only selected those regions whose volumes may confer protection or risk of relapse. It may well be that some of these regions have a completely average volume in patients with bipolar disorder. Still, a larger or smaller volume may confer protection.

Interestingly, previous studies have found increased middle temporal and decreased cerebellar volumes in antipsychotic-naive individuals with first-episode schizophrenia.46,47 Therefore, it could be that this region's increase is associated with a psychotic-related higher manic relapse risk.

An important finding of our study is that the machine learning algorithms estimated the risk better when we combined clinical variables and structural MRI data. We want to highlight that this finding does not mean that models based only on clinical data cannot successfully estimate the risk of manic relapse. Indeed, some previous clinical models using larger sample sizes have successfully done it.15–17 However, we show that the combination of clinical and MRI data improves the estimations. This finding is very relevant because the paradigm of many previous MRI machine learning papers was to use MRI data to predict clinical data (e.g., the diagnosis). Conversely, we propose a new paradigm: to use brain imaging to help clinics. Our findings are humble and need replication, but they show that this new paradigm is viable.

The overlap between clinical and MRI variables detected in the descriptive univariate analysis and those selected by the machine learning algorithm was only partial. One reason for these discrepancies may be the potential effects of confounders in the univariate analysis. For example, both unusual thought content and excitement symptoms seemed to increase the risk of manic relapse. However, individuals with high unusual thought content in our cohort also tended to have high excitement (Spearman rho=0.341, p=0.016), and a bivariate analysis including both symptoms showed that excitement did not increase relapse risk when controlling for unusual thought content. Therefore, it is unsurprising that the machine learning algorithm seldom selected it. Other differences between the descriptive univariate analysis and the machine learning results may relate to their different methodological nature. The former selects the variables using a conventional threshold (p<0.05). In contrast, the latter selects those variables that optimize the risk estimation model. The variables included in the machine learning model may not need to be “statistically significant”; conversely, the algorithms may not select “statistically significant” variables if their inclusion does not improve the risk estimation. It may be the case, for instance, that between two strongly correlated variables or voxels, the algorithm tends to select one and not the other, even if the latter is also associated with higher risk. Therefore, we suggest considering neither the descriptive univariate analysis nor the machine learning analysis as explicative. The univariate analysis is descriptive, and it does not consider confounding effects. At the same time, the machine learning analysis only aims to estimate the risk. Also relevantly, the p-values of the descriptive analysis only refer to the relationship between symptoms/volumes and observed relapses in the same data. In contrast, the p-value of the machine learning analysis refers to the relationship between the relapse risk estimated from the symptoms/volumes and the observed relapses in new data.

An obvious way to optimize the accuracy of the risk estimations is by increasing the sample size. For instance, as noted earlier, a previous study with a much larger sample size successfully predicted the risk of relapse exclusively using clinical data.15,16 Therefore, we strongly suggest future studies have large cohorts or that groups combine the existing longitudinal data to have larger sample sizes, as done in the ENIGMA Bipolar Disorder working group.48

We also speculate that future studies with more complex algorithms may yield more accurate risk estimations. Deep learning, for example, could detect complex relationships between features that our Cox lasso regression did not detect. However, any complex algorithm would require a much larger sample size, which is not readily achievable in a longitudinal study. We also speculate that we might have achieved more accurate risk estimations combining information from different MRI sequences. For instance, diffusion MRI may provide finer details about white matter structure. However, we aimed only to use the structural MRI many physicians already ask for to rule out tumors and other organic pathology. And in any case, risk estimations only based on baseline data might hardly achieve very high accuracy because relapses also depend on life events and stressors that may happen after the assessment.49

If replicated, the risk estimation model could provide the clinician with helpful information to optimize the patient's follow-up and thus reduce the risk of manic relapse. Given the relevance and severity of such relapses, the economic cost of a structural MRI may seem negligible. However, it could be that, for many reasons, the information provided by the model does not translate into a reduction of the relapse risk compared to not using models or using models based only on clinical data. Future trials should investigate the effectiveness, e.g., via the number of patients needed to scan to prevent relapse, and derive whether the overall strategy is cost-effective.

This study has some limitations. First, we used a relatively simple machine-learning algorithm. More complex algorithms would likely be more accurate, but they require a substantially larger sample size. Second, we collected the data at a particular time: when the current manic episode remits. However, we have reported earlier that the manic relapse risk changes dynamically in time, e.g., it progressively decreases if the patient does not have relapses.10 Thus, our model would still provide a reasonable estimation of the risk at the beginning of the follow-up, but this risk could be re-estimated at each follow-up visit (e.g., depending on new life events). Third, clinicians may prefer deciding the frequency of follow-up visits and medication based on proximal clinical variables such as sleep disturbance or increasing activity. However, we suggest not choosing between model-based risk estimations or proximal clinical variables but considering them all. Fourth, we only collected a set of clinical and MRI variables. Other clinical and MRI variables may further help estimate the manic relapse risk. Fifth, the discrimination and calibration assessments at different follow-up times assumed that patients lost to follow-up had not relapsed. We thus suggest that these specific assessments be taken with some caution. Finally, our sample was large enough to provide adequate statistical power but not to conduct subgroup analyses. Future studies may investigate whether different subgroups of patients may benefit from different models.

We must also remind the strengths. First, we used standard clinical variables and structural MRI. These data are often collected during clinical assessment, increasing the approach's feasibility. Second, this is a longitudinal study, which is rare in the context of neuroimaging.48,50 And in addition, we conducted frequent follow-up visits (at 2, 4, 6, 9, 12, 15, and 18 months). Finally, we based on the results of previous rigorous methodological studies to optimize MRI data processing avoiding data torturing and bias.