How far is clinical assessment from the bullseye? Using MEmind to compare clinical assessment with self-assessment in patients with depression and anxiety diagnosis

Abstract

Background and objectives

Technology based assessments are being used for screening and monitoring in a wide scope of medical specialties, including mental health field. Depression and anxiety are common disorders in which e-health tools can be useful. We aimed to compare clinician assessment of illness severity in patients with depression and anxiety diagnosis with computer-based self-assessment within 24 h of clinician evaluation via MEmind (www.memind.net), a novel web-tool.

Methods

From May 2014, adult patients attended in outpatient settings in Fundación Jiménez Diaz Psychiatry Department were registered in MEmind, a web tool designed for psychiatric assessment. During the recruitment, clinicians use CGI-S for patient assessment via MEmind and provide patients a code and password to use the web-tool. We selected those patients diagnosed with depression and/or anxiety who connected within 24 h of the clinical visit and complete in the web page GHQ and WHO-5 scales. We calculated a bivariate correlation for CGI-S, WHO-5 and GHQ-12.

Results

Of the 231 participants, 157 (68%) were diagnosed with anxiety disorders and 74 (32%) with depression. Using the Spearman Rho test for correlation, we found a low correlation between CGI-S and total WHO-5 (r = −0.192; p = 0.006) and between CGI-S and total GHQ-12 (r = 0.211; p = 0.002) and a good correlation between total WHO-5 and total GQH-12 (r = −0.606; p = 0.000).

Conclusions

We found a low correlation between clinician assessment and patients’ self-reports within 24 h of clinician evaluation. Factors that potentially influenced the degree of correlation related with patients, clinicians, measurements and technology are discussed.

Background

Technological developments and participatory health initiatives are expanding the scope of medicine from a traditional focus on disease cure to a personalized preventive approach.¹ Increasingly technology based assessments are being used for screening and monitoring in a wide scope of medical specialties. With this in mind, MEmind was developed to help clinicians optimize and personalize clinical psychiatric assessment and treatment.² Thereby, MEmind allows to improve communication among patients, support network and mental-health professionals; to monitor doctor's drugs prescription habits^3,4; and monitoring patients through ecological momentary assessment (EMA).⁵

Anxiety and depression are common disorders, in fact more than 350 million people worldwide are affected by depression, making it the biggest cause of disability.⁶ In 2010, the estimated number of persons affected by anxiety disorders and unipolar depression in Europe was 69.1 million and 30.3 million, respectively. Furthermore, depression represents 7.2% of the overall burden of disease, with 4,320,400 disability adjusted life years lost.⁷

An early diagnosis and treatment would result in improved personal functioning and reduce long-term costs.⁸ Screening instruments for early diagnosis and monitoring are rarely used consistently; a technological approach could facilitate a consistent use. Combining validated screening instruments with the resourcefulness of tablet and phone applications that allow for EMA approaches offers the possibility to tackle these issues in a variety of settings, including primary care and specialty services.

In turn, clinical assessment, the tool to early diagnosis and optimal treatment in mental health, currently relies mostly on retrospective self-reports. The latter are inconsistent with interviewer judgments in as many as 60% of patients and correlate only modestly with informant reports (from clinicians or 7 friends/relatives).⁹ Patients, caregivers and doctors may have differing perceptions of illness and yet all potentially influence the evolution of that illness.¹⁰ Even high-contact clinician ratings are retrospective, subject to recall bias, time-limited and mostly occur in a clinical setting, which might in itself influence the patients report and not reflect the patient's state in their actual environment.

Specifically, in cases of depression and anxiety, the agreement between patient self-assessment and clinician evaluation is far from be perfect. For depression, different studies find discrepancies in severity of illness depending on the rater (patient or clinician), with patients overrating their illness when compared with clinicians¹¹ whereas others find good correlation.^9,10 For anxiety, a good correlation is more frequently found.^12,13

Ecological momentary assessment, by contrast, allows acquisition of self-reported information in real-time, maximizing accuracy and avoiding recall bias¹⁴ and in this field is where we used the MEmind technology. Closely related to this, it is important to realize how new technologies are changing the doctor-patient relationship, nowadays it seems that “people are more honest with their phones than with their doctors”.¹⁵ Indeed, evidence supports that people are more forthcoming on online health questionnaires regarding sensitive areas of importance in psychiatry, such as past traumatic events like sexual abuse, substance abuse or suicidal thoughts and/or behavior.^16,17

In this study our aim was to compare clinician assessment of illness severity in patients with depression and anxiety diagnosis with computer-based self-assessment within 24 h of clinician evaluation.

Methods

Participants and setting

Patients were recruited from psychiatric outpatient facilities within the catchment area of Fundación Jiménez Díaz General Hospital in Madrid. This hospital is part of the National Health Service and provides medical coverage to 850,000 people. From May 2014 onwards all clinicians working at the six mental health centers of the catchment area were encouraged to use the MEmind Wellness Tracker systematically in their clinical activity, after receiving specific training in its use.

Out of the total registered on the MEmind platform in the first year of use, 231 patients were included in the study. Inclusion criteria were based on timing of assessment and diagnosis, we included patients who submitted their self-assessment within 24 h of clinician assessment who were diagnosed with depression (ICD-10 codes F32 to F39) and/or anxiety (ICD-10 codes F40, F41 and F43).

Exclusion criteria were illiteracy, refusal to participate, current imprisonment, being under guardianship and emergency situations during which the patient's state of health did not allow for a written informed consent.

Materials: web tool and questionnaires

MEmind is available at www.memind.net and is compatible with Smartphones, Tablets and computers with any operating system. The MEmind application has two interfaces, one for clinicians (the electronic health record view) and another for patients (the EMA view).

The electronic health record view was designed for clinician use during medical, psychological or nurse practitioner visits. It was designed to capture data from standard psychiatric assessment including sociodemographic, diagnostic, treatment information as well as nurse practitioner annotations (e.g. vital signs and anthropometric measurements) organized in different tabs (Fig. 1). Additionally, care providers can add information to the basic evaluation using a large customizable choice of relevant scales or notes. For this study we used sociodemographic data and the Clinical Global Impression-Severity scale (CGI-S).¹⁸

Fig. 1.

MEmind Health Record View (P100).

The EMA view, designed for patients, consisted of three tabs with the following headings: (1) How are you today?; (2) General Health Questionnaire, and (3) Notes (Fig. 2). The first tab How are you today? included questions on eating and sleeping as well as the WHO (Five) Well-Being Index (WHO-5).¹⁹ The second tab General Health Questionnaire consisted of the 12-Item General Health Questionnaire (GHQ-12).^20,21 Finally, the third tab Notes allows free-text, but was not used in this study.

Fig. 2.

MEmind patient's view.

CGI-S rates the psychiatrist's impression of the severity of psychopathology ranging from 1 (Normal, not at all ill) to 7 (Among the most extremely ill patients). WHO-5 is a scale with five items on the subjective quality of life based on positive mood, vitality and general interest. We used the percentage score method: a percentage score of 0 represents worst possible whereas a score of 100 represents best possible quality of life. GHQ-12 is the instrument most extensively used for screening common mental disorders, is composed of six positively phrased items^1,3,4,7,8,12 and six negatively phrased items.^2,5,6,9–11 Different scoring methods have been proposed for the GHQ items, we used standard GHQ–0011 scoring. According this method, for positive items 0 was given for answers “more than usual” and “same as usual” and 1 for answers “less than usual” and “much less than usual”; and for negative items, 0 for answers “not at all” and “no more than usual” and 1 for answers “rather more than usual” and “much more than usual”. All questionnaires were completed in Spanish.

Study procedure

Patients were informed about the study by the clinician during regular clinical visits. If the patient agreed to participate following written informed consent he/she was registered in the web tool and received username and password. During the visit, the clinician completed the CGI as well as other items in MEmind. Clinical diagnoses followed ICD-10 criteria.²² Diagnoses were made after reviewing all available information, including medical records and clinical interviews with patient and relatives.

Once registered with MEmind, patients were able to connect to the EMA interface freely (no instructions were given regarding when and how often to connect). We selected those patients who connected and entered data (GHQ and WHO-5) within 24 h of the clinical visit.

Ethics and data protection

The study was conducted in compliance with the Declaration of Helsinki and approved by the local ethics committee. All participants gave written informed consent. Data protection was ensured and an external auditor guaranteed that security measures met the Organic Law for data protection standards at a high protection level.

Statistical analysis

We analyzed data using the SSPS version 22.0 package. First of all, with Kolmogorv–Smirnov test, we demonstrated a normal distribution for WHO-5 and a non-normal distribution for GHQ-12 and CGI-S. A descriptive analysis of sample characteristics was followed by a bivariate correlation (Spearman Rho test) for CGI-S, WHO-5 and GHQ-12.

Results

Descriptive findings

Out of the total number of patients registered by clinicians in MEmind during the first year of use in our outpatient facilities, 1288 used the EMA interface, of which 1106 did so within 24 h of the medical consultation. Of these, 231 were diagnosed with depression or/and anxiety disorders and were selected for inclusion in this study.

Of the 231 participants, 145 (62.8%) were women. Age of participants ranged between 18 and 72 years, with mean age of 43.7 years (sd = 12.2). One hundred and seven (68%) were diagnosed with anxiety disorders and 74 (32%) with depression.

Regarding the scales, CGI-S median score (25th and 75th percentiles) was 3 (3–4), the mean WHO-5 score was 44.83 (sd = 22.2) and the median GHQ-12 score (25th and 75th percentiles) was 4 (1–8) (for details on individual scale items see Fig. 3 and Tables 1 and 2).

Fig. 3.

ICG-S measured by the clinician.

Table 1.

WHO-5 scores for individual items.

WHO-5 item		Mean (sd)
1	I have felt cheerful and in good spirits	47.14 (26.1)
2	I have felt calm and relaxed	45.88 (26.7)
3	I have felt active and vigorous	45.23 (28.6)
4	I woke up feeling fresh and rested	40.02 (30.4)
5	My daily life has been filled with things that interested me	45.89 (27.6)

Table 2.

GHQ-12 scores for individual items.

GHQ-12 items		Score = 1
		N	Percentage
1	Able to concentrate	148	64.1
2	Loss of sleep over worry	130	56.3
3	Playing a useful part	134	58
4	Capable of making decisions	147	63.6
5	Felt constantly under strain	125	54.1
6	Couldn’t overcome difficulties	124	53.7
7	Able to enjoy day-to-day activities	132	57.1
8	Able to face problems	139	60.2
9	Feeling unhappy and depressed	133	57.6
10	Losing confidence	133	57.6
11	Thinking of self as worthless	150	64.9
12	Feeling reasonable happy	148	64.1

Correlation between clinician assessment of severity and self-reported scales

Using the Spearman Rho test for correlation, we found a low correlation between CGI-S and total WHO-5 (r = −0.192; p = 0.006) and between CGI-S and total GHQ-12 (r = 0.211; p = 0.002) and a good correlation between total WHO-5 and total GQH-12 (r = −0.606; p = 0.000).

Discussion

We found a low correlation between clinician assessment of severity illness and patients’ self-reports using screening questionnaires within 24 h of clinician evaluation. Ideally, self-assessment would be equivalent to clinical assessment or at least have a good correlation; the results of this study indicate that in fact they do not. Several factors that potentially influenced the degree of correlation in this study are discussed below.

Factors related with patient

The time factor may have reduced the degree of correlation in different ways: Firstly, the time bias that a relatively broad time window of 24 h introduces, especially when considering the circadian variation of symptoms over time. Secondly, the assessment sequence i.e. the fact that all patients selected self-assessed after having seen the clinician may well have influenced their symptoms e.g. levels of anxiety or even just their symptom reporting. Also important is that users were using MEmind for the first time and this might also have influenced symptom reporting and anxiety levels.

The self-selection bias needs to be considered: only patients who actually completed the questionnaire within 24 h were included in the study. This might well include those who are more severely ill or simply more anxious about their symptoms, ultimately affecting levels of symptom reporting and in turn correlation. Alternatively it might have been those who felt unheard during clinicians visit and thus had an urge to convene the severity of their symptoms.

The variation in setting in which the self-evaluation took place: hospital, home or public transport to list a few may well influence patients’ responses and attitudes toward the doctor. In line with this and also relevant is whether patients completed self-assessment on their own or in company of others.

Factors related with clinicians

The question of whether clinicians are failing to collect patient information accurately springs to mind. In the process of a clinical encounter a clinicians’ role, among many, is to interpret idioms of distress to understand the complaint and diagnose. This process is inherently flawed and inevitably leads to a certain loss of information regarding the subjective symptom experience of patients. The holistic data collection system offered by MEmind and other such technologies might well serve to bridge this gap in current clinical practice.

More practical limitations inherent to the use of MEmind in the clinical setting as it is relatively time-consuming for clinicians during consultation complicate optimal data collection, the time point at which clinicians entered patients data might have varied possibly leading to recall bias.

Factors related with measures

Previous studies have questioned the validity of CGI. In a study evaluating the validity of the CGI-I and CGI-S as outcome measures in clinical trials, Forkmann et al.²³ found: (1) No strong evidence for the validity of neither of them; and (2) Congruence between CGI ratings from patients’ and staff's perspective was not convincing. They concluded that it could not be assumed that the view of the patient on the severity of his impairment was fully represented by therapist or team ratings. In fact they advocate for the incorporation of multiple self- and clinician-reported scales into the design of clinical trials in addition to CGI in order to gain further insight into CGI's relation to the patients’ perspective. This finding could partly explain the reduced correlation observed between patient and clinician rating.

Factors related with technology

The role technology plays and its influence on peoples’ behavior remains to be better understood. Patients might have had preconceptions regarding the implications of their entering of data electronically and so have altered their responses. However some evidence suggest that data collection over apps and other technologies apart from being more ecological and timely also enable people to respond more honestly.¹⁵

The strength of and need for a more holistic and integrative evaluation system such as the one MEmind offers becomes tangible when you consider our findings. Information goes amiss when evaluating patients purely from clinician visits and this affects patient management. Considering the impact on suicidal behavior, for example, the importance of the discrepancy between self and observer-rated depressive symptoms becomes more concrete. In a study by Tsujii et al. patients with mild major depressive disorder who overrated their depression severity as compared with clinicians’ ratings were more likely to have a history of suicide attempts.¹¹

The results of this study, which was part of the development of MEmind, reinforce the value of using a powerful novel tool for efficient data collection from a very large sample. In fact the size of the sample is one of the strengths of this study. MEmind is an EMA tool designed for the comprehensive evaluation of mental conditions; with easy access through any device with Internet connection (a mobile App will soon be available). Not only does MEmind have important implications for research in mental health, but also promises to be an effective aid in clinical practice.

Limitations inherent to the use of MEmind in the clinical setting is the fact that it is relatively time-consuming for clinicians during consultation, which may complicate optimal data collection.

In conclusion, our results highlight the importance of holistic evaluation systems that take patients and clinician assessments into account. Our web tool -MEmind- is a promising tool for this purpose. The experience gained using it has served to advance our understanding of the effects of using such a technology and become aware of the different factors that should be considered and potentially controlled for in research with these technologies.

Conflicts of interests

The authors have no conflict of interest to declare.

Funding

This work is partially supported by: Instituto de Salud Carlos III fondos FEDER (ISCIII PI16/01852), Delegación del Gobierno para el Plan Nacional de Drogas (20151073) and American Foundation for Suicide Prevention (AFSP) (LSRG-1-005-16).

Acknowledgements

The authors acknowledge the involvement of Ivan de la Calle (Cabaro S.L.) in the development of the MEmind program.