Men who have sex with men (MSM) are at a high risk for HIV infection. While pre-exposure prophylaxis (PrEP) is an effective oral preventive strategy, its success is largely dependent on consistent medication adherence.
ObjectiveThe aim of this study was to develop the machine learning web application and evaluate the performance in predicting PrEP adherence.
MethodsThe PrEP prospective cohort study of the MSM population conducted in Western China from 2019 to 2023, and we collected adherence data and personal characteristics data from 747 MSM. Predictor variables were screened and the performance of several machine learning methods in predicting nonadherent behaviors were compared.
ResultsA total of 11 candidate variables were screened that predicted nonadherent behaviors. We developed and evaluated five machine learning models that performed well in predicting adherence. Attitudes of male sexual partners, self-efficacy, HIV testing, number of male sexual partners, and risk perception were the most important predictors of adherence. The optimal prediction model was displayed in a shiny web application for online calculation of the probability of occurrence of nonadherent behaviors among MSM.
ConclusionsMachine learning performed well in predicting nonadherent behaviors among MSM. An interactive and intuitive web application can help identify individuals who may have nonadherent behaviors, resulting in improved medication adherence and increased prevention efficacy.
Acquired immune deficiency syndrome (AIDS) is a chronic infectious disease caused by the human immunodeficiency virus (HIV). In recent years, the number of men who have sex with men (MSM) who are infected with HIV has shown a rapid growth trend, and MSM have become a high-risk population for HIV infection (Wang et al., 2022). Based on previous reports in the literature, MSM have been found to have a significantly higher likelihood of HIV infection compared to the general population, with some studies indicating it could be around 20 times higher (Bao et al., 2021; Kellie et al., 2018). In China, the prevalence of HIV infection among the MSM population has shown a rising trend over the years, with the proportion of new HIV infections increasing from 2.5 % to 25.5 % (Hu et al., 2019; Qin et al., 2017). To address these challenges and reduce the rate of new HIV infections among high-risk population, it is important to prioritize HIV/AIDS prevention efforts.
Currently, oral Pre-exposure prophylaxis (PrEP) has been shown to be a safe and effective antiretroviral-based bioprevention approach for HIV (Li et al., 2023). The Chinese Expert Consensus on Drug Use for HIV Pre-Exposure Prophylaxis, with editions released in 2020 and 2023, offers guidance on the use of medications for implementing PrEP (Xu et al., 2020; Zhang et al., 2023). However, PrEP has not been widely promoted in China, and the overall level of PrEP utilization among Chinese MSM is low and the price of the drug is expensive (Jing et al., 2019; Yan et al., 2013). Furthermore, it is worth noting that the efficacy of PrEP is highly dependent on adherence (Claire et al., 2022). Therefore, it is particularly important to conduct studies on PrEP adherence. It not only helps to improve the effectiveness of PrEP, but also provides a scientific basis for the development of more effective interventions and policies. Our study on adherence was based on this context, which helped to improve the efficacy of PrEP in the Chinese MSM.
So far, despite the fact that researchers have given PrEP adherence in the MSM population a great deal of attention, the majority of studies have concentrated on exploring the influencing factors of adherence (Andrew et al., 2021; Laura et al., 2021; Luana et al., 2021). Few studies on the prediction of adherence have been reported. Meanwhile, there was no further exploration and excavation in analyzing methods. Given the limitations of PrEP adherence in terms of research content and analytical methods, refining medication-taking strategies and establishing predictive models for identifying nonadherent behaviors could help improve adherence and increase PrEP efficacy. The developed PrEP adherence prediction tool could be a useful clinical service to identify high-risk populations and prioritize resources.
In digital epidemiology, machine learning algorithms are a developing field that may enhance predictive modeling and decision support (Topol, 2019). These methods can incorporate more covariates in large datasets, deal with complex relationships between predictors and outcomes, and achieve high precision (Lee et al., 2018). In China, few studies have used machine learning methods to predict PrEP adherence in the MSM population, and our study will fill this gap. As a result, the applied research on predictive modeling of PrEP adherence was still in the preliminary stage and needed to be better explored and developed. Additionally, the application of predictive modeling is crucial. The ways in which the results of machine learning-based predictions can assist managers or clinicians in a broader range of scenarios deserve significant attention. Mobile health (mHealth) technologies are effective and cost-efficient strategies that can improve individual and public health (Roman et al., 2017). Based on this technology, due to its ubiquity and ease of use on mobile devices, it can be used not only for reminding and recording, but also for expanding application scenarios in combination with modeling (Liu et al., 2020; Roman et al., 2020). Many previous studies have applied machine learning to web applications to enable the prediction of survival, prognosis, and disease progression, which is clinically valuable and significant (Karabacak et al., 2024; Mert et al., 2024; Yan et al., 2024). The user-friendly web application provides an intuitive and easy-to-use platform that makes the application of predictive models more convenient. At the same time, through the Internet, web applications can be accessed anytime and anywhere, increasing their usability and usefulness. The development of an interactive and intuitive web application for predicting adherence levels in MSM population and helping to design individualized interventions would play an important role in improving PrEP adherence.
Therefore, the aim of this study was to develop the PrEP adherence prediction model using different machine learning methods and to develop an interactive and intuitive web application. The resulted machine-learning predictive model for PrEP adherence could be part of an integrated decision-support tool that facilitates the identification of high-risk individuals for nonadherence and personalized prevention strategies in PrEP.
MethodsStudy setting, population, and designOur study was a PrEP prospective cohort study of the MSM population conducted in Western China (Chongqing, Sichuan, and Xinjiang) from 2019 to 2023 (Trial registration: Chinese Clinical Trial ChiCTR190026414). The study population were HIV-negative MSM. The eligible MSM were recruited through collaboration with local Non-governmental organizations (NGOs) and peer referral. Inclusion criteria for study participants included: assigned male sex at birth; age between 18 and 65 years old; HIV-negative; self-reported sexual behavior with a male sexual partner in the last six months; willingness to use PrEP drugs under the supervision of clinicians and to comply with follow-up schedules; and signing of an informed consent form.
The researchers enrolled eligible study participants in the PrEP prospective cohort study and distributed PrEP drugs (Lamivudine and Tenofovir Disoproxil Fumarate tablets) free of charge. Study participants were asked to take PrEP drugs orally once daily, one tablet (300 mg/tablet) at a time. Every 3 months, study participants were informed by the administrator to go to the study center for face-to-face follow-up and a new round of PrEP drugs were dispensed. A total of 4 follow-up visits were conducted, at months 3, 6, 9, and 12.
OutcomesStudy participants self-reported their adherence each time they attended a follow-up visit. Study participants were asked, "In the last two weeks, have you had a missed dose? How many days were missed?" Participants answered "yes" or "no" and the number of days missed (0–14 days). Adherence is equal to the percentage of days on medication. Previous literature suggested that adherence of ≥80 % could be defined as good adherence (David et al., 2018; Haberer et al., 2013; Ira et al., 2016).
PredictorsIn total, we measured 31 variables of MSM personal characteristics (Supplementary Table 1). After enrollment in the PrEP cohort, study participants were randomized into the reminder group and the no-reminder group. Participants in the reminder group received daily reminder messages based on the mobile app; participants in the no-reminder group who took PrEP drug daily did not receive reminder messages. Reminders were sent through the WeChat App (one of the most popular social media apps in China) at regular intervals to remind users to take their PrEP drugs. We named this variable "reminder system" to explore whether the use of the reminder system as a potential predictor of adherence was effective.
The main HIV-related characteristics of MSM individuals included HIV knowledge score, HIV testing and counseling, risk perception, and number of male sexual partners. Of these, the HIV knowledge score was measured by the knowledge scale consisting of 13 questions related to HIV (Diclemente et al., 1986; Koopman et al., 1990). These questions included some assessments of HIV knowledge, where participants answered “True”, “False”, or “Don't know”. A correct answer was scored as 1 point, while an incorrect answer or “Don't know” was scored as 0 points, resulting in a total scale score ranging from 0 to 13. A score of ≥11 indicated a high level of HIV knowledge (Qu et al., 2018).
Variables related to PrEP primarily included attitude of male sexual partners and self-efficacy. Attitude of male sexual partners were measured by a question, "How do you think your male partners would react if they knew you were taking PrEP?" Participants chose to categorize the variables by answering "positive", "neutral", and "negative". MSM were asked about “I am confident that I can remember to take my medicine on time” as a measure of their self-efficacy and were categorized into high and low levels.
Model development and validationData on adherence at the first follow-up (month 3) were taken as the outcome variable (≥80 % was defined as high adherence), and data on personal characteristics of MSM at baseline were taken as predictor variables. We used five prediction methods, including a traditional Logistic regression model (LR) and four machine learning algorithms. The machine learning algorithms were Decision tree (DT), Support vector machine (SVM), Random forest (RF), and Extreme gradient boosting machine (XGBoost). The data were preprocessed before being analyzed. First, the multiple imputation method was used to fill in the missing values of each predictor variable, which was performed using SAS software. Secondly, LASSO regression was used to screen for potential predictors and to make a preliminary comparison of the differences between the screened variables in the high and low adherence groups. This step was performed using the "glmnet" package in the R software.
We split the data into the 80 % training set and the 20 % testing set. The SMOTE algorithm was used to balance the data, and the balanced training data were brought into the five prediction models for training and hyperparameter optimization. The modeling results with the best performance were saved, applied to the testing set data, and cross-validated with ten folds. The metrics for model evaluation included Area under the curve of receiver operating characteristic (AUC-ROC), Accuracy, Sensitivity, Specificity, Positive predictive value (PPV), Negative predictive value (NPV) and F1-Score. Our study used AUC-ROC as the primary evaluation metric and Sensitivity as the secondary evaluation metric. The DeLong test was used to compare the differences between the AUC-ROCs to find the model with the best predictive performance. The training and testing processes of the models were realized by using the "tidymodels" package in R software.
The trained optimal model was applied to 6-month, 9-month, and 12-month data as an internal validation to assess the predictive effectiveness of the model. Meanwhile, for the optimal model, the importance of its variables was ranked and the partial dependence plots of the top 5 variables were created to explain the effects of the variables. The predictions of the optimal model were also displayed in a shiny web application, which enabled online probability estimation and interactive computation. This step was realized using the "shiny" package in R software. The flowchart of the study was shown in Fig. 1.
Research flowchart
NGOs: Non-governmental organizations; LASSO: Least absolute shrinkage and selection operator; LR: Logistic regression; DT: Decision tree; SVM: Support vector machine; RF: Random forest; XGBoost: Extreme gradient boosting machine; ROC: Receiver operating characteristic curve; PPV: Positive predictive value; NPV: Negative predictive value; PrEP: Pre-exposure prophylaxis; MSM: Men who have sex with men.
In the optimal model, we trained the model by bringing each predictor variable into the model and compared the resulting univariate AUC-ROC with the AUC-ROC of the optimal model. Secondly, for the subgroups of adherence, we also set the criteria to 90 %, 100 %, and 60 %, respectively, and brought them into the optimal model. In addition, previous findings have shown that men who have sex with multiple men (MSMM) have a higher risk of HIV infection than other men (Jiaxiu et al., 2021). We categorized MSM with two or more male sexual partners as MSMM and the rest as Non-MSMM. On the one hand, both MSMM and Non-MSMM were applied to the optimal model to evaluate the prediction effect; on the other hand, separate prediction models were built for MSMM and Non-MSMM to evaluate the prediction effect. The performance of the two prediction methods in MSMM and Non-MSMM was compared.
ResultsDescription of demographic informationA total of 747 MSM participated in the first follow-up, and their basic demographic information were described in Supplementary Table 2. According to the results of the descriptive analysis, most of the recruited MSM were over 25 years old, mainly urban population, Han ethnicity, and highly educated. 57.56 % of MSM had high adherence.
Variable screeningThe process and results of the LASSO screening were shown in Fig. 2. According to the results presented in Fig. 2, a total of 11 variables were filtered out.
Univariate analysis for different adherence groups
The 11 variables screened were compared between the two groups (Supplementary Table 3). It was observed that the variable “Reminder system” was not statistically different between the high and low adherence groups (p = 0.065), while the rest of the variables showed significant differences between the two groups.
Model constructionThe screened 11 variables were brought into the model as independent variables, split into the training set (80 %) and the testing set (20 %), and balanced using the SMOTE algorithm (Supplementary Table 4 for the comparison of the training and testing sets). LR, DT, SVM, RF and XGBoost methods were used to train and predict the model (setting low adherence as a positive result, in which the predicted result was the nonadherent MSM). The evaluation metrics of each prediction model on the testing set were shown in Table 1. According to the prediction results of the model, RF performed best on the ROC-AUC metric, with a value of 0.704 (95 % CI=0.627–0.781); RF performed best on the Sensitivity metric, with a value of 67.44 %.
Evaluation metrics for each predictive model on the testing set.
LR: Logistic regression; DT: Decision tree; SVM: Support vector machine; RF: Random forest; XGBoost: Extreme gradient boosting machine; AUC-ROC: Area under the curve of receiver operating characteristic; PPV: Positive predictive value; NPV: Negative predictive value.
The results of the AUC-ROC comparison of the five prediction models were shown in Table 2. It was observed that the AUC-ROC of the RF prediction model (AUC-ROC=0.704, 95 %CI=0.627–0.781) outperformed the traditional Logistic regression model (AUC-ROC=0.660, 95 %CI=0.579–0.741), and the XGBoost model (AUC-ROC=0.675, 95 %CI=0.595–0.755).
Results of AUC-ROC comparison of five predictive models.
| Models | LR | DT | SVM | RF | XGBoost |
|---|---|---|---|---|---|
| LR | – | 0.917 | 0.983 | <0.05 | 0.492 |
| DT | 0.917 | – | 0.924 | 0.142 | 0.587 |
| SVM | 0.983 | 0.924 | – | 0.248 | 0.723 |
| RF | <0.05 | 0.142 | 0.248 | – | <0.05 |
| XGBoost | 0.492 | 0.587 | 0.723 | <0.05 | – |
LR: Logistic regression; DT: Decision tree; SVM: Support vector machine; RF: Random forest; XGBoost: Extreme gradient boosting machine; AUC-ROC: Area under the curve of receiver operating characteristic.
Fig. 3-A illustrated the AUC-ROC results for the five predictive model. RF performed optimally in predicting PrEP adherence in the MSM population. Fig. 3-B illustrated the results of the five prediction models in the ten-fold cross-validation. All five prediction models showed stability in the ten-fold cross validation. The 6-month, 9-month, and 12-month follow-up data were brought into the trained RF model (Fig. 3-C). The ranked order of the variables for the optimal model was shown in Fig. 3-D. Based on the partial dependence plots of top 5 most important variables (Fig. 3-E), MSM with negative attitudes of male sexual partners, low self-efficacy, not having been tested for HIV, not having a male sexual partner, and having low risk perception were more likely to engage in nonadherent behaviors.
Summary of model prediction results, variable importance and partial dependence plot. A AUC-ROC results for five prediction models. B Results of the five predictive models in ten-fold cross-validation. C Results of bringing 6-month, 9-month, and 12-month follow-up data to the optimal model. D Ranking of variable importance for the optimal model. 3-E Partial dependence plot of the top 5 ranked variables.
LR: Logistic regression; DT: Decision tree; SVM: Support vector machine; RF: Random forest; XGBoost: Extreme gradient boosting machine; CV: Cross validation; AUC: Area under the curve; ROC: Receiver operating characteristic curve; HIV: Human immunodeficiency virus; AIDS: Acquired immune deficiency syndrome.
The predictions of the optimal model were eventually displayed in a shiny web application for online estimation of probabilities and interactive computation (Fig. 4), which was available at: https://lauralin1104.shinyapps.io/RFshiny/.
Sensitivity analysis
Based on the sensitivity analyses, we obtained that (1) the inclusion of a single variable as a predictor variable in the predictive model was not as good as the full-variable model (Supplementary Table 5). (2) The optimal model performed well in predicting different adherence categorization criteria (Supplementary Fig. 1). (3) The optimal model could be used to directly predict adherence in different populations (Supplementary Table 6).
DiscussionOur findings indicated that the machine learning approaches were accurate in predicting PrEP adherence. The most useful predictors were routine and easy to collect. The results of the sensitivity analyses showed that the trained optimal model could be used in different categorization criteria for adherence, as well as applied to different populations of MSM. The routine collection of these data and the accuracy of their predictive results suggested that machine learning approaches may be of value in routine PrEP medication management services. In addition, machine learning predictive modeling was applied to the web application to help identify nonadherent individuals in the MSM population, which can help improve medication adherence and increase prevention efficacy in high-risk population.
In our study, the MSM in Western China had an adherence rate of 57.56 % in the use of PrEP. Although this rate is relatively high, it is still not sufficient to ensure optimal protection against HIV, highlighting the need for further interventions to improve adherence. Therefore, timely identification and intervention are necessary for MSM who were nonadherent during PrEP. Adherence of study participants was measured using self-report. Previous studies have shown that this method may result in inaccurate estimation of adherence due to recall bias and social desirability bias (Buscher et al., 2011; Williams et al., 2013). However, it was also noted in the literature that self-reported data from the participants were reliable (Marins et al., 2019). The previous study of the PrEP program from a clinical setting, researchers found a significant correlation between self-reported adherence and medication levels, demonstrating the feasibility and validity of self-reports for measuring PrEP adherence (Montgomery et al., 2017). Compared to other methods of measuring adherence, such as measuring Tenofovir-diphosphate (TFV-DP) concentration in dried blood spots (DBSs) (Goldwirt et al., 2021), electronic pill cap (Atukunda et al., 2022), drug level monitoring in hair and urine (Hannaford et al., 2021), self -reported adherence was more convenient and quicker to use for initial assessment of early clinical adherence and was also more easily available in any setting. Meanwhile, compared with the classical HIV/STIs prediction model, the AUC values of the individual models for predicting HIV ranged from 0.698 to 0.763 (Bao et al., 2021). The mean value of the AUC of our RF prediction model for adherence in ten-fold cross-validation was 0.757, indicating that RF performed well and reliably in predicting adherence in the MSM population. It was evident that the predictive model of PrEP adherence that we developed using MSM individual characteristic data, with variables that were easy to collect in daily routine, was feasible and valuable in practical applications.
Our findings indicated that attitudes of male sexual partners were a significant variable in predicting PrEP adherence in the MSM population and ranked first. MSM whose male sexual partners' attitudes were negative were more likely to engage in nonadherent behaviors. This was an interesting finding because few studies in China have related the attitudes of male sexual partners to PrEP adherence in the MSM population. In a study conducted with same-sex male couples, 26.7 % reported that their sexual partners did not support the use of PrEP, and 57.7 % did not believe that PrEP would be beneficial to them or their sexual partners (Kahle et al., 2020). However, in China, influenced by traditional culture, it is very difficult to conduct surveys among male couples. The cultural and social environment in China has low acceptance of MSM, and many MSM individuals face significant pressure and stigma from their families, society, and workplaces (Zhou et al., 2024). This environment may lead MSM individuals to have more conservative and resistant attitudes towards openly discussing and adopting sexual behavior-related preventive measures, such as PrEP. Previous research showed that the use of peer education intervention can encourage MSM to access HIV prevention information, counseling, and testing (Duan et al., 2013). Perceived peer use of PrEP was positively correlated with personal use of PrEP (Mike et al., 2020). It was concluded that MSM's male sexual partners, as the manifestation of an intimate peer relationship, were likely to influence their interest, willingness, and adherence to PrEP, which highlighted the need for tailored dyadic (users and male sexual partners) interventions. At the same time, future study should consider the impact of cultural background on the health behaviors and attitudes of MSM individuals. Cross-cultural comparative studies can be conducted to further explore the awareness and acceptance of PrEP among MSM population in different cultural contexts.
Self-efficacy in MSM population was also one of the important predictors of their adherence behavior. Previous studies mentioned that adherence self-efficacy was associated with PrEP knowledge and confidence in HIV prevention effectiveness (Sun et al., 2021). In addition, previous study in Mississippi Southern MSM showed that greater self-efficacy for taking PrEP was more likely to be adherent to PrEP (Whiteley et al., 2021). This was consistent with our findings. At the same time, HIV testing as one of the predictive variables for adherence suggested that we need to focus on the population who have not been tested for HIV, and these MSM may be at a higher risk of infection. HIV testing and PrEP were potentially complementary tools that could be combined to enhance the ability of at-risk individuals to prevent HIV infection (Mujugira et al., 2022). In our study, the number of male sexual partners and risk perception were shown to be significantly associated with PrEP adherence, which was also consistent with previous research findings (Lebouché et al., 2016; Wood et al., 2019). Our study used the variable reminder system to predict adherence in the MSM population. Previous studies have found that forgetting to take medication was the most common objective cause of nonadherence, accounting for 70.21 % (Qu et al., 2018). Our findings suggested that APP-based daily reminder mechanisms were associated with adherence in the MSM population. Furthermore, the effectiveness of mobile health (mHealth) intervention in promoting adherence has been observed in various populations. For instance, previous studies have found that digital health interventions in relation to obesity management have the potential to achieve relatively high rates of adherence (Jakob et al., 2024). Meanwhile, mHealth can also be used to support self-management of cancer pain (Wu et al., 2024). It highlights the importance of mHealth, with customizable reminders, tailored content, and personalized services that will help increase the effectiveness of digital health interventions in promoting healthy behaviors.
Strengths and limitationsAll predictors in our model were self-reported variables, so our predictive tools may have real-life applications. Based on the shiny web application we developed, a user-friendly, internet-based self-assessment tool could potentially help the MSM population identify nonadherent behaviors during PrEP use in a clinical setting. This allowed clinicians or administrators to identify individuals at high risk for nonadherent behaviors and provide timely intervention and medication management. In the clinical setting, we believe that this tool would help clinicians categorize high-risk individuals and guide the use of PrEP more effectively. By being more alert to individuals with potential nonadherent behaviors, clinicians can offer personalized support and interventions, such as more frequent follow-ups, counseling, and reminders. In addition, this tool could be integrated into electronic health records (EHR) systems to streamline the process of identifying and monitoring high-risk individuals. For example, alerts could be generated for healthcare providers when a patient is flagged as high-risk, prompting immediate action. Moreover, risk assessment of individuals provides quantitative information about the overall risk of PrEP nonadherence behaviors in the MSM population, which is crucial for behavioral and epidemiologic surveillance. This data can inform public health strategies and policies aimed at improving adherence rates, such as targeted education campaigns and community-based support programs.
There were some limitations to our study. Firstly, as a sensitive population, MSM attach great importance to privacy issues. Since the investigation in this study was in the form of a face-to-face questionnaire and all variables were self-reported information, some sensitive issues may not be authentically represented. More emphasis should be placed on privacy protection for sensitive populations in future investigations. Secondly, our study data sources were subjective and lacked objective measures. For example, measures of adherence lacked objective data. However, for objective measures of adherence, for instance, measuring blood levels in the user's body, which were invasive, costly, and time-consuming, were not necessarily widely accepted and used on a large scale by the MSM in the clinic. Self-reported methods had the advantage of being convenient and quick to use and could be widely used in the clinical setting as a measurement tool for initial screening. In addition, during further validation of the model, we found that the predictive efficacy of the optimal model performs well when 6 months of data were brought in. However, the predictive efficacy decreased with the 9 and 12 months of data. We analyzed that this may be related to loss to follow-up (LFU) among MSM in the late follow-up period. Previous studies have also shown a higher risk of PrEP LFU among younger and less educated MSM (Amanda et al., 2022). These variables were disproportionately lost during follow-up and may have affected the predictive efficacy of adherence. Currently, our application is designed for online visualization only, and we are actively considering implementing more functions in a future update to enhance the usability of our application. In the future, if the opportunity arises, we can investigate the effects of the model's predictions on other populations, thereby extending the utility of the web application. Finally, our study used only a few of the most commonly used machine learning algorithms for adherence prediction, and other methods could be explored in future research. Since the model was constructed only in the Western region of China, further exploration and validation were needed to extend the optimal model to the whole country to make it universally applicable. Meanwhile, fuller external validation should be conducted in the future.
ConclusionIn conclusion, our study demonstrated that machine learning algorithms excel in predicting nonadherent behaviors among MSM using PrEP. This innovative online assessment tool represents a substantial advancement over traditional methods by offering real-time, personalized risk assessments and interventions. It stands out by leveraging predictive analytics to proactively address nonadherence, which is crucial for improving health outcomes in a vulnerable population. The tool's ability to dynamically adapt and learn from new data positions it as an invaluable asset in the public health arsenal, particularly in the monitoring and management of PrEP adherence among MSM in Western China. Future research should explore expanding this tool to other high-risk populations to enhance its generalizability and public health impact. In addition, integrating new predictive variables, such as psychosocial factors and behavioral patterns, could further refine its accuracy and utility. This continuous development and application of machine learning technologies hold promise for advancing public health strategies and optimizing PrEP adherence.
