General Information
Conflict of Interest
- yoda_project_coi_form_for_data_requestors_2018_turecki
- robert190621_-_yoda_conflict_of_interest_form
- coi_disclosureform_kp
- conflict_of_interest_joseph_mehltretter
- 617_caitrin_updated_signed
- mail_-_dua_confirmation
- signed_david_yoda_project_coi_form_for_data_requestors_2018_updated2
Request Clinical Trials
Associated Trial(s): What type of data are you looking for?:Request Clinical Trials
Data Request Status
Status: Unknown - RevokedResearch Proposal
Project Title: Machine learning prediction of remission in patients given augmented treatment in major depression
Scientific Abstract:
Background: Globally, clinical depression affects over 320 million people and is the leading cause of disability worldwide. While numerous treatment options do exist, most patients spend months to years undergoing an arduous trial-and-error process before finding a treatment that works for them.
Objective: We will train machine learning models to help determine when adjuncts to antidepressants would be useful in an effort to reduce time to recovery. All models will be published and made open source and not directly commercialized.
Study design: We will use patient data to train our deep learning model to predict the outcome of methylphenidate hydrochloride and risperidone augmentation of SSRI/SNRI treatment for treatment-resistant Major Depressive Disorder.
Participants: Patients diagnosed with major depressive disorder who have failed to respond to at least one antidepressant treatment and were given risperidone or methylphenidate augmentation.
Main outcome measures: Models will be trained to maximize specific model metrics: AUC, PPV, NPV, sensitivity, specificity. These models are then tested to see if they provide projected improvement in population remission on a held out sample of the data and if this is significantly different from random allocation in a series of bootstrapped samples.
Statistical analyses: To predict different clinical outcomes, our custom high-level deep learning pipeline, Vulcan, supports deep learning, random forest, and other models as well as a feature selection pipeline (i.e variance thresholding, recursive feature elimination).
Brief Project Background and Statement of Project Significance:
Major depressive disorder occurs in an estimated 10%-15% of the population, and remains one of the highest public health concerns. While there is a range of pharmacological treatment options, nearly one third of patients fail to respond to adequate doses of antidepressant agents. Treatment-resistant depression (TRD) is defined as major depressive disorder (MDD) with symptoms that fail to respond to treatment with at least two different classes of antidepressant medications (Al-Harbi, 2012). These patients require further treatment that may involve switching antidepressant medication, combining different antidepressants, the addition of a medication which is not an antidepressant, or somatic therapies (Philip, Carpenter, Tyrka, & Price, 2010). This is done on a trial-and-error basis, which results in a long and difficult process for the patient.
Both risperidone, an antipsychotic agent, and methylphenidate, a stimulant used to treat ADHD, have been used as options for augmentation of antidepressant monotherapy for TRD. In general, risperidone augmentation has been found to be more effective than placebo, although results are mixed. Methylphenidate has been found to be effective in elderly patients, but results are less positive in other populations (Barowsky & Schwartz, 2006; Philip et al., 2010). It is clear that these augmentation strategies are not effective for everyone, and further investigation is required in order to determine who may benefit the most from these treatments. Due to the complexity of TRD, and the heterogeneity of symptoms, and medication side effects, there is likely a complex interaction of various factors that impact the efficacy of methylphenidate and risperidone augmentation.
In order to unravel these interactions, a machine learning approach that can interpret data sets with a large number of variables may be the most effective solution to the challenge of predicting efficacy of treatment augmentation. To address this challenge, we have developed our pipeline, Vulcan AI, which provides a comprehensive pipeline for high-dimensional data visualization, data preprocessing, rapid modular network prototyping, training, evaluation, and model interpretability. We use a deep neural network to analyze our data, allowing us to capture the complex, non-linear relationships likely to be present in psychiatric data (e.g. mediation and moderation effects, which are often unknown a priori). When data is present in insufficient quantities for deep learning to be useful, Vulcan can be used to implement random forests, regression, and gradient boosting. By training the algorithm with data that includes the outcomes of antidepressant augmentation with methylphenidate and risperidone augmentation, we should be able to identify patients with TRD that will benefit from these treatments, reducing the arduous trial-and-error process inherent in the treatment of depression that does not respond to first line monotherapy.
Specific Aims of the Project:
The aims of this project are two-fold: 1) improving our machine learning model to include prediction of augmentation treatment utility and 2) to improve understanding of subtypes of treatment resistant depression (TRD). We aim to expand our model to include treatment efficacy predictions for patients who have been unresponsive to classical antidepressant medication. By training our model with data that includes the outcomes of antidepressant augmentation with risperidone or concerta for treatment resistant depression, we hope that we will be able to effectively identify patients with TRD who will likely benefit from this treatment. Ideally, this would reduce the arduous trial-and-error process that these patients typically go through.
Through the training process, our algorithm will also identify a range of factors that it has identified as important for predicting outcome. This knowledge, in and of itself, would add to the scientific literature surrounding treatment resistant depression, by providing a starting point for classifying different subtypes of TRD based on symptom features that may determine response to treatment augmentation with antipsychotics or stimulants. It allows us to identify features that not only predict outcome but also act as mediators or moderators.
Study Design:
What is the purpose of the analysis being proposed? Please select all that apply.: Preliminary research to be used as part of a grant proposal Develop or refine statistical methods Research on clinical prediction or risk prediction
Software Used: I am not analyzing participant-level data / plan to use another secure data sharing platform
Data Source and Inclusion/Exclusion Criteria to be used to define the patient sample for your study: All patients with primary diagnoses of major depression and without bipolar disorder will be included.
Primary and Secondary Outcome Measure(s) and how they will be categorized/defined for your study:
Accuracy of remission prediction as per the area under the receiver operating curve (AUROC); remission will be defined as a subthreshold score on the main outcome questionnaire used in the study. We will also look at our model?s ability to produce projected improvements on the population remission rate.
Other measures will include the model's sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).
Main Predictor/Independent Variable and how it will be categorized/defined for your study: We do not define a priori predictors; these will be selected by our feature selection pipeline.
Other Variables of Interest that will be used in your analysis and how they will be categorized/defined for your study: We do not define a priori predictors; these will be selected by our feature selection pipeline.
Statistical Analysis Plan:
As noted, we are using machine learning methods- mostly deep learning, but also random forests, logistic regression, and gradient boosting. We plan to first analyze the data from the two studies we are requesting and the study protocols to determine if the studies can be pooled or not, based on whether or not the study protocols were sufficiently similar and if similar data was collected between the two studies. After this, we will pass the data (pooled or separately, depending on if the data can be pooled) through our feature selection pipeline, which will use recursive feature elimination, variance thresholding, and feature importance thresholding to select the features most predictive of remission. Following this we will train four models: a deep learning model, a random forest model, a logistic regression, and a gradient boosting approach, and analyze resulting model metrics to determine which approach provides the highest AUC. Data will be split into training and test splits, which models trained and internally cross-validated (k= 10) prior to being tested on a held-out sample. Another held-out sample will be used to measure model predictions regarding population remission rate. This step consists of a naive, hypothetical analysis followed by a conservative, non-hypothetical analysis. The naive analysis consists of predicting remission rates with different treatments for each patient in the held-out dataset, and then averaging their sample remission rate and comparing it to the population remission rate. This provides a blue-sky estimate of model utility in differential treatment benefit prediction, but remains hypothetical. We then produce 1000 bootstrapped samples and look at the patients who happened to be randomized to the treatment our model suggests they should have received and see if these patients do better in terms of remission rate compared to the general patient population; this allows for us to determine if the model could significantly improve remission rates.
A NOTE ON USE OF MACHINE LEARNING- REPLY TO REVIEWER COMMENTS
Thank you for the question about the sample size and dimensionality. With respect to dimensionality, we have had success in building models with good predictive and differential predictive performance using models with less than 50 features- in fact between 17 and 31. As these are mostly symptom scales, which will be present in this data, we do believe that we will be have sufficient dimensionality. We have also found that we can use machine learning analyses on smaller than usually anticipated datasets without overfitting- with the ability to fit models to datasets with ~600 people. The 2 trials listed here are likely too small in isolation for deep learning, and as such our plan is to use less advanced, more traditional machine learning algorithms to start- beginning with linear regression (for continuous outcomes like final score) or logistic regression (using remission/non-remission) and then moving to gradient boosting and random forests. We will try and see at that point, once we have a few good traditional baseline models, how deep learning perform, and also how some of our previous deep learning algorithms trained on larger datasets (including one with an augmentation treatment) performs when predicting this data (we will not mix the data from this project with data from previous projects). After we publish the models from this project in an open-source manner, we will also explore federated learning approaches- how to have models trained on different data but in the same problem space work together to provide predictions. As such our work will still be able to employ machine learning approaches, but these will be selected with the sample size and risk of overfitting in mind.
Narrative Summary: Globally, clinical depression affects over 320 million people and is the leading cause of disability worldwide. As such a widespread disorder that is incredibly symptomatically homogenous, two thirds of patients fail to respond to the first treatment and require numerous treatment changes. As such, the treatment selection process in depression is essentially an educated ?guess and check? approach. Using machine learning, we aim to personalize this process by using patient data to predict the most effective treatments for each individual so that patients can reach recovery faster. In particular, we aim to predict which patients will benefit the most from the addition of adjuvant treatments.
Project Timeline: Assuming a 12 month timeline, we will spend the first three months familiarizing ourselves with the data and building initial models, as well as installing needed packages on the online portal (which we have already checked is possible). We will then spend two months refining and finalizing the models. We will have completed the draft of our manuscript three months later (8 months into the project) and will circulate to co-authors and submit it by month 9 of the project; that way, we still have three months to use the data to complete any revisions.
Dissemination Plan: We plan to include all of our findings, in manuscripts to be submitted to peer-reviewed journals. Some appropriate journals may include JAMA Psychiatry, American Journal of Psychiatry, and Translational Psychiatry. In addition, our code for Vulcan AI is open source, available on GitHub (https://github.com/Aifred-Health/Vulcan). We will also be disseminating the final models via a Git-like service and will be submitting our work to conferences. We also generally post all of our work on pre-print servers.
Bibliography:
Al-Harbi, K. S. (2012). Treatment-resistant depression: therapeutic trends, challenges, and future directions. Patient Preference and Adherence, 6, 369?388. https://doi.org/10.2147/PPA.S29716
Barowsky, J., & Schwartz, T. L. (2006). An Evidence-Based Approach to Augmentation and Combination Strategies for. Psychiatry (Edgmont), 3(7), 42?61.
Philip, N. S., Carpenter, L. L., Tyrka, A. R., & Price, L. H. (2010). Pharmacologic Approaches to Treatment Resistant Depression: A Re-examination for the Modern Era. Expert Opinion on Pharmacotherapy, 11(5), 709?722. https://doi.org/10.1517/14656561003614781