Research Proposal
Background: Schizophrenia and bipolar disorder are severe major mental disorders of which the prevalence is 1.1% and 2.6% among adults in the United States (US), respectively [9, 12]. Many randomized clinical trials reveal that antipsychotic drugs are helpful to treat patients with schizophrenia or bipolar disorder. However, given that participants in small randomized trials are often not representative of a general population it is questionable whether the results can be applied to individuals with these disorders across the US.
Objective: To develop and assess new statistical methods for generalizing treatment effects estimated from network meta-analysis for bipolar disorder and schizophrenia to the US populations of individuals with these disorders.
Study Design: We will perform network meta-analyses with individual-level participant data after standardizing subjects in each trial to look like the target population of interest.
Participants: Adult participants with a DSM-IV diagnosis of schizophrenia or Bipolar I disorder will be included in our analyses.
Main Outcome Measure(s): Our primary efficacy outcomes will be the change scores on the Positive and Negative Syndrome Scale total score and the Young Mania Rating Scale for participants with schizophrenia and bipolar disorder, respectively. Our primary safety outcome will be suicide/self-injury related adverse events.
Statistical Analysis: We will implement propensity score methods to generalize each randomized trials and then will perform Bayesian random-effect network meta-analyses to combine across trials.
Randomized clinical trials (RCTs) are generally considered the gold standard for investigating the efficacy and safety of treatment on outcome [1]. Synthesizing results from all accessible RCTs investigating a treatment compared to placebo, called a meta-analysis, provides a comprehensive understanding of the treatment effect by accounting for heterogeneity of treatment effects across trials [5]. Over the past few years, researchers have developed network meta-analysis methods to compare multiple treatments at one time and find the best treatment in terms of efficacy and safety even if those treatments were not directly compared in each trial [6, 7, 10, 11]. More recently, advances in data sharing systems and computing power enable us to integrate all available individual participant-level data (IPD) [8, 13]. The primary merit of combining IPD is that we can incorporate participant characteristics such as demographics and baseline severity of illness into the analysis, which can help enhance the field of personalized medicine. However, it is difficult to answer a question such as “can we make the same decisions to other people in a population of interest?” In other words, “can the results of RCTs be generalized to broader groups of individuals?”
Suppose, for example, that we want to generalize the effect of antipsychotic drugs to the adult population in the US with schizophrenia. Results from a single RCT are not enough to transfer the results to the population because the sample of the trial might not represent the population well. To compensate for this limitation, we can combine results from multiple RCTs. Since different RCTs may enroll different subsets of the population, combined subjects from multiple RCTs might represent the target population better than subjects from a single trial. However, results from standard network meta-analysis may be misleading because they pool trials without considering heterogeneous characteristics of subjects across trials (e.g., some trials may have mostly older participants while others mostly have younger participants.) While methods have been developed to generalize from one trial to a target population [3, 14, 15], they have not been extended to meta-analysis settings [2]. In this project, we aim to draw better inferences that combine information from the existing RCTs and account for the fact that the subsets in each trial may be different from the target population.
This research will provide novel methods to estimate population treatment effects in network meta-analysis and will apply those methods to existing RCTs for schizophrenia and bipolar disorder. As a result, this research will lead to a better understanding of the gap between well-designed randomized studies and real world settings, and thus change the way researchers interpret scientific findings to be more relevant to general clinical practice.
Aim 1. Assess heterogeneity of treatment effects across studies, countries, and patient characteristics using network meta-analysis methods with individual-level participant data that pool information across studies.
Aim 2. Evaluate the representativeness of each RCT and develop propensity score weighting methods for estimating population average treatment effects from network meta-analysis.
Aim 3. Apply the methods from Aim 2 to the two sets of trials for (1) schizophrenia, and (2) bipolar disorder to estimate treatment effects and compare these estimates to those estimated from non-generalized network meta-analysis.
We will carry out this project as a career transition award (K99), supported by the National Institute of Mental Health. The K99 award supports postdoctoral researchers who pursue being independent researchers (i.e., tenure-track faculty). As the K99 award supports projects up to 5 years: 2-year mentored phase and 3-year independent research phase, we plan to accomplish Aims 1 and 2 in the first 2 years under supervision of mentors and then Aim 3 in the next 3 years independently.
We will include all RCTs investigating antipsychotic treatments for schizophrenia and bipolar disorder to perform network meta-analysis with individual-level participant data. We will also use individual-level participant covariates including demographics (e.g., gender and age), family history of mental disorder, and baseline mental illness severity information. The primary endpoint will be the most commonly reported follow-up time across studies in the 6 to 13 week window. We will consider study participants who are male or female, aged older than 18 with a Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (DSM-IV) diagnosis of schizophrenia or bipolar I disorder.
Based on the Clinical Study Report Synopsis, partially available online, we identify 20 trials for schizophrenia and 12 trials for bipolar disorder. The 20 trials for schizophrenia will provide evidence regarding the comparative effectiveness and safety of paliperidone, paliperidone palmitate, risperidone, and risperidone long acting injection. The 12 trials for bipolar disorder will allow assessment of comparative effectiveness and safety of paliperidone, risperidone, risperidone LAI, and topiramate.
Our primary efficacy outcomes will be the change scores from baseline on the Positive and Negative Syndrome Scale (PANSS) total score and the Young Mania Rating Scale (YMRS) for schizophrenia and bipolar disorder, respectively. Based on Clinical Study Report Synopsis, all trials for schizophrenia measured the PANSS total score and all trials for bipolar disorder measured the YMRS. We thus can conduct network meta-analyses using the same outcome measure across studies.
Our primary safety outcome will be suicide/self-injury related adverse events.
The main predictors include participants’ characteristics:
(1) Demographics: age, gender, nationality etc.,
(2) Clinical information: weight, body mass index, duration of mental illness, drug or alcohol abuse, etc., (3) Family history: the number of family members having mental illness, paternal age, etc., and
(4) Severity of symptoms: baseline score for primary outcome.
These predictors will be used to assess treatment effect heterogeneity across trials and similarity between a trial and the target population of interest.
We will also use some secondary efficacy outcomes including Clinical Global Impression Scale and Personal and Social Performance Scale. For secondary safety outcomes, we will consider treatment-emergent adverse events, electrocardiograms and extrapyramidal symptoms rating scales.
For Aim 1, we will first investigate the extent of the evidence around treatment effect heterogeneity across studies in two sets of trials: treatments for (1) schizophrenia, and (2) bipolar disorder. It is important to investigate treatment effect heterogeneity across studies because generalizability is most threatened by treatment effect heterogeneity. We will perform network meta-analysis with IPD under a Bayesian framework to combine results from multiple RCTs. The models will include random effects to capture treatment effect heterogeneity across trials. We will also include treatment by covariate interactions to assess treatment effect heterogeneity across different patient characteristics.
For Aim 2, we will first apply the existing generalizability methods to each trial and re-estimate a generalized treatment effect for each trial. The existing generalizability methods use propensity scores to estimate the probability of participating in each trial given a set of covariates. Weights based on these propensity scores can then be used to weight subjects in each trial to look like the target population of interest. The trials, once equated in this way, can then be compared in a meta-analysis in a more principled way, to allow inferences about the effects of the treatment in the target population.
Finally, for Aim 3, we will apply the proposed generalizability methods for network meta-analysis to trials for schizophrenia and bipolar disorder. These generalized treatment effect estimates will be compared with the treatment effect estimates from non-generalized network meta-analysis.
This project will help researchers generalize, apply, and implement findings from network meta-analysis of randomized clinical trials to target populations. The network meta-analysis enables us to borrow strength across multiple studies to gain more power, resulting in obtaining reliable and precise effect estimates. By generalizing results of network meta-analysis, we will translate the results into appropriate policy and practice decisions.
This project will be submitted to the National Institute of Mental Health as a K99 application, due February 12, 2016. We expect the results of the review in mid-2016. We aim to accomplish our Aim 1 in the first year after we have data access. At the end of Year 1, the first paper will be prepared for publication. We will report all results to the YODA project and ask to renew the data use agreement. We will complete Aim 2 in Year 2 and the second paper will be prepared for publication at the end of Year 2. Again, the results will be reported to the YODA project as requested and we hope to analyze the data for an additional 3 more years to carry out Aim 3.
The target audience of our manuscripts will be psychiatrists, biostatisticians, and epidemiologists. Possible journals for methodological work include the Journal of the American Statistical Association, Journal of the Royal Statistical Society, Annals of Applied Statistics, and Statistics in Medicine. Possible journals for clinical findings include the American Journal of Psychiatry, JAMA Psychiatry, and Psychiatric Services. Our results will be presented in major statistics and psychiatry conferences such as Eastern North American Region, Joint Statistical Meetings, and American Psychiatric Association Annual Meeting.
1. Baker SG and Kramer BS. (2003). Randomized trials, generalizability, and meta-analysis: graphical insights for binary outcomes. BMC medical research methodology, 3(1), 10.
2. Bonell C, Oakley A, Hargreaves J, Strange V, and Rees R. (2006). Research methodology: Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ: British Medical Journal, 333(7563), 346.
3. Cole SR and Stuart EA. (2010). Generalizing evidence from randomized clinical trials to target populations the actg 320 trial. American journal of epidemiology, 172(1), 107-115.
4. Cosgrove VE and Suppes T. (2013). Informing DSM-5: biological boundaries between bipolar I disorder, schizoaffective disorder, and schizophrenia. BMC medicine, 11(1), 127.
5. DerSimonian R and Laird N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177–188.
6. Hong H, Carlin BP, Shamliyan T, Wyman JF, Ramakrishnan R, Sainfort F, and Kane RL. (2013). Comparing Bayesian and frequentist approaches for multiple outcome mixed treatment comparisons. Medical Decision Making, 33(5), 702-714.
7. Hong H, Chu H, Zhang J, and Carlin BP. (2015). A Bayesian missing data framework for generalized multiple outcome mixed treatment comparisons. Research Synthesis Methods. In press.
8. Hong H, Fu H, Price KL, and Carlin BP. (2015). Incorporation of individual patient data in network meta-analysis for multiple continuous endpoints, with application to diabetes treatment. Statistics in Medicine, 34(20), 27942819.
9. Kessler RC, Chiu WT, Demler O, and Walters EE. (2005). Prevalence, severity, and comorbidity of twelve-month DSM-IV disorders in the National Comorbidity Survey Replication (NCS-R). Archives of General Psychiatry, 62(6), 617-27.
10. Lu G, Ades AE. (2004). Combination of direct and indirect evidence in mixed treatment comparisons. Statistics in Medicine, 23(20), 3105–3124.
11. Lumley T. (2002). Network meta‐analysis for indirect treatment comparisons. Statistics in medicine, 21(16), pp. 2313-2324.
12. Regier DA, Narrow WE, Rae DS, Manderscheid RW, Locke BZ, and Goodwin FK. (1993). The de facto mental and addictive disorders service system. Epidemiologic Catchment Area prospective 1-year prevalence rates of disorders and services. Archives of General Psychiatry, 50(2), 85–94.
13. Riley RD, Lambert PC, and Abo-Zaid G, (2010). Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ: British Medical Journal, p. 340:c221.
14. Stuart EA, Cole SR, Bradshaw CP, and Leaf PJ. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369-386.
15. Stuart EA, Bradshaw CP, and Leaf PJ. (2015). Assessing the generalizability of randomized trial results to target populations. Prevention Science, 16(3), 475-485.