Skip to main content


Research Proposal

Project Title: 
Identifying heterogeneous treatment effects from canagliflozin: development and validation of models for HbA1c reduction and adverse event risk
Scientific Abstract: 

Background: Identifying heterogeneous treatment effects (HTEs) is important for the treatment of type 2 diabetes, particularly because medications chosen after metformin monotherapy pose both potential benefits (e.g., hemoglobin A1c reduction, cardiovascular disease risk reduction), and potentially serious risks (adverse events such as urogenital infections).
Objective: To identify HTEs from canagliflozin.
Study design: Development and validation of risk models for reduction in HbA1c, reduction in atherosclerotic cardiovascular disease, and increase in probability of serious adverse events.
Participants: N = 5,971 from eight randomized, double-blind canagliflozin trials in YODA.
Main outcome measures: Absolute percentage point decrease in HbA1c at 52 weeks; and absolute probability of serious adverse event at 52 weeks.
Statistical analysis: A limited subset of theory-based potential predictor variables for HTEs have been chosen for potential inclusion in the models. Gradient forest analysis will be performed using the pre-randomization values of these potential predictors. Gradient forest analysis develops multivariate models for HTEs in each outcome measure, based on repeated cross-validation of decision trees that are constructed to explain variation in observed treatment effect (absolute percentage reduction in HbA1c, absolute risk increase in serious adverse event rate) between study arms among patient subgroups. A 75% stratified random sample across all trials will be used for derivation and internal cross-validation, with the remaining

Brief Project Background and Statement of Project Significance: 

A goal of precision medicine is to identify patients more likely to experience benefit or harm from a given therapy (heterogeneous treatment effects, HTEs). HTEs are difficult to identify through typical univariate subgroup analyses, which have limited statistical power (1–3). Additionally, clinical care is not well-informed by univariate analyses (e.g., if males experience benefit but older people experience harm, how should a practitioner counsel an older male?). Consequently, multivariate HTE modeling has been recommended to advance personalized decision-making (4–6), but poses the risk of generating false positive results with multiple testing.
Recently, machine learning methods—particularly gradient forest analysis (7)—have aided identification of HTEs. Gradient forest analysis can separate trial populations into subgroups characterized by multiple simultaneous characteristics, using cross-validation and P-value correction to reduce false positives (7,8). We have adapted the gradient forest method to help identify HTEs when pooling data across trials with different study designs, including trials with differing medication dosage, co-occurring medications, or control groups, using principles from network meta-analysis (9–11) (NMA). The technique can create new risk prediction tools from individual participant data, while accounting for diversity between studies. This application will be the first use of the technique, to our knowledge, to clinical data; we have applied the method to non-network meta-analysis setting from single trials, but only simulated pooled analyses to establish non-bias and low false-positive rates (8).
Estimating HTEs for new glycemic agents for type 2 diabetes is particularly important, as individualizing glycemic treatment is now recommended (12), but how best to individualize treatment remains unclear. Prior NMAs suggest that newer diabetes drugs present large potential benefits and large potential risks (13,14). Canagliflozin, a sodium glucose co-transporter 2 (SGLT-2) inhibitor, increases glucose excretion in urine, significantly reducing HbA1c and associated disease complications (15,16). But canagliflozin also presents increased risk of adverse events including urogenital infections, bony fractures, and lower limb amputations (15,17). Those receiving the most benefit from canagliflozin in terms of reduced HbA1c were not those experiencing serious adverse events in published trials to date—suggesting that HTE models may be clinically helpful to distinguish high-benefit from high-risk patients (15). Canaglifozin had greater A1c reduction than almost any other new diabetes medicines. In spite of that, the risk of limb amputation in particular may make it too high-risk for clinical use. Therefore, identifying which populations are lower versus higher benefit and lower versus higher risk is of clinical importance.  
Hence, our development of HTE models in this study may advance scientific knowledge about the development of benefit/risk models to personalize medical therapies. The study may also add to generalizable knowledge for treatment of type 2 diabetes.

Specific Aims of the Project: 

Study Objective:
To develop and validate predictive models for individualized estimation of canagliflozin HTEs on each of two outcome measures: absolute percent reduction in hemoglobin A1c, and absolute risk increase for a serious adverse event.

Specific hypothesis to be tested:
Pre-randomization participant characteristics chosen based on prior theory (specific demographics, vital signs, laboratory biomarkers, and baseline medication use) can separate participants who experience lower from higher absolute percentage point reduction in hemoglobin A1c (%), and participants who experience lower from higher absolute risk increase in serious adverse events when taking canagliflozin.

What is the purpose of the analysis being proposed? Please select all that apply.: 
New research question to examine treatment effectiveness on secondary endpoints and/or within subgroup populations
New research question to examine treatment safety
Data Source and Inclusion/Exclusion Criteria to be used to define the patient sample for your study: 

All participants in YODA’s randomized, double-blind trials including canagliflozin, with at least 52 weeks follow-up, will be included. We anticipate N = 5,971 participants with type 2 diabetes, at least 18 years of age, comparing canagliflozin at any dosage to placebo or other diabetic agents, with co-administration of other diabetic agents in both the intervention and control group.

Main Outcome Measure and how it will be categorized/defined for your study: 

Absolute percentage point reduction in hemoglobin A1c (%), defined as a continuous measure with exact hemoglobin A1c reduction in each canagliflozin treatment arm versus control arm between week 0 and week 52.
Absolute risk increase in each of two serious adverse events (two separate outcomes of urogenital infection, and lower limb amputation), defined as probability of the serious adverse event by week 52 in each canagliflozin treatment arm versus control arm.

Main Predictor/Independent Variable and how it will be categorized/defined for your study: 

Randomization to canagliflozin treatment group (dummy variable 1/0).

Other Variables of Interest that will be used in your analysis and how they will be categorized/defined for your study: 

Age (in years), Sex (male/female), Race/ethnicity (White/Black/Other), baseline systolic and diastolic blood pressure (mmHg), baseline fasting lipids (total, HDL, LDL and triglycerides in mg/dL), baseline body mass index (kg/m^2), baseline estimated glomerular filtration rate by MDRD equation (mL/min/1.73m^2), baseline hemoglobin A1c (%), baseline fasting plasma glucose (mg/dL), prior history of neuropathy or diabetic ulcer, prior history of urogenital infection.

Statistical Analysis Plan: 

Descriptive analysis will include summary statistics of the above variables of interest by treatment arm within and across all trials
Multivariate non-parametric analysis will involve gradient forest analysis, which proceeds in four steps. First, 75% of the pooled individual participant data across all included trials will be divided in half randomly, with an equal number of canagliflozin and control arm participants in each of the two data subsets (the remaining 25% of the data will be held out for interval validation). Second, variables from the class of predictor variables of interest will be chosen by randomly sampling subsets of potential predictors for HTEs (listed above), to construct a decision-tree made of those predictors that could split the first of the two subsamples of data into subgroups with higher and lower treatment effect. Treatment effect is defined as the absolute difference in hemoglobin A1c, ASCVD or serious adverse event probability between the canagliflozin and control group arms, with effect modifiers included for the individual study, canagliflozin dosage, co-occurring medications, and whether the control arm is an active treatment (glimepiride or sitagliptin) rather than placebo (9–11). Subgroups are required to be >5% of the overall pooled study sample. Third, once the initial decision tree is constructed from the first subsample of data, the values of each predictor that define branches in the decision tree are refined using the second subsample of data, so that the final subgroups at the bottom of the tree (“leaves” of the tree) have maximum between-group differences and minimum within-group differences in treatment effect. Refinement in the second data subset reduces the influence of outliers, and helps produce unbiased HTE estimates (7). The overall approach is repeated 4,000 times from the first step, to produce a “forest” of trees by repeated random resampling of the data (cross-validation). No change in estimated variable importance is typically observed beyond 4,000 trees (7), but this will be empirically assessed to determine if a higher number of trees is necessary. Variable importance is defined as the frequency with which a given variable was incorporated into a tree at the first, second, and further split points (i.e., a variable can change positions between trees, but variable selection for each position is tracked to monitor its importance). The significance of the interaction term between subgroup and therapy arm will be tested using the q-value correction approach, which will correct to a P<0.05 threshold for the empirical probability of obtaining false-positive HTE when performing multiple tests (18); subgroups with significance by the q-value threshold will be maintained. After the forest is constructed and cross-validated, the summary (average) decision tree that placed those variables of highest importance at each split point among the forest of trees will be identified.
To assess performance of the summary decision tree, absolute risk difference in the probability of each outcome will be calculated between the canagliflozin and control arms within each subgroup (leaf) of the trial population, and across the subgroups (nonparametric Jonckherre test for trend across subgroups). Although there are no formal power analyses for causal forest procedures, prior simulations suggest that at least 10 events per predictor variable should be observed in the pooled control arms (>130 events) (19); there were over three times as many events for each of the two severe adverse event outcomes among the included trial participants.
In sensitivity analyses, the decision tree will be reconstructed using just the subset of trials in which canagliflozin was compared to placebo (6 of the 8 trials, N = 3,672), and to separately analyze persons with canagliflozin 100mg and with 300mg to identify effectiveness of the effect modifier terms.

Narrative Summary: 

In this study, we seek to develop and validate risk models for estimating: (i) decrease in hemoglobin A1c (HbA1c), and (ii) increase in serious adverse event risk from canagliflozin, using individual participant data from randomized controlled trials. Multivariate risk models have the potential to identify subgroups of patients that have a greater probability of benefit or of harm from a given therapy (heterogeneous treatment effects, HTEs). Here, we plan to identify HTEs through methods that aim to reduce the chance of false-positive associations, and produce unbiased effect estimates when a medication has been compared at varying dosages with different co-occurring medications.

Project Timeline: 

Anticipated project start date: November 1, 2017
Analysis completion date: February 31, 2017
Date manuscript drafted: April 31, 2017
First submission for publication: June 31, 2017
Date results reported back to YODA: June 31, 2017

Dissemination Plan: 

Anticipated products: Peer-reviewed journal publication
Target audience: primary care, internal medicine, and endocrinology colleagues
Potentially suitable journal for submission: The Lancet Diabetes & Endocrinology


1. VanderWeele TJ, Knol MJ. Interpretation of subgroup analyses in randomized trials: heterogeneity versus secondary interventions. Ann Intern Med. 2011 May 17;154(10):680–3.
2. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials. JAMA Intern Med [Internet]. 2017 Feb 13 [cited 2017 Feb 21]; Available from:
3. Basu S, Sussman JB, Hayward RA. Detecting Heterogeneous Treatment Effects to Guide Personalized Blood Pressure Treatment: A Modeling Study of Randomized Clinical Trials. Ann Intern Med. 2017 Jan 3;154(10):680–3.
4. Burke JF, Hayward RA, Nelson JP, Kent DM. Using Internally Developed Risk Models to Assess Heterogeneity in Treatment Effects in Clinical Trials. Circ Cardiovasc Qual Outcomes. 2014 Jan 1;CIRCOUTCOMES.113.000497.
5. Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010 Aug 12;11:85.
6. Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006 Apr 13;6:18.
7. Athey S, Imbens G. Recursive partitioning for heterogeneous causal effects. Proc Natl Acad Sci. 2016 Jul 5;113(27):7353–60.
8. Baum A, Scarpa J, Bruzelius E, Tamler R, Basu S, Faghmous J. Targeting weight loss interventions to reduce cardiovascular complications of type 2 diabetes: a machine learning-based analysis of heterogeneous treatment effects in The Look AHEAD Trial. Lancet Diabetes Endocrinol. 2017;epub ahead of print.
9. Cleophas TJ, Zwinderman AH. Network Meta-analysis. In: Modern Meta-Analysis [Internet]. Springer; 2017 [cited 2017 Sep 1]. p. 145–155. Available from:
10. Dagne GA, Brown CH, Howe G, Kellam SG, Liu L. Testing moderation in network meta-analysis with individual participant data. Stat Med. 2016 Jul 10;35(15):2485–502.
11. Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. Med Decis Making. 2013;33(5):607–617.
12. American Diabetes Association. 6. Glycemic Targets. Diabetes Care. 2017 Jan 1;40(Supplement 1):S48–56.
13. Palmer SC, Mavridis D, Nicolucci A, Johnson DW, Tonelli M, Craig JC, et al. Comparison of clinical outcomes and adverse events associated with glucose-lowering drugs in patients with type 2 diabetes: a meta-analysis. Jama. 2016;316(3):313–324.
14. Shehab N, Lovegrove MC, Geller AI, Rose KO, Weidle NJ, Budnitz DS. US Emergency Department Visits for Outpatient Adverse Drug Events, 2013-2014. JAMA. 2016 Nov 22;316(20):2115–25.
15. Neal B, Perkovic V, Mahaffey KW, de Zeeuw D, Fulcher G, Erondu N, et al. Canagliflozin and Cardiovascular and Renal Events in Type 2 Diabetes. N Engl J Med. 2017 17;377(7):644–57.
16. Cefalu WT, Leiter LA, Yoon K-H, Arias P, Niskanen L, Xie J, et al. Efficacy and safety of canagliflozin versus glimepiride in patients with type 2 diabetes inadequately controlled with metformin (CANTATA-SU): 52 week results from a randomised, double-blind, phase 3 non-inferiority trial. The Lancet. 2013 Sep 14;382(9896):941–50.
17. Fadini GP, Avogaro A. SGTL2 inhibitors and amputations in the US FDA Adverse Event Reporting System. Lancet Diabetes Endocrinol [Internet]. 2017 [cited 2017 Sep 1]; Available from:
18. Storey JD, Taylor JE, Siegmund D. Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: A unified approach. J R Stat Soc Ser B. 2004;66:187–205.
19. Wager S, Athey S. Estimation and inference of heterogeneous treatment effects using random forests. J Am Stat Assoc [Internet]. 2017 [cited 2017 Jun 22];(just-accepted). Available from:

General Information

How did you learn about the YODA Project?: 

Request Clinical Trials

Associated Trial(s): 
What type of data are you looking for?: 
Individual Participant-Level Data, which includes Full CSR and all supporting documentation

Data Request Status

Change the status of this request: 
Incomplete Not Reviewed