Skip to main content

2020-4417

Research Proposal

Project Title: 
Vibration of effects in pooled analyses of Canagliflozin vs placebo, randomized, double blind trials
Scientific Abstract: 

background: Vibration of effect (VoE) describes the extent to which an effect may change under multiple distinct analyses, such as different model specifications in epidemiological research (1): any analysis involves a universe of possible analytical choices that might influence the effects observed.
objective: In this case study, we will explore VoE in pooled analyses of clinical trials focusing on clinical trials evaluating the efficacy and safety of Canagliflozin in the treatment of type 2 diabetes mellitus.
study design: case study
participants: Studies were identified and selected from the YODA project platform. Preliminary searches identified 13 study datasets fulfilling our inclusion criteria.
Main outcome measure: Effect sizes will be expressed in terms of: Mean difference for HbA1c; Hazard ratio for Major Cardio-vascular Events (this will be analyzed as survival data / the first event will be considered); Hazard ratio for Serious Adverse Events (this will be analyzed as survival data / the first event will be considered);
Statistical analysis: HbA1c measures, MACE and serious adverse events, will be analyzed in 3 different analyses. All these analyses will be performed for different timepoints. In each pooled analysis, data will be pooled using a one‐stage IPD meta‐analysis approach. We will compute the distribution of point estimates of these effect sizes and their corresponding p values under various analytical scenarios defined by the combination of studies.

Brief Project Background and Statement of Project Significance: 

Vibration of effect (VoE) describes the extent to which an effect may change under multiple distinct analyses, such as different model specifications in epidemiological research (1): any analysis involves a universe of possible analytical choices that might influence the effects observed. More recently, such a phenomenon has been identified in the field of evidence synthesis, with the case study of indirect comparisons of naltrexone versus nalmefene (2) two similar drugs used in the treatment of alcohol use disorders. This is of concern because evidence synthesis is generally considered as a very reliable domain resulting in high grade evidence. There is therefore a need to explore VoE for various methods of evidence synthesis, such as for instance pooled analyses.

Indeed, performing pooled analyses of randomized controlled trials (i.e. non-exhaustive quantitative syntheses that pool individual patient data (IPD) from several independent trials exploring similar research questions) is a common practice in therapeutic research. For instance, for the antidepressant duloxetine, 43 pooled-trials publications were found for 30 papers which explored the efficacy of duloxetine in a clinical trial for the treatment of major depression (3). Pooled analyses often explore secondary analyses (e.g. subgroup effects) or examine new questions (3)(4). In other situations, the increased sample size resulting from pooling different studies may help to solve power issues of the initial studies. However, in contrast with individual patient data meta-analysis, pooled analyses are not exhaustive of all the available studies retrieved in a systematic review and may be prone to p-hacking by selecting a posteriori combination of studies based on their results (3). And indeed, the use of pooled analyses has been described as a strategy used by pharmaceutical firms to “hide” negative studies results by publishing these studies with the positive ones (5). In addition, a posteriori selection of clinical trials to be pooled may result in differences in observed results and the multiplicity of possibilities it involves implies a certain degree of VoE.

We decided to explore this question in the field of Type 2 diabetes mellitus. There is still a tension in this field about the clinical value of the drugs that reduce chronic hyperglycemia. While there is no doubt about efficacy of these drugs on the surrogate marker of HBA1C levels, there is still a heated debate about their impact on clinical outcomes including cardiovascular ones (6). The case study of rosiglitazone even suggests that this drug was associated with an increase in Serious adverse events and more specifically a cardiovascular risk, especially for heart failure events (7). Canaglifozin is such a drug used for glycemic control and data of its clinical development program are shared on YODA platform making any analysis of VoE possible. We therefore designed this study to explore VoE in pooled analyses of clinical trials evaluating the efficacy and safety of Canagliflozin in the treatment of type 2 diabetes mellitus.

Specific Aims of the Project: 

In this case study, we will explore VoE in pooled analyses of clinical trials focusing on clinical trials evaluating the efficacy and safety of Canagliflozin in the treatment of type 2 diabetes mellitus. We will use IPD from trials data that are available for other researchers within the YODA (Yale University Open Data Access) project (https://yoda.yale.edu/).
We will perform all possible pooled analyses comparing Canaglifozin versus Placebo to explore how much three different outcomes can change across the universe of all possible combinations of trials. Our evaluation will focus on efficacy (HbA1c and MACE, i.e. Major Cardiovascular Events) and one safety (SAE, i.e. Serious Adverse Events) outcome.

Our hypotheses are that VoE will not be observed for HbA1c while it will be observed for both MACE and SAE. There will be no specific statistical test for this as the existence of VoE is defined by very simple descriptive parameters described in the analysis section.

What is the purpose of the analysis being proposed? Please select all that apply.: 
Research on clinical trial methods
Software Used: 
R
Data Source and Inclusion/Exclusion Criteria to be used to define the patient sample for your study: 

Studies will be selected if they meet the following criteria:
 Patient population:
o At least 18 years old;
o Diagnosis of type 2 diabetes mellitus;
o All sexes are accepted;
o All BMI are accepted;
o All screening HbA1c concentration are accepted;
 Intervention:
o Canagliflozin at a dose between 100 and 300mg;
o This could be used in association with another treatment or not;
 Comparator:
o Placebo only;
 Outcomes:
o HbA1c (primary or secondary outcomes);
o Major cardiovascular events (MACE) during all the study period (that can be retrieved within the serious adverse events);
o Serious Adverse Events during all the study period;
 Study design:
o Randomized, controlled, double blind trials.

Main Outcome Measure and how it will be categorized/defined for your study: 

Effect sizes will be expressed in terms of:
 Mean difference for HbA1c;
 Hazard ratio for Major Cardio-vascular Events (this will be analyzed as survival data / the first event will be considered);
 Hazard ratio for Serious Adverse Events (this will be analyzed as survival data / the first event will be considered);

We will compute the distribution of point estimates of these effect sizes (ESs) and their corresponding p values under various analytical scenarios defined by the combination of individual studies. Pooled analyses will be considered to be “statistically significant” if the ES is associated with a p value < 0.05. The presence of a “Janus effect” will be investigated by calculating the 1st and 99th percentiles of the distribution of the ES (8). A Janus effect will be defined as an ES which is in the opposite direction between the 1st and 99th percentiles of pooled analyses. It demonstrates the presence of substantial VoE (8). Our hypotheses are that VoE will not be observed for HbA1c while it will be observed for both MACE and SAE.

Main Predictor/Independent Variable and how it will be categorized/defined for your study: 

Effect sizes will be expressed in terms of:
 Mean difference for HbA1c;
 Hazard ratio for Major Cardio-vascular Events (this will be analyzed as survival data / the first event will be considered);
 Hazard ratio for Serious Adverse Events (this will be analyzed as survival data / the first event will be considered);

Statistical Analysis Plan: 

HbA1c measures, MACE and serious adverse events, will be analyzed in 3 different analyses. All these analyses will be performed for different timepoints: (12 [or closest date], 18 [or closest date], 26 [or closest date] and 52 weeks [or closest date]).

In each pooled analysis defined above, data will be pooled using a one‐stage IPD meta‐analysis approach that will analyze all the patient‐level data from all the trials in a single step, using a hierarchical (random effects) model that accounts for the clustering of patients within studies.

Effect sizes will be expressed in terms of:
 Mean difference for HbA1c;
 Hazard ratio for Major Cardio-vascular Events (this will be analyzed as survival data / the first event will be considered);
 Hazard ratio for Serious Adverse Events (this will be analyzed as survival data / the first event will be considered);

We will compute the distribution of point estimates of these effect sizes (ESs) and their corresponding p values under various analytical scenarios defined by the combination of individual studies. Pooled analyses will be considered to be “statistically significant” if the ES is associated with a p value < 0.05. The presence of a “Janus effect” will be investigated by calculating the 1st and 99th percentiles of the distribution of the ES (8). A Janus effect will be defined as an ES which is in the opposite direction between the 1st and 99th percentiles of pooled analyses. It demonstrates the presence of substantial VoE (8). Our hypotheses are that VoE will not be observed for HbA1c while it will be observed for both MACE and SAE.

All analyses will be performed using R (https://www.r-project.org/). The results will be presented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) format (http://www.prisma-statement.org/). The data and code will be shared on the Open Science Framework (https://osf.io/). The protocol will be registered on the Open Science Framework (https://osf.io/) before starting the study.

Narrative Summary: 

In this case study, we will explore VoE in pooled analyses of clinical trials focusing on clinical trials evaluating the efficacy and safety of Canagliflozin in the treatment of type 2 diabetes mellitus. We will use IPD from trials data that are available for other researchers within the YODA (Yale University Open Data Access) project (https://yoda.yale.edu/).
We will perform all possible pooled analyses comparing Canaglifozin versus Placebo to explore how much three different outcomes can change across the universe of all possible combinations of trials. Our evaluation will focus on efficacy (HbA1c and MACE, i.e. Major Cardiovascular Events) and one safety (SAE i.e. Serious Adverse Event)

Project Timeline: 

Project start : May 2020
Data request : September 2020
Analyses start : March 2021
Project finish : September 2021

Dissemination Plan: 

We will try to publish this case study as a case study article in a medical journal as BMJ.

Bibliography: 

1. Ioannidis JPA. Why Most Discovered True Associations Are Inflated. Epidemiology. 2008 Sep;19(5):640–648.
2. Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Medicine. 2019 Sep 16;17(1):174.
3. Spielmans GI, Biehn TL, Sawrey DL. A Case Study of Salami Slicing: Pooled Analyses of Duloxetine for Depression. PPS. 2010;79(2):97–106.
4. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nature Human Behaviour. 2017 Jan 10;1(1):1–9.
5. de Vries Y, Roest A, Turner E, Jonge P. Hiding negative trials by pooling them: A secondary analysis of pooled-trials publication bias in FDA-registered antidepressant trials. Psychological Medicine. 2018 Sep 28;49:1–7.
6. Bejan-Angoulvant T, Cornu C, Archambault P, Tudrej B, Audier P, Brabant Y, et al. Is HbA1c a valid surrogate for macrovascular and microvascular complications in type 2 diabetes? Diabetes & Metabolism. 2015 Jun 1;41(3):195–201.
7. Wallach JD, Wang K, Zhang AD, Cheng D, Nardini HKG, Lin H, et al. Updating insights into rosiglitazone and cardiovascular risk through shared data: individual patient and summary level meta-analyses. BMJ [Internet]. 2020 Feb 5 [cited 2020 Jul 26];368. Available from: https://www.bmj.com/content/368/bmj.l7078
8. Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology. 2015 Sep 1;68(9):1046–58.

Supplementary Material: 

General Information

How did you learn about the YODA Project?: 
Colleague

Request Clinical Trials

Associated Trial(s): 
What type of data are you looking for?: 
Individual Participant-Level Data, which includes Full CSR and all supporting documentation

Data Request Status

Change the status of this request: 
Ongoing