General Information
Conflict of Interest
- yoda_project_coi_form_for_data_requestors_2019_siebert
- yoda_project_coi_form_for_data_requestors_2019_florian_naudet
- yoda_project_coi_form_for_data_requestors_2019_ar210329
- yoda_project_coi_form_for_data_requestors_2019gaba
Request Clinical Trials
Associated Trial(s):- NCT02417064 - A Randomized, Double-blind, Multicenter, Active-controlled Study to Evaluate the Efficacy, Safety, and Tolerability of Fixed Doses of Intranasal Esketamine Plus an Oral Antidepressant in Adult Subjects With Treatment-resistant Depression
- NCT02497287 - An Open-label, Long-term, Safety and Efficacy Study of Intranasal Esketamine in Treatment-resistant Depression
- NCT02493868 - A Randomized, Double-blind, Multicenter, Active-Controlled Study of Intranasal Esketamine Plus an Oral Antidepressant for Relapse Prevention in Treatment-resistant Depression
Request Clinical Trials
Data Request Status
Status: PublishedResearch Proposal
Project Title: Data-sharing and re-analysis for main studies assessed by the European Medicines Agency - a cross-sectional study on EPARs
Scientific Abstract:
Background: Since the launch of data sharing initiatives, the scientific literature has lacked studies attesting to the effectiveness of data sharing and its impact on the reproducibility of therapeutic research.
Objective: Assess inferential reproducibility (i.e., Individual Patient Data is available and qualitatively similar conclusions can be drawn from a reanalysis of the original trials) for main RCTs assessed by the EMA.
Study design: Cross-sectional study
Participants: Primary outcomes of 62 randomly selected main studies will be reanalyzed. These are the main studies on new medicines and biosimilars given approval by the European Commission after 1st of January 2017 marked as main studies in the European Assessment Reports.
Main outcome: The primary outcome of this study is the proportion of studies where the conclusions were reproduced. We will also describe and compare the main outcomes, p-values and effect sizes in the re-analyses, and the analyses reported in the EPARs, the study reports and the publications, and we will describe discrepancies.
Statistical Analysis: Using only the protocol and the data, re-analyses will implement the following different steps: 1/identification of the primary outcome (identification of outcome switching), 2/definition of the analysis population, 3/construction of the SAP, 4/re-analysis of the primary outcome.
Brief Project Background and Statement of Project Significance:
The EMA aimed to be a pioneer in transparency, in November 2010, it decided to share every piece of documentation received, in the wake of the first version of policy 0043 (1). As part of its transparency policy, the EMA publishes European Public Assessment Reports (EPAR) after the European Commission?s decision on the specific medicines. These reports include, amongst other documents, results of main trials (2). On October 2nd 2014 the EMA released its policy 0070 on ?publication of clinical data for medicinal products for human use?(3). The agency describes a two-step approach. From 1st of January 2015 clinical reports on medicines submitted for Marketing Authorization have been published. A second step includes the publication of IPD. A date for the implementation of this step still needs to be fixed. However, as a result of Brexit and the relocation of the EMA to the Netherlands, further developments and renovation have been stopped for the moment (4, 5). Efforts are therefore still needed to reach full transparency in the EMA.
Up to now, there has been no clear picture of the inferential reproducibility of the main studies in European Assessment Reports. These main studies are used to assess the efficacy of new medical products which will influence the lives of millions of people in the European Union. If data availability were to prove low it should urge the EMA to implement an even stronger data-sharing policy. If reanalyses of available data show low reproducibility, it would argue for independent re-analyses at the time of the approval. On the other hand, if there were to be no issues in terms of reproducibility, it would reinforce the confidence one can have in EMA?s transparency concerning processes and decisions.
In 2019, the EU Clinical Trial Regulation 536/2014 will come into force and will further extend the boundaries of data-sharing and transparency in the EMA as well as within the European Union. The EU Portal and Data Base is being developed, creating a single-entry point for submitting clinical trials in the EU. A further advantage is that not only information about clinical trials included in Marketing Authorization Applications (MAA) can be found on the portal, but the aim is to have data about every single trial conducted in the European Union, whether or not it is part of an MAA (6). Our research could help to highlight the interest of the future regulation.
Specific Aims of the Project:
As part of a global research program on reproducibility in therapeutic research (ReiTheR, funded by the French National Research Agency), we designed the present cross-sectional study to assess inferential reproducibility (i.e. when IPD is available, whether qualitatively similar conclusions can be drawn from a reanalysis of the original trials) for main studies assessed by the EMA.
Our hypothesis is that for most trials (> 95%) for which we obtain the data, the results observed on the primary outcome will be fully reproducible. However, although we planned one year for data collection, we are aware that after this time some data will still not be available and thus not be re-analyzable. Nevertheless, the worst-case scenario for precision estimates is that 50% of the studies will be analyzable and reproduced.
Study Design:
What is the purpose of the analysis being proposed? Please select all that apply.: Confirm or validate previously conducted research on treatment effectiveness
Software Used: R
Data Source and Inclusion/Exclusion Criteria to be used to define the patient sample for your study:
2 researchers independently identified all main studies on new medicines and biosimilars given approval by the European Commission after 1st of January 2017 marked as main studies in the European Assessment Reports. 62 of these studies were randomly sampled ensuring a precision of 12% to estimate our primary outcome, the proportion of studies where the conclusions are reproduced.
Following main study documents will be collected by making a request to the sponsor: 1) IPD, 2) data analysis plan, 3) unpublished and/or published study protocols with any date-stamped amendments 4) all the following dates: date of the last visit of the last patient, date of database lock (if available) and date of study unblinding, 5) unpublished and/or published (scientific article) study reports.
Reanalyses of the primary outcome(s) of each study will be performed by one researcher, on the basis of a dossier prepared by his team.
Data extraction, eligibility criteria are detailed in the protocol attached to this request.
Primary and Secondary Outcome Measure(s) and how they will be categorized/defined for your study:
The Primary outcome is the proportion of studies where the conclusions were reproduced (yes/no; reproduced and reproduced with verification, as defined in the protocol attached). In case of a divergence for two or more co-primary outcomes in the same study (i.e., one analysis is reproduced and not the other(s)) the different co-primary outcomes will be described independently but the whole study will be considered as not reproduced.
Strategy for reanalyses, procedure to assess reproducibility, outcomes are detailed in the protocol attached to this request.
Main Predictor/Independent Variable and how it will be categorized/defined for your study:
Re-analyses of the primary outcome(s) of each requested study will be performed. These requested studies are those extracted from different EPARs.
All results of these analyses will be reported in terms of each study?s 1/ conclusion (positive or negative), 2/ p-value, 3/ effect size (and details about the outcome), 4/ changes from the initial protocol.
These results will first be compared with the results of the analyses reported in the EPARs and, if these are not available, with the study reports, and again if not available, with the publications. All results from all available documents will be gathered (EPARs, study reports & publications) and will be presented in the results section.
Because interpreting an RCT involves clinical expertise, and cannot be reduced to solely quantitative factors, an in-depth discussion between two researchers not involved in the re-analysis , based on both quantitative and qualitative (clinical judgment) factors will enable a decision on whether the changes in results described quantitatively could materialize into a change in conclusions.
Other Variables of Interest that will be used in your analysis and how they will be categorized/defined for your study: The strategy for reanalyses, procedure to assess reproducibility, outcomes and data anlysis are detailed in the protocol attached.
Statistical Analysis Plan:
Reanalyses of the primary outcome(s) of each study will be performed by one researcher, on the basis of a dossier prepared by his team. This reviewer will have no access to study reports, journal publications, statistical analysis plan, or analytical code, in order to ensure that the analysis is as blind as possible to the primary analysis. This reviewer will be instructed not to try to find these documents or the published report.
Using only the protocol and the data, re-analyses will implement the following different steps: 1/ identification of the primary outcome (identification of outcome switching), 2/ definition of the analysis population, 3/ construction of the SAP, 4/ re-analysis of the primary outcome. Any change identified between the first version of the protocol and the version used for the re-analysis of the primary outcome will be tracked and described.
The ICH guidelines will be used to replace missing items, insufficient information in the documents provided by sponsors when developing the analysis plan for each study included. This analysis plan will be recorded on OSF for each study. Differences between the reanalysis SAP and the original study analysis plan will be also described
An independent senior statistician will be available to discuss any difficult aspect or choice in the analysis plan before the re-analysis, so as to choose the most consensual analyses (e.g. Intention to Treat population for a superiority trial).
We will perform a descriptive analysis of the characteristics of the requested studies. This will include counts, percentages and their associated 95% confidence intervals (CIs).
Effect estimates in the different studies will be expressed as standardised mean differences (SMDs) and their associated 95% confidence intervals. For binary outcomes, odds ratios and their 95% CIs will be calculated and converted into the standardised mean difference.
In order to compare the results of our re-analyses with the original results, the following steps will be implemented: 1/ We will compare the statistical significance in the form of the p-value. If different, the result will be considered as not reproducible. If not different, 2/ we will qualitatively compare effect sizes and their respective 95% CIs. In case of +/- 0.10 points difference in point estimates (expressed as standardised mean differences), the difference will be discussed with a clinician in order to assess its clinical significance.
All analyses will be performed using the open source statistical software R (R Development Core Team). The code will be made public on the Open Science Framework, as well as a file summarizing the process to retrieve all data-sets.
Narrative Summary: Since the launch of data sharing initiatives, the scientific literature has lacked studies attesting to the effectiveness of data sharing and its impact on the reproducibility of therapeutic research.This study is a cross-sectional study aiming to assess inferential reproducibility (i.e. Individual Patient Data is available and qualitatively similar conclusions can be drawn from a reanalysis of the original trials) for main RCTs assessed by the European Medicine Agency (EMA). With this study, we aim to establish evidence on data sharing initiatives and precisely for pivotal trials used by the EMA.
Project Timeline:
From January 2020 until January 2021 we will collect and select relevant trials and initiate all data requests. Collation and re-analysis of the data will take place from January 2020 to November 2021.
Expected date of data analysis completion: November 2021
Expected date of manuscript completion: February 2022
Dissemination Plan: The study protocol was accepted at BMC Medicine as a registered report: the research protocol was peer-reviewed before the actual research takes place. Being accepted, the editors undertake to publish the completed study if the protocol is validated even if there are statistically negative findings (study hypothesis not verified). This approach is expected to reduce issues such as publication bias. In addition, the protocol is pre-registered on the Open Science Framework.
Bibliography:
1. European Medicines Agency. European Medicines Agency policy on access to documents. 2018 October 04 [cited 2019 July 05]. Available from: https://www.ema.europa.eu/en/documents/other/policy/0043-european-medici….
2. European Medicines Agency. Guide to information on human medicines evaluated by EMA 2017 June 19 [Available from: https://www.ema.europa.eu/en/documents/other/guide-information-human-med….
3. European Medicines Agency. European Medicines Agency policy on publication of clinical data for medicinal products for human use 2019 March 21 [Available from: https://www.ema.europa.eu/en/documents/other/european-medicines-agency-p….
4. European Medicines Agency. Clinical data publication 2019 [cited 2019 June 24]. Available from: https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/cl….
5. Doshi P. EMA scales back transparency initiatives because of workload. BMJ. 2018;362:k3513.
6. European Commission. Clinical trials – Regulation EU No 536/2014 2019 [cited 2019 June 23]. 497 Available from: https://ec.europa.eu/health/human-use/clinical-trials/regulation_en.