Cro, S;
Forbes, G;
Johnson, NA;
Kahan, BC;
(2020)
Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials.
BMC Medicine
, 18
, Article 137. 10.1186/s12916-020-01590-1.
Preview |
Text
s12916-020-01590-1.pdf - Published Version Download (687kB) | Preview |
Abstract
BACKGROUND: Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. METHODS: A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication. RESULTS: Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n = 31, 35%) and analysis population (n = 28, 31%), followed by the use of covariates (n = 23, 26%) and the approach for handling missing data (n = 16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. CONCLUSIONS: Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.
| Type: | Article |
|---|---|
| Title: | Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials |
| Location: | England |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1186/s12916-020-01590-1 |
| Publisher version: | https://doi.org/10.1186/s12916-020-01590-1 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
| Keywords: | P-hacking, Randomised controlled trials, Statistical analysis, Statistical analysis plan, Transparency |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology > MRC Clinical Trials Unit at UCL |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10101060 |
Archive Staff Only
 |
View Item |
