Selective reporting clinical trials

Any discrepancies between the authors were resolved through discussion, until consensus was reached. One author extracted details of the characteristics and results of the empirical cohort studies. Information on the main objectives of each empirical study was also extracted, and the studies were separated according to whether they related to selective reporting or discrepancies between sources.

Selective reporting of analyses was defined as when the reported analyses had been selected from multiple analyses of the same data for a given outcome. A discrepancy was defined as when information was absent in one source but reported in another, or when the information given in two sources was contradictory.

Data extraction was checked by another author. No masking was used, and disagreements were resolved through discussion. In the absence of a recognised tool to evaluate the methodological quality of the empirical studies eligible for this review, we developed and used three criteria to assess methodological quality:.

Two authors independently assessed these items for all studies. An independent assessor Matthew Page was invited to assess one study [6] because the first author was directly involved in its design.

Any discrepancies were resolved through a consensus discussion with a third reviewer not involved with the included studies. This review provides a descriptive summary of the included empirical cohort studies. We refrained from any statistical combination of the results from the different cohorts because of the differences in their design. The search strategy identified records. After duplicates were removed, records were screened, and were excluded. Full texts were accessed for 56 articles.

Twenty-one articles were excluded after assessment of their full text: two were reviews; one was not in English; one was reported in abstract form only and the study was never undertaken; three were not empirical studies; three included only non-RCTs; and 11 did not consider analysis-related issues.

Two ongoing studies four publications were also identified [7] — [10]. One study [7] , [9] , [10] included protocols and related journal publications approved by six research ethics committees from to in Switzerland, Germany, and Canada. The aim of the study was to determine the agreement between planning of subgroups, interim analyses, and stopping rules and their reporting in subsequent publications. A conference abstract was identified in which the authors assessed RCTs submitted to the European Medicines Agency for marketing approval and assessed selective reporting of analyses [8].

The authors were contacted for further details. Twenty-two studies in 31 publications containing a total of 3, RCTs were included [6] , [11] — [31].

All 22 included studies investigated discrepancies, and although several of these studies considered the statistical significance of results, none investigated selective reporting bias. Study characteristics are presented in Tables 1 — 3. Included RCTs were published between and The source of funding of RCTs was not considered in 11 of the cohort studies. In ten studies, industry funded a median of Details of the methodological quality of the included studies are presented in Table 4.

We had methodological concerns about five studies because data extraction was completed by only one person or only a sample was checked by a second author.

Eleven studies were high quality, as data extraction was completed by at least two people. Six studies were rated as uncertain quality, as information regarding data extraction was not provided. We had methodological concerns about one study because positive and negative findings were not defined, while 11 studies were of high quality, with clear definitions of positive and negative findings.

Defining a positive or negative finding was not a study objective in ten studies. One study [18] generated methodological concerns because some secondary analyses were not reported, and the study report stated only that no difference was observed between positive and negative trials. The remaining 21 studies were of high quality, with all comparisons and outcomes stated in the study methods reported in full. We did not have access to any protocols for the empirical studies in order to make a more comprehensive assessment of how each study performed for this methodological quality item.

Eight studies investigated discrepancies in statistical analyses [6] , [15] , [17] , [20] , [23] , [25] , [27] , [28]. Table 5 summarises the results of these studies. Five studies considered whether an intention-to-treat analysis or per protocol analysis was specified or reported [6] , [20] , [23] , [25] , [28]. Melander et al. Rosenthal and Dwan found discrepancies contradictions and information absent between trial registry entries and publications in the reporting of outcomes at different time points [6].

Three studies investigated discrepancies in the handling of missing data [15] , [23] , [27] Table 7. Chan et al.

Rising et al. Two studies investigated discrepancies in the use of continuous and binary versions of the same underlying data [6] , [27] Table 9. Turner et al. A third study, Riveros et al. Rosenthal et al. Twelve studies investigated discrepancies in subgroup analyses [6] , [11] — [15] , [18] , [19] , [21] , [22] , [26] , [30] Table The majority considered whether subgroup analyses were pre-specified or post hoc.

Assmann et al. The fourth study found subgroup analyses reported in only seven RCTs, and no details had been included in the trial registry entries for six of these RCTs, while the seventh had no discrepancies [6]. Although 11 studies looked at funding as a study characteristic, only two considered the relationship between discrepancies and funding.

Sun et al. Rosenthal and Dwan found no statistically significant differences in discrepancy rates of primary and secondary outcomes between registry entries and final reports of industry-sponsored versus non—industry-sponsored trials [6]. Twenty-two cohort studies of RCTs were included in this review that examined discrepancies between information given in different sources.

None of the included studies examined the selective reporting bias of analyses in RCTs that would arise if analyses were reported or concealed because of the results. Such an assessment may prove to be difficult without access to statistical analysis plans and trial datasets to determine the statistical significance of the results for the analyses that were planned and for those that were reported. The majority of studies 12 focussed on the reporting of subgroup analyses.

None of the included studies provided any detail on the reasons for inconsistencies. A number of studies commented on whether or not reported subgroup analyses were pre-specified. While this may not be seen as a comparison, we reported the findings for these studies within this review because post hoc decisions about which subgroups to analyse and report may be influenced by the findings of those or related analyses.

There are likely to be many other selective inclusion and reporting mechanisms for which there is no current empirical evidence, and a more complete categorisation is provided elsewhere [4].

The methodological concerns that were observed in the included studies were not critical, and they should not impact importantly on the interpretation of the results of the studies. These discrepancies may be due to reporting bias, errors, or legitimate departures from a pre-specified protocol.

Reliance on the source documentation to distinguish between these reasons may be inadequate, and contact with trial authors may be necessary. In terms of selective reporting of outcomes, a previous study interviewed a cohort of trialists about outcomes that were specified in trial protocols but not fully reported in subsequent publications [33].

Three of the included studies [11] , [17] , [25] were included in a previous Cochrane methodology review that was restricted to studies that compared any aspect of trial protocols or trial registry entries to publications [5]. The conclusions of the current review and the previous review are similar in that discrepancies were common, and reasons for them were rarely reported in the original RCTs. This current review focussed on analyses only, and included studies that compared different pieces of trial documentation or details within a trial publication.

In accordance to International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use E 9 guidance Statistical Principles for Clinical Trials [34] , procedures for executing the statistical analysis of the primary, secondary, and other data should be finalised before breaking the blind on those analyses.

The availability of a trial protocol or separate statistical analysis plan is of prime importance for inferring whether the results presented are a selected subset of the analyses that were actually done and whether there are legitimate reasons for departures from a pre-specified protocol. Many leading medical journals, e.

In order to ensure transparency, protocols and any separate analysis plans for all trials need to be made publicly available, along with the date that the statistical analysis plan was finalised, details of reasons for any changes, and the dates of those changes.

Additional analyses suggested by peer reviewers when a manuscript is submitted for publication should be judged on their own merits, and any additional analyses that are included in the final paper should be labelled as such. Whilst evidence-based guidelines exist for researchers to develop high-quality protocols for clinical trials e. Systematic reviewers need to ensure they access all possible trial documentation, whether it is publicly available or obtained from the trialists, in order to assess the potential for selective reporting bias for analyses.

The Cochrane risk of bias tool is currently being updated, and the revised version will acknowledge the possibility of selective analysis reporting in addition to selective outcome reporting. Selective analysis reporting generally leads to a reported result that may be biased, so sits more naturally alongside other aspects of bias assessment of trials, such as randomisation methods, use of blinding, and patient exclusions. Selective outcome reporting may lead either to bias in a reported result e.

There are to date no readily accessible data on selective reporting bias of analyses in cohorts of RCTs. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials RCTs. Methods and findings: A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.

Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies containing 3, RCTs published between and were included. Twenty-two studies reported on discrepancies between information given in different sources. Find articles by R Kush. Find articles by P Lumbiganon. Find articles by D Moher. Find articles by F Rockhold. Find articles by I Sim. Find articles by E Wager. Author information Article notes Copyright and License information Disclaimer.

Corresponding author. Correspondence to Davina Ghersi e-mail: tni. Received Apr 4; Accepted Apr 8. All rights reserved. This article has been cited by other articles in PMC. Background When researchers embark on a clinical trial, they make a commitment to conduct the trial and to report the findings in accordance with basic ethical principles.

What is a finding? The type and amount of information required will depend on the nature of the audience and how it will use the information received but will have four key elements: 1. Methodological context The user needs to know what the original plans for the trial were and how it was actually conducted. Population context The user needs to consider to whom the results of the trial can be applied. A result for each outcome A clinical trial will usually consider several outcomes, each of which might be measured in multiple ways or at multiple time points, and there might be more than one way to analyse each of these measures.

Public availability Clinical trial results should be available to everyone, regardless of where they live. Conclusion People making decisions about health care need access to knowledge derived from the findings of clinical trial research. Footnotes Competing interests: None declared. References 1. Declaration of Helsinki. Publication and related biases. Health Technol Assess. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.

Horton R.