Umbrella reviews: a useful study design in need of standardisationBMJ 2022; 378 doi: https://doi.org/10.1136/bmj.o1740 (Published 05 August 2022) Cite this as: BMJ 2022;378:o1740
- Xiaoting Shi, doctoral student,
- Joshua D Wallach, assistant professor
Systematic reviews and meta-analyses, which identify and synthesise evidence from individual studies, are often believed to provide an overview of the best available evidence on a specific research question. In epidemiology, however, systematic reviews and meta-analyses typically focus on individual exposure to outcome relationships, which can fail to capture all potentially related exposures or outcomes across an entire field. Moreover, concerns have consistently been raised about the growing number of overlapping and conflicting reviews.12 These limitations emphasise the need for a study design that can potentially provide a higher level synthesis of summary level evidence.12
Umbrella reviews, which are also known as overviews of systematic reviews or systematic reviews of meta-analyses, summarise the spread and strength of associations reported in previously conducted systematic reviews and meta-analyses.3 They can consider numerous exposures and outcomes; provide an assessment of the impact of sample size, heterogeneity, and hints of bias on summary associations; and evaluate the quality of individual systematic reviews and meta-analyses. These evaluations, which have increased in popularity over the past decade,4 are particularly useful in fields where a large number of reviews have already been conducted.
In our recent umbrella review published in BMJ Medicine, we identified and summarised all associations reported in meta-analyses on environmental exposures and non-Hodgkin’s lymphoma (NHL).5 Although many exposures (such as dietary, clinical, lifestyle, chemical, and occupational factors) have been the focus of separate meta-analyses, little is known about the accumulated evidence across a range of potential environmental exposures and NHL subtypes. Across 85 meta-analyses reporting 257 unique environmental exposure-NHL associations, we found that most meta-analyses were low quality and presented either non-significant or weak evidence.6 Only one association—history of coeliac disease and risk of NHL—was classified as presenting convincing evidence. Although our study suggests the need for improving not only primary studies but also evidence synthesis in this field, it also highlights several challenges of conducting umbrella reviews without uniform handbooks and reporting guidelines.
Firstly, it can be challenging to select an individual meta-analysis when there are overlapping meta-analyses for the same exposure-outcome relationships. While some umbrella reviews select the largest or most recent meta-analyses,78910 others prioritise those with the greatest precision11 or the highest quality.12 Some umbrella reviews even go as far as updating the individual searches from each eligible meta-analysis.12 In our evaluation, we selected a single association from the largest meta-analyses on each topic, even though there may have been more recent or higher quality meta-analyses. We selected this approach given the large number of identified associations.
Secondly, individual meta-analyses often report multiple associations for different exposure contrast levels (such as exposed versus unexposed, high versus low levels of exposure, or dose-response), which can make it difficult to select and summarise only one association. When designing our study, we found that while some umbrella reviews justified why certain comparisons were selected, others primarily selected exposed versus unexposed comparisons.1314 In our evaluation, we prioritised the associations from exposed versus unexposed comparisons. When these comparisons were not reported, however, we also recorded any associations from comparisons of high versus low levels of exposures. Although this approach may not have captured the complexities of all higher exposure levels, our objective was to provide a manageable overview of all reported exposures across a large field.
Thirdly, different methodological approaches can be used in umbrella reviews to assess the credibility of individual associations from meta-analyses, including the role of statistical significance, sample size, heterogeneity, and certain biases.10131516 Many umbrella reviews, including our own, used the same methods to conduct the analyses for each of these characteristics.817 However, the current methods could be modified (such as standardising all associations using different meta-analytical methods),1819 which could ultimately impact how evidence is classified.
Overall, our experience suggests that there are opportunities to improve the design, conduct, and reporting of umbrella reviews, to help ensure that these studies are rigorous and reproducible. Unlike traditional systematic reviews and meta-analyses, which have more established methodology and reporting guidelines,2021 the recommendations for umbrella reviews are disjointed, with separate efforts outlining various concerns and recommendations.2223242526 The Cochrane Handbook for Systematic Reviews of Interventions and Joanna Briggs Institute Manual provide recommendations for umbrella reviews of interventions,2728 but additional resources are needed to accommodate different scenarios. Together, these efforts could help standardise approaches, minimise the need for authors to make subjective decisions, and ultimately reduce the number of overlapping umbrella reviews that are conducted using different methodological approaches.29
Competing interests: In the past 36 months XS was supported by the China Scholarship Council and the Yale Graduate School of Arts and Sciences. JDW currently receives research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing, from the Food and Drug Administration, and from the National Institute on Alcohol Abuse and Alcoholism under award K01AA028258.