Abstract
Background and objectives: Systematic reviews of clinical studies aim to compile best available evidence for various diagnosis and treatment options. This study assessed the methodologic quality of all systematic reviews relevant to the practice of nephrology published in 2005.
Design, setting, participants, & measurements: We searched electronic databases (Medline, Embase, American College of Physicians Journal Club, Cochrane) and hand searched Cochrane renal group records. Clinical practice guidelines, case reports, narrative reviews, and pooled individual patient data meta-analyses were excluded. Methodologic quality was measured using a validated questionnaire (Overview Quality Assessment Questionnaire). For reviews of randomized trials, we also evaluated adherence to recommended reporting guidelines (Quality of Reporting of Meta-Analyses).
Results: Ninety renal systematic reviews were published in year 2005, 60 of which focused on therapy. Many systematic reviews (54%) had major methodologic flaws. The most common review flaws were failure to assess the methodologic quality of included primary studies and failure to minimize bias in study inclusion. Only 2% of reviews of randomized trials fully adhered to reporting guidelines. A minority of journals (four of 48) endorsed adherence to consensus guidelines for review reporting, and these journals published systematic reviews of higher methodologic quality (P < 0.001).
Conclusions: The majority of systematic reviews had major methodologic flaws. The majority of journals do not endorse consensus guidelines for review reporting in their instructions to authors; however, journals that recommended such adherence published systemic reviews of higher methodologic quality.
Physicians make better clinical decisions when they understand the circumstances and preferences of their patients and combine their personal experience with clinical evidence supporting the available options (1). The public also expects their physicians to take up new research findings in a timely way (2). Systematic reviews of clinical studies aim to compile the best available evidence for various diagnosis and treatment options (3), and the number of systematic reviews has increased in the past decade (4). Physicians rate systematic reviews as more relevant than original articles for clinical practice (5), presumably because reading a systematic review is a more efficient way of keeping up to date.
A well-conducted review summarizes the best available evidence for a specific question, highlights the strengths and weaknesses of the evidence, and identifies new focuses for future research (3). A meta-analysis, when used appropriately, can achieve statistical power to detect an overall effect that was not observed in each of the smaller primary studies.
The quality of a systematic review depends on the quality of the primary studies cited in the review. The results can also be influenced by the methods used to compile and interpret the primary studies and by inappropriately pooling the data. The review team can potentially arrive at the wrong conclusion (6,7). In an attempt to prevent this problem, guidelines for the reporting of systematic reviews have been published. Quality of Reporting of Meta-Analyses (QUOROM) guidelines were published in 1999, describing recommended methods for conducting and reporting a systematic review of randomized, controlled trials (8), and Meta-analysis of Observational Studies in Epidemiology (MOOSE) consensus guidelines were published in 2000 (9); however adherence to these guidelines has been variable. Previous studies in the fields of anesthesia (10), general surgery (11), critical care (12), emergency medicine (13), and oncology (14) showed that a majority of systematic reviews contained major methodologic flaws.
To date, no study has examined whether systematic reviews of higher methodologic quality are more likely to be published in journals that endorse reporting guidelines. We examined this issue in this analysis of recently published systematic reviews relevant to the practice of nephrology. We also determined where renal systematic reviews are published and their quality and other factors associated with high-quality reviews.
Materials and Methods
Reviews Eligible for Analysis
We defined a systematic review as a full-text article that attempted to answer a focused clinical question by examining previously published evidence identified through a protocol-driven process. Case reports, clinical practice guidelines, narrative reviews, and pooled individual patient data meta-analyses in the absence of a comprehensive review of the literature all were excluded.
Systematic reviews were deemed relevant to nephrology when they described one of the following content areas: Kidney failure; renal transplantation; chronic kidney disease; glomerular disease; reflux nephropathy; metabolic, acid/base, or water disturbances; procedures involving patients with kidney disease; or care of patients with kidney disease. To provide information on the current state of systematic reviews, we considered only reviews that were published in 2005.
When the same review was published by the authors in two or more different sources (e.g., Cochrane Database and a journal), we abstracted the data from both sources separately (occurred three times). Although we recognize that these are not completely independent observations, this was done for three reasons. First, in this study, we wanted to characterize where renal systematic reviews are published. Excluding a review because of duplication would counter this purpose. Second, we wanted to investigate whether the source of publication influenced the reporting and/or the methodologic quality of a systematic review, and duplicate reviews offered the potential to examine this issue. Finally, choosing to exclude one of the duplicate sources would have been an arbitrary decision.
Finding Renal Reviews
A systematic literature search of Medline (OVID), Experta Medica (Embase), American College of Physicians Journal Club, Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effectiveness for the year 2005 (DARE) for the year 2005 was performed by an experienced librarian and was supplemented by a hand search of Cochrane Renal Group records of non-Cochrane meta-analyses. Language restrictions were not used.
Duplicates, protocols, and conference abstracts were excluded. The search strategy included the terms renal, kidney, dialysis, hemodialysis, hemofiltration, continuous ambulatory peritoneal dialysis, continuous cycler-assisted peritoneal dialysis, glomerulonephritis, GFR, proteinuria, end-stage renal disease, nephritis, nephrosis, nephropathy, nephrotic, and nephrotoxicity as combination of keywords and corresponding subject headings in thesaurus tree structure (MeSH and EMTREE) and was refined and tested for each database.
Two reviewers independently evaluated the eligibility of each citation, and the full-text article was retrieved when either reviewer considered the citation potentially relevant. Two reviewers independently evaluated the eligibility of each full-text article, with disagreements resolved by consensus.
Data Abstraction
Two reviewers independently abstracted data from all reviews that met the eligibility criteria, and disagreements were resolved by consensus. The source of publication was classified into three groups: Journals specializing in nephrology or transplant medicine, journals not specializing in nephrology or transplant medicine, and Cochrane reviews. The US National Library of Medicine Medline journal subject terms were used to determine the primary discipline of each journal. The journal impact factor was obtained from the 2005 edition of the Institute for Scientific Information Journal Citation Reports.
We abstracted the type of analytic methods used in the study (meta-analysis, meta-regression, qualitative synthesis), study type (diagnosis, prognosis, treatment, etiology/harm, economics), search methods used to identify primary literature, language restrictions used, and the sources of funding. The topic of the systematic reviews was classified into one of five specified renal categories: Acute kidney injury, chronic kidney disease or glomerulonephritis, chronic hemodialysis or peritoneal dialysis, kidney transplantation, and pediatric nephrology. The conclusions reached were classified into three groups: Majority (≥50%) of included studies demonstrated sufficient evidence of significant effect, majority (≥50%) demonstrated sufficient evidence of no significant effect, or majority (≥50%) demonstrated insufficient evidence to make a conclusion.
The quality of each systematic review was evaluated using a previously validated instrument: Overview Quality Assessment Questionnaire (OQAQ) (15). Systematic reviews of randomized, controlled trials were further assessed for adherence to QUOROM. We then searched the website of each journal that published at least one relevant systematic review to establish whether it recommended the use of any consensus guidelines for reporting of systematic reviews (e.g., QUOROM, MOOSE) in its instructions to potential authors. The date of implementation of guidelines was confirmed via correspondence with editors of each journal. The data abstractors were blinded to journal endorsement status.
Overview Quality Assessment Questionnaire
The OQAQ is the only validated tool for assessing the methodologic quality of systematic reviews (15). It was designed to evaluate whether the authors of a systematic review conducted a comprehensive search, minimized bias in the selection of primary studies to be included for analysis, evaluated the primary literature, and pooled the results appropriately.
OQAQ is composed of 10 questions. The first nine questions have set answers of “yes,” “no,” or “partially/can't tell” and are designed to assess different aspects of methodologic quality. The 10th item is an assessment of the overall methodologic quality of the systematic review on a scale of 1 to 7. A score of ≤3 indicates major methodologic flaws that have a strong potential for creating bias in the results of a systematic review.
To reduce interrater variability, we used the modified version to the OQAQ as described in past studies (12,13,16,17). This addendum does not change the questions in the OQAQ; rather, it adds guidelines for each question to standardize interpretation.
QUOROM Consensus Statement
The QUOROM statement was published in 1999 (7). It is a set of consensus guidelines developed to standardize and improve reporting quality of meta-analyses of randomized, controlled trials to minimize bias. It has also been used in the past to evaluate the quality of systematic reviews (12,14,16–18).
The QUOROM checklist is composed of 18 points. It evaluates the reporting quality of all parts of a systematic review, including the title, abstract, introduction, methods, results, and the conclusion. Much like the CONSORT guidelines for randomized, controlled trials, the QUOROM guidelines ensure adequate reporting in all aspects of a systematic review, thereby allowing the reader to judge the quality of the presented evidence.
Data Analysis
Reviewer agreement was quantified by using the κ statistic. To determine which factors were associated with higher methodologic quality, an ANOVA was used for continuous outcomes and a χ2 used for categorical outcomes. Factors that were significant in univariate analysis were examined in linear and logistic regression. P ≤ 0.01 was considered significant. Citations were managed using Reference Manager 11.0 (ResearchSoft, Carlsbad, CA). Data were abstracted using an electronic form on Microsoft Excel 2000 (Microsoft Corp., Redmond, WA). Data analysis was performed using SPSS 14.0 (SPSS, Chicago, IL) and SAS 9.1 (SAS Institute, Cary, NC).
Results
Study Selection
We screened a total of 591 citations. A total of 109 full-text articles were retrieved, and the eligibility of each full-text article was evaluated (Figure 1). The chance-corrected agreement was good between two independent reviewers who evaluated the eligibility of each review (κ = 0.71). We included two reviews that initially seemed nonrelevant from the title (19,20), because they included important outcomes that influence renal care. We excluded three reviews published by the Cochrane Renal Group (21–23); two reviews were not deemed relevant to our definition of renal care, because one review examined the treatment of uncomplicated urinary tract infection (21), and the other examined treatment of acute renal colic (22); the third review studied emergency interventions for acute hyperkalemia from all causes (23) but did not perform any analysis of the data in the context of a population with renal disease.
Selection of systematic reviews.
Characteristics of Nephrology-Related Systematic Reviews
A total of 90 renal systematic reviews were analyzed (all reviews presented in Table 1) (19,20,24–111). The majority of reviews were published in journals that do not specialize in nephrology or transplant medicine (n = 47; 52%), and a substantial proportion were published as Cochrane reviews (n = 18; 20%). Non-nephrology nontransplant journals and the Cochrane Database published the majority of systematic reviews in all five renal subject areas (Table 2).
Included studiesa
Number of systematic reviews by publication type and by topic in nephrology
Journals that specialize in transplant medicine or nephrology focused on topics relating to dialysis and kidney transplantation and published few reviews on acute kidney injury, chronic kidney disease, and glomerulonephritis. No systematic reviews related to pediatric nephrology were published by journals that specialize in nephrology or transplant medicine.
On average, 23 primary studies were included in the final analysis of each systematic review, with a range of 0 to 155 studies; however, a significant proportion of systematic reviews (n = 16; 11%) did not clearly indicate the number of primary studies included in the analysis. One systematic review included no articles in its analysis (87); however, because it was designated as a systematic review and not a protocol by the publishing source, it was included in the analysis.
Quality of Nephrology-Related Systematic Reviews
Interrater agreement on the presence of major flaws was good (κ = 0.71). The OQAQ scores were also poor with an average score of 6.0 out of 9.0 (SD 2.9; range 0.0 to 9.0), indicating incomplete reporting of methodologic techniques used in the review. The majority of nephrology-related systematic reviews (49 [54%] of 90) contained major methodologic flaws (Table 3).
Quality of systematic reviews
There were two common methodologic flaws. The majority of reviews (44 of 90 reviews) failed to assess the methodologic quality of the studies that they included in their analysis. An even bigger proportion of reviews (51 of 90 reviews) failed to show evidence that they attempted to minimize their bias during the process of selection of studies for analysis by having only one person select eligible studies, or they lacked a priori criteria for inclusion. Factors that were independently associated with higher methodologic quality were a lack of language restriction in the literature search, an attempt to contact primary study authors, and an explicit statement regarding sources of funding (Table 3).
The number of authors (P = 0.17) or funding by industry (P = 0.30) did not significantly correlate with study quality. The final conclusions reached (presence of association, absence of association, or insufficient evidence as judged independently by M.M. and H.T.-P.) did not have any relationship to the quality of the systematic reviews (P = 0.35). Affiliation with a nephrology department showed a trend toward an association with a higher methodologic quality (P = 0.05), but it did not reach our predetermined significance level of P = 0.01.
Overall, many nephrology-related systematic reviews of randomized, controlled trials did not adhere to QUOROM guidelines (Table 4), with an average score of 11.9 out of 18.0 (SD 4.6; range 0.0 to 18.0). A minority (four of 48) of journals recommended adherence to consensus guidelines for reporting of systematic reviews (e.g., QUOROM, MOOSE) in the instructions to authors (112–115); however, these journals published systematic reviews of randomized, controlled trials that had a significantly higher methodologic quality as measured by OQAQ when compared with journals that did not recommend adherence to guidelines (see Table 4).
Quality of systematic reviews of randomized, controlled trialsa
Discussion
Well-conducted systematic reviews aim to compile the best available evidence for various diagnosis and treatment options. We assessed the methodologic quality of 90 systematic reviews in the field of nephrology published in 2005. Only one quarter of these reviews were published by journals that specialize in nephrology or transplant medicine, another half were published in other journals, and the remainder were published as Cochrane reviews. The majority of systematic reviews had major methodologic flaws as measured by a validated instrument. Consensus guidelines for reporting of data in systematic reviews (QUOROM and MOOSE) were published in 1999 and 2000, respectively, but a majority of journals still do not endorse these guidelines in their instructions to authors; however, journals that recommended adherence to these guidelines published systemic reviews of higher methodologic quality.
Findings and Implications for Clinicians, Researchers, Publishers, and Librarians
The methodologic quality of many nephrology-related systematic reviews was poor. This could indicate that the methods used in systematic reviews were inappropriate and the conclusions of such reviews possibly biased. At the same time, the reviews did not adhere strictly to reporting guidelines, indicating suboptimal reporting quality. Clinicians and researchers should carefully appraise the methods used in a scientific review before giving any weight to the results.
Our study indicates that journals that endorsed reporting guidelines published systematic reviews of higher methodologic quality. Adoption of such guidelines by all journal publishers could help to improve the quality of published reviews.
The conclusion of any systematic review is limited by the reliability of estimates provided by the included primary articles; however, a majority of the systematic reviews failed to acknowledge this fact with no assessment of the methodologic quality of included studies. If the readers are to rely on review conclusions, then it is essential that they be given enough information to evaluate the quality of the supporting data.
Scientific research articles must include enough information to allow an independent party to repeat and verify the results. The method by which relevant information is compiled can be subjective. If there are no a priori inclusion criteria and if only a single person arbitrarily determines the entire selection process of primary evidence, the results may not replicate if the process was to be repeated independently. This was the most common critical flaw of the systematic reviews that we examined.
Independent factors that were associated with higher methodologic quality were a lack of language restriction in the literature search, an attempt to contact primary study authors, and an explicit statement regarding sources of funding. These points are not part of the OQAQ, but they are labor-intensive (e.g., translation of articles published in a foreign language). This may indicate that study authors who were willing to commit more effort to their project generally produced higher quality research.
Reviews published in the Cochrane Database were of significantly higher quality compared with the reviews published in journals that did not endorse consensus guidelines. The difference in quality may be due to RevMan software provided by Cochrane specifically for the creation of systematic reviews. This software has a built-in methodologic framework for conducting reviews, which may prompt review teams to adopt better methodologic quality.
Only a minority of reviews were published in journals that specialize in nephrology or transplant medicine, and the nonspecialty journals published articles on a broad range of topics. This is an important point for clinicians, researchers, and librarians, because a review of evidence based solely on subspecialty journals will result in missed studies.
Strengths, Limitations, and Future Directions
Although reporting guidelines such as QUOROM were published more than 7 years ago, this is the first study to demonstrate that the endorsement of such guidelines by journal editors is associated with significantly higher methodologic quality. This is also the first study to evaluate the sources and the quality of systematic reviews relevant to the clinical practice of nephrology. We used a comprehensive method to identify all renal systematic reviews for the most recent complete year. The inclusion of studies and the abstraction of data were performed in duplicate to minimize any bias in these tasks.
With respect to the quality of systematic reviews, our findings concur with previous articles that examined the quality of systematic reviews in the fields of anesthesia (10), general surgery (11), critical care (12), and emergency medicine (13). All of them found that a majority of systematic reviews contained major methodologic flaws. It is interesting that another common conclusion was that a large proportion of systematic reviews in a given specialty are not published in the subspecialty-related journals (10,12); however, none of these studies examined the association between the endorsement of reporting guidelines by a journal and the quality of the published articles.
The search was performed by an experienced librarian, but it remains conceivable that we missed a systematic review relevant to nephrology, particularly studies in which renal outcomes were secondary to the main question. Many important quality measures for the rigorous conduct of a systematic review (e.g., statistical tests for quantitative measurement of publication bias) are not considered in the OQAQ; however, currently, it is the only validated tool for the assessment of methodologic quality of systematic reviews.
Although the quality of the reviews was measured, we did not assess whether the flaws present in the studies were due to poor reporting at the level of the study authors or journal publishers or to poor methods used by the study authors. This could be investigated in the future by contacting the publishers and study authors. Other studies previously indicated that reporting often differs from the actual study method (116); however, regardless of the reason, the poor quality of the systematic reviews as judged by the reader makes these reports less credible for use in clinical practice. Finally, we assessed the quality of systematic reviews published in the course of a single year to provide an assessment of the current state of affairs. Although there may be changes in the quality of literature from year to year, that there were large deficiencies in methodologic quality of reviews published during a recent period of time should serve as a call to action.
It is not known whether poor methodologic quality influences the final conclusions reached by a nephrology-related systematic review, and this would be an important area of further study. A minority of reviews were published by renal journals. This may reflect a possible publishing bias and warrants further consideration.
Conclusions
Although there are examples of high-quality systematic reviews in nephrology, most could be improved. Funding bodies and authors need to make efforts to improve the quality of published reviews, to raise confidence for using these reports in patient care. Journal endorsement of consensus guidelines such as QUOROM may also be an important tool for improving the quality of systematic reviews.
Disclosures
None.
Acknowledgments
This project was by supported by the Kidney Foundation of Canada. A.X.G. was supported by a Clinician Scientist Award from the Canadian Institutes for Health Research. All work was performed independent of the funding bodies.
We thank Salina Chen, Arsh Jain, John Johnson, Anna Mathew, Salimah Shariff, Meghan Vlasschaert, Robert Yang, Ann Young, Ann McKibbon, Nancy Wilczynski, and the reviewers for helpful comments.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
- Received October 16, 2007.
- Accepted February 11, 2008.
- Copyright © 2008 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Cited By...
- No citing articles found.