Abstract
Writing grants that are subsequently funded is an integral part of the process of patient-oriented research. A catalogue of common deficiencies that are identified in the grant review process can yield valuable insights into the process of grant writing. This article provides the authors’ opinion on the common pitfalls in the current patient-oriented research applications that if identified before submission can lead to a stronger application. The authors participated in the review of clinical research grants to the National Kidney Foundation and catalogued the weaknesses of the grants that were reviewed and discussed. The top five reasons identified with grants were problems with study design (76%); statistical issues (34%); general issues such as ownership of the work, mentor, and environment (29%); weak hypothesis (24%); and problems with the research question, such as novelty or lack of creation of new data (24%). Patient-oriented research grants that have strong mentoring, are hypothesis driven, and have a strong study design that addresses sample size, analysis, and confounding factors have an increased chance of yielding high-quality research and, therefore, successful funding.
The fragile state of patient-oriented research is exemplified by few clinical investigators and fewer programs to learn patient-oriented research, which perhaps is responsible for even fewer clinical trials that examine outcomes that make a difference to patient care (1). Peer-review processes tend to be less favorable for clinical research than for laboratory research in general, perhaps because of concerns about human subject safety (2). Writing grants that get funded is an integral part of the process of patient-oriented research. One approach to refining the process of grant writing is to participate in the process of grant review. However, invitation to these committees is often limited to researchers with a track record of funding or publications. Nevertheless, deliberations of these committees can yield valuable insights into the research capabilities of the community of nephrologists who seek grant support to conduct patient-oriented research.
The National Kidney Foundation (NKF) annually reviews grants to fund fellows, junior faculty (“young investigators”), and established clinical scientists in the area of patient-oriented research. We report on the common strengths and weaknesses of applications during one NKF grant cycle to provide recommendations for study design during study planning and grant writing. These recommendations should improve the quality of grants that are submitted for potential funding and, most important, the quality of patient-oriented research that addresses problems in renal medicine.
We participated in the review of grants to NKF that were limited to patient-oriented research in Chicago, IL, on February 24 and 25, 2005. The strengths and weaknesses of each grant that was reviewed and discussed by the scientific review committee were noted prospectively by one of us (R.A.) and subsequently categorized into broad categories related to the design of clinical investigations. Thus, the report that follows is the collective opinion of the committee. The number of grants that had deficiencies in each of these areas was counted and tabulated. Because the stage of training may relate to deficits in certain areas of the grants, we compared the proportion of grants with specific characteristics by submission category (fellowship, young investigator, or clinical scientist). We also compared submitted grants from institutions with and without K-30 clinical research training awards. Finally, the characteristics of successful applications were compared with those that were not funded. The sample size was limited to the grants reviewed by one committee. Formal hypothesis testing, therefore, is not reported. Nevertheless, the lessons provided by these observations may be useful for those who intend to conduct patient-oriented research.
Number, Characteristics, and Outcomes of Grants
We reviewed 41 grants, three of which were found to be ineligible because of the nature of the research. Thus, 38 grants were reviewed further, 21 of which were fellowship applications, 10 of which were young investigator applications, and seven of which were clinical scientist applications. Table 1 shows the rank-ordered results of the deficits that were found in the grant applications. The most common deficiency was a problem with study design, identified in 29 (76%) of the grants. Among problems with study design, the following were the top five issues: Unclear or inadequate study design, lack of consideration of confounding variables, lack of or an inadequate control group, lack of rigorous definitions of disease state or end points, and lack of focus in the research plan. Statistical issues were identified in more than one third of the grants; these related to inadequate power, lack of power calculations, and a missing or inadequate analytical plan.
Rank order of deficits in clinical grants submitted to the NKFa
General issues that do not fall into standard categories but are discussed by research committees were the next most common problem. In particular, for the young investigator and clinical scientist awards, it was unclear in four instances whether the work was that of the potential grantee or of the mentor. The qualifications of the mentors or the investigators were suboptimal in three instances. The fourth major problem was that the research was not hypothesis driven; methods used were not detailed or defined or were inadequate; or the question was not novel, or no new data were being created.
Among these problems, it is surprising that none was related clearly to the stage of the investigator’s career with the sole exception of the statistical flaws (Table 2). None of the clinical scientist awards was identified to have statistical issues, whereas 40% of the young investigator awards and 43% of the fellowship applicants had statistical flaws. Finally, the scores awarded by the primary and secondary reviewers were not different for the three levels of investigators.
Deficits identified by class of application
All but seven grants came from institutions with existing K-30 awards. None of the problems enumerated in Table 1 were different between institutions with or without K-30 awards. with the exception of statistical issues. Six (86%) of seven grants without K-30 awards had statistical issues, whereas only seven (23%) of 31 grants with K-30 programs had statistical flaws.
Ten grants were funded, four of which were fellowship awards, three of which were young investigator awards, and three of which were clinician scientist awards with a commitment of $1,070,000. The characteristics of the 10 grants that were funded were compared with 28 that were unsuccessful. The characteristics of grants with an excellent score (2 or less) were compared with those with a less favorable score (>2), and the results were similar to those reported for funded applications (data not shown). Although the purpose of this report was not to analyze comprehensively and model the characteristics associated with successful funding, in general, grants that were successful or were scored well had fewer errors. It is clear that the grants that were funded also had errors, but they were not deemed severe enough to be of a quality that would preclude funding. Grant applications are awarded a score on the basis of the overall impression by the committee; therefore, it is difficult to dissect which characteristics influenced the overall score.
Lessons Learned
This report provides a list of common problems that were identified in a recent NKF review cycle of clinical research grants. Three of 41 grants submitted did not fall within the scope of clinical research as defined by the Nathan Report, i.e., studies that deal directly with patients (3). Other submissions that met the definition of eligible clinical research included epidemiologic studies, outcomes research, or health services research. The Clinical Scientist Award is designed to support translational research or research that leads to novel therapeutic interventions. The definition of clinical research from the Nathan Report excludes in vitro studies that use human tissues but do not deal directly with patients. Examples that meet the definition of clinical research include epidemiologic studies, outcomes research, and health services research (3). Therefore, potential grantees would save time and effort if they follow the instructions for eligibility before writing their grants.
A review of Table 1 provides an opportunity for potential grant applicants and their mentors to identify deficiencies that may exist in their research plans and, therefore, their grant applications. Several excellent texts provide a detailed outline for the process of designing clinical research (4,5).
Successful patient-oriented research (POR) study research applications need to address the following:
Is the proposal hypothesis driven?
Do the preliminary data or previously published data suggest that the hypothesis is sound?
Is the hypothesis novel, and will the findings make a contribution to the field?
Is the study design appropriate to answer the hypothesis?
Is the analysis plan sound, and can the project be completed in the time requested?
Does the study have human subject committee approval?
Are the mentor and the institution both committed to the candidate and the research program?
In the case of fellowship and “young investigator” awards, will the applicant gain valuable experience, regardless of the findings of the proposed study?
Study design issues were the major problem for many studies. Statistical issues continue to plague the grants. Although the sample size for comparison was small, it seemed that established investigators were particularly cognizant of statistical issues, whereas fellows and younger, less established investigators may be less familiar with analytic issues or focused more on the project’s interest than on feasibility. As such, mentors of the younger applicants should pay particular attention to this area. Grants that had a qualified statistician and an analytical plan fared better in general. Grants that were submitted from institutions without K-30 awards had more statistical flaws. Institutions with and without K-30 awards differ in many ways, so causality cannot be inferred. However, young investigators from institutions without local statistical support might benefit from external review before grant submission.
For the young investigator and established clinical scientist awards, there should be no scientific overlap between the support requested from NKF and that received from another agency. Candidates should be encouraged to submit applications to multiple agencies, but mentors and trainees should not “share” grants or jointly submit the same grant to two agencies with different individuals designated as principal investigator.
In summary, the rigorous conduct of patient-oriented research requires careful attention to study design, especially statistical issues; the research proposed should be hypothesis driven and move the field forward. Patient-oriented research grants that are hypothesis driven; have an adequate study design, sample size, and analytical plan; discuss the confounding factors and ways to deal with them; are performed in clinical research centers; and have strong institutional and mentor support have a greater chance of succeeding.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
- Copyright © 2006 by the American Society of Nephrology
In this issue
