What factors are really important in the valuation of NIH grant applications?

Approximately seven years ago in 2009, the NIH introduced a new series of criteria by which NIH grant applications could be “fairly” reviewed. At that time, five distinct review criteria, namely Significance, Applicant, Environment, Approach and a brand-new criterion, Innovation, were introduced. Reviewers of NIH grant applications were then required to comment on each of these five criteria and provide a relative merit score ranging from 1 to 10. Each application was also assigned an overall Impact Factor. According to the NIH, the Impact Factor should ““Reflect [a reviewer’s] assessment of the likelihood that the proposed research will have a sustained, powerful influence on the research fields involved.”  (http://grants.nih.gov/grants/peer_review_process.htm)  In other words, this factor should reflect the reviewer’s overall opinion of the relative quality of the application as a whole.

Since that time, the NIH has, on occasion, undertaken efforts to evaluate scores assigned to NIH R01 grant applications with respect to both the overall Impact Factor and its relationship to the various review criteria. The initial study was undertaken by the NIGMS in 2010 in which 360 R01 applications to that Institute were analyzed via Full Regression and Principal Component analysis. The results of this study provided strong evidence that, of the five review criteria, Innovation and Approach most closely correlated with overall Impact Score while the least correlation was obtained for the Environment review criterion. A follow-up study, published in the following year by the NIH Office of Extramural Research provided additional evidence in support of these conclusions with the Approach criterion clearly providing the best correlation with Impact Factors, and Significance a close second.

Well, the NIH has recently decided that five years is an appropriate period of time by which the validity of these early findings regarding the five distinct review criteria needed to be revisited. In this respect, a recently completed new, but relatively similar in terms of questions asked, study by the NIH Office of Extramural Research has just been completed and the findings summarized in the latest issue of the NIH Extramural Nexus (July 22, 2016; edited by Michael Lauer). Once again, and perhaps not surprisingly, the Approach section (correlation value 0.84) was documented to have the highest correlative value with Overall Impact, with Significance (correlation value 0.68) falling closely behind, followed by Innovation (0.61), Applicant (0.51) and Environment (0.44), in essence repeating and reinforcing the findings of the earlier studies. For the full report see PLOS-One: http://dx.doi.org/10.1371/journal.pone.0155060.g001

Collectively, all of these studies underscore the critical importance of both Approach and Significance in the preparation of NIH grant applications to any NIH Institute/Center. In this respect, we have recently emphasized in several Facebook/GWSW blog posts the rather significant changes in NIH grant applications that have profoundly influenced the way in which applicants need to write their proposals. (See, for example, “Forthcoming Significant Changes in NIH Applications” April 6, 2016 and “Three new NIH Reviewer Questions for Applicants”, May 2, 2016). It is our considered opinion that the majority of NIH applicants have not taken these new NIH criteria, concerning overall Scientific Premise, Critical Review of the Literature and Preliminary Data with respect to strengths and weaknesses, and documentation of Key Reagents, to heart. This is likely to doom many otherwise meritorious applications.

In anticipation of these forthcoming changes, early this spring, GWSW personnel undertook the commitment to make significant changes to The Grant Application Writer’s Workbook NIH Edition that was published in April, 2016. This extensively revised Workbook provides very detailed instructions on how applicants can effectively deal with these new NIH requirements. They truly require a total rethinking of how to write the Significance subsection of the application and key components of the Approach subsection. These changes are not trivial and we would encourage you to peruse the sample pages available on the Workbook webpage. If you were to have any doubt about how to address these major changes, information in this Workbook will provide you with the detailed step-by-step approach needed to place your application as one fully in compliance with these new NIH requirements. This is, of course, never a bad position to be in when submitting grant applications in this highly competitive environment!