NIH Impact Factor 26, (34%) NOT FUNDED. What do I do now?
Unfortunately, the above peer reviewer evaluation of a first submission of a grant application occurs all too often, invariably leaving the applicant with the difficult decision of whether to resubmit the proposal. Obviously, there is no easy answer to this question and there are a number of factors that should be considered (discussed in detail in our versions of The Grant Application Writer’s Workbook), including the opinions of trusted colleagues, potential ease of responding to reviewers’ concerns, extent of additional pilot data that might be required, perceived level of enthusiasm of the Program Officer, and his/her recommendation. While many of these are “best bet judgments” as to decisions regarding resubmission, it never hurts to at least consider hard data. In this respect, the NIH has recently weighed in on this in a recent issue of the NIH Extramural Nexus by Mike Lauer entitled “Resubmissions Revisited: Funded Resubmission Applications and their Initial Peer Review Scores” (February 17, 2017).
Dr. Lauer and his colleagues analyzed more than 83,000 unsolicited A0 R01 applications submitted over a four-year period from 2012 through 2016. About 69,000 (or 83%) were “Type 1” (de novo) applications, and 14,000 (or 17%) were “Type 2” (or competing renewal) applications. The data were analyzed in terms of both initial funding based on impact score, and the influence of initial impact score on the likelihood of an A1 application being funded. The data, in addition to addressing the key question which will be the focus of this blog post, provide some rather interesting findings. First, with respect to initial impact score on the initial funding, not surprisingly, approximately two-thirds of the applications with an initial impact score between 10 and 30 were funded. (For these and all of the data discussed here, rates are slightly higher for Type 2 applications relative to Type 1 applications.) Success rates for applications with impact scores between 30 and 40 were significantly less, but still appreciable (about 10%). None of these findings would be particularly surprising.
Perhaps somewhat less anticipated is that relatively few applications within the impact score between 40 and 50 were also funded (about 1-2%). Remarkably, there were even a handful (less than a dozen) of applications with impact scores of 50-90 that were funded based on the initial A0 application. The fact that applications with what would universally be perceived as less than encouraging impact scores would be funded brings up an important point that is frequently not fully appreciated by NIH applicants. Namely, the critical importance of programmatic relevance cannot be overemphasized. NIH applications are evaluated on two independent set of criteria that are independently derived. Scientific merit is determined by Study Sections (and reflected by impact scores), whereas programmatic relevance is determined by Program Officers (based upon programmatic needs and relevance). In this respect, the most likely explanation for applications with priority scores >40 being approved for funding would be that they were evaluated as having exceptionally high programmatic relevance.
For those applications that received impact scores that did not merit funding, the obvious question would be whether to resubmit. Once again, the majority of the data reported by Dr. Lauer are not particularly surprising, with likelihood of resubmission showing a direct relationship with initial impact score. Likelihood of resubmission decreased gradually with increasing impact score (10-30 at ~85%, 30-40 at ~75%, 40-50 at ~65%, 50-90 at ~45% and non-discussed at ~30%). What these data seem to indicate is that impact scores directly influence the level of applicants’ enthusiasm for resubmission of the initial A0 application. In terms of potential resubmission; however, while there is only a significant difference between Type 1 versus Type 2 at the higher impact scores, the differential widens as the initial impact score increases, perhaps reflecting increasing angst among previously-funded applicants relative to new applicants.
In terms of success rates, these also closely reflect the likelihood of resubmission findings. In this respect, the data regarding the likelihood of funding of the A1 application based upon impact score of the A0 application are as follows: 10-30 at ~65%, 30-40 at ~ 55%, 40-50 at ~35%, 50-90 at ~25% and unscored at ~15%. What all this means is that the likelihood that you will ever be funded based upon the content of the A0 application is such that the initial application must receive a pretty good score. If the initial application receives an impact score in the 10-30 range, you have about an 85% chance of being funded; if the initial impact score was unscored, Dr. Lauer estimates that the likelihood that a new application will ever receive funding is about 1½% and for a competitive renewal application that percentage is about 10%.
While these data can be used to assess overall probabilities, the ultimate decision will be a judgment call among you, your Program Officer, and respected colleagues in whom you have trust.