Response to NIH RFI on Interim Research Products

The NIH has posted an RFI on including “preprints and interim research products” in NIH applications and reports. Many others have provided responses to this RFI that they have shared publicly, including thoughtful responses from David Mobley, Steven Floor, and others that have been collected by ASAPBio. Here is my response, written very quickly on a train ride back from the NIH.

Types of interim research products your or your organization create/and or host

My laboratory in a mixed wet/dry laboratory. Interim research projects my laboratory both creates and uses are:

  1. Feedback on what are considered to be interim research products, and how they are used in your field

Insight on how particular types of interim research products might impact the advancement of science

There is great value in many interim research products. Peer-reviewed publications are not error-free and endowed with value simply by virtue of the fact that they have passed through the peer review process—to think otherwise is nothing short of “magical thinking” that does nothing but damage our field and hinder progress. The current publication peer review system is highly flawed, and may do little to ultimately enhance ultimate research quality or accuracy. Instead, the main factors governing the utility of research products (interim and final) are:

The timeliness factor is not to be underestimated. In particular, it is critical to consider the overall cost of delaying release of a publication and its data by a lengthy multi-month review process.

Feedback on potential citation standards

Should the NIH consider trying to define their own standard, as they have with PMIDs or PMCIDs, I would encourage them to first read this XKCD comic:

Insight on the possible need and potential impact of citing interim products on peer review of NIH applications

For early-career scientists (where time to receiving first NIH funding is a major factor in career progression) or any time-sensitive research, it is critical that the NIH allow interim products to be cited in NIH applications as evidence of productivity leading up to grant submissions.

Advice on how NIH reviewers might evaluate citations of interim research products in applications

NIH reviewers should evaluate interim research products the same way they should evaluate any research project: By the quality, utility, and significance of the scientific results presented in it. How else are would they evaluate research products? By looking to see if they are published in Nature or Science? Because if that is what is currently happening, we have a major problem not with how reviewers should evaluate interim research projects, but how reviewers evaluate all research products.

Any other relevant information

Rather than delicately approaching the issue of pre-publication products (instead of wholeheartedly embracing them), the NIH should develop a strategy to cope with the impending demise of the journal enterprise, and whether the NIH can do more to accelerate this process. Elsevier alone has an annual publishing revenue of $25.2B, which is nearly the size of the NIH extramural budget. Elimination of the flawed peer-review journal system and establishment of an NIH-supported post-publication review or rating system that rapidly disseminated NIH-funded scientific results would save an enormous amount of money and drastically increase scientific productivity.