The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it—that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is more accurately predicted by peer-review recommendations or discretion by program staff and institute leaders. Using data from a prestigious training program at the National Institute of Health (NIH), the Ruth L. Kirschstein National Research Service Award (NRSA), we provide evidence on the efficacy of peer review. We find that, despite all current claims to the contrary, the existing peer-review system works as intended. It more closely predicts high-quality science and future research independence than discretion. We discover also that regression discontinuity, the econometric method typically used to examine the effect of scientific funding, does not fit many scientific-funding models and should only be used with caution when studying federal awards for science.
Authors
- Acknowledgements & Disclosure
- Corresponding Author: Misty L. Heggeness, U.S. Census Bureau, 4600 Silver Hill Road, Washington, DC 20746 (e-mail: misty.heggeness@gmail.com). This paper reports the results of research and analysis undertaken by the National Institutes of Health (NIH) staff, consultants, and contractors. We acknowledge generous support and feedback from Lisa Evans, Henry Khachaturian, Kay Lund, Sherry Mills, Matthew Notowidigdo, Walter T. Schaffer, and Jennifer Sutton. This work would not have been possible without the amazing skills and assistance of Frances D. Carter-Johnson, Maria I. Larenas, Ryan Pfirrmann-Powell, and Ying Zeng within the NIH Division of Biomedical Research Workforce Programs, as well as participants from the 2018 Association for Public Policy Analysis and Management (APPAM) fall research conference and seminar participants at Iowa State University. We appreciate the time and candor given to us by various NIH institute program officers and training directors who so graciously agreed to be interviewed in an effort to help us truly understand the NRSA funding process on the ground. We also give very special thanks to Pierre Azoulay, Hyungjo Hur, Ming Lei, Jon Lorsch, Julie Mason, Michael Spittel, and Jonathan Wiest for taking the time to provide critical feedback. This analysis began while Misty L. Heggeness was employed at the National Institutes of Health. Donna Ginther acknowledges financial support from NIH Division of Biomedical Research Workforce (DBRW) (Contract No. HHSN276201300089U) to NETE Solutions. Any errors in this paper are the sole responsibility of the authors. The views presented here do not necessarily reflect any official position of the National Institutes of Health, U.S. Census Bureau, the Federal Reserve System, the Federal Reserve Bank of Minneapolis, or the University of Kansas. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
- DOI
- http://dx.doi.org/10.3386/w26841
- Published in
- United States of America
