Empirical researchers frequently rely on normal approximations in order to summarize and communicate uncertainty about their findings to their scientific audience. When such approximations are unreliable, they can lead the audience to make misguided decisions. We propose to measure the failure of the conventional normal approximation for a given estimator by the total variation distance between a bootstrap distribution and the normal distribution parameterized by the point estimate and standard error. For a wide class of decision problems and a class of uninformative priors, we show that a multiple of the total variation distance bounds the mistakes which result from relying on the conventional normal approximation. In a sample of recent empirical articles that use a bootstrap for inference, we find that the conventional normal approximation is often poor. We suggest and illustrate convenient alternative reports for such settings.
Authors
- Acknowledgements & Disclosure
- We acknowledge funding from the National Science Foundation (SES-1654234, SES-1949047) and the Semester Undergraduate Program for Economics Research at Harvard. We thank our dedicated research assistants for their contributions to this project, as well as the many authors who helped us to work with their code, data, and bootstrap replicates. We thank Anna Mikusheva, Jon Roth, and participants at the Princeton Day of Statistics for helpful comments. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. Jesse M. Shapiro Shapiro has, in the past, been a paid visitor at Microsoft Research New England and a paid consultant for FutureOfCapitalism, LLC. Shapiro has been paid for writing by the New York Times. Shapiro's spouse has a disclosure statement posted at https://emilyoster.net/about/.
- DOI
- https://doi.org/10.3386/w32038
- Published in
- United States of America
