Methods Matter: P-Hacking and Causal Inference in Economics

Abel Brodeur, Nikolai Cook and Anthony Heyes

No 11796, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: The economics 'credibility revolution' has promoted the identification of causal relationships using difference-in-differences (DID), instrumental variables (IV), randomized control trials (RCT) and regression discontinuity design (RDD) methods. The extent to which a reader should trust claims about the statistical significance of results proves very sensitive to method. Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and p-hacking is a substantial problem in research employing DID and (in particular) IV. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.

Keywords: research methods; causal inference; p-curves; p-hacking; publication bias (search for similar items in EconPapers)
JEL-codes: A11 B41 C13 C44 (search for similar items in EconPapers)
Pages: 28 pages
Date: 2018-08
New Economics Papers: this item is included in nep-hme, nep-hpe and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (33)

Published - published as 'Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics' in: American Economic Review, 2020, 110 (11), 3634-3660

Downloads: (external link)
https://docs.iza.org/dp11796.pdf (application/pdf)

Related works:
Working Paper: Methods Matter: P-Hacking and Causal Inference in Economics (2018)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp11796

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().