Nothing Special   »   [go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/102.html
   My bibliography  Save this paper

Heterogeneity in Effect Size Estimates: Empirical Evidence and Practical Implications

Author

Listed:
  • Holzmeister, Felix
  • Johannesson, Magnus
  • Böhm, Robert
  • Dreber, Anna
  • Huber, Jürgen
  • Kirchler, Michael
Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confi dence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confi dence intervals. We illustrate that heterogeneity of this magnitude-unless properly accounted for-has severe implications for statistical inference with strongly increased rates of false scientifi c claims.

Suggested Citation

  • Holzmeister, Felix & Johannesson, Magnus & Böhm, Robert & Dreber, Anna & Huber, Jürgen & Kirchler, Michael, 2024. "Heterogeneity in Effect Size Estimates: Empirical Evidence and Practical Implications," I4R Discussion Paper Series 102, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:102
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/283318/1/I4R-DP102.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:grz:wpsses:2021-08 is not listed on IDEAS
    2. Nate Breznau & Eike Mark Rinke & Alexander Wuttke & Hung H. V. Nguyen & Muna Adem & Jule Adriaans & Amalia Alvarez-Benjumea & Henrik K. Andersen & Daniel Auer & Flavio Azevedo & Oke Bahnsen & Dave Bal, 2022. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 119(44), pages 2203150119-, November.
    3. Katherine L. Milkman & Linnea Gandhi & Mitesh S. Patel & Heather N. Graci & Dena M. Gromet & Hung Ho & Joseph S. Kay & Timothy W. Lee & Jake Rothschild & Jonathan E. Bogard & Ilana Brody & Christopher, 2022. "A 680,000-person megastudy of nudges to encourage vaccination in pharmacies," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 119(6), pages 2115126119-, February.
    4. Katherine L. Milkman & Dena Gromet & Hung Ho & Joseph S. Kay & Timothy W. Lee & Pepi Pandiloski & Yeji Park & Aneesh Rai & Max Bazerman & John Beshears & Lauri Bonacorsi & Colin Camerer & Edward Chang, 2021. "Megastudies improve the impact of applied behavioural science," Nature, Nature, vol. 600(7889), pages 478-483, December.
    5. Amanda Kvarven & Eirik Strømland & Magnus Johannesson, 2020. "Comparing meta-analyses and preregistered multiple-laboratory replication projects," Nature Human Behaviour, Nature, vol. 4(4), pages 423-434, April.
    6. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    7. Joel Ferguson & Rebecca Littman & Garret Christensen & Elizabeth Levy Paluck & Nicholas Swanson & Zenan Wang & Edward Miguel & David Birke & John-Henry Pezzuto, 2023. "Survey of open science practices and attitudes in the social sciences," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    8. Dorothy Bishop, 2019. "Rein in the four horsemen of irreproducibility," Nature, Nature, vol. 568(7753), pages 435-435, April.
    9. Regina Nuzzo, 2014. "Scientific method: Statistical errors," Nature, Nature, vol. 506(7487), pages 150-152, February.
    10. Philip B. Stark, 2018. "Before reproducibility must come preproducibility," Nature, Nature, vol. 557(7707), pages 613-613, May.
    11. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    12. Christoph Huber & Anna Dreber & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Utz Weitzel & Miguel Abellán & Xeniya Adayeva & Fehime Ceren Ay & Kai Barron & Zachariah Berry & Werner Bönte , 2023. "Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(23), pages 2215572120-, June.
    13. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    14. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    15. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    16. Christopher J. Bryan & Elizabeth Tipton & David S. Yeager, 2021. "Behavioural science is unlikely to change the world without a heterogeneity revolution," Nature Human Behaviour, Nature, vol. 5(8), pages 980-989, August.
    17. Nate Breznau & Eike Mark Rinke & Alexander Wuttke & Hung H. V. Nguyen & Muna Adem & Jule Adriaans & Amalia Alvarez-Benjumea & Henrik K. Andersen & Daniel Auer & Flavio Azevedo & Oke Bahnsen & Dave Bal, 2022. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty," Decision Analysis, INFORMS, vol. 119(44), pages 2203150119-, November.
    18. Eric-Jan Wagenmakers & Alexandra Sarafoglou & Balazs Aczel, 2022. "One statistical analysis must not rule them all," Nature, Nature, vol. 605(7910), pages 423-425, May.
    19. repec:nas:journl:v:115:y:2018:p:2584-2589 is not listed on IDEAS
    20. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    21. Rotem Botvinik-Nezer & Felix Holzmeister & Colin F. Camerer & Anna Dreber & Juergen Huber & Magnus Johannesson & Michael Kirchler & Roni Iwanir & Jeanette A. Mumford & R. Alison Adcock & Paolo Avesani, 2020. "Variability in the analysis of a single neuroimaging dataset by many teams," Nature, Nature, vol. 582(7810), pages 84-88, June.
    22. Kristijan Armeni & Loek Brinkman & Rickard Carlsson & Anita Eerland & Rianne Fijten & Robin Fondberg & Vera E Heininga & Stephan Heunis & Wei Qi Koh & Maurits Masselink & Niall Moran & Andrew Ó Baoill, 2021. "Towards wide-scale adoption of open science practices: The role of open science communities," Science and Public Policy, Oxford University Press, vol. 48(5), pages 605-611.
    23. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    24. Katherine L. Milkman & Mitesh S. Patel & Linnea Gandhi & Heather N. Graci & Dena M. Gromet & Hung Ho & Joseph S. Kay & Timothy W. Lee & Modupe Akinola & John Beshears & Jonathan E. Bogard & Alison But, 2021. "A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 118(20), pages 2101165118-, May.
    25. Brian C. Martinson & Melissa S. Anderson & Raymond de Vries, 2005. "Scientists behaving badly," Nature, Nature, vol. 435(7043), pages 737-738, June.
    26. A. Delios & E.G. Clemente & T. Wu & H. Tan & Y. Wang & M. Gordon & D. Viganola & Z. Chen & A. Dreber & M. Johannesson & T. Pfeiffer & E.L. Uhlmann & A.M.A. Al-Aziz & A.T. Abraham & J. Trojan & M. Adam, 2022. "Examining the Generalizability of Research Findings from Archival Data," Post-Print hal-04452695, HAL.
    27. Uri Simonsohn & Joseph P. Simmons & Leif D. Nelson, 2020. "Specification curve analysis," Nature Human Behaviour, Nature, vol. 4(11), pages 1208-1214, November.
    28. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    29. Wicherts, Jelte M. & Veldkamp, Coosje Lisabet Sterre & Augusteijn, Hilde & Bakker, Marjan & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2016. "Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking," OSF Preprints umq8d, Center for Open Science.
    30. Balazs Aczel & Barnabas Szaszi & Alexandra Sarafoglou & Zoltan Kekecs & Šimon Kucharský & Daniel Benjamin & Christopher D. Chambers & Agneta Fisher & Andrew Gelman & Morton A. Gernsbacher & John P. Io, 2020. "A consensus-based transparency checklist," Nature Human Behaviour, Nature, vol. 4(1), pages 4-6, January.
    31. Raphael Silberzahn & Eric L. Uhlmann, 2015. "Crowdsourced research: Many hands make tight work," Nature, Nature, vol. 526(7572), pages 189-191, October.
    32. Balazs Aczel & Barnabas Szaszi & Alexandra Sarafoglou & Zoltan Kekecs & Šimon Kucharský & Daniel Benjamin & Christopher D. Chambers & Agneta Fisher & Andrew Gelman & Morton A. Gernsbacher & John P. Io, 2020. "Author Correction: A consensus-based transparency checklist," Nature Human Behaviour, Nature, vol. 4(1), pages 120-120, January.
    33. Uri Simonsohn & Joseph P. Simmons & Leif D. Nelson, 2020. "Publisher Correction: Specification curve analysis," Nature Human Behaviour, Nature, vol. 4(11), pages 1215-1215, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    2. Christoph Huber & Anna Dreber & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Utz Weitzel & Miguel Abellán & Xeniya Adayeva & Fehime Ceren Ay & Kai Barron & Zachariah Berry & Werner Bönte , 2023. "Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(23), pages 2215572120-, June.
    3. Dreber, Anna & Johannesson, Magnus, 2023. "A framework for evaluating reproducibility and replicability in economics," I4R Discussion Paper Series 38, The Institute for Replication (I4R).
    4. Eric-Jan Wagenmakers & Alexandra Sarafoglou & Sil Aarts & Casper Albers & Johannes Algermissen & Štěpán Bahník & Noah Dongen & Rink Hoekstra & David Moreau & Don Ravenzwaaij & Aljaž Sluga & Franziska , 2021. "Seven steps toward more transparency in statistical practice," Nature Human Behaviour, Nature, vol. 5(11), pages 1473-1480, November.
    5. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    6. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.
    7. Kaiser, Tim & Lusardi, Annamaria & Menkhoff, Lukas & Urban, Carly, 2022. "Financial education affects financial knowledge and downstream behaviors," Journal of Financial Economics, Elsevier, vol. 145(2), pages 255-272.
    8. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    9. Anna Dreber & Magnus Johannesson & Yifan Yang, 2024. "Selective reporting of placebo tests in top economics journals," Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
    10. Abel Brodeur & Nikolai Cook & Carina Neisser, 2024. "p-Hacking, Data type and Data-Sharing Policy," The Economic Journal, Royal Economic Society, vol. 134(659), pages 985-1018.
    11. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    12. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
    13. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).
    14. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    15. Adler, Susanne Jana & Röseler, Lukas & Schöniger, Martina Katharina, 2023. "A toolbox to evaluate the trustworthiness of published findings," Journal of Business Research, Elsevier, vol. 167(C).
    16. Slichter, David & Tran, Nhan, 2023. "Do better journals publish better estimates?," MPRA Paper 118433, University Library of Munich, Germany.
    17. Doucouliagos, Hristos & Hinz, Thomas & Zigova, Katarina, 2022. "Bias and careers: Evidence from the aid effectiveness literature," European Journal of Political Economy, Elsevier, vol. 71(C).
    18. Sebastian Bachler & Armando Holzknecht & Jürgen Huber & Michael Kirchler, 2024. "From Individual Choices to the 4-Eyes-Principle: The Big Robber Game revisited among Financial Professionals and Students," Working Papers 2024-04, Faculty of Economics and Statistics, Universität Innsbruck.
    19. Vigren, Andreas & Pyddoke, Roger, 2020. "The impact on bus ridership of passenger incentive contracts in public transport," Transportation Research Part A: Policy and Practice, Elsevier, vol. 135(C), pages 144-159.
    20. Gechert, Sebastian & Mey, Bianka & Opatrny, Matej & Havranek, Tomas & Stanley, T. D. & Bom, Pedro R. D. & Doucouliagos, Hristos & Heimberger, Philipp & Irsova, Zuzana & Rachinger, Heiko J., 2023. "Conventional Wisdom, Meta-Analysis, and Research Revision in Economics," EconStor Preprints 280745, ZBW - Leibniz Information Centre for Economics.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:102. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.