Nothing Special   »   [go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/cgd/wpaper/336.html
   My bibliography  Save this paper

Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336

Author

Listed:
  • Lant Pritchett, Justin Sandefur
Abstract
In this paper we examine how policymakers and practitioners should interpret the impact evaluation literature when presented with conflicting experimental and non-experimental estimates of the same intervention across varying contexts. We show three things. First, as is well known, non-experimental estimates of a treatment effect comprise a causal treatment effect and a bias term due to endogenous selection into treatment. When non-experimental estimates vary across contexts any claim for external validity of an experimental result must make the assumption that (a) treatment effects are constant across contexts, while (b) selection processes vary across contexts. This assumption is rarely stated or defended in systematic reviews of evidence. Second, as an illustration of these issues, we examine two thoroughly researched literatures in the economics of education—class size effects and gains from private schooling—which provide experimental and non-experimental estimates of causal effects from the same context and across multiple contexts. We show that the range of “true” causal effects in these literatures implies OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Third, we show that in important cases in economics, parameter heterogeneity is driven by economy- or institution-wide contextual factors, rather than personal characteristics, making it difficult to overcome external validity concerns through estimation of heterogeneous treatment effects within a single localized sample. We conclude with recommendations for research and policy, including the need to evaluate programs in context, and avoid simple analogies to clinical medicine in which “systematic reviews” attempt to identify best-practices by putting most (or all) weight on the most “rigorous” evidence with no allowance for context.

Suggested Citation

  • Lant Pritchett, Justin Sandefur, 2013. "Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336," Working Papers 336, Center for Global Development.
    • Handle: RePEc:cgd:wpaper:336
    as

    Download full text from publisher

    File URL: http://www.cgdev.org/sites/default/files/context-matters-for-size_0.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. Chin, Aimee, 2005. "Can redistributing teachers across schools raise educational attainment? Evidence from Operation Blackboard in India," Journal of Development Economics, Elsevier, vol. 78(2), pages 384-405, December.
    3. Wo[ss]mann, Ludger & West, Martin, 2006. "Class-size effects in school systems around the world: Evidence from between-grade variation in TIMSS," European Economic Review, Elsevier, vol. 50(3), pages 695-736, April.
    4. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    5. Esther Duflo, 2001. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment," American Economic Review, American Economic Association, vol. 91(4), pages 795-813, September.
    6. Guildo W. Imbens, 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation," American Economic Review, American Economic Association, vol. 93(2), pages 126-132, May.
    7. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    8. Miguel Urquiola & Eric Verhoogen, 2009. "Class-Size Caps, Sorting, and the Regression-Discontinuity Design," American Economic Review, American Economic Association, vol. 99(1), pages 179-215, March.
    9. Duflo, Esther & Dupas, Pascaline & Kremer, Michael, 2015. "School governance, teacher incentives, and pupil–teacher ratios: Experimental evidence from Kenyan primary schools," Journal of Public Economics, Elsevier, vol. 123(C), pages 92-110.
    10. Lant Pritchett & Michael Woolcock & Matt Andrews, 2013. "Looking Like a State: Techniques of Persistent Failure in State Capability for Implementation," Journal of Development Studies, Taylor & Francis Journals, vol. 49(1), pages 1-18, January.
    11. Shahrukh Rafi Khan & David Kiefer, 2007. "Educational Production Functions for Rural Pakistan: A Comparative Institutional Analysis," Education Economics, Taylor & Francis Journals, vol. 15(3), pages 327-342.
    12. Michael Clemens & Claudio Montenegro & Lant Pritchett, 2008. "The Place Premium: Wage Differences for Identical Workers across the U.S. Border," Working Papers 148, Center for Global Development.
    13. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    14. G. M. Arif & Najam Us Saqib, 2003. "Production of Cognitive and Life Skills in Public, Private, and NGO Schools in Pakistan," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 42(1), pages 1-28.
    15. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    16. Brown, Philip H. & Park, Albert, 2002. "Education and poverty in rural China," Economics of Education Review, Elsevier, vol. 21(6), pages 523-541, December.
    17. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    18. Elizabeth M. King & Claudio E. Montenegro & Peter F. Orazem, 2012. "Economic Freedom, Human Rights, and the Returns to Human Capital: An Evaluation of the Schultz Hypothesis," Economic Development and Cultural Change, University of Chicago Press, vol. 61(1), pages 39-72.
    19. M. Niaz Asadullah, 2005. "The effect of class size on student achievement: evidence from Bangladesh," Applied Economics Letters, Taylor & Francis Journals, vol. 12(4), pages 217-221.
    20. Cristian Pop-Eleches, 2010. "The Supply of Birth Control Methods, Education, and Fertility: Evidence from Romania," Journal of Human Resources, University of Wisconsin Press, vol. 45(4), pages 971-997.
    21. Rodrik, Dani, 2008. "The New Development Economics: We Shall Experiment, but How Shall We Learn?," Working Paper Series rwp08-055, Harvard University, John F. Kennedy School of Government.
    22. Paul W. Glewwe & Eric A. Hanushek & Sarah D. Humpage & Renato Ravina, 2011. "School Resources and Educational Outcomes in Developing Countries: A Review of the Literature from 1990 to 2010," NBER Working Papers 17554, National Bureau of Economic Research, Inc.
    23. Lant Pritchett & Salimah Samji & Jeffrey S. Hammer, 2012. "It's All about MeE: Using Structured Experiential Learning ('e') to Crawl the Design Space," WIDER Working Paper Series wp-2012-104, World Institute for Development Economic Research (UNU-WIDER).
    24. Lee, Valerie E. & Lockheed, Marlaine E., 1989. "The effects of single-sex schooling on student achievement and attitudes in Nigeria," Policy Research Working Paper Series 206, The World Bank.
    25. Alan B. Krueger, 1999. "Experimental Estimates of Education Production Functions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 497-532.
    26. Lant Pritchett & Michael Woolcock & Matt Andrews, 2013. "Looking Like a State: Techniques of Persistent Failure in State Capability for Implementation," Journal of Development Studies, Taylor & Francis Journals, vol. 49(1), pages 1-18, January.
    27. repec:pri:rpdevs:hammer_its_all_about_me is not listed on IDEAS
    28. repec:unu:wpaper:wp2012-63 is not listed on IDEAS
    29. Bacolod, Marigee P. & Tobias, Justin L., 2006. "Schools, school quality and achievement growth: Evidence from the Philippines," Economics of Education Review, Elsevier, vol. 25(6), pages 619-632, December.
    30. Card, David, 2001. "Estimating the Return to Schooling: Progress on Some Persistent Econometric Problems," Econometrica, Econometric Society, vol. 69(5), pages 1127-1160, September.
    31. Daniel Suryadarma & Asep Suryahadi & Sudarno Sumarto & F. Halsey Rogers, 2006. "Improving Student Performance in Public Primary Schools in Developing Countries: Evidence from Indonesia," Education Economics, Taylor & Francis Journals, vol. 14(4), pages 401-429.
    32. Gomes-Neto, Joao Batista & Hanushek, Eric A, 1994. "Causes and Consequences of Grade Repetition: Evidence from Brazil," Economic Development and Cultural Change, University of Chicago Press, vol. 43(1), pages 117-148, October.
    33. Tessa Bold, Mwangi Kimenyi, Germano Mwabu, Justin Sandefur, 2011. "The High Return to Private Schooling in a Low-Income Country- Working Paper 279," Working Papers 279, Center for Global Development.
    34. Pritchett, Lant H. & DEC, 1994. "Desired fertility and the impact of population policies," Policy Research Working Paper Series 1273, The World Bank.
    35. Monazza Aslam, 2003. "The Determinants of Student Achievement in Government and Private Schools in Pakistan," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 42(4), pages 841-876.
    36. Harriet Nannyonjo, 2007. "Education Inputs In Uganda : An Analysis of Factors Influencing Learning Achievement in Grade Six," World Bank Publications - Books, The World Bank Group, number 6758.
    37. Miguel Urquiola, 2006. "Identifying Class Size Effects in Developing Countries: Evidence from Rural Bolivia," The Review of Economics and Statistics, MIT Press, vol. 88(1), pages 171-177, February.
    38. repec:pri:rpdevs:deaton_instruments_randomization_learning_all_04april_2010 is not listed on IDEAS
    39. Michaelowa, Katharina, 2001. "Primary Education Quality in Francophone Sub-Saharan Africa: Determinants of Learning Achievement and Efficiency Considerations," World Development, Elsevier, vol. 29(10), pages 1699-1716, October.
    40. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    41. Glewwe, Paul, et al, 1995. "An Eclectic Approach to Estimating the Determinants of Achievement in Jamaican Primary Education," The World Bank Economic Review, World Bank, vol. 9(2), pages 231-258, May.
    42. Alexander Tabarrok, 2013. "Private Education In India: A Novel Test Of Cream Skimming," Contemporary Economic Policy, Western Economic Association International, vol. 31(1), pages 1-12, January.
    43. Marshall, Jeffery H., 2009. "School quality and learning gains in rural Guatemala," Economics of Education Review, Elsevier, vol. 28(2), pages 207-216, April.
    44. Lant Pritchett & Salimah Samji & Jeffrey Hammer, 2012. "It’s All About MeE: Using Structured Experiential Learning (‘e’) to Crawl the Design Space," CID Working Papers 249, Center for International Development at Harvard University.
    45. Behrman, Jere R. & Khan, Shahrukh & Ross, David & Sabot, Richard, 1997. "School quality and cognitive achievement production: A case study for rural Pakistan," Economics of Education Review, Elsevier, vol. 16(2), pages 127-142, April.
    46. Bedi, Arjun S & Marshall, Jeffrey H, 1999. "School Attendance and Student Achievement: Evidence from Rural Honduras," Economic Development and Cultural Change, University of Chicago Press, vol. 47(3), pages 657-682, April.
    47. Hsieh, Chang-Tai & Urquiola, Miguel, 2006. "The effects of generalized school choice on achievement and stratification: Evidence from Chile's voucher program," Journal of Public Economics, Elsevier, vol. 90(8-9), pages 1477-1503, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. What do 600 papers on 20 types of interventions tell us about how much impact evaluations generalize? Guest post by Eva Vivalt
      by Development Impact Guest Blogger in Development Impact on 2014-11-10 06:36:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.
    2. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    3. Hanushek, Eric A., 2021. "Addressing cross-national generalizability in educational impact evaluation," International Journal of Educational Development, Elsevier, vol. 80(C).
    4. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    5. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    6. Grossman, Guy & Humphreys, Macartan & Sacramone-Lutz, Gabriella, 2020. "Information Technology and Political Engagement: Mixed Evidence from Uganda," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 82(4), pages 1321-1336.
    7. Corduneanu-Huci, Cristina & Dorsch, Michael T. & Maarek, Paul, 2021. "The politics of experimentation: Political competition and randomized controlled trials," Journal of Comparative Economics, Elsevier, vol. 49(1), pages 1-21.
    8. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2015. "The gold standard for randomized evaluations: from discussion of method to political economy," Working Papers DT/2015/01, DIAL (Développement, Institutions et Mondialisation).
    9. Luis Andres & Christian Borja-Vega & Crystal Fenwick & Ronald Gomez-Suarez & Jaime De Jesus Filho, 2018. "A Brief Summary of Global WASH Interventions," World Bank Publications - Reports 29868, The World Bank Group.
    10. Muller, Sean, 2014. "Randomised trials for policy: a review of the external validity of treatment effects," SALDRU Working Papers 127, Southern Africa Labour and Development Research Unit, University of Cape Town.
    11. Andrews, Matt & Pritchett, Lant & Woolcock, Michael, 2017. "Building State Capability: Evidence, Analysis, Action," OUP Catalogue, Oxford University Press, number 9780198747482.
    12. Karthik Muralidharan & Venkatesh Sundararaman, 2013. "Contract Teachers: Experimental Evidence from India," NBER Working Papers 19440, National Bureau of Economic Research, Inc.
    13. Emmy De Buck & Karin Hannes & Hans Van Remoortel & Thashlin Govender & Axel Vande Veegaete & Alfred Musekiwa & Vittoria Lutje & Margaret Cargo & Hans‐Joachim Mosler & Philippe Vandekerckhove & Taryn Y, 2016. "PROTOCOL: Approaches to Promote Handwashing and Sanitation Behaviour Change in Low‐ and Middle Income Countries: A Mixed Method Systematic Review," Campbell Systematic Reviews, John Wiley & Sons, vol. 12(1), pages 1-46.
    14. Muralidharan, Karthik & Das, Jishnu & Holla, Alaka & Mohpal, Aakash, 2017. "The fiscal cost of weak governance: Evidence from teacher absence in India," Journal of Public Economics, Elsevier, vol. 145(C), pages 116-135.
    15. Clive Bell & Lyn Squire, 2017. "Providing Policy Makers with Timely Advice: The Timeliness-Rigor Trade-off," The World Bank Economic Review, World Bank, vol. 31(2), pages 553-569.
    16. Florent Bedecarrats & Isabelle Guérin & François Roubaud, 2017. "L'étalon-or des évaluations randomisées : du discours de la méthode à l'économie politique," Working Papers ird-01445209, HAL.
    17. Ngoc Thi Minh Tran & Michael P. Cameron & Jacques Poot, 2019. "What are migrants willing to pay for better home country institutions?," Letters in Spatial and Resource Sciences, Springer, vol. 12(3), pages 257-268, December.
    18. Rebecca Stone & Thomas de Hoop & Andrea Coombes & Pooja Nakamura, 2020. "What works to improve early grade literacy in Latin America and the Caribbean? A systematic review and meta‐analysis," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(1), March.
    19. Daido Kido, 2022. "Distributionally Robust Policy Learning with Wasserstein Distance," Papers 2205.04637, arXiv.org, revised Aug 2022.
    20. Das, Ashis & Friedman, Jed & Kandpal, Eeshani, 2014. "Does involvement of local NGOs enhance public service delivery ? cautionary evidence from a Malaria-prevention evaluation in India," Policy Research Working Paper Series 6931, The World Bank.
    21. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    22. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    23. Carinne Brody & Thomas de Hoop & Martina Vojtkova & Ruby Warnock & Megan Dunbar & Padmini Murthy & Shari L. Dworkin, 2015. "Economic Self‐Help group Programs for Improving Women's Empowerment: A Systematic Review," Campbell Systematic Reviews, John Wiley & Sons, vol. 11(1), pages 1-182.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pritchett Lant & Sandefur Justin, 2014. "Context Matters for Size: Why External Validity Claims and Development Practice do not Mix," Journal of Globalization and Development, De Gruyter, vol. 4(2), pages 161-197, March.
    2. Woolcock, Michael, 2013. "Using Case Studies to Explore the External Validity of 'Complex' Development Interventions," Working Paper Series rwp13-048, Harvard University, John F. Kennedy School of Government.
    3. Miguel Urquiola, 2015. "Progress and challenges in achieving an evidence-based education policy in Latin America and the Caribbean," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-30, December.
    4. Behrman, Jere R., 2010. "Investment in Education Inputs and Incentives," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4883-4975, Elsevier.
    5. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers WR-587, RAND Corporation.
    6. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers 587, RAND Corporation.
    7. Sudipto Mundle, 2018. "Fifty years of Asian experience in the spread of education and healthcare," WIDER Working Paper Series wp-2018-97, World Institute for Development Economic Research (UNU-WIDER).
    8. Mundle, Sudipto, 2018. "Development of Education and Health Services in Asia and the Role of the State," Working Papers 18/239, National Institute of Public Finance and Policy.
    9. Harounan Kazianga & Leigh Linden & Ali Protik & Matt Sloan, 2015. "Impact Evaluation of Burkina Faso's BRIGHT Program: Design Report," Mathematica Policy Research Reports c0250cd3f27d448ea70d909c3, Mathematica Policy Research.
    10. Sudipto Mundle, 2018. "Fifty years of Asian experience in the spread of education and healthcare," WIDER Working Paper Series 97, World Institute for Development Economic Research (UNU-WIDER).
    11. Cuesta, Ana & Glewwe, Paul & Krause, Brooke, 2016. "School infrastructure and educational outcomes: a literature review, with special reference to Latin America," LSE Research Online Documents on Economics 123331, London School of Economics and Political Science, LSE Library.
    12. Woolcock, Michael, 2013. "Using Case Studies to Explore the External Validity of 'Complex' Development Interventions," Working Paper Series rwp13-048, Harvard University, John F. Kennedy School of Government.
    13. Karthik Muralidharan & Venkatesh Sundararaman, 2013. "Contract Teachers: Experimental Evidence from India," NBER Working Papers 19440, National Bureau of Economic Research, Inc.
    14. Marco Manacorda, 2012. "The Cost of Grade Retention," The Review of Economics and Statistics, MIT Press, vol. 94(2), pages 596-606, May.
    15. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    16. Niklas Jakobsson & Mattias Persson & Mikael Svensson, 2013. "Class-size effects on adolescents' mental health and well-being in Swedish schools," Education Economics, Taylor & Francis Journals, vol. 21(3), pages 248-263, July.
    17. Maximilian Bach & Stephan Sievert, 2019. "Birth Cohort Size Variation and the Estimation of Class Size Effects," Discussion Papers of DIW Berlin 1817, DIW Berlin, German Institute for Economic Research.
    18. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.
    19. Masino, Serena & Niño-Zarazúa, Miguel, 2016. "What works to improve the quality of student learning in developing countries?," International Journal of Educational Development, Elsevier, vol. 48(C), pages 53-65.
    20. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.

    More about this item

    Keywords

    external validity; treatment effects; policy evaluation; causal inference;
    All these keywords.

    JEL classification:

    • D04 - Microeconomics - - General - - - Microeconomic Policy: Formulation; Implementation; Evaluation
    • I2 - Health, Education, and Welfare - - Education
    • O2 - Economic Development, Innovation, Technological Change, and Growth - - Development Planning and Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cgd:wpaper:336. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Publications Manager (email available below). General contact details of provider: https://edirc.repec.org/data/cgdevus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.