Abstract
Propensity score matching and weighting methods are applied to balance covariates and reduce selection bias in the analysis of observational study data, and ultimately estimate a treatment effect. We wish to evaluate the impact of a Supplemental Instruction (SI) program on student success in an Introductory Statistics course. In such student success studies, propensity score methods have been applied successfully for evaluating a binary treatment, namely whether a student attending SI performs better or worse than a student who does not attend SI. However, in this setting, we also want to draw inferences on the dose-response relationship, namely how does the number of times a student attends SI impact performance in the course. In this paper, we introduce generalized propensity scores (GPS) for analyzing such continuous treatment. We extend recent developments in GPS analyses from the personalized learning literature for evaluating SI engagement on student success. As part of the exposition, we provide a brief review of generalized propensity scores, compare our proposed GPS approaches, present guidelines on how these methods can be applied to educational data, and present R code and illustration for practitioners to use as a template in educational data mining applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Austin, P.C. (2008). Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score. Pharmacoepidemiology and Drug Safety, 17, 1202–17. https://doi.org/10.1002/pds.1673.
Austin, P.C. (2009). Balance diagnostics for comparing the distribution ofbaseline covariates between treatment groups in propensity-score matchedsamples. Statistics in Medince, 28, 3083–3107.
Austin, P. C., & Stuart, E.A. (2015). Moving towards best practice when using inverse probability of treatment weighting (iptw) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661–3679. https://doi.org/10.1002/sim.6607.
Babcock, R., & Georgiou, J. (2020). Comparison of two online courseware instructional methods using propensity score matching. Journal of Effective Teaching in Higher Education.
Dawson, P., van der Meer, J., Skalicky, J., & Cowley, K. (2014). On the effectiveness of supplemental instruction: a systemic review of supplemental instruction and peer-assisted study sessions literature between 2001 and 2010. Review of Educational Research, 84(4), 609–639.
Dehejia, R.H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and statistics, 150–161.
Fletcher, R. (2013). Newton-like methods. In Practical methods of optimization (pp. 44–79). Wiley. Retrieved from https://doi.org/10.1002/9781118723203.ch3.
Fong, C., Hazlett, C., & Imai, K. (2018). Covariate balancing propensity score for a continuous treatment:Application to the efficacy of political advertisements. The Annals of Applied Statistics, 160–167.
Fuente, A.D.L. (2000). Mathematical methods and models for economists. Cambridge University Press. https://doi.org/10.1017/CBO9780511810756.
Greer, J., & Mark, M. (2016). Evaluation methods for intelligent tutoring systems revisited. International Journal of Artificial Intelligence in Education, 26. https://doi.org/10.1007/s40593-015-0043-2.
Greifer, N. (2020). Weightit:Weighting for covariate balance in observational studies [Computer software manual]. Retrieved from https://CRAN.R-project.org/package=WeightIt (R package version 0.10.1).
Guarcello, M.A., Levine, R.A., Beemer, J., Frazee, J.P., Laumakis, M.A., & Schellenberg, S.A. (2017). Balancing student success:Assessing supplemental instruction through coarsened exact matching. Technology, Knowledge and Learning, 22(3), 335–352. https://doi.org/10.1007/s10758-017-9317-0.
Hirano, K., & Imbens, G.W. (2004). The propensity score with continuous treatments. In A. Gelman X.-L. Meng (Eds.) Applied bayesian modeling and causal inference from incomplete-data perspectives (pp. 78–83). New York: Wiley.
Imai, K., & Ratkovic, M. (2013). Covariate balancing propensity score. Journal of the Royal Statistical Society:Series B (Statistical Methodology), 76(1), 243–263. https://doi.org/10.1111/rssb.12027.
International center for supplemental instruction. (2014). Supplemental Instruction Supervisor Manual.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: Springer.
Kluve, J., Schneider, H., Uhlendorff, A., & Zhao, Z. (2012). Evaluating continuous training programmes by using the generalized propensity score. Journal of the Royal Statistical Society Series A, 175 (2), 587–617. Retrieved from https://ideas.repec.org/a/bla/jorssa/v175y2012i2p587-617.htmldoi:j.1467-985X.2011.01000.x.
Martin, D., & Arendale, D. (1993). Covariate balancing propensity score. Supplemental Instruction: Improving First-year Student Success in Highrisk Courses (2nd ed.)
McCaffrey, D.F., Ridgeway, G., & Morral, A.R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Statistics in Medince, 9, 403–425.
Mousavi, A., Schmidt, M., Squires, V., & Wilson, K. (2020). Assessing the effectiveness of student advice recommender agent (sara): the case of automated personalized feedback. International Journal of Artificial Intelligence in Education.
Powell, M.G., Hull, D.M., & Beaujean, A.A. (2020). Propensity score matching for education data:Worked examples. The Journal of Experimental Education, 88, 145–164. https://doi.org/10.18637/jss.v042.i08https://doi.org/10. https://doi.org/10.18637/jss.v042.i0818637/jss.v042.i08.
R Core Team. (2020). R:A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from https://www.R-project.org/.
Robins, J.M., Hernan, M.A., & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 550–560.
Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
Rosenbaum, P.R., & Rubin, D.B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 150–161.
Sales, A.C., & Pane, J.F. (2020). The effect of teachers reassigning students to new cognitive tutor sections. In Proceedings of the 13th international conference on educational data mining. International Educational Data Mining Society.
Stuart, E., King, G., Imai, K., & Ho, D. (2011). Matchit:nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42. https://doi.org/10.18637/jss.v042.i08.
Wilde, E.T., & Hollister, R. (2007). How close is close enough? evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477.
Wilke, M.C., Levine, R.A., Guarcello, M.A., & Fan, J. (2020). Estimating the optimal treatment regime for student success programs. Behaviormetrika, 48, 309–343.
Zhu, Y., Coffman, D., & Ghosh, D. (2015). A boosting algorithm for estimating generalized propensity scores with continuous treatments. Journal of Causal Inference, 25–40.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research was supported in part by NSF grant 1633130.
Appendix
Appendix
Assessing covariate balance for negative binomial regression weighting of Algorithm ??. The plot on the right shows the balance between students who did not attend any SI session and students who attend. The plot on the left shows the balance between students who went to 4 or more SI sessions and students who attend less than 4 SI sessions
Assessing covariate balance for covariate balancing GPS of Algorithm 2. For each covariate, the plots show standardized mean difference before and after weighting (black and blue dots respectively). The plot on the right shows the balance between students who did not attend any SI session and students who attend. The plot on the left shows the balance between students who went to 4 or more SI sessions and students who attend less than 4 SI sessions
Assessing covariate balance for AACC boosting of Algorithm 2. For each covariate, the plots show standardized mean difference before and after weighting (black and blue dots respectively). The plot on the right shows the balance between students who did not attend any SI session and students who attend. The plot on the left shows the balance between students who went to 4 or more SI sessions and students who attend less than 4 SI sessions
Bootstrap confidence interval on treatment effect for students who attend only SI and MSLC, but not Stat recitation, for each method (ACCC in red, CBGPS in green, NB in blue). The dots represent the average treatment effect, the bars span the range of the confidence interval; the results are based on 500 bootstrap replicates
Bootstrap confidence interval on treatment effect for students who attend only SI and Stat recitation, but not MSLC, for each method (ACCC in red, CBGPS in green, NB in blue). The dots represent the average treatment effect, the bars span the range of the confidence interval; the results are based on 500 bootstrap replicates
Bootstrap confidence interval on treatment effect for students who attend only SI, MSLC, and Stat recitation, for each method (ACCC in red, CBGPS in green, NB in blue). The dots represent the average treatment effect, the bars span the range of the confidence interval; the results are based on 500 bootstrap replicates
Rights and permissions
About this article
Cite this article
Shao, L., Levine, R.A., Guarcello, M.A. et al. Estimating a Dose-Response Relationship in Quasi-Experimental Student Success Studies. Int J Artif Intell Educ 33, 155–184 (2023). https://doi.org/10.1007/s40593-021-00280-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-021-00280-0