evidence review

w i l l t e c h n o lo gy t r a n s f o r m e d u c at i o n
for th e b e t te r?
This publication summarizes a forthcoming academic review paper on education technology,
“Upgrading Education with Technology: Insights from Experimental Research.”

ov e rv i e w a n d p o l i c y i s s u e s It is important to step back and understand how technology

can help—or in some cases hinder—student learning. In this
In recent years, there has been widespread excitement around executive summary, we synthesize the experimental literature
the transformative potential of technology in education. In on technology-based education interventions, focusing on
the United States alone, spending on education technology literature from developed countries.5 We share key results and
has exceeded $13 billion.1 Programs and policies to promote highlight areas for future inquiry.
the use of education technology (or “ed tech”)—including
hardware distribution, educational software, text message
campaigns, online courses, and more—may expand access
to quality education, support students’ learning in innovative
ways, and help families navigate complex school systems.
However, the rapid development of education technology in the
United States is occurring in a context of deep and persistent Technology for Education Consortium. “How School Districts Can Save (Billions) on
1

Edtech.” Accessed December 20, 2018. https://marketbrief.edweek.org/wp-content/

inequality.2 Depending on how programs are designed, uploads/2017/03/How_School_Districts_Can_Save_Billions_on_Edtech.pdf.
how they are used, and who can access them, education
technologies could alleviate or aggravate existing disparities. 2
Reardon, Sean, Demetra Kalogrides, and Kenneth Shores.“The Geography of Racial/
Ethnic Test Score Gaps.” CEPA Working Paper No.16-10. Stanford Center for
Education Policy Analysis, Stanford, CA, 2018.
While access to computers and internet is expanding,
approximately five million school-age children still do not have
3
Pew Research Center. “Digital divide persists even as lower-income Americans make
gains in tech adoption.” Accessed December 20, 2018. http://www.pewresearch.
a broadband internet connection at home,3 putting them at org/fact-tank/2017/03/22/digital-divide-persists-even-as-lower-income-americans-
a disadvantage for homework assignments, access to online make-gains-in-tech-adoption/.
resources, and digital literacy development. Low-income 4
Bulman, George and Robert Fairlie. “Technology and Education.” Handbook of the
students and students of color in particular disproportionately Economics of Education 5 (2015): 239-280.
lack access to technology.4 5
This policy brief also references studies from developing countries when relevant.

of Education.
1
“How School Districts Can Save (Billions) on Ed Tech.” 2017. Technology 5
This policy brief also references studies from developing countries when relevant.
for Education Consortium. https://marketbrief.edweek.org/wp-content/
uploads/2017/03/How_School_Districts_Can_Save_Billions_on_Edtech.pdf
p ove r t y a c t i o n l a b.o r g
2
Reardon et al., 2018.

3
“Digital divide persists even as lower-income Americans make gains in tech adoption.”
k e y l e s so n s

Initiatives that expand access to computers and internet

alone generally do not improve kindergarten to 12th
grade students’ grades and test scores, but do increase
computer usage and improve computer proficiency.

Educational software designed to help students

develop particular skills at their own rate of progress
have shown enormous promise in improving learning
outcomes, particularly in math. There is some evidence
to suggest that these programs can boost scores by the
same amount as effective tutoring programs, yet more
research is needed to fully understand the underlying
mechanisms for why certain educational software
programs are more effective than others.

Technology-based nudges that encourage specific,

one-time actions—such as text message reminders
to complete college course registrations—can
have meaningful, if modest, impacts on a variety of
education-related outcomes, often at low costs.

Technology-enabled social psychology interventions—

such as growth mindset interventions—can have
significant and meaningful effects relative to their low
costs, but these effects tend to be small and effective
only for specific groups of students.

Combining online and in-person instruction can work

as well as traditional in-person only classes, which
suggests blended learning may be a cost-effective
approach for delivering instruction. Students in online-
only courses, however, tend to perform worse than
students in in-person-only courses.

Many novel applications of technology to education,

such as the use of interactive whiteboards or virtual
reality, attract wide interest from school administrators
but have not yet been rigorously evaluated for their
efficacy. More research is needed to help identify which
products boost student learning and reduce, rather than
widen, existing inequalities in education.

cover photo: shutterstock .com

photo: shutterstock .com

2 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
m e t h o d o lo gy

We share evidence from 126 randomized evaluations and

regression discontinuity designs, grouped together as
experimental evidence in this publication. We included
papers if they were high-quality evaluations conducted in a
developed country and tested interventions that utilized some
form of technology to improve learning-related outcomes.
Randomized evaluations from developing countries are not
formally included in this review, although they are mentioned
when relevant to the broader discussion of how technology
impacts learning.

r i gorous m e thodolog i es to es ti m ate

c aus a l i m pac t

Randomized evaluations—when properly implemented—

are generally considered the strongest research design
for quantitatively estimating average causal effects. Our
review also chose to include regression discontinuity photo: shutterstock .com

studies with large samples and well-defined thresholds

because they produce estimated program effects identical
to randomized evaluations for participants at a particular
cutoff.6 table 1. standard deviations

effect size interpretation7

m e a s u r i n g i m pac t 0.10 standard deviations 50th percentile

to 54th percentile
Comparing results across different studies can be difficult,
especially when studies conducted in different contexts 0.20 standard deviations 50th percentile
measure different outcomes—or even use different tests to to 58th percentile
look at the same outcome. While these differences can never
be completely eliminated, we can contextualize results using a 0.30 standard deviations 50th percentile
to 62nd percentile
roughly comparable unit called a standard deviation. Standard
deviations can give us a sense of the general size of impact
0.40 standard deviations 50th percentile
across contexts (see table 1).
to 66th percentile

Regression discontinuity designs (RDDs) are quasi-experiments that identify a

6

well-defined cutoff threshold. The cutoff threshold defines a change in eligibility or

program status for those above it—for instance, the minimum test score required for
a student to be eligible for financial aid. It may be plausible to think that treatment
status is ‘as good as randomly assigned’ among the subsample of observations that fall
just above and just below the threshold. The jump in an outcome between those just
above and those just below the threshold can be interpreted as the causal effect of 7
This chart says that an intervention with effect size of 0.10 standard deviations moves
the intervention in question for those near the threshold. Berk et al. 2010; Cook and a student who scored at the 50th percentile up to the 54th percentile, for example.
Wong 2008; Shadish et al. 2011. This interpretation assumes a normal distribution.

p ove r t y a c t i o n l a b.o r g 3
r e s u lt s Laptop distribution also increased computer skills. Computer
skills rose more meaningfully for minorities, women, lower-
I. Supplying computers and internet alone generally do not income, and younger students.14 More research is needed to
improve students’ academic outcomes, but do increase determine whether these results would successfully replicate
computer usage and improve computer proficiency. to other contexts.

Disparities in access to information and communication Broadly, programs to expand access to technology have
technologies can exacerbate existing educational inequalities. been effective at increasing use of computers and improving
Students without access at school or at home may struggle computer skills.15 Though perhaps intuitive, this is noteworthy
to complete web-based assignments and may have a hard given the logistical challenges of technology distribution,
time developing digital literacy skills. Ever since technology’s the potential reluctance of students and educators to adopt
incorporation in the classroom took off during the 1990s, technology into daily practice, and the increasing importance
governments and other stakeholders have invested heavily in of digital literacy skills.
technology distribution and subsidy initiatives to expand access.8
At the same time, increasing access to technology may have Evidence base: 13 experimental papers
adverse impacts on academic achievement, for example if
students end up using technology only for recreational purposes.
II. Educational software (or “computer-assisted learning”)
When it comes to academic achievement, computer programs designed to help students develop particular
distribution and internet subsidy programs generally did skills have shown enormous promise in improving
not improve grades and test scores at the K-12 level. In the learning outcomes, particularly in math.
United States, the Netherlands, and Romania, distributing
free computers to primary and secondary students did not Targeting instruction to meet students’ learning levels has
improve—and sometimes harmed—test scores.9 In studies been found to be effective in improving student learning, but
that found negative results, researchers find suggestive large class sizes with a wide range of learning levels can make
evidence that family rules regarding computer use and it hard for teachers to personalize instruction.16 Software has
homework appear to mitigate some of the negative effects.10 the potential to overcome traditional classroom constraints by
customizing activities for each student. Educational software–
Experimental studies conducted in developing countries have, or “computer-assisted learning”–programs range from light-
for the most part, come up with similar results.11 However, touch homework support tools to more intensive interventions
one program in China that combined computer distribution that re-orient the classroom around the use of software. Most
with educational software boosted test scores, suggesting educational software that have been evaluated experimentally
distributing hardware while sharing specific learning tools help students practice particular skills through “personalized
may be a promising approach.12 tutoring” approaches.17

At the postsecondary level, computer distribution programs Computer-assisted learning programs have shown enormous
appear to be more promising, although evidence comes mainly promise in improving academic achievement, especially in
from one randomized evaluation at a community college. math. Of all thirty studies of computer-assisted learning
Distributing laptops to low-income students at a northern programs, twenty reported statistically significant positive
California community college had modest but positive effects effects.18 Fifteen of the twenty programs found to be effective
on passing rates, graduation rates, and likelihood of taking a
transfer course for a four-year college, at least in part because
it saved time previously spent accessing computer labs.13
14
Ibid.

15
Fairlie and Robinson 2013.

8
White House Office of the Press Secretary. “President Obama Announces 16
Banerjee et al. 2007; Banerjee et al. 2016.
ConnectALL Initiative.” Accessed December 21, 2018. https://obamawhitehouse.
archives.gov/the-press-office/2016/03/09/fact-sheet-president-obama-announces- 17
Kulik and Fletcher 2015.
connectall-initiative.
18
Barrow et al. 2009; Beal et al. 2013; Campuzano et al. 2009; Deault et al. 2009;
9
Fairlie and Robinson 2013; Leuven et al. 2007; Malamud and Pop-Eleches 2011. Hegedus et al. 2015; Kelly et al. 2013; Mitchell and Fox 2001; Morgan and Ritter
2002; Pane et al. 2014; Ragosta 1982; Ritter et al. 2007; Roschelle et al. 2010;
10
Malamud and Pop-Eleches 2011. Roschelle et al. 2016; Schenke et al. 2014; Singh et al. 2011; Snipes et al. 2015; Tatar
et al. 2008; Wang and Woodworth 2011; Wijekumar et al. 2012; and Wijekumar et al.
11
Beuermann et al. 2015; Cristia et al. 2012; Piper et al. 2016. 2014 report positive effects in at least one treatment arm. Borman et al. 2009; Cabalo
et al. 2007; Cavalluzzo et al. 2012; Dynarski et al. 2007; Faber and Visccher 2018;
12
Mo et al. 2015. Pane et al. 2010; Rouse and Krueger 2004; Rutherford et al. 2014; and Van Klaveren
et al. 2017 do not report positive effects. Pane 2014 only finds positive impacts on
13
Fairlie and London 2012. math outcomes in the second year.

4 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
photo: shutterstock .com

were focused on improving math outcomes.19 A study of a When it comes to computer-assisted reading programs, the
math program that enabled students to control the motions evidence was limited and showed mixed results. A program
of animated characters by building or editing mathematical that taught students a technique for breaking down texts
functions showed the largest effect sizes of any large-scale boosted middle school reading comprehension scores by 0.2
study included in the review—0.63 and 0.56 standard deviation to 0.53 standard deviations,21 demonstrating that computer-
improvements in math scores for seventh and eighth graders, assisted learning has the potential to support students in
respectively.20 While other studies of computer-assisted math literacy development as well as in math.
programs demonstrated more modest effects, they continued
to show promise. A number of these programs adapted
instruction to meet student needs by leveraging artificial com pute r - a s s i s te d le a r n i n g
intelligence and machine learning. Other effective programs
provided timely feedback to students and shared data on An evaluation of a supplementary math homework
student performance with teachers to inform their approach. program in Maine boosted average scores by 0.18
standard deviations despite requiring less than thirty to
forty minutes per week.22 This program gives students
feedback and guidance as they work through math
problems and sends student data to teachers to help them
meet students’ needs. This program had a positive effect
on student achievement, with a significantly larger effect
size for students at or below the median.

Note that this program required access to a laptop or

a tablet—programs that expand access to technology
(described in section I) may sometimes be necessary to
generate the positive effects associated with computer-
assisted learning (described in section II).

19
Barrow et al. 2009; Beal et al. 2013; Hegedus et al. 2015; Kelly et al. 2013; Morgan Evidence base: 30 experimental papers
and Ritter 2002; Pane et al. 2014; Ragosta 1982; Ritter et al. 2007; Roschelle et
al. 2010; Roschelle et al. 2016; Schenke et al. 2014; Singh et al. 2011; Snipes et al.
2015; Tatar et al. 2008; Wang and Woodworth 2011. Pane 2014 only finds positive
impacts on math outcomes in the second year. Campuzano et al. 2009 did not focus
exclusively on math outcomes and is therefore not included in this count. Wijekumar et al. 2012; Wijekumar et al. 2014.
21

20
Roschelle et al. 2010. 22
Roschelle et al. 2016.

p ove r t y a c t i o n l a b.o r g 5
figure 1. computer- assisted learning: impact on student learning in math

Reasoning Mind adaptive math program 0.00

(Wang and Woodworth 2011)

DreamBox adaptive math program 0.14

(Wang and Woodworth 2011)

Adaptive CAL program compared against a static one 0.00

across multiple subjects (Van Klaveren et al. 2017)

ASSISTments online math homework support 0.40

(Singh et al. 2011)
digital tutoring progr ams

ASSISTments online math homework support

(Roschelle et al. 2016) 0.18

Cognitive Tutor math 0.36

(Ritter et al. 2007)

Cognitive Tutor Algebra I 0.00

(Pane et al. 2014)
0.20

Cognitive Tutor Geometry -0.19

(Pane et al. 2010)

Cognitive Tutor Algebra I 0.29

(Morgan and Ritter 2002)

ASSISTments online math homework support 0.56

(Kelly et al. 2013)

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6

estimated impacts on math performance ( as reported in standardized effect sizes)

Year 1 Cohort Note: This graph only includes studies that looked exclusively at math software. Studies that looked at both math
(where applicable) and reading programs, including Campuzano et al. 2009 and Dynarski et al. 2007, are not included for this reason.
These two Department of Education studies evaluated roughly a dozen computer-assisted learning programs and over
Year 2 Cohort two years. The studies found a general pattern of null effects. However multiple programs are aggregated together in
(where applicable) some of the analyses, and the multi-program design generally makes it difficult to interpret these results in the context
of the other studies discussed here.

6 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
 figure 1. computer- assisted learning: impact on student learning in math (continued)

Cognitive Tutor’s Bridge to Algebra program 0.00

digital tutoring progr ams

(Cabalo et al. 2007)

AnimalWatch web-based math tutoring program 0.30

(Beal et al. 2013)

I Can Learn © aka “Interactive Computer Aided Natural 0.17

Learning” program for pre-algebra (Barrow et al. 2009)

School of One middle school math program 0.00

whole - school

(Rockoff 2015)
integr ation

Kentucky Virtual Schools hybrid program for Algebra 1 0.00

(Cavalluzzo et al. 2012)

Spacial-Temporal (ST) Math 0.14

(Schenke et al. 2014)

Spacial-Temporal (ST) Math 0.00

(Rutherford et al. 2014)
simulation progr ams

SimCalc interactive math software for 8th grade 0.56

(Roschelle et al. 2010)

SimCalc interactive math software for 7th grade 0.63

(Roschelle et al. 2010)

SimCalc interactive math software 0.35

(Hegedus et al. 2015)*

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6

estimated impacts on math performance ( as reported in standardized effect sizes)

Year 1 Cohort
(where applicable)

Year 2 Cohort
(where applicable) * Standardized effect size backed out using post-test mean and standard deviation.

p ove r t y a c t i o n l a b.o r g 7
photo: shutterstock .com

8 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
III. Technology-based nudges—such as text message Technology can make this communication easier, faster,
reminders—can have meaningful, if modest, impacts and more systematic. Programs to facilitate school-parent
on a variety of education-related outcomes, often at communication—including sending grades and attendance
extremely low costs. information and sharing personalized feedback—have shown
promising results. Eight of ten studies focused on improving
Technology can be used to help address systematic biases in school-family information flows demonstrated positive
decision-making and other psychological factors that lead to effects on student GPAs, test scores, assignment scores,
unintended outcomes, like high school graduates not enrolling and/or attendance.26
in college as a result of missing financial aid deadlines. Low-
cost interventions like text message reminders can successfully
support students and families at each stage of schooling. m es s ag i n g m at te r s i n sc hool- fa m i ly
com mu n i c ati on
Early Childhood and Elementary: Programs to Increase Literacy and
Learning at Home (7 experimental papers) Keep your school community in mind when selecting and
designing programs.
Young children do better in school if their parents have
encouraged and participated in learning activities at home.23 • Identify barriers to student engagement to assess
However, parents—especially low-income parents dealing with whether this approach makes sense in your context.
high stress, limited resources, and time constraints at home— • Choose communication methods that parents can
do not always regularly dedicate time to these activities. access easily, and select opt-out rather than opt-in
programs where possible.
Text messages with reminders, tips, goal-setting tools, and • Use language and translation options in schools
encouragement can increase parental engagement in learning with parents who are English Language Learners.
activities, such as reading with their children. For example, a
preschool program in San Francisco that texted suggestions Personalized feedback and specific action items can
to parents of small, easy tasks, provided encouragement, and increase student engagement.
sent reminders increased parental engagement and boosted Think carefully about the tone and messaging to
children’s literacy scores (with effect sizes ranging from 0.21 parents as family-school communication can affect
to 0.34 standard deviations).24 While a similar standardized student-teacher relationships.
program in San Francisco kindergartens showed no impact, texts
to parents with specific recommendations matched to each
kindergartener’s reading level showed substantial benefits.25

Middle and High School: Programs to Facilitate School-Parent

Communication (13 experimental papers)
In middle and high schools in the U.S., the role of parents
typically shifts away from direct activities with children and
toward encouraging teenagers’ effort in school. Schools can
help parents support their children by providing families with
information about their children’s performance.

23
Levine, Susan C., Linda Suriyakham, Meredith Rowe, Jenellen Huttenlocher, &
Elizabeth Gunderson. 2010. “What Counts in the Development of Young Children’s
Number Knowledge?” Developmental Psychology 46: 1309-1319; Price 2010; Sénéchal
and LeFevre 2002.
26
Bergman 2015; Bergman 2016; Bergman and Chan 2017; Bergman et al. 2018;
24
York and Loeb 2018. Bergman and Rogers 2016; Kraft and Dougherty 2013; Kraft and Rogers 2015; and
Rogers and Feller 2016 found positive effects. Balu et al. 2016 and Bergman and Hill
25
Doss et al. 2018. 2018 did not find positive effects.

p ove r t y a c t i o n l a b.o r g 9
Transitioning to College: Programs to Support the College Application Social Psychology Interventions: Programs to Develop
Process, Financial Aid, and Enrollment (19 experimental papers) Resilience, Confidence, and Positive Learning Attitudes
As students near the end of high school, they have the (15 experimental papers)
opportunity to pursue further education. However, the Students’ educational performance can be heavily affected by
college application process can be complex and overwhelming. emotions, beliefs, and attitudes. Technology-enabled social
Technology-based programs to personalize support and share psychology interventions aim to alleviate psychological barriers
reminders on specific tasks may help smooth this process. and cultivate confidence and positive learning attitudes. A
common social psychology intervention, for example, is to
While interventions that provided generic information on reinforce the idea that intelligence is not fixed and rather can
education tax credits or financial aid did not increase college grow through hard work.32
enrollment in the U.S.,27 programs that provided timely,
specific, and personalized information were more consistently Despite promising evidence from small-scale studies, large-scale
effective. In particular, programs that leveraged technology studies have found that technology-enabled social psychology
to suggest specific action items, streamline financial aid interventions do not improve academic outcomes on average,
procedures, and/or provide personalized support boosted although they can lead to meaningful effects under some
college application and enrollment rates28 and encouraged circumstances.33 These effects tend to be concentrated within
better-informed financial aid decisions.29 For example, subsamples and, even then, tend to be quite small.34 In some
personalized text messages increased college matriculation by cases where social psychology interventions did not improve
3.3 percentage points among students who had been accepted academic outcomes, they did have a positive impact on
to and planned to attend Georgia State University.30 This psychological outcomes, for example, the likelihood of
program sent reminders based on specific incomplete required taking academic risks.35 Findings from studies so far have
tasks and leveraged artificial intelligence to automate responses generated hints that certain students may benefit more from
to common student questions. Programs like this one can social psychology interventions. For instance, those who start
reduce the proportion of students who register for college but out further behind in terms of academic performance and/or
then do not show up. Programs that combined technology with social-psychological attitudes tend to respond better to social
in-person supports also improved financial aid receipt, college psychology interventions. However, the current evidence is far
matriculation, and college persistence.31 from sufficient to state this conclusively.

Evidence base: 54 experimental papers

27
Bergman et al. 2016; Darolia 2016; Hyman 2018; Page et al. 2016. Note that the
evidence from outside the United States shows that information interventions can
lead to positive effects on related outcomes, including views of higher education
and knowledge of financial aid. See Oreopoulos and Dunn 2013 and Dinkelman and
Martinez 2014.
27
Bergman et al. 2016; Darolia 2016; Hyman 2018; Page et al. 2016. Note that the evidence
from outside the United States shows that information interventions can lead to
28
Castleman and Page 2015; Castleman and Page 2016. Oreopoulos and Petronijevic
positive effects on related outcomes, including views of higher education and knowledge 2017 found text-based advising was not effective for first year college students in Canada.
of financial aid. See Oreopoulos and Dunn 2013 and Dinkelman and Martinez 2014. 29
Barr et al. 2016; Bird et al. 2017; Castleman and Page 2015; Castleman and Page 2016.
28
Castleman and Page 2015; Castleman and Page 2016. Oreopoulos and Petronijevic 2017 30
Page and Gehlbach 2017.
found text-based advising was not effective for first year college students in Canada.
Bettinger et al. 2012; Castleman et al. 2012; Castleman and Meyer 2016;
31
29
Barr et al. 2016; Bird et al. 2017; Castleman and Page 2015; Castleman and
Oreopoulos and Ford 2016.
Page 201620162016A.
32
Snipes et al. 2012.
30
Page and Gehlbach, 2017.
33
Pauneksu et al. 2015; Yeager et al. 2016.
Bettinger et al. 2012; Castleman et al. 2012; Castleman and Meyer 2016;
31

Oreopoulos and Ford 2016. 34

Ibid.
photo: shutterstock .com
35
Unkovic et al. 2016; Forsyth et al. 2007.

10 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
photo: shutterstock .com

IV. Online courses are developing a growing presence only, found that student performance was lower in online
in education, but the limited experimental evidence courses. It is possible that students taking online courses
suggests that online courses lower student academic may struggle with the lack of accountability or miss out
achievement compared to in-person courses. However, on motivating relationships with instructors and peers.
students perform similarly in courses with both in-person Nonetheless, students generally performed similarly—and
and online components compared to traditional face-to- in some cases better—in courses that included both a face-
face classes. In Massive Open Online Courses (MOOCs), to-face component and an online component and in courses that
behavioral interventions (like the mindset interventions were entirely face-to-face.38
described in section III) increased course persistence and
completion rates. One study did find that offering 8th grade students the option
to enroll in an online algebra course in schools where a standalone
Since their emergence in the 1990s, online courses have algebra class was not offered improved algebra achievement
developed a growing presence in education. Proponents of and also increased the likelihood of participation in an advanced
conventional online courses and massive open online courses math course sequence in high school.39 However, it is possible
(MOOCs) highlight their potential to reduce costs and improve that students would have learned even more had they taken an
access. Post-secondary students who enroll in conventional in-person algebra course rather than an online course.
online programs tend to be more likely to face educational
disadvantages compared to students in traditional programs.36 One study assessed whether online programs expand access
to students who would not otherwise enroll in a degree
Conventional Online Courses (17 experimental studies) programs, finding that Georgia Tech’s online master’s program
in computer science did expand access, especially among mid-
Conventional online courses—taught as part of entirely online career prospective students.40
degree programs or degree programs that include online or
partially online courses—have grown in popularity in the
last decade. However, in four of six studies37 that directly
compared the impact of taking a course online versus in-person
38
Alpert et al. 2016; Bowen et al. 2014; Esperanza et al. 2016; Foldnes 2016;
Harrington et al. 2015; Joyce et al. 2015. Wozny et al. 2018. Positive effects from
blended learning were found only in three of the four studies that specifically tested
the flipped classroom model, which reverses traditional instruction by delivering
content that is typically taught in the classroom at home via the internet (Esperanza
et al. 2016; Foldnes 2016; Wozny et al. 2018.)
Deming et al. 2015.
36

Heppen et al. 2011.

39

37
Alpert et al. 2016; Figlio 2013; Heppen et al. 2012; Keefe 2003; Poirier and
Feldman 2004; Zhang 2005. 40
Goodman et al. 2016.

p ove r t y a c t i o n l a b.o r g 11
photo: shutterstock .com

Massive Open Online Courses (MOOCs) (11 experimental studies) Experimental research on MOOCs has focused primarily
Offering open access and unlimited participation, MOOCs on whether and how behavioral interventions can improve
have the potential to reach many more students in a more MOOC completion rates and extend coverage to students
diverse range of contexts than conventional online courses. with limited educational opportunities. Interventions to
Millions of students are enrolled in MOOCs worldwide.41 increase completion rates through mindset interventions
MOOCs have the potential to provide access to high-quality (like those discussed in section III) have typically increased
coursework to students with fewer educational opportunities, persistence. Seven of the nine studies evaluating these types of
but enrollment and success rates are highly skewed toward interventions found positive effects from at least one treatment
populations with more financial resources.42 Broadly speaking, arm.44 For example, information on performance relative to
MOOCs face very low completion rates.43 peers,45 commitment devices to limit distractions,46 planning
prompts,47 and writing exercises aimed at increasing a sense of
belonging48 boosted completion rates.

Evidence base: 28 experimental papers

44
Banerjee and Duflo 2016; Davis et al. 2017; Kizilcec et al. 2014; Kizilcec et al.
2017; Lamb et al. 2015; Martinez 2014A; Martinez 2014B; Patterson 2015;
Yeomans and Reich 2017. Banerjee and Duflo 2016 and Kizilcec et al. 2014
do not find positive effects.

45
Davis et al. 2017; Martinez 2014A.
41
Shah 2018. Accessed January 11, 2019. https://www.edsurge.com/news/2018-01-22-
a-product-at-every-price-a-review-of-mooc-stats-and-trends-in-2017. 46
Patterson 2015.

42
Hansen and Reich 2015. Yeomans and Reich 2017.
47

43
Banerjee and Duflo 2014. 48
Kizilcec et al. 2017.

12 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Access to Technology Carter et al. (2016) Prohibiting use of computers during a college economics class

Access to Technology Faber et al. (2015) Differences in broadband connection speeds

Access to Technology Fairlie (2012A) One-to-one laptop distribution

Access to Technology Fairlie (2012B) One-to-one laptop distribution

Access to Technology Fairlie (2015) One-to-one laptop distribution

Access to Technology Fairlie and Bahr (2018) One-to-one laptop distribution

Access to Technology Fairlie and Grunberg (2014) One-to-one laptop distribution

Access to Technology Fairlie and Kalil (2017) One-to-one laptop distribution

Access to Technology Fairlie and London (2012) One-to-one laptop distribution

Access to Technology Fairlie and Robinson (2013) One-to-one laptop distribution

Access to Technology Goolsbee and Guryan (2006) E-Rate, subsidy for internet in schools

Access to Technology Leuven et al. (2007) Subsidies for computers and software in under-resourced schools

Access to Technology Malamud and Pop-Eleches Euro 200 program, subsidy for low-income families with
(2011) schoolchildren to buy computers

Computer-Assisted Learning Barrow et al. (2009) I Can Learn© aka “Interactive Computer Aided Natural Learning”
program for pre-algebra and algebra

Computer-Assisted Learning Beal et al. (2013) AnimalWatch web-based math tutoring program

Computer-Assisted Learning Borman et al. (2009) Fast ForWord computer-based language and reading training
program

Computer-Assisted Learning Cabalo et al. (2007) Cognitive Tutor's Bridge to Algebra program

Computer-Assisted Learning Campuzano et al. (2009) 16 types of software products for math and reading

Computer-Assisted Learning Cavalluzzo et al. (2012) Kentucky Virtual Schools hybrid program for Algebra 1

Computer-Assisted Learning Dynarksi et al. (2007) 16 types of software products for math and reading

Computer-Assisted Learning Deault et al. (2009) ABRACADABRA web-based literacy program

Computer-Assisted Learning Faber and Visccher (2018) Snappet digital formative assessment tool focused on spelling

Computer-Assisted Learning Hegedus et al. (2015) SimCalc interactive math software

p ove r t y a c t i o n l a b.o r g 13
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Computer-Assisted Learning Kelly et al. (2013) ASSISTments online math homework support

Computer-Assisted Learning Mitchell and Fox (2001) DaisyQuest and Daisy's Castle reading game

Computer-Assisted Learning Morgan and Ritter (2002) Cognitive Tutor Algebra I

Computer-Assisted Learning Pane et al. (2010) Cognitive Tutor Geometry

Computer-Assisted Learning Pane et al. (2014) Cognitive Tutor Algebra I

Computer-Assisted Learning Ragosta (1982) Cognitive Tutor for math

Computer-Assisted Learning Ritter et al. (2007) Cognitive Tutor for math

Computer-Assisted Learning Rockoff (2015) School of One middle school math program

Computer-Assisted Learning Roschelle et al. (2010) SimCalc interactive math software

Computer-Assisted Learning Roschelle et al. (2016) ASSISTments online math homework support

Computer-Assisted Learning Rouse and Krueger (2004) Fast ForWord computer-based language and reading training
program

Computer-Assisted Learning Rutherford et al. (2014) Spatial-Temporal (ST) Math

Computer-Assisted Learning Schenke et al. (2014) Spatial-Temporal (ST) Math

Computer-Assisted Learning Singh et al. (2011) ASSISTments online math homework support

Computer-Assisted Learning Snipes et al. (2015) Elevate summer math program

Computer-Assisted Learning Tatar et al. (2008) SimCalc interactive math software

Computer-Assisted Learning Van Klaveren et al. (2017) Adaptive CAL program compared against a static program
across multiple subjects

Computer-Assisted Learning Wang and Woodworth (2011) (1) DreamBox math program; (2) Reasoning Mind math program

Computer-Assisted Learning Wijekumar et al. (2012) ITSS (Intelligent Tutoring for Structure Strategy) program for
reading and language

Computer-Assisted Learning Wijekumar et al. (2014) ITSS (Intelligent Tutoring for Structure Strategy) program for
reading and language

Behavioral Interventions Cortes et al. (2018) Text messaging program to nudge parents of kindergarteners to
(Early Childhood) engage in literacy activities with children

Behavioral Interventions Doss et al. (2018) Text messaging program to nudge parents of kindergarteners to
(Early Childhood) engage in literacy activities with children

Behavioral Interventions Kraft and Monti-Nussbaum Parents texted to encourage engagement in activities to
(Early Childhood) (2017) counteract summer learning loss

14 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Behavioral Interventions Kraft and Rogers (2015) Parents texted on student behavior/performance
(Early Childhood)

Behavioral Interventions Mayer et al. (2015) Texting program to promote learning engagement of
(Early Childhood) Head Start parents

Behavioral Interventions Meuwissen et al. Text2Learn, a mobile phone texting program for low-income
(Early Childhood) parents of preschoolers

Behavioral Interventions York and Loeb (2018) Text messaging program to nudge preschool parents to engage
(Early Childhood) in literacy activities with children

Behavioral Interventions Balu et al. (2016) Automated text messages to parents of high school students
(Primary/Secondary) informing about absence

Behavioral Interventions Bergman (2015) Automated texts to parents about performance

(Primary/Secondary)

Behavioral Interventions Bergman (2016) Learning Management System (parents have access to an online
(Primary/Secondary) portal with child's classes, grades, assignments, etc.)

Behavioral Interventions Bergman and Chan (2017) Automated texts to parents about performance
(Primary/Secondary)

Behavioral Interventions Bergman et al. (2018) Providing regular information to families about their child’s
(Primary/Secondary) academic progress in one arm and supplementing with home
visits on skills-based information in a separate arm

Behavioral Interventions Bergman and Hill (2018) Publishing teacher ratings online
(Primary/Secondary)

Behavioral Interventions Bergman and Rogers (2016) Text message to parents regarding their child’s academic
(Primary/Secondary) performance, including grades, upcoming tests and
missing assignments

Behavioral Interventions Bursztyn and Jensen (2015) Two interventions: (1) performance leaderboard into computer-
(Primary/Secondary) based high school courses; (2) Complimentary access to an
online SAT preparatory course. Sign-up forms differed randomly
across students only in whether they said the decision would be
kept private from classmates

Behavioral Interventions Fryer (2016) Provided free cellular phones and daily information about the link
(Primary/Secondary) between human capital and future outcomes via text message in
one treatment and minutes to talk and text as an incentive in a
second treatment

Behavioral Interventions Kraft and Dougherty (2013) Parents texted about student behavior/performance
(Primary/Secondary)

Behavioral Interventions Kraft and Rogers (2015) Parents texted about student behavior/performance
(Primary/Secondary)

p ove r t y a c t i o n l a b.o r g 15
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Behavioral Interventions McGuigan et al. (2012) Information campaign about the costs and benefits of pursuing
(Primary/Secondary) post compulsory education

Behavioral Interventions Rogers and Feller (2016) One of three personalized message information treatments
(Primary/Secondary) throughout the school year

Behavioral Interventions Barr et al. (2016) Text messaging campaign prompting loan applicants at
(Post-secondary) a large community college to make informed and active
borrowing decisions

Behavioral Interventions Bergman et al. (2016) E-mails and letters to potential/prospective/current college
(Post-secondary) students about financial aid/incentives

Behavioral Interventions Bettinger et al. (2012) FAFSA assistance during tax filing
(Post-secondary)

Behavioral Interventions Bird et al. (2017) Nudges for early FAFSA filing through Common App
(Post-secondary)

Behavioral Interventions Castleman et al. (2012) Providing college counseling to low income students during the
(Post-secondary) summer through email, text message, and in-person consultation

Behavioral Interventions Castleman and Meyer (2016) A text messaging campaign to provide lower-income college
(Post-secondary) students with simplified information, encouragement, and access
to one-on-one advising

Behavioral Interventions Castleman and Page (2015) Text messages to reduce summer melt
(Post-secondary)

Behavioral Interventions Castleman and Page (2016A) Text messages to improve FAFSA re-filing for sophomore year
(Post-secondary)

Behavioral Interventions Castleman and Page (2016B) Text messages to improve enrollment tasks
(Post-secondary)

Behavioral Interventions Chande et al. (2015) Texting motivational messages and organizational
(Post-secondary) reminders to students

Behavioral Interventions Darolia (2016) Letters e-mailed to students regarding financial aid
(Post-secondary)

Behavioral Interventions Hyman (2018) Mailing letters with web address to college information website
(Post-secondary)

Behavioral Interventions Ksoll et al. (2014) Mobile phone-based adult education program (Cell-Ed)
(Post-secondary)

Behavioral Interventions O’Connell and Lang (2018) Personalized email reminders encouraging out-of-class study
(Post-secondary)

Behavioral Interventions Oreopoulos and Dunn (2013) 3-minute video and opportunity to use financial aid calculator
(Post-secondary)

16 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Behavioral Interventions Oreopoulos and Ford (2016) Application assistance with technology incorporated into the high
(Post-secondary) school curriculum

Behavioral Interventions Oreopoulos and Petronijevic Text-based advising

(Post-secondary) (2017)

Behavioral Interventions Page et al. (2016) FAFSA texting program

(Post-secondary)

Behavioral Interventions Page and Gehlbach (2017) Text message reminders and assistance with matriculation
(Post-secondary) requirements during the summer before freshman year for
students who were accepted and plan to attend college

Behavioral Interventions Smith et al. (2018) Software that sends a “grade nudge,” a personalized message to
(Post-secondary) each homework assignment regarding the student's current grade

Behavioral Interventions Forsyth et al. (2007) Self-esteem bolstering intervention

(Social Psychology)

Behavioral Interventions Good et al. (2003) E-mail mentorship by college students who encouraged middle
(Social Psychology) school students to view intelligence as malleable or to attribute
academic difficulties in the seventh grade to the novelty of the
educational setting

Behavioral Interventions Harackiewicz et al. (2012) Three-part intervention (two brochures mailed to parents and a
(Social Psychology) website) highlighting the usefulness of STEM courses

Behavioral Interventions Morisano et al. (2010) Goal-setting program

(Social Psychology)

Behavioral Interventions Paunesku et al. (2015) Growth-mindset and sense-of-purpose interventions

(Social Psychology)

Behavioral Interventions Oreopoulos et al. (2018) Choose-Your-Own-Challenge online modules designed to teach
(Social Psychology) students effective learning behaviors and adaptive perspectives

Behavioral Interventions Oreopoulos et al. (2018) Online planning exercise with information and guidance to
(Social Psychology) create a weekly schedule containing sufficient study time and
other obligations

Behavioral Interventions Unkovic et al. (2016) Personalized emails encouraging graduate students to apply
(Social Psychology) for a conference

Behavioral Interventions Walton et al. (2015) Social-belonging intervention to protect students’ sense
(Social Psychology) of self-belonging

Affirmation-training intervention to help students manage

stress related to social marginalization

Behavioral Interventions Yeager et al. (2013) 6-session intervention that taught an incremental theory
(Social Psychology) (a belief in the potential for personal change) through
Cyberball electronic game

p ove r t y a c t i o n l a b.o r g 17
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Behavioral Interventions Yeager et al. (2014) A malleable (incremental) theory of personality—the belief that
(Social Psychology) people can change

Behavioral Interventions Yeager et al. (2014) Promoting a prosocial, self-transcendent purpose

(Social Psychology)

Behavioral Interventions Yeager et al. (2016A) Growth mindset interventions during the transition to high
(Social Psychology) school: Qualitative inquiry and rapid, iterative, randomized
“A/B” experiments were conducted to inform intervention
revisions for this population

Behavioral Interventions Yeager et al. (2016B) “Lay theory” intervention that explains the meaning of
(Social Psychology) commonplace difficulties before college matriculation

Behavioral Interventions Yeager et al. (2017) A program teaching a growth mindset of intelligence
(Social Psychology)

Online Learning Alpert et al. (2016) Face-to-face versus blended versus purely online course content

Online Learning Bowen et al. (2014) Blended instruction versus face-to-face only

Online Learning Deming et al. (2016) Resume audit of fictitious resumes varied by for-profit vs. public,
online vs. brick-and-mortar

Online Learning Esperanza et al. (2016) Flipped classroom model

Online Learning Figlio (2013) Online lectures

Online Learning Foldnes et al. (2016 ) Flipped classroom model

Online Learning Goodman et al. (2016) Online Master of Science in Computer Science

Online Learning Harrington et al. (2015) Flipped classroom model

Online Learning Joyce et al. (2015) One class/week (blended) versus two classes/week (face-to-face)

Online Learning Heppen et al. (2011) Online Algebra I course

Online Learning Heppen et al. (2012) Online algebra courses for credit recovery

Online Learning Keefe (2003) Two studies: (1) lecture and interaction online versus traditional
face-to-face; (2) interaction versus regular lecture experience

Online Learning Jackson and Makarin (2018) Teacher access to online off-the-shelf quality lessons and support
to promote their use

Online Learning Poirier and Feldman (2004) Traditional face-to-face versus online course

Online Learning Wozny et al. (2018 Flipped classroom model

Online Learning Zhang (2005) The interactive e-classroom component of the Learning By Asking
system versus traditional face-to-face classrooms

18 A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b
a p p e n d i x : e va luat i o n s i n c lu d e d i n t h i s r e v i e w

intervention t ype rese archers progr a m details

Online Learning Zhang et al. (2006) Interactive video, non-interactive video and without video
learning environments

Massive Open Online Courses Banerjee and Duflo (2014) MOOC sign-up deadline

Massive Open Online Courses Banerjee and Duflo (2016) (1) Option to commit to structured study time; (2) Self-efficacy
messages; (3) Tutoring services in groups of 20

Massive Open Online Courses Davis et al. (2017) A personalized feedback system that facilitates social comparison
of current students with previously successful learners

Massive Open Online Courses Davis et al. (2018) MOOC-based Adaptive Retrieval Practice System, which delivers
quiz questions from prior course units

Massive Open Online Courses Kizilcec et al. (2014) “Collectivist,” “individualist,” or “neutral” emails sent to MOOC
participants to encourage forum participation

Massive Open Online Courses Kizilcec et al. (2017) Mindset interventions addressing social identity threat using a
“value relevance affirmation” exercise and a "social-belonging
intervention”

Massive Open Online Courses Lamb et al. (2015) Self-assessment questions aimed at improving forum
participation for MOOC students: (1) a self-participation check;
(2) discussion priming; and (3) discussion preview emails

Massive Open Online Courses Martinez (2014A) Emails informing students of their relative position in the course:
(1) a “positive” one telling how many students recipients did
better than; and (2) a "negative" one stating how other students
outperformed the recipient

Massive Open Online Courses Martinez (2014B) E-mails on the negative correlation between procrastination
and achievement

Massive Open Online Courses Patterson (2015) (1) A commitment device where students pre-commit to time limits
on distracting Internet activities; (2) a reminder tool by time spent
on distracting websites; (3) a focusing tool that allows students to
block distracting sites on the course website

Massive Open Online Courses Yeomans and Reich (2017) Open-ended planning prompts asking students to describe any
specific plans they made to engage with course content and
complete assignments on time

p ove r t y a c t i o n l a b.o r g 19
 • In what ways does education technology reduce—or
co n c lus io n s
widen—disparities in education?
Amidst the excitement and sizeable investment in education • What are the impacts of education technology on different
technology, we aim to step back and take stock of what we types of learners?
currently know from the experimental evidence:
• What types of learning activities can be effectively delivered
Simply providing students with access to computer through education technology?
technology yielded largely mixed results. At the K-12
level, giving a child a computer may have limited impacts • Which components of effective education technology
on learning outcomes, but generally improves computer programs are most important for student learning?
proficiency and other cognitive outcomes. Distributing
computers may have a more direct impact on learning • What are the long-term impacts of education technology
outcomes at the postsecondary level. on student achievement?
Computer-assisted learning shows considerable promise. • What are the replicability and scalability of programs that
Potentially due to its ability to personalize instruction, have been found to be effective?
computer-assisted learning can be quite effective in helping
students learn, particularly with math. More research is
• How should teachers and classrooms interact with
needed to understand which components of computer- education technology?
assisted learning most contribute to effective programs, • What is the cost-effectiveness of technology-driven programs
how best to offer them, and which types of learning
activities are best suited for software-based instruction.
relative to other effective approaches in education?

Evaluations of technology-enabled behavioral Technology is developing at an astonishing pace—rapid

interventions also generally find positive effects across advances in artificial intelligence and machine learning have
all stages of schooling, although the impacts are generally
small. Yet given their low cost, behavioral interventions
already reshaped many aspects of daily life. Against this
like large-scale text message campaigns may be a cost- backdrop, promising uses of education technology have the
effective way to support students, families, and schools. potential to support massive inroads in learning. Yet, far more
research is necessary to help determine which of these myriad
While technology-enabled social psychology education technologies are worth pursuing.
interventions can have significant effects, impacts are
generally small and specific to certain groups of students.

Though online learning courses have exploded in a b o u t j - pa l

popularity over the last decade, we found that relative
to courses with some degree of face-to-face teaching, The Abdul Latif Jameel Poverty Action Lab (J-PAL) is a global
students taking online-only courses may experience
research center working to reduce poverty by ensuring that
negative learning outcomes.
policy is informed by scientific evidence. Anchored by a
Going forward, we encourage additional research to further network of 171 affiliated professors at universities around
explore the potential role of education technology in schools, the world, J-PAL conducts randomized impact evaluations to
identify interventions that expand opportunity, and evaluate answer critical questions in the fight against poverty.
how underlying mechanisms can advance learning.

for further re ading

a r e a s f o r f u t u r e r e s e a rc h This evidence review is an executive summary of work by Maya

Escueta, Andre Nickow, Phil Oreopoulos, and Vincent Quan:
These results highlight technology’s potential to improve
learning, especially when used to overcome existing “Upgrading Education with Technology: Insights from
constraints in instruction and learning. Though more research Experimental Research” (forthcoming)
is needed before recommending broad-scale adoption,
computer-assisted learning and technology-enabled behavioral “Education Technology: An Evidence-Based Review”
interventions emerge as two particularly promising areas.
Moving forward, a key goal will be to understand how these
technologies can bridge gaps in educational access and reduce,
rather than widen, disparities in learning. Building off what Evidence Review Author: Sophie Shank | Editor: Vincent Quan
we now know, researchers and education practitioners have a Designer: Elizabeth Bond
major opportunity to study critical open questions about the Suggested Citation: J-PAL Evidence Review. 2019. “Will Technology
impact of technology in education: Transform Education for the Better?” Cambridge, MA: Abdul Latif Jameel
Poverty Action Lab.