Azu Etd 14024 Sip1 M PDF
Azu Etd 14024 Sip1 M PDF
Azu Etd 14024 Sip1 M PDF
by
__________________________
Copyright © Deborah Schneider 2015
DOCTOR OF PHILOSOPHY
2015
2
THE UNIVERSITY OF ARIZONA
GRADUATE COLLEGE
As members of the Dissertation Committee, we certify that we have read the dissertation
prepared by Deborah Schneider, titled The Effects of the Use of an ICT-Based Reading
Final approval and acceptance of this dissertation is contingent upon the candidate’s
I hereby certify that I have read this dissertation prepared under my direction and
3
STATEMENT BY THE AUTHOR
This dissertation has been submitted in partial fulfillment of the requirements for
Brief quotations from this dissertation are allowable without special permission,
4
ACKNOWLEDGEMENTS AND DEDICATION
I wish to acknowledge the kind and constant efforts of my singularly brilliant committee
chair, Dr. Nancy Mather, without whose support, encouragement, and mentorship (plus
no small amount of door opening) none of this would have been possible.
I also extend my sincere gratitude to the other members of my committee. Prof. Dr.
Shirin Antia's earned authority, warmth, wisdom, and intelligence are unmatched and
provide an ideal model to her students. Prof. Dr. Carl Liaupsin, who leads by quiet
example, taught me that simplest is often best: My manuscripts will not be judge by their
My most sincere thanks go to Prof. Dr. Debora Levine and Dr. Lesli Doan, who shared
with me their wealth of statistical knowledge and helped me to build a bit of my own.
talented fellow students, and Alex Chambers, Retina Bauschatz, and Merdyth Bauer.
I wish also to recognize my husband, Dr. Tobias Schneider, who regularly juggled two
toddlers and ten time zones to ensure that I had the time and ability to complete this
endeavor. Merci mille fois, mon amour. Je t'aime plus que la vie elle-meme.
Finally, I wish to dedicate this dissertation to my mother, Anne Richardson, whose quick
mind and unslakable thirst for knowledge have served always as my inspiration. Had she
been granted a small fraction of the opportunity I have been so fortunate to enjoy, I know
there would now be a veritable alphabet of letters trailing her name. Thanks, Mom.
5
TABLE OF CONTENTS
PAGE
LIST OF TABLES.............................................................................................................12
LIST OF FIGURES...........................................................................................................14
ABSTRACT.......................................................................................................................15
Conceptual Framework..............................................................................20
Research Question.................................................................................................22
Statement of Hypotheses............................................................................22
Participants.................................................................................................23
Method.......................................................................................................24
Limitations.................................................................................................24
Definitions of Terms..............................................................................................26
Multimodal Instruction..................................................................31
Formative Feedback.......................................................................32
Interactivity....................................................................................33
Mastery Learning...........................................................................34
6
Research Related to ICT-Based Reading Interventions.............................35
Purpose.......................................................................................................51
Prior Research............................................................................................51
Rationale....................................................................................................56
Research Questions....................................................................................57
Method...................................................................................................................58
Inclusion Criteria.......................................................................................58
Selection Process.......................................................................................62
Instrument..................................................................................................63
Rating Procedure........................................................................................64
Results....................................................................................................................65
Description of Participants.............................................................67
Comparison Condition(s)...................................................69
Outcome Measures.........................................................................71
Data Analysis.................................................................................72
7
Studies that Did Not Meet Quality Standards............................................75
Discussion..............................................................................................................80
Summary of Findings.................................................................................80
Limitations.................................................................................................83
Participants.....................................................................................85
Participants Demographics............................................................85
Assignment to Groups....................................................................86
Intervention Settings......................................................................87
Materials....................................................................................................87
MVRC Overview...............................................................87
Measures....................................................................................................90
WJ IV ACH........................................................................91
TOSWF-2...........................................................................91
Fidelity Measures...........................................................................93
8
Product Usage Logs...........................................................93
Behavioral Observations....................................................94
Intervention Procedure...............................................................................94
Intervention Instruction..................................................................97
Comparison Instruction..................................................................97
Teacher Training............................................................................97
Assessment Procedure...............................................................................98
Independent Variable...................................................................100
Dependent Variables....................................................................100
Demographic Variables...............................................................100
Data Analysis...........................................................................................101
Rationale......................................................................................101
Test Procedure.............................................................................103
Confirmatory Measures...............................................................104
Measurement of Data...................................................................105
9
Missing Data................................................................................105
Data Screening.............................................................................106
Identification of Covariates..........................................................112
Data Analysis...........................................................................................112
Test Procedure.............................................................................112
Multivariate Tests........................................................................114
Univariate Tests...........................................................................115
Fidelity of Implementation......................................................................122
Behavioral Observations..............................................................122
Purpose................................................................................................130
Research Question...............................................................................131
Method.................................................................................................................131
Participants...................................................................................131
Participants Demographics..........................................................132
Assignment to Groups..................................................................133
10
Intervention Settings....................................................................133
Materials..................................................................................................134
Measures..................................................................................................134
Fidelity Measures.........................................................................136
Behavioral Observations..................................................136
Intervention Procedure.............................................................................137
Intervention Instruction................................................................137
Comparison Instruction................................................................138
Teacher Training..........................................................................138
Assessment Procedure.............................................................................139
Data Analysis...........................................................................................140
Measurement of Data...................................................................140
Missing Data................................................................................140
Identification of Covariates..........................................................141
Test Procedure.........................................................................................141
11
Tests of Equality of Variance and Covariance............................142
Multivariate Tests........................................................................142
Univariate Tests...........................................................................143
Fidelity of Implementation......................................................................147
Discussion............................................................................................................148
Limitations...............................................................................................150
RESEARCH (AMEER).......................................................................................153
DESCRIPTIONS.................................................................................................158
REFERENCES................................................................................................................172
12
LIST OF TABLES
PAGE
Table 4. Descriptive Statistics by Condition for Each Dependent Variable and Interval-
Level Covariate....................................................................................................108
13
Table 21. Pairwise Comparisons for Reading Fluency....................................................122
Table 22. Student Engagement Behavior per the Planned Activity Check.....................123
Table 29. Student Engagement Behavior per the Planned Activity Check.....................148
14
LIST OF FIGURES
PAGE
15
ABSTRACT
(four classrooms in each school) in two public elementary schools in the southwestern
United States. Examiners obtained reading achievement data for each participating
student. Pre- and post-test measures included tests of the Woodcock-Johnson Tests of
Achievement (WJ IV ACH), as well as the Test of Silent Word Reading Fluency
determine whether there were significant mean differences in (a) non-word reading, (b)
real word reading, (c) non-word spelling, (d) real word spelling, and/or (e) reading
fluency post-test achievement scores favoring students assigned to use the MVRC online
reading intervention, once the effects of differences in pre-test achievement scores and
relevant demographic variables had been accounted for. Analyses revealed a significant
main effect (λ = .668, F [5, 161] = 16.014, p < .001, multivariate η2 = .332) of the
result which was confirmed across three of the study’s dependent variables: real word
spelling (F[1, 165] = 16.341, p < .001, multivariate η2 = .090), non-word spelling (F[1,
165] = 29.212, p < .001, multivariate η2 = .150), and reading fluency (F[1, 165] = 58.348,
16
CHAPTER ONE: INTRODUCTION
Literacy and its component skills, the ability to read with fluency and
comprehension and write fluently and coherently, are essential to educational attainment
across domains: they “[bridge] the gap between learning to read and reading to learn”
(Duke, Bennett-Armistead, & Roberts, 2003, p. 226) and provide the key that opens the
however, has not yet achieved its potential in ensuring that as many Americans as
possible enjoy the benefits of literacy. The findings of the National Assessment of Adult
Literacy revealed that 43% of adults in the United States scored at basic or below basic
levels in prose literacy, or the ability to understand, summarize, make simple inferences,
determine cause and effect, and recognize an author’s purpose when presented with texts
of moderate density (Kutner, Greenberg, Jin, Boyle, Hsu, & Dunleavy, 2007). Results of
the National Assessment of Educational Progress (Grigg, Daane, Jin, & Campbell, 2003)
et al., 2003). Research suggests that once children have reached this point in their
education, when the focus of instruction has shifted from learning to read to reading to
learn (Duke et al., 2003), they are at increased risk for academic failure (Felton & Pepper,
1995; Juel, 1988), often struggling to acquire the content knowledge necessary for
academic success.
and social success. This conclusion is well supported in the literature. Kennely and
17
scores and school dropout, and researchers have consistently found that youngsters with
O’Brien, & Langhinrichsen-Rohling, 2007), placing them at increased risk for future
The vast majority of children at risk for illiteracy can be taught to read with fluency
Schuster, Yaghoub-Zadeh, & Shanahan, 2001; Snow, Griffin, & Burns, 2005). In
phonics has been shown to positively affect the reading and writing abilities of students
with reading-related challenges (Ehri et al., 2001; Hatcher, Hulme, & Snowling, 2004;
achievement (Savage et al., 2013), and such instruction often requires little or no direct
intervention on the part of the classroom teacher (Bishop & Edwards Santoro, 2006).
“generally with an underlying expectation that student learning can improve … through
supportive skill instruction with practice” (Cassady & Smith, 2005, p. 362). This
sentiment was mirrored in the National Reading Panel’s (NPR) report of 2000, which
allowing students greater opportunity to “interact instructionally with text” than typically
18
offered by conventional instruction alone (Ch. 6, p. 8).
and computer technologies are (a) explicit, systematic instruction in the sound-symbol
correspondences of spoken and written language (Camilli et al., 2003; Ehri et al., 2001;
Torgerson et al., 2006), (b) multimodal instruction to promote recall and retention (Low
& Sweller, 2005; Moreno & Mayer, 2007), (c) formative feedback to guide learning and
activate prior knowledge (Narciss, 2013), (d) interactivity to promote attention and
engagement (Sims, 2000, 2003), and (e) opportunities for mastery learning to enhance
achievement (Guskey, 2007, 2012). The question of whether or not ICT-based reading
interventions have actually leveraged the potential advantages of the medium, however,
that ICT-based beginning reading programs generally have provided inconsistent and
unsystematic instruction (Edwards Santoro & Bishop, 2010; Grant et al., 2012).
2013) and inadequately researched (Blok, Oostdam, Otter & Overmaat, 2002; Kulik,
2003; Slavin, Lake, Chambers, Cheung, & Davis, 2009; Torgerson & Zhu, 2003),
particularly with regard to studies involving participants aged eight years and younger
research design is employed to evaluate the efficacy of the MindPlay Virtual Reading
19
Statement of the Problem
With some notable exceptions (e.g., Macaruso, Hook, & McCabe, 2006;
McMurray, 2013; Savage et al., 2013; Savage, Abrami, Hipps, & Deault, 2009; Savage,
Erten, Abrami, Hipps, Comaskey, & van Lierop, 2010), relatively little high quality
Kulik, 2003; Slavin et al., 2009; Torgerson & Zhu, 2003). Prominent voices in the field
have suggested that teachers and education authorities remain wary of adopting any ICT-
based reading program until it has a consistent base of high quality evidentiary support
(Slavin et al., 2009; Torgerson, 2007; Torgerson & Zhu, 2003).1 Through the present
study, the author wishes to fill a gap in the existing ICT-based beginning reading
intervention literature, while addressing issues of research design and intervention quality
In the present study, the author relates the results of quantitative research designed
to evaluate the efficacy of MindPlay Virtual Reading Coach (MVRC), a sequential, code-
focused online reading intervention, when used to supplement regular reading instruction
multivariate data analyses and statistical controls for differences in pre-test achievement
1
Among the recommendations of Torgerson (2007) and Torgerson and Zhu (2003) was the conduct of
randomized controlled trials (RCTs). Owing to the small sample size in the present study, it was not
20
Conceptual Framework
The conceptual framework for the present study is based on five research-supported
premises. First, failure to develop strong reading skills in early elementary school has
impacts tend to intensify as children progress through school (Felton & Pepper, 1995;
Juel, 1988). Second, the vast majority of children can be taught to read with fluency and
meet individual needs (Ehri et al., 2001; Hatcher, Hulme, & Snowling, 2004; Torgerson
et al., 2006). Third, systematic instruction in code-based skills has been shown to
positively affect the reading ability of both typically developing students and those with
reading-related challenges (Ehri et al., 2001; Snow et al., 2005). Fourth, sequential ICT-
based reading interventions of sufficient duration and intensity can and do improve
beginning reading achievement (Macaruso et al., 2006; Savage et al., 2009; Savage et al.,
2010; Savage et al., 2013), even among students with reading-related challenges
(McMurray, 2013). Fifth, ICT-based reading interventions whose content and delivery
instructional design are likely to be of greatest benefit to students (Savage et al., 2013).
The primary objective of the author of the present study was to contribute to the
Reading Coach (MVRC) (MindPlay Educational Software for Reading, 2015), an ICT-
21
provided by a classroom teacher. MindPlay Virtual Reading Coach offers highly
NRP (2000). The MVRC software provides multisensory learning, engaging students
visually and auditorily, in order to strengthen associations between learned content (Kast,
Meyer, Vögeli, Gross, & Jäncke, 2007) and reduce memory demands on individual
cognitive systems (Low & Sweller, 2005). Immediate formative feedback is provided to
students while they interact with program content, rather than simple corrective feedback,
as formative feedback has been shown to increase retention and decrease demands on
to 90%) of initial concepts and skills before new concepts and skills are introduced,
ensuring that students do not have gaps in foundational knowledge and promoting
The secondary objective of the author was to ensure the conformity of the present
study with the highest standards for design, analyses, and reporting in educational
research. Therefore, the author elected to align the present study with the indicators for
(2005) and published by the Council for Exceptional Children. The present study, as
designed, meets all of the relevant2 essential quality indicators and five of the desirable
quality indicators, thus satisfying the criteria for high quality research.
2
One essential quality indicator was not relevant to the present study, as it applied only to studies involving
populations presenting with disabilities or learning difficulties.
22
Importance of the Present Study
A critical issue of national significance is ensuring that all students have optimal
opportunities to develop the reading skills necessary to succeed in school and in life. The
present study employs high quality design and data analytic techniques, and an
intervention grounded in evidence-based theory and best practices for the promotion of
literacy development among beginning readers. It will yield important findings regarding
sequenced mastery model of instruction, and it will contribute to the knowledge base
educators and administrators with information critical to the selection of effective ICT-
Research Question
In the present study, the following research question was addressed: Are there
significant mean differences in (a) non-word reading, (b) real word reading, (c) non-word
spelling, (d) real word spelling, and/or (e) reading fluency post-test achievement scores of
students assigned to use the MVRC online reading intervention in addition to regular
condition, once the effects of differences in pre-test achievement scores and relevant
Statement of Hypotheses
The author of the present study has identified the following alternative hypothesis
(HA) related to the overall effects of the intervention: Statistically significant main effects
23
of the intervention favoring the treatment group will be detected in overall reading and
spelling achievement. The author has identified the following alternative hypotheses
Participants
two public elementary schools in the southwestern United States. Of those, 107 were
comparison condition. Owing to attrition and illness, 39 participants had incomplete data
cases, and the comparison condition comprised 81, representing 170 complete cases, or
81.34% of original cases. While overall data loss was below 20%, data loss did not
impact each condition equally, as the treatment group retained 83.12% of original cases,
while the comparison group retained 79.41% of original cases, producing a differential
24
attrition rate of 3.71%.
Method
instruction in the classroom setting. Participants assigned to the treatment group used the
software for 30 minutes each day, Monday through Thursday, for a total of two hours per
week throughout the regular school year (mid-September through mid-April), with the
exception of holidays, school functions, and mandatory state testing days. Participating
scripted program of reading and language arts instruction but did not receive the MVRC
intervention. During the time allotted for the MVRC intervention, students assigned to
from their classroom teachers consistent with the scripted curriculum and routine
participating student. Pre- and post-test measures included the Test of Silent Word
Reading Fluency, Second Edition (TOSWRF-2; Mather, Hammill, Allen & Roberts,
2014) and four tests of the Woodcock Johnson Tests of Achievement, Fourth Edition (WJ
Limitations
limitations. Chief among these is the lack of equivalence across demographic factors
between the treatment and comparison groups. While the author employed statistical
controls to correct for the differences, further research with random assignment to groups
25
or a non-post-hoc matched design should be performed. Also absent was observational
participating classrooms was modest, this information would have been helpful in
focused tasks of reading fluency, word spelling, and word reading. Each of these tasks
measured aspects of decoding or encoding, but none assessed the synthesis of these skills
decoding ability is essential to fluent reading, the ultimate goal of reading is the
Definitions of Terms
26
To facilitate reading of the present study, definitions of relevant terms follow:
between the sounds of speech and the letters of written words in an alphabetic language
Blend: a combination of two or more letters used to represent closely but distinctly
27
syllable
Full alphabetic phase: the stage of alphabet learning in which children have
Graphemes: the symbols with which the sounds of spoken language are
Interactivity: the way in which content is driven by and adaptive to user activity
28
single syllable
language
Partial alphabetic phase: the stage of alphabet learning in which children have
in spoken language
spoken language
Reading failure: failure to acquire basic reading and spelling skills by grade three
29
through increasingly difficult blocks of content
syllable
30
CHAPTER TWO: REVIEW OF LITERATURE
This chapter has two main sections: a review of literature and a critical synthesis
of experimental and quasi-experimental research. In the first section, the author presents
a brief review of literature related to the design and delivery of ICT-based beginning
delivery are discussed, as are models for the acquisition and development of phonetic and
present study, the author focuses on a specific form of instructional ICT, sequential
Santoro, 2006, p. 62). Such interventions may be used to supplement reading instruction
(Savage et al., 2009; Savage et al., 2010; Savage et al., 2013), low performers (Cassady
& Smith, 2005; Macaruso et al., 2006), and those with learning difficulties (McMurray,
2013).
31
Proposed Advantages of ICT-Based Reading Interventions
(a) multimodal instruction using purposefully linked visual and auditory content, (b)
formative feedback to guide learning and activate prior knowledge, (c) interactivity to
promote engagement and meaningful learning, and (d) opportunities for mastery learning
presentation of content in more than one sensory mode. The term is used to refer to
instruction that incorporates various sensory modalities (e.g., visual, auditory, tactile), but
for the purpose of the present study it will refer only to visual and auditory modalities,
consistent with research in cognitive psychology (Clark & Paivio, 1991; Fadel, 2008).
The multimodal (i.e., visual and auditory; Clark & Paivio, 1991) instruction provided by
2008). It is hypothesized that instruction using both visual and auditory modalities
reduces cognitive load by splitting processing tasks between visual and auditory
processing mechanisms (Clark & Paivio, 1991; Low & Sweller, 2005). When visual and
cognitive processing are reduced, and retention is improved (Moreno & Mayer, 2007).
three separate trials, the authors found that recall was significantly better for picture-
sound pairs than for single modality items, leading the authors to conclude that linking
32
Multimodal instruction can also serve to clarify content: if a learner does not
understand a concept when presented in one modality (e.g., text [visual]), an additional
representation of that concept in another modality (e.g., speech [auditory]) might improve
preferences, thus increasing satisfaction with learning and instruction (Sankey, Birch, &
reading ability, even among students with dyslexia. As an example, Kast, Meyer, Vögeli,
Gross, and Jäncke (2007) provided three months of training using a computer-based
to build up the memory strength of graphemes and phonemes” (p. 357) to Swiss children
knowledge or behavior and improve learning (Shute, 2008). The major aim of formative
feedback is to increase the knowledge and skill of the learner by directing and facilitating
immediate feedback that goes beyond simple correction to provide strategically useful
33
load by activating existing knowledge and directing learners to effective strategies, thus
can improve task motivation (Narciss, 2004), retention (Moreno, 2004), and achievement
(Moreno, 2004; Narciss, 2004). Formative feedback is especially beneficial for novice or
struggling learners, who are more likely to become cognitively overwhelmed during
learning (Shute, 2008). As an example, Mioduser, Tur-Kaspa and Leitner (2001) found
that Israeli preschool students deemed at risk for learning disabilities who received an
a loosely defined term (Rose, 1999) that can be used to characterize both conventional
and computer-based models of instruction. For the purpose of the present study,
however, interactivity will refer simply “to those functions and/or operations made
available to the learner to enable them to work with content material presented in a
minimize boredom because the learner must stay alert and on-task in order to interact
34
needs and strengths (Sims, 2003). Interactive instruction can be particularly beneficial to
novice and struggling learners (Sims, 2003). As an example, Segers and Verhoeven
segmentation, and oral blending. While the intervention was provided, on average, only
15 minutes per week during the intervention period, treatment group students enjoyed
immediately following the intervention and upon follow-up testing in the first grade.
intervention offered to grade one students (N = 166) found statistically significant gains
in reading achievement among students who completed a total of only five hours of
which students are required to demonstrate mastery of initial instructional content to a set
criterion before more challenging content is presented (Guskey, 2012). Mastery learning
achievement and attitudes toward learning (Guskey, 2007), it can be difficult to fully
35
however, are able to deliver highly individualized learning that adapts to students needs
The benefits of mastery learning appear to be greatest for struggling learners; however, a
across grades and across content (Kulik, Kulik, & Bangert-Drowns, 1990). An example
of the beneficial effects of mastery learning is provided by Vaughn, Serido, and Wilhelm
(2006).3 Vaughn et al. (2006) reported the results of a randomized controlled trial (N =
hours of instruction of ICT-based reading instruction each week, while students assigned
to the comparison group received supplementary literacy activities consistent with routine
classroom instruction. The authors employed a pre-post design and performed analyses
both on post-test scores and gain scores. While statistically significant differences in
vocabulary, comprehension, and overall reading ability favoring students assigned to the
treatment condition were detected, gains were greatest among English language learners
(ELLs) and students designated struggling readers. Measures of effect size were not
provided; however, the percentage of treatment group students reading below grade level
fell by nearly half from pre-test to post test (from 64.7% to 33.1%).
3
This study (Vaughn et al., 2006) was not included in the formal synthesis that follows this review of
literature, as it was not subjected to peer review.
36
unresolved (Edwards Santoro & Bishop, 2010; Grant et al., 2012), and ICT-based reading
instruction remains poorly theorized and inadequately researched (Savage et al., 2013),
particularly concerning studies involving participants aged eight years and younger
(Lankshear & Knobel, 2003). This dearth of quality research is exemplified by the
analysis of over forty studies designed to evaluate the efficacy of ICT-based reading
interventions in promoting early literacy achievement, Blok et al. (2002) found only three
studies (Barker and Torgersen [1995] and two sub-studies contained in Foster, Erickson,
language reading intervention. While the authors (Barker & Torgesen, 1995; Foster et
favoring participants in the treatment group, the software used (iterations of DaisyQuest)
is now obsolete. All of the other studies included in the meta-analysis performed by Blok
et al. (2002) were either conducted in a language other than English or involved
presentation of text, speech feedback, or virtual flashcards. Though Blok et al. (2002)
reported a modest corrected overall effect size estimate of +0.19 across studies, they
cautioned that this result should be interpreted with care, as many of the studies evaluated
were poorly designed, and several compared the effects of multiple software applications,
The caution urged by Blok et al. (2002) appears well founded in the light of the
results of other syntheses, whose authors applied more stringent inclusion criteria (e.g.,
requiring experimental designs or matched quasi-experimental designs with both pre- and
37
post-test measures). In a 2003 synthesis of the randomized controlled trial (RCT)
literature in the field, Torgerson and Zhu identified a dozen studies meeting inclusion
criteria. Of those, only one (Mitchell & Fox, 2001) included a sequential, code-based
ICT-based reading intervention delivered to participants in the early primary grades, and
the authors of that study reported no statistically significant effect of the intervention.4
Science Foundation and performed by Kulik in 2003. While the author (Kulik, 2003)
only three of those showed statistically significant positive effects favoring treatment
group participants, and all three were published in the early 1990s and evaluated the
In a more recent synthesis, Slavin, Lake, Chambers, Cheung, and Davis (2009)
criteria; however, only eight were peer-reviewed studies including a now extant
grade three. Of those eight, four were embedded in a larger study commissioned by the
Pendeleton, 2009). Of the four interventions evaluated by Campuzano et al. (2009), none
achievement. Unlike Campuzano et al. (2009), the authors of three of the four remaining
studies reviewed by Slavin et al. (2009) reported statistically significant positive effects
4
While Mitchell and Fox (2001) found no significant differences favoring the ICT-based reading
intervention group when compared to comparable teacher-delivered intervention group, they did find
significant differences favoring the ICT-based reading intervention group when compared to a no-treatment
control.
38
of their respective interventions, with effect sizes ranging from +0.17 to +1.05. While
were not found favoring the treatment group in measures of word reading (Paterson,
Jacobs Henry, O'Quin, Ceprano, & Blue, 2003), lacked a pre-test measure for that
construct, putting its findings concerning reading achievement into question. Notably,
both of the still extant software packages for which the authors provided statistically
significant evidentiary support, Waterford Early Learning Program (Cassady & Smith,
2005; Tracey & Young, 2005) and Lexia Learning Systems software (Macaruso, Hook,
& McCabe, (2006), involved a strong emphasis on systematic phonics instruction and the
studies might be variables associated with the design and execution of the research. In
their 2002 synthesis, Blok et al. noted that design-related variables accounted for the
majority of variance across the studies they evaluated and observed that many of the
the highest quality (e.g., Waterford Early Learning Program), findings have been
Campuzano et al., 2009) contradicting the findings of smaller studies (e.g., Cassady &
While RCT designs have been hailed as the gold standard for educational research
(Slavin et al., 2011; Torgerson & Zhu, 2003; Torgerson, 2007), an RCT design is not a
39
guarantee of methodological strength. As an example, the large-scale RCT performed by
Campuzano et al. (2009),5 which included a tepid evaluation of the Waterford Early
experimental control over product use, inconsistent evaluation methods and instruments,
unavailability of fidelity data, and an absence of measures closely aligned to the study’s
interventions. While the largely favorable research on the same product conducted by
Cassady and Smith (2005) and Tracey and Young (2007) was by no means perfect,
superior.
high quality design also exist in the ICT intervention literature. In their generally
positive evaluations of Lexia Reading software, Macaruso et al. (2006) and McMurray
interventions. Similarly, both Macaruso et al. (2006) and McMurray (2013) provided and
duration of program usage and students’ achievement. The findings of Macaruso et al.
(2006) and McMurray (2013) were mirrored by those of several carefully designed and
online reading program explicitly grounded in developmental models and theory derived
5
This study (Campuzano et al., 2009) contained data from the second cohort of a larger study performed by
Dynarski et al. (2009).
6
In the case of Cassady and Smith (2005), the authors failed to take into account possible cumulative
effects of a teacher professional development intervention being offered simultaneously, and in the case of
Tracey and Young (2007), the authors found modest statistically significant between groups at pre-test,
which they failed to account for in their analyses.
40
from contemporary reading research (Savage et al., 2009; Savage et al., 2010; Savage et
al., 2013).
the issue of content, the findings of the National Reading Panel (NRP; 2000) identified
five key elements of literacy instruction for which there was substantial evidentiary
understanding of the sound system of spoken language; (2) phonics, knowledge of the
correspondence between speech sounds and the orthographic (spelling) patterns in written
language; (3) fluency, the speed and accuracy of reading; (4) vocabulary, knowledge of
the body of words used in a particular language; and (5) comprehension, the ability to
understand the message and meaning of written text. Of these five elements, two are
and how speech sounds (phonemes) are denoted in written text form the foundation for
both decoding (reading) and encoding (spelling) in alphabetic languages (Ehri, 1998;
National Reading Panel, 2000). Both phonological awareness, the umbrella skill under
which phonemic awareness is found (Badian, 2001; Pufpaff, 2009), and phonics (Adams,
7
While phonemic awareness and phonological awareness are sometimes used interchangeably,
phonological awareness denotes the understanding of the sound system of spoken language more
generally; phonemic awareness refers to the ability to detect and manipulate the smallest perceptually
distinct units of sound (e.g., the ability to discriminate between /p/ and /b/ in the words pat and bat) in a
spoken language (Pufpaff, 2009).
41
1990; Camilli et al., 2003; Ehri et al., 2001; Torgerson et al., 2006) comprise a complex
overstate. The National Early Literacy Panel (NELP; 2008), in a comprehensive review
of high quality research in early literacy, found that phonological awareness and
speech sounds [phonemes] and written letters [graphemes]) were the two best predictors
designed to evaluate the efficacy of systematic phonics interventions, Ehri et al. (2001)
concluded that explicit instruction in “[s]ystematic phonics helped children learn better
than all forms of control instruction” (p. 393), and that it was effective not only in
providing beginning reading instruction, but also in preventing and remediating reading
difficulties. These findings were largely mirrored in a replication (of the study performed
by Ehri et al. [2001]) conducted by Camilli et al. (2003) two years later. While Camilli et
al. (2003) found that even non-systematic phonics instruction benefited learners, the
authors noted that “the advantage of systematic phonics instruction over some phonics
literature, Torgerson et al. found that “[s]ystematic phonics instruction within a broad
(p. 8), which extended both to typically developing learners and those at risk for reading
difficulties.
42
For code-based skills to be taught effectively, however, they must be introduced in
phonological and alphabetic skills has been well defined in the literature, yet there
remains a “need for progression within ICT activities that reflects these qualitative
changes in development” (Savage et al., 2013, p. 312). A brief overview of two research-
based models for the development of phonological and alphabetic skills follows.
sequence for the acquisition of phonological skills derived from an extensive review of
research in early literacy. These skills progress from the detection and production of
words (e.g., <cow‧boy>), and then the syllabification of multisyllabic words (e.g.,
<mi‧tten>). After these phonological skills have been acquired, development progresses
ability to isolate and manipulate individual speech sounds. The developmental sequence
of phonemic awareness skills progresses from blending of sounds (e.g., /k/ /æ/ /t/ to cat),
to sound-to-word matching (e.g., Does cat end with the /t/ sound?), to word-to-word
matching (e.g., Does cat begin with the same sound as cap?), to phoneme isolation (e.g.,
What is the beginning sound in cat?), to phoneme segmentation (e.g., Say cat one sound
at a time.), to phoneme deletion (e.g., Say cat without the /t/ sound.), to phoneme
substitution (e.g., Say cat with the /p/ sound instead of the /t/ sound.), and finally to
phoneme reversal (e.g., Say cat backwards, so the beginning sound is at the end and the
ending sound is at the beginning [i.e., tack]). It is important to note, however, that there
43
is individual variation in the acquisition and development of phonological skills, and the
research suggests that children do not develop skills in a rigid sequence, but in often
overlapping stages (Invernizzi & Tortorelli, 2013); therefore, individual skills need not be
taught in a lockstep fashion, but in a responsive manner that is adaptive to the needs of
introduction of the symbol system of the language, a modified Latin alphabet, in the case
children use insights derived from their knowledge of the alphabet to identify the sounds
in spoken and written words (Invernizzi & Tortorelli, 2013). In her seminal work on the
development of alphabetic skills based on her own research and that of others in the field.
In the first of the four phases, the pre-alphabetic phase, learners have not yet begun to
form cognitive connections between letters and the sounds that represent them. By the
second phase, characterized as partial alphabetic, however, children have begun to form
example, may be used to infer letter sounds (e.g., T begins with the /t/ sound, or the initial
sound in its letter name). Invernizzi and Tortorelli (2013) noted that in this phase, letters
whose initial sounds correspond to the phonemes they most typically produce (e.g., B,
/b/; D, /d/; K, /k/) should be taught first, followed by letters whose terminal sounds
correspond to the phonemes they most typically produce (e.g., F, /f/; M, /m/; N, /n/), and
finally, letters whose names do not provide consistent phonetic cues (e.g., C, W).
44
taught before letters whose graphical representations are easily confused, and letter
sounds that occur most frequently should be taught before those that occur least
develop from partial alphabetic to full alphabetic knowledge, or the ability to recognize
and segment all common sounds within a one-syllable word (Ehri, 2005). This phase is
vowels and digraphs (i.e., a pair of characters used to write a single phoneme [e.g., ch]).
Full alphabetic knowledge can then be built upon to develop facility with more complex,
chunked units of written text, including blends (two consonant phonemes articulated in
close succession within the same syllable [e.g., st]), onsets (the initial phonological unit
in a syllable) and rimes (the terminal phonological unit of a syllable, including the vowel
and any consonants that follow it), morphological units (grammatically meaningful units
of language [e.g., -ing]), and other common orthographic patterns, until a learner has
reduces demands on memory (i.e., phonological units are chunked, rather than
individually decoded, and many whole words are represented in memory) and forms the
basis for fluent and accurate reading, as well as improved reading comprehension and
to individual needs (Invernizzi & Tortorelli, 2013). The presentation and delivery of
45
content are therefore matters of particular concern when instruction is provided by
experts in the field, Grant et al. (2012) created a developmentally sequenced taxonomy of
reading skills with which to evaluate ICT-based preschool and early primary reading
software. The taxonomy was unusual in its comprehensiveness, covering skills and sub-
skills related to (a) concepts about print, (b) alphabetic knowledge, (c) phonological
awareness, (d) phoneme-grapheme relationships, (e) phonics, (f) syntactic awareness, (g)
decoding, (h) fluency, and (i) comprehension. Using their novel taxonomy, Grant et al.
reading skills. Of those, only five included instruction in synthetic phonics, and none
included instruction in phonemic segmentation. The authors reported that among the
consistent across the software levels or in congruence with the reading taxonomy
expectations,” and “fewer skills than expected were being taught through the software
programs” (Grant et al., 2012, p. 333). Typical weaknesses in the software included a
Edwards Santoro and Bishop (2010). In 2006, Bishop and Edwards Santoro created a
46
framework for evaluating beginning reading software that took into consideration both
content and delivery, providing research-supported criteria for evaluating three distinct
aspects of interface design and four distinct aspects of instructional design, as well as the
the model proposed by Bishop and Edwards Santoro (2006) focused on the quality of (a)
aesthetics, the way texts, graphics, animations, and sounds are designed to enhance user
experience; (b) operational support, the way the program provides direct and indirect
support to users, allowing them to use the program with minimal help from teachers or
other adults; and (c) interactions, the way in which content is driven by and adaptive to
user activity. As to instructional design, Bishop and Edwards Santoro (2006) identified8
(a) systematicity, the degree to which instruction comprises “cycles that progress
hierarchically through increasingly difficult blocks of content” (p. 62); (b) instructional
learning; (c) assessment, the degree to which user performance is tracked, evaluated, and
used to inform and adapt goals and content; and (d) motivation, the degree to which the
reviewed sample did not meet research-based criteria for interface, instructional design,
8
Category names have been altered slightly (e.g., Systematic to Systematicity) to preserve grammatical
parallelism.
9
Because the model proposed by Bishop and Edwards Santoro (2006) focused exclusively on beginning
reading interventions for at-risk readers, criteria for evaluating program content were limited, including
only (a) phonological skills, including phonemic manipulation and segmentation, and (b) alphabetic skills,
including phoneme-grapheme relationships and decoding. For a more comprehensive taxonomy of reading
skills as they apply to ICT-based reading interventions, see Grant et al. (2012).
47
and beginning reading content …” (p. 114). Among their findings was a negative
correlation between quality of interface design and quality of content, as well as a lack of
programs Edwards Santoro and Bishop reviewed offered any degree of explicit
instruction in code-based skills, and none “scored above ‘neutral’ for the effective
practice activities indicator, ‘The program uses informative and instantaneous feedback
messages to support content learning’” (2010, p. 112), suggesting considerable room for
Though the results of the reviews of software performed by Grant et al. (2012) and
Edwards Santoro and Bishop (2010) paint an unflattering picture of ICT-based beginning
reading interventions generally, neither Grant et al. (2012) nor Edwards Santoro and
Bishop (2010) evaluated all of the interventions commonly used in the public schools.
Examples of widely used programs that were not included in either review are
HeadSprout, iReady, MVRC, Plato Focus, and Waterford Early Learning Program. It
should also be noted that both Grant et al. (2012) and Edwards Santoro and Bishop
(2010) found considerable variation in quality among the programs they reviewed, and
their appraisals of ICT-beginning reading software were not universally negative. Grant
et al. (2012), for example, praised several programs for providing students with
learning, multiple opportunities for practice, and identified achievement goals. Similarly,
Edwards Santoro and Bishop (2010) identified a modest number of programs providing
both high quality content and instructional design, noting that the best among them
“[supplied] adequate opportunities for learners to practice new skills, [presented] the
48
same requirements in embedded activities as presented in the instructional sequence,
[required] mastery of skills before moving onto new skills, and [moved] systematically
While reviews of software of the types performed by Grant et al. (2012) and
Edwards Santoro and Bishop (2010) serve to identify software consistent with
theoretically informed models for the content and design of beginning reading
efficacy. In the subsequent section, the author presents a critical synthesis of the
kindergarten through grade three. The purpose of this synthesis was to determine
whether sufficient evidence existed to support any information and computer technology-
49
ICT-Based Reading Interventions: A Critical Synthesis of the Literature
In the information age, strong reading skills are key to unlocking opportunities for
educational and professional advancement (Eisenberg, Lowe, & Spitzer, 2004). As the
economy moves further from a model based on the manufacture of physical goods toward
Bank, 2004), it is critical that all students have optimal opportunities to develop the
has not yet achieved its potential in ensuring that as many Americans as possible enjoy
adults sponsored by the National Center for Education Statistics, ninety-three million
adults in the United States scored at basic or below basic levels in prose literacy, or the
ability to read and comprehend continuous texts, at the time of sampling in 2003 (Kutner,
Greenberg, Jin, Boyle, Hsu, & Dunleavy, 2007). This group comprised over forty
percent of the total adult population in the United States -- four in ten Americans who
were unable to understand, summarize, make simple inferences, determine cause and
effect, or recognize an author’s purpose when presented with texts of moderate density
While the negative impacts of illiteracy are perhaps most severe for older youth and
young adults (Kennelly & Monrad, 2007; Rutherford, Bullis, Anderson, & Griller-Clark,
illiteracy are found early in childhood (McCardle, Scarborough, & Catts, 2001). Children
who struggle in the beginning stages of reading development are likely to face increasing
50
obstacles to the development of literacy and its component skills as they grow older
(Spira, Bracken, & Fischel, 2005). Those who have not mastered basic reading skills by
grade three, when the focus of instruction usually shifts from learning to read to reading
to learn (Duke, Bennett-Armistead, & Roberts, 2003), are beset by academic problems
that tend to intensify in the later grades. Because they lack the ability to read fluently and
with comprehension, such students often fail to gain conceptual knowledge from written
texts and consequently fall further behind their literate peers with each passing year
(ICT) has been widely advanced as a means by which to promote reading achievement
(Savage et al., 2013). However, little has been done to “tie computer-mediated reading
pedagogical models for technology” (Savage et al., 2013, p. 310), and the field remains
primary grades (Lankshear & Knobel, 2003). There is also a lack of evaluative research
examining the evidence base supporting the use of ICT-based beginning reading
programs. A handful of systematic reviews have been performed, but none has focused
the present synthesis, the author presents an up-to-date evaluative synthesis of the
literature in sequential ICT-based reading instruction in the early primary grades. The
primary objective of the author is to identify high quality ICT-based beginning reading
interventions that can be used at scale in typical school settings. The secondary objective
of the author is to determine which features, if any, are common to high quality ICT-
51
based programs of beginning reading instruction.
Purpose
The purpose of this synthesis is threefold: (a) to build upon previous research by
including articles that have been published in the past several years; (b) to narrow the
scope of the research by selecting for inclusion only those studies focusing on sequential,
limited utility in typical educational settings; and (c) to increase the level of scrutiny to
which the articles are subjected by evaluating them using the Anchored Matrix for
Prior Research
between 1990 and 2002, Blok, Oostdam, Otter and Overmaat (2002) identified a total of
42 studies meeting inclusion criteria. Of those, only two (Barker & Torgesen, 1995;
Foster, Erickson, Foster, Brinkman, and Torgesen, 1994) involved a sequential, code-
obsolete. While Blok et al. (2002) calculated a modest overall effect size of +0.19 across
the studies they analyzed, they cautioned that many of the studies were poorly designed,
and design-related variables, such as the time of pretesting and the language of
instruction, accounted for the majority of variance. In another meta-analysis, Soe, Koki,
and Chang (2000) found 17 studies performed between 1982 and 1999 that met their
broad inclusion criteria, but only two of those included participants in the early primary
52
grades, and none included a sequential ICT-based reading intervention offering
systematic instruction in code-based skills. Furthermore, the mean effect size across the
financed by the National Science Foundation, found a total of nine controlled studies of
reading software programs; however, only three of those showed significant effects of
their respective interventions, and the median effect of the “nine studies was to raise
students reading scores by 0.06 standard deviations, a trivial increment” (p. v). Similarly,
Torgerson and Zhu (2003) identified very few (12) ICT-based intervention studies
meeting inclusion criteria. Of those studies, only one included a condition in which
narrowly-focused fluency or comprehension activities), and its authors (Mitchell & Fox,
2001) reported that the intervention (iterations of DaisyQuest) was not superior to
struggling readers in kindergarten through fifth grade, Slavin, Lake, Davis, and Madden
however, only five of those studies had been published within the prior decade, and all
nine of the others investigated now obsolete software programs. Of the more recently
published studies, only two focused on students in the early primary grades (i.e.,
kindergarten through grade three), and the authors of only one of those (Macaruso, Hook,
& McCabe, 2006) reported positive effects of the intervention.11 In another recent
10
It is worth noting that Mitchell and Fox (2001) found significant differences favoring the ICT-based
reading intervention group when compared to a no-treatment control.
11
The study failing to report positive effects was the large-scale RCT performed by Campuzano, Dynarski,
Agodini, and Rall (2009), which is discussed in greater depth in the subsequent paragraph.
53
synthesis of the reading intervention literature, Slavin, Lake, Chambers, Cheung, and
selection criteria; however, six are now well over fifteen years old, and three were not
peer reviewed. Of the remaining ten, four were embedded in a larger study
Agodini, & Rall, 2009). The study included both a first grade and a fourth grade cohort,
subsets of each of which were subjected to one of four different reading interventions.
While the study was a large randomized controlled trial (RCT), it had important
et al. (2009) were evaluated individually until the second year of the project because “in
the first year, the study operated under the guideline that individual product effects would
not be reported” (p. 37); however, “classroom observations were not conducted [by
Campuzano et al. (2009)] in the second year” (p. 37), and implementation data were not
Unlike Campuzano et al. (2009), the authors of five of the six remaining studies
reviewed by Slavin et al. (2009) reported positive effects of their respective interventions,
with effect sizes ranging from +0.17 to +0.71. Notably, all five were published after
whole-class instruction (Chambers et al., 2006; Chambers et al., 2008), rather than
individual ICT-based reading instruction; however, three others (Cassady & Smith, 2005;
54
Macaruso et al., 2006; Tracey & Young, 2007) involved code-based primary reading
Similarly, the authors of a few studies published within the past ten years have
an example, Englert, Zhao, Collings, and Romig (2005) examined the effects of an online
participants was small (N = 31), and the study did not have a true control in either of its
phases. In another more robustly designed study, Chambers et al. (2011) found
among struggling readers in grade one, but not among students in grade two. Two other
Chambers et al. (2008a), low achieving first graders who participated in the computer-
graders who received ICT-based small group tutoring and other technological
measures of reading achievement (Chambers et al., 2008b). The intervention upon which
both studies focused, however, was as a targeted program in code-focused oral and
written language skills, allowing a single trained tutor to work with several students at
12
The study whose authors (Paterson, Jacobs Henry, O'Quin, Ceprano, & Blue, 2003) did not report
significant effects of the intervention was a mixed-methods quasi-experimental design that lacked a pre-test
measure of decoding. Therefore, any statistical inferences derived from reported post-test decoding scores
are of limited validity.
55
once. Chambers et al. explained “[r]ather than replacing the tutor, the program was
designed to increase program fidelity by assisting tutors and students in each of the three
trained tutor, and it was designed to be used within the context of a larger teacher-
delivered scripted reading curriculum, limiting its value to teachers in schools that have
The authors of several recent studies have also found statistically significant
favoring the treatment group, particularly among students with the lowest pretest scores.
In results of a 2011 follow-up study using the same intervention, Macaruso and Rodman
reported that preschoolers in both treatment and control classes showed improvements in
gains than those in the control group, particularly in measures of phonological awareness,
in which the differences achieved statistical significance. Similarly, Mitchell and Fox
(2001) found that struggling kindergarten and first-grade students who received five
13
It should be noted that the curriculum, Success for All, in which the tutoring program was embedded has
significant evidentiary support. (See Slavin & Madden [2012] for a good overview of the program and its
evidentiary base.) Therefore, this observation is not intended to diminish the program or its utility in any
way.
14
While generally well-designed, these studies were not eligible for inclusion in the present synthesis due
to a lack of measures of decoding administered pre- and post-test, or other disqualifying design-related
factors.
56
hours of training on an ICT-based reading intervention (DaisyQuest) providing
they did not outperform students whose teachers delivered comparable instruction.
Rationale
instruction have been mixed, their findings point to improvements in the quality of
reading instructional software over time. Of the syntheses of literature in which the
authors examined only studies published ten or more years ago, the authors determined
that the evidence in support of ICT-based reading instruction was unpromising (Blok et
al. 2002), negligible (Kulik et al., 2003; Soe et al., 2000), or insufficient (Torgerson &
Zhu, 2003). The authors of a more recent synthesis (Slavin et al., 2009), however, found
interventions delivered to students in the early primary grades. The results of this
intervention studies (Chambers et al., 2008a; Chambers et al., 2008b; Chambers et al.,
kindergarten students (Macaruso & Walker 2008; Macaruso & Rodman, 2011), whose
intervention literature (Slavin et al., 2011; Slavin et al., 2009) did not formally evaluate
the quality of the research that they examined. While their selection criteria excluded
57
designs without a control or comparison condition, as well as designs whose measures or
procedures were likely to be biased or lack validity, some of the studies that Slavin et al.
(2009) reviewed for their synthesis (e.g., those embedded in the larger study performed
guidelines suggested by Gersten et al. (2005) and published by the Council for
ICT-based reading interventions, but none were designed to critically evaluate the state of
the guidelines suggested by the What Works Clearinghouse [What Works Clearinghouse,
2008], CONSORT [Shultz, Altman, & Moher, 2010] or the Council for Exceptional
Children [Gersten et al., 2005]). Furthermore, the majority of these syntheses (Blok et
al., 2002, Kulik et al., 2003; Soe et al., 2000; Torgerson & Zhu, D., 2003) are now over a
decade old. Given the rapid pace of innovation in the field and the positive trajectory of
synthesis is warranted.
Research Questions
In support of this purpose, the author sought to answer the following research
questions:
58
(2) What features, if any, are common to high quality ICT-based programs of
Method
Inclusion Criteria
To be included in this synthesis, articles must have satisfied the following criteria:
units at the level of randomization, included both pre- and post-test data
for all key measures, with at least one standardized measure of decoding
randomization, included post-test data for all key measures for all
59
(3) Participant Criteria
three
or research institution
teaching)
60
iii. Designed to deliver English-language reading instruction
vi. Replicable in realistic school settings (i.e., Programs that could not be
refereed academic journals because such articles had been subjected to peer review and
outlets. While this decision surely introduced a degree of publication bias into the
analysis, the author felt that the peer review process was a necessary check on the
potential for experimenter bias inherent in a field rife with commercially commissioned
and independently published research. Owing to the rapid pace of change in instructional
technology, the author chose to examine only articles published since January 01, 2000.
This decision was bolstered by a comparison of the results of older syntheses (Blok et al.,
2002, Kulik et al., 2003; Soe et al., 2000), whose authors found few, if any, studies
61
the early elementary grades, to those of a more recent synthesis (Slavin et al., 2009),
The author chose to include only articles relating the results of experimental or
criterion falls short of the gold standard of randomized controlled trials suggested by the
assignment to groups is often difficult, if not unethical or impossible (Odom et al., 2005).
The author excluded studies that included only students in language learning settings, as
such studies’ results were likely to be impacted by the participants’ linguistic status. The
authors included only studies involving an English language reading intervention because
orthographic depth varies from language to language, and the depth of a language’s
orthography has been shown to impact the ease and speed with which children acquire
As the early mastery of basic reading skills is essential to later school success
(Spira, Bracken, & Fischel, 2005), a decision was made to examine only studies
Likewise, the author chose to evaluate only interventions that included sequential
elementary grades (National Reading Panel, 2000). The author also elected to include
through other distribution channels, and would not require a degree of personnel
62
commitment likely to create barriers to implementation in realistic school settings (i.e.,
programs that could not be used without extensive, direct instructional assistance from a
teacher or tutor).
Because the aim of the author was to identify ICT-based reading interventions
that could be used in school settings with minimal direct intervention on the part of
classroom teachers, certain of the interventions included in previous syntheses did not
qualify for analysis in the present study. Furthermore, the author also chose to exclude
et al. (2009). This research performed by Campuzano et al. (2009) had several important
use, inconsistent evaluation methods and instruments, unavailability of fidelity data, and
Selection Process
search process and reference chasing. Preliminary searches were conducted using the
AND computer; reading AND *experiment AND Internet; reading AND *experiment
AND online; reading AND *experiment AND software; reading AND intervention AND
computer; reading AND intervention AND Internet; reading AND intervention AND
online; and reading AND intervention AND software. Search results were narrowed by
grade level, design, and publication type. Following database searches, the author
63
reviewed the abstracts of the articles referenced in the reviews of literature and meta-
analyses examined for this synthesis, as well as articles referenced in studies selected via
online search. Articles whose abstracts were consistent with selection criteria were added
appropriate, review. This process yielded a total of seven articles consistent with
Instrument
To examine selected research, the author employed the quality indicators identified
by Gersten et al. (2005) and published by the Council for Exceptional Children.
Gersten’s model was chosen because it was designed with the particular constraints and
demands of education research in mind. It has been widely adopted in the field and has
been cited in numerous other syntheses (e.g., Browder, Wakeman, Spooner, Ahlgrim-
Delzell, & Algozzine, 2006; Chard, Ketterlin-Geller, Baker, Doabler, & Apichatabutra,
indicators, 11 of which they deemed to be essential to research and eight of which they
experimental study to be considered of acceptable quality, it must meet all but one of the
essential quality indicators and at least one of the desirable quality indicators. For a
but one of the essential quality indicators and at least four of the desirable quality
acceptable quality studies, or two high quality studies, that support the practice.
64
While Gersten et al. (2005) identified essential and desirable quality indicators for
experimental and quasi-experimental research, they did not provide an instrument with
anchored intervals by which to assess research for the presence or absence of quality
indicators. For the purpose of this synthesis, therefore, a rubric with anchored intervals
was created. This rubric, characterized as the Anchored Matrix for Evaluating
Research in Special Education (2005). The rubric was designed such that articles could
be assessed for the degree to which they demonstrated the presence of both essential and
desirable quality indicators. For each of the essential quality indicators, an anchored
scale was created, with values ranging from 0 to 3. A value of 0 corresponds to the
content that exceeds the specified quality indicator. Article content that meets but does
not exceed a specified quality indicator corresponds to a value of 2, and article content
that approaches but does not meet a specified quality indicator corresponds to a value of
1. A scale without an intermediate point was chosen to eliminate the possibility of a no-
Rating Procedure
The author carefully evaluated each of the articles included in this synthesis using
the rubric presented in Appendix A. After rating, the results of each study’s evaluation
were reviewed to determine whether the study could be considered of acceptable or high
quality per the guidelines established by Gersten et al. (2005). Studies achieving a score
of two or higher for any criterion were considered to have met the specified quality
65
indicator. Studies meeting at least 10 of the essential quality indicators and at least four
of the desirable quality indicators were deemed of high quality; studies meeting at least
10 of the essential quality indicators and at least one of the desirable quality indicators
were deemed of acceptable quality; and studies meeting fewer than ten of the essential
quality indicators, or studies meeting at least ten of the essential quality indicators but
none of the desirable quality indicators were deemed not to have met the quality
indicators for acceptable or high quality research. In cases in which a particular quality
indicator was inapplicable to a study (e.g., information confirming disability status for
research in which individuals with disabilities did not participate), it was excluded from
Results
Following a careful search using the selection procedures outlined in the Method
section of this article, a total of seven articles meeting selection criteria were identified.
Three of those were accounted for in the syntheses performed by Slavin et al. (2009) and
Slavin et al. (2011). The other four were either not included in the syntheses performed
by Slavin et al. (2009) and Slavin et al. (2011) or published in 2011 or thereafter.
66
Table 1
Overview of Studies
Study Design N Intervention Grade Measures Duration
Cassady & Quasi- 93 Waterford 1 Terra Nova An average of
Smith experimental Early Reading 30 hours over
(2005) Learning (CBT/ one semester
Program McGraw-
Hill, 1997)
Macaruso Quasi- 179 Lexia 1 GMRT 64 sessions of
et al. Experimental Learning unspecified
(2006) duration
McMurray Quasi- 106 Lexia 6-7 GRT II Twenty weeks
(2013) experimental Learning years of unspecified
old duration
Savage et Experimental 144 ABRA 1 GRADE; An average
al. (2009) CTOPP, of 13 hours
tests of the over 12 weeks
WJ III ACH
Savage et Quasi- 60 ABRA 1 GRADE; An average of
al. (2010) experimental CTOPP, 16 hours over
tests of the 13 weeks
WJ III ACH
Savage et Experimental 1,067 ABRA K-2 GRADE; Approximately
al. (2013) CTOPP, 20 hours
DIBELS
Tracey & Quasi- 265 Waterford K TERA-2 Two semesters
Young experimental Early (half of of sessions of
(2007) Learning participants) unspecified
Program duration
67
Evaluation for Essential Quality Indicators
identified by Gersten et al. (2005) is provided, along with a discussion of the degree to
which each of the articles satisfy those criteria. Gersten et al. (2005) organized their
outcome measures, and data analysis. Those same categories are used to organize the
following section.
that might impact generalization or replication. This is particularly important in the case
of research that involves students with learning or functional disabilities, students from
intervention providers and examiners are comparable across conditions. Such rigor is
studies. Of the seven articles reviewed for this synthesis, six (Cassady & Smith, 2005;
Macaruso et al., 2006; McMurray, 2013; Savage et al., 2013; Savage, Abrami, Hipps, &
Deault, 2009; Savage, Erten, Abrami, Hipps, Comaskey, & van Lierop, 2010) met all of
the quality indicators for the description of participants with a score of adequate or
68
ample. In no case did the authors exclusively focus on students with learning or
achievement or dyslexia. Similarly, in two of the studies (Cassady et al., 2005; Macaruso
et al., 2006), the authors focused at least some of their analyses on students performing at
the bottom of the distribution. In the case of Macaruso et al. (2006), the data from
students identified at risk based on Title I status were analyzed separately,15 and Cassady
and Smith provided separate analyses by quartile. In each of the cases where the authors
examined the data by quartile or risk status (Cassady & Smith, 2005; Macaruso et al.,
In the research performed by Cassady and Smith (2005), Macaruso et al. (2006),
Savage et al. (2009), Savage et al. (2010), and Savage et al. (2013) relevant demographic
data were reported, and comparisons were performed to ensure that no significant
students with the most significant deficits for assignment to the treatment group;
baseline variables. Modest significant differences between groups were detected only in
one measure (of two) of a single variable, picture vocabulary. To account for this
difference, the author performed regression analyses to determine the proportion of the
variance in outcomes for which it was responsible. In the research performed by Tracey
and Young (2007), however, significant pretest differences favoring the nonintervention
group were found on the Test of Early Reading Ability, Second Edition (TERA-2;
15
It should be noted that Macaruso et al. (2006) excluded data derived from students with documented
learning disabilities, owing to the students’ uneven distribution across groups.
69
Wheeler, 1999), but these differences were not addressed or controlled for in subsequent
data analyses.
Because each of the studies reviewed for this synthesis included interventions
delivered primarily by computer, with little active participation from teachers or other
modest. Therefore, the author of the present study chose to eliminate this criterion from
her analyses. Nevertheless, the authors of all but one of the studies reviewed for this
training to implement their respective interventions, and the authors of all of the studies
educational interventions and curricula, as well as the conditions and contexts in which
replicable precision, as are features of the learning environments and contexts in which
those interventions take place. Also of importance are measures designed to assess
services provided to control participants might introduce confounds into study data or
analyses. If relevant details of the intervention and control conditions are omitted from
comparison condition was an area of weakness across the majority of studies. Only one
70
of the studies (Savage et al., 2013) analyzed for this synthesis satisfied all of the quality
indicators for the implementation of the intervention and description of the comparison
condition(s) identified by Gersten et al. (2005). The authors of all of the studies (Cassady
& Smith, 2005; Macaruso et al., 2006; McMurray, 2013; Savage et al., 2009; Savage et
al., 2010, Savage et al., 2013, Tracey & Young, 2007) described their respective
interventions in sufficient detail; however, the majority of the studies’ authors (Cassady
& Smith, 2005; Macaruso et al., 2006; McMurray, 2013; Savage et al., 2009; Savage et
al., 2010, Tracey & Young, 2007) failed to adequately document the nature of services
some degree the nature of typical classroom reading instruction across conditions, but
only one (Savage et al., 2013) observed and specifically documented the instruction
The authors of all of the studies (Cassady & Smith, 2005; Macaruso et al., 2006;
McMurray, 2013; Savage et al., 2009; Savage et al., 2010, Savage et al., 2013, Tracey &
Young, 2007) provided at least some documentation of the duration and intensity of their
respective interventions. Both Macaruso et al. (2006) and McMurray (2013) implied that
the reading software was used unevenly across participants. McMurray (2013)
achievement attributable to this factor; however, she did not provide documentation of
the mean use of the intervention or the range in intervention use, only stating that it was
used for 20 weeks between pre- and post-testing. Macaruso et al. (2006), by contrast,
provided detailed usage data, including mean use of the intervention and the range in
intervention use. Furthermore, the authors (Macaruso et al., 2006) performed analyses to
71
determine the relationship between gain scores and number of skill units completed by
generalizable skills have been taught, are essential to high quality research. Measures
with established validity and reliability are preferable to those of uncertain reliability and
validity, and multiple measures are necessary to ensure that sufficient data exist to
minimum, pre- and post-test measures must be performed within a temporal proximity
sufficiently close to the intervention to demonstrate that any significant differences are
Studies varied as to the quality of outcome measures. Four of the seven articles
(Macaruso et al., 2006; Savage et al., 2009; Savage et al., 2010; Savage et al., 2013)
satisfied all the criteria for outcome measures outlined by Gersten et al. (2005). In the
case of Savage et al. (2009), Savage et al. (2010), and Savage et al. (2013), the authors
balance between broad and narrow measures of reading achievement. While Macaruso et
al. (2006) and McMurray (2013) both used only a single instrument to measure reading
achievement, each was nationally normed and well aligned to its respective intervention.
16
It is worth noting that this analysis might be misleading, as McMurray (2013) reported that the lowest
performers in her sample sometimes became stuck within a particular skill unit, despite high levels of
program usage.
72
In the case of Macaruso et al. (2006), individual test data were reported, giving measures
and Smith (2005), however, used only state standardized testing data to measure reading
analyses performed by Tracey and Young (2007) were unusual in that only half of
participants were given a nationally normed measure of reading achievement. The other
half were given a test of auditory processing and a test of reading abilities created by the
publisher of the intervention. While the standardized test (TERA-2; Wheeler, 1999)
offered to half the participants contained both broad and narrow measures, only
All of the studies (Cassady & Smith, 2005; Macaruso et al., 2006; McMurray,
2013; Savage et al., 2009; Savage et al., 2010, Savage et al., 2013, Tracey & Young,
2007) employed pre-post designs; in only one study (Savage et al., 2009), however, did
the authors perform follow-up testing to capture any enduring effects of the intervention.
While follow-up testing is desirable, it is not essential to high quality research per the
Data analysis. The quality of data analysis is key to accurately identifying any
essential that analyses be carefully linked to research question(s) and that the unit of
analysis is consistent with the unit of group assignment. Research must demonstrate that
the assumptions of statistical tests have been met or otherwise addressed. Furthermore,
researchers should report not only inferential statistics but also measures of statistical
73
power and the magnitude of effects. These data are important to determining not only the
statistical significance of any treatment effects, but also their practical significance.
The quality of data analysis varied considerably from study to study. Three of the
studies (Cassady & Smith, 2005; McMurray, 2013; Tracey & Young, 2007) failed to
satisfy at least one of the quality indicators for data analysis outlined by Gersten et al.
(2005). The authors of the other four articles (Macaruso et al., 2006; Savage et al., 2009;
Savage et al., 2010; Savage et al., 2013) performed sound analyses and provided
measures of effect size for all key findings. In the case of Savage et al. (2010) and
data using pre-test scores as covariates. Macaruso (2006) provided inferential analyses
only for composite scores,17 while Savage et al. (2010) evaluated group differences on
comparisons. Savage et al. (2009) used a similar procedure, following which adjusted
scores were used for planned comparisons across the three groups. As for Savage et al.
(2013), achievement test data were analyzed using hierarchical linear modeling (HLM)
variance on gains for a single measure with post-hoc regression analyses to determine the
proportion of variance in reading scores accounted for by various factors), effect sizes
were not reported. In the case of Cassady and Smith (2005), effect sizes were provided
for key measures; however, the authors failed to account for an important potential
confound in their research design. Rather than a traditional randomized controlled trial or
74
analysis, evaluating the relative achievement of two consecutive cohorts of first-grade
students. The first cohort of students did not receive any ICT-based reading intervention,
regular classroom instruction. At the same time, participating teachers were involved in
benefits accrued to the second cohort were a result of the ICT-based reading intervention,
performed by Tracey and Young (2007), significant differences were found between
groups on pre-test achievement scores; however, those differences were not controlled for
in the univariate analysis of variance on gains, and measures of effect size were not
reported. Results of evaluations for the presence of the quality indicators identified by
75
Table 2
Study Quality
Outcome Data Total High
Study Participants Conditions
Measures Analysis Score Quality
Cassady and 2/2 2/3 1/2 2/3 7/10 Yes
Smith (2005)
Macaruso et 2/2 2/3 2/2 3/3 9/10 Yes
al. (2006)
McMurray 2/2 1/3 1/2 2/3 7/10 No
(2013)
Savage et al. 1/1 2/3 2/2 3/3 8/9 Yes
(2009)
Savage et al. 1/1 2/3 2/2 3/3 8/9 Yes
(2010)
Savage et al. 1/1 3/3 2/2 3/3 9/9 Yes
(2013)
Tracey and 0/1 2/3 1/2 1/3 4/9 No
Young
(2007)
Three studies analyzed for this synthesis (Cassady & Smith, 2005; McMurray,
2013; Tracy & Young, 2007) did not meet the standards for high or acceptable quality
research outlined by Gersten et al. (2005). While Cassady and Smith (2005), Tracey and
Young (2007), and McMurray (2013) reported statistically significant positive effects of
performed by Cassady and Smith (2005) is notable in an important regard: Cassady and
Smith (2005), like Macaruso et al. (2006), found that the lowest performing participants
intervention studies analyzed for this synthesis (Savage et al., 2009; Savage et al., 2010;
76
Savage et al., 2013) were student gains evaluated by quartile or disability status;
however, such analyses might have yielded valuable findings concerning factors affecting
differential response.
The study performed by McMurray (2013) approaches the standards for high
quality research outlined by Gersten et al. Were a more detailed follow-up report
published by the author (McMurray, 2013), the cumulative weight of the studies
Lexia a promising practice per the guidelines established by Gersten et al. (2005).
Four of the seven studies (Macaruso et al., 2006; Savage et al., 2009; Savage et
al., 2010; Savage et al., 2013) analyzed for this synthesis met the criteria for high quality
research identified by Gersten et al. (2005). Notably, all but one (Macaruso et al., 2006)
of the high quality studies were conducted by the same research team and involved the
same ICT-based reading software. Though three of the high quality studies (Macaruso et
al., 2006; Savage et al., 2009; Savage et al., 2010) failed to satisfy one of the essential
criteria, they satisfied a large number of the desirable criteria and thus met the standards
for high quality research (Gersten et al., 2005), minor weaknesses notwithstanding. It is
worth noting that in the study that satisfied all of the quality criteria (Savage et al., 2013),
the authors explicitly stated their intention to perform high quality research and
& Moher, 2010) standards as the framework for their study design, analyses, and
reporting.
77
In one high quality study (Macaruso et al., 2006), instruction using Lexia Phonics
Based Reading and Lexia Reading SOS (Strategies for Older Students) software was
adaptive, and sequential instruction in code-based skills. The authors (Macaruso et al.,
in five schools in an urban school district. While the authors failed to find statistically
significant differences favoring the treatment group after one school year, they did detect
correspondences and measures of word reading favoring students eligible for Title I
services. In fact, at post-test, the achievement of Title I students assigned to the treatment
group was comparable to that of students not eligible for Title I services.
In three other high quality studies (Savage et al., 2009; Savage et al., 2010;
distributed by Canada’s Centre for the Study of Learning and Performance (CSLP) (n.d.
a) was provided to participants in the treatment condition. While the intervention was the
same across three of the high quality studies (Savage et al., 2009; Savage et al., 2010;
Savage et al., 2013), the designs employed by the authors varied somewhat from study to
study. In the oldest of the studies (Savage et al., 2009), the authors compared student
word analysis, text comprehension, and fluency; however, the participants in one
78
treatment condition completed phoneme-based synthetic phonics activities using
based analytic phonics activities using ABRACADABRA (Savage et al., 2009). Students
control condition in word blending, elision, word attack, and reading comprehension at
the immediate posttest (Savage et al., 2009). At the delayed posttest, students in the
analytic phonics treatment condition experienced significant gains relative to the control
condition in word blending, elision, rapid object naming, word attack, and reading
significant gains relative to the control condition in word blending, elision, and reading
were identified based on participating teachers’ self-reported stage of technology use and
instructional technology purposefully and systematically, but did little to integrate that
technology into other forms of instruction; and adaptation, in which teachers used
other classroom instruction to support the skills learned through the use of instructional
technology (Savage et al., 2010). Though the trial was short in duration and involved
only a small number of participants (N = 60), the authors nevertheless found statistically
79
significant positive effects of the intervention across measures (Savage et al., 2010).
Notably, however, these effects were present only for participants in the adoption and
underperformed those in the control condition, suggesting that teachers’ comfort and
achievement outcomes.
In the most recent study performed by Savage et al. (2013), the authors conducted
phonological blending, phoneme segmentation fluency, sight word reading, and letter-
CPSL reports weighted effect sizes of +0.396 for alphabetic knowledge, +0.187 for
reading fluency, and +0.340 for reading comprehension (Centre for the Study of Learning
and Performance, n.d. b). These are non-trivial effects, suggesting that the
80
al., 2009; Savage et al., 2010; Savage et al., 2013) satisfy the criteria for the identification
research-supported practice.
Discussion
Summary of Findings
The author of this synthesis sought to answer the following research questions: (a)
a research-supported practice? and (b) What features, if any, are common to high quality
there must be at least four acceptable quality studies, or two high quality studies, that
support the practice. The results of the author’s analyses suggest that while there is an
communication, assessment, and parent tools (Centre for the Study of Learning and
Performance, n.d. a). The instruction component offers analytic and synthetic phonics,
81
word-level practice, letter-sound recognition, word segmenting, modeled reading and
expression, sentence writing, and story sequencing, among other activities. Teachers are
charged with selecting developmentally appropriate starting points for their students;
however, instruction builds sequentially based on student performance (Centre for the
Study of Learning and Performance, n.d. a). The professional development component of
the ABRACADABRA software includes lesson plans, video clips, and animated
into their own classroom instruction (Centre for the Study of Learning and Performance,
n.d. a). Teachers can also share their questions, anecdotes, remarks and lesson plans
using the communication component of the ABRACADABRA software (Centre for the
provides teachers with activity statistics and error reports to facilitate planning and
remediation (Centre for the Study of Learning and Performance, n.d. a). Finally, the
activities, and information to help parents engage their children in reading (Centre for the
While the results of this synthesis point to a promising practice in the area of ICT-
based reading interventions, the base of high quality research in the field nevertheless
research, the author found very few studies meeting selection criteria. Clearly, therefore,
82
research, should be performed to evaluate the ICT-based reading interventions most
commonly used in American and Commonwealth public schools. This research should
be geared not only toward product evaluation, but also to the identification of optimal
instructional practices. Particular attention should be paid to the duration and intensity of
instructional integration and treatment fidelity among teachers. Finally, research should
be performed in accordance with an established framework for high quality research and
reporting (e.g., the guidelines suggested by the What Works Clearinghouse [What Works
Clearinghouse, 2008], CONSORT [Shultz et al., 2010] or the Council for Exceptional
The results of this synthesis provide only modest implications for practice. At
charge to teachers in Canada (Centre for the Study of Learning and Performance, n.d. a);
however, the author of this synthesis was able to find no comparably-researched program
instruction, and at-home learning and reinforcement. Also notable is the fact that the
83
appropriate instruction in phonics and other code-based skills, for which there is strong
Limitations
This synthesis has several limitations. The search procedures employed by the
author were by no means exhaustive, and articles meeting selection criteria may have
been overlooked. Furthermore, the selection criteria used by the author were narrow in
scope, and only articles published in refereed journals were considered for inclusion.
excluded from analysis. Moreover, the research evaluated for this synthesis was rated by
It should also be noted that for the purpose of this synthesis, qualitative designs and
mixed-methods or quasi-experimental designs that did not include both pre- and post-test
measures of decoding, as well as controls for differences between, groups were excluded.
Nevertheless, many of these studies offer important insights about not only the efficacy
of interventions, but also their usability, acceptability, and social validity. Furthermore,
they often serve to elucidate the conditions and contingencies most likely to promote a
program’s success.
84
CHAPTER THREE: METHOD
In this chapter, the method by which the present study was conducted is
discussed. The composition of the sample and methods for controlling for differences
between groups are outlined and explained. The content and delivery of the intervention
are described in detail, and representative screen shots are provided. The conduct of
study measures and the qualification of examiners are described. Instruments are
discussed, including the validity, reliability, and suitability to the present study. Finally,
the research design and approach to data analysis are detailed and defended.
The purpose of the present study was to determine whether there were significant
mean differences in (a) non-word reading, (b) real word reading, (c) non-word spelling,
(d) real word spelling, and/or (e) reading fluency post-test achievement scores of students
assigned to use the MVRC online reading intervention in addition to regular classroom
once the effects of differences in pre-test achievement scores and relevant demographic
primary means of data analysis. All data collection activities were supervised and
coordinated by a third-year doctoral student in special education. The resulting data set
was analyzed after having undergone de-identification, including the removal of all
85
Participants and Settings
classrooms in two public elementary schools in the southwestern United States. Of those,
107 were assigned to the treatment condition, and 102 were assigned to a business-as-
comprised 89 complete cases, and the comparison condition comprised 81, representing
170 complete cases, or 81.34% of original cases. While overall data loss was below
20%, data loss did not impact each condition equally, as the treatment group retained
83.12% of original cases, while the comparison group retained 79.41% of original cases,
and within the same postal code. At the time of testing, 81.6% of students in the district
were identified as Hispanic, 12.1% as white, 4.9% as Native America, 2.7% as African
American, and 0.6% as Asian or Pacific Islander. Over nine-in-ten (93.2%) students
were eligible to receive free or reduced price school meals, and 6.2% were homeless.
(ELL), and an additional 5.3% had been reclassified as fluent English speakers.18
The percentage of students in the sample who had a documented disability tracked
18
This reclassification process occurred when students achieved a satisfactory level of English-language
proficiency per standardized testing.
19
Data were derived from district publications. A reference to the source of this data has not been
included, so as to protect the identity of participating schools.
86
very closely with district demographics, at 12.3%. Sample demographics, however,
varied markedly from district demographics in ELL status, as 37.4% of sample group
members were classified as ELLs, and an additional 8.8% were reclassified as fluent
English speakers. Data concerning ethnicity and markers of socioeconomic status were
not made available at the individual case level in order to protect the privacy of
participants.
routines, random assignment at the student level was impossible. Assignment to groups
was therefore performed at the school level. One school, including each of its two
participating classrooms, was assigned to the treatment condition, and the other school,
including each of its two participating classrooms, was assigned to the comparison
condition. The school whose third grade students (the youngest grade tested) performed
more poorly (39% pass rate in reading) relative to the other (53% pass rate in reading) on
the state standardized test of in the 2012-2013 school year was assigned to the treatment
condition, and the school that performed better (53% pass rate in reading) relative to the
other (39% pass rate in reading) on the state standardized test of reading in the 2012-2013
school year was assigned to the comparison condition.20 Group assignment was
performed in this manner in order to ensure that any demographic advantages would
online reading instruction in their schools’ computer labs during a fixed period each day,
Monday through Thursday. No other students were permitted in the lab during treatment,
20
Data were derived from district publications. A reference to the source of this data has not been
included, so as to protect the identity of participating schools.
87
and the environment was made as reasonably free of distractions as possible.
Reading personnel visited the school sites and confirmed the adequacy of the schools’
Materials
Software for Reading, 2015) intervention is a multi-component reading and language arts
curriculum delivered via the Internet. It contains explicit instruction in (1) phonemic
awareness, (2) phonics, (3) fluency, (4) vocabulary, and (5) comprehension in alignment
with the recommendations of the National Reading Panel (NRP) (2000), as well as
consistent with the models of alphabetic and phonological development proposed by Ehri
(2005) and Pufpaff (2009) and satisfies the large majority of the content criteria outlined
by Grant et al. (2012), as well as all of the instructional and interface design criteria
MVRC overview. To access the MVRC program, participants log into an online
account, in which their interactions with the software and progress toward reaching
achievement targets are logged and retained (MindPlay Educational Software for
Reading, 2012). When students first access the program, they are immediately directed
to a diagnostic assessment, which is used for tracking and placement. The full diagnostic
assessment is repeated three times each academic year, and comprehensive progress
monitoring assessments are delivered every 14 days. All assessments are computer
88
adaptive: items delivered to a student increase or decrease in difficulty in response to that
student’s performance.
mastery. Mastery level activities include phonemic awareness, phonics, and grammar
assessment scores, and content within each lesson is adaptive in response to student
performance. Instruction and remediation within the activities are delivered via video
clips featuring the student’s assigned reading coach. Reading coaches include speech and
language pathologists (SLPs) and other specially trained individuals of diverse age and
ethnicity, and scripts and other instructional content were developed with the assistances
“fone” when asked by the reading coach to type the word “phone” will see a video clip in
which the reading coach advises the student that the /f/ sound can have different
spellings. If the student again fails to spell the word correctly, a video clip in which the
reading coach reminds the student that in words of Greek origin the /f/ sound is spelled
increasing difficulty, while the phonics and grammar and meaning components contain
five lesson sets of increasing difficulty. Each lesson set contains up to 20 different
presentations of the lesson content. If, based on the results of internal assessments, a
student fails to master at least 90% of the content assigned for a lesson, the lesson will be
represented in a different format until all of the lesson’s alternative presentations are
89
exhausted. The MVRC intervention allocates instructional time in relation to student
performance such that skills and activities in which a student demonstrates the greatest
proficiency receive the least instructional time, while skills and activities in which a
student demonstrates the least proficiency receive the greatest instructional time.
level activities, (s)he begins proficiency level activities, including fluency and vocabulary
lessons. Embedded within these lessons are structured spelling and comprehension
activities, in addition to lessons in the core skills associated with each lesson. The time
objectives, less time is devoted to proficiency level activities. Figure 1, used with
permission, outlines MVRC’s lesson sequence and assessment checkpoints. Note that all
phonemic awareness activities must be successfully completed, and the student must
achieve mastery in required phonics activities, before the introduction of grammar and
90
Figure 1. MVRC lesson sequence and assessment checkpoints. Taken with
Figure 1. MVRC lesson sequence and assessment checkpoints. Taken with
permission from the MindPlay Virtual Reading Coach Resource Guide (MindPlay
Educational Software for Reading, 2012).
Measures
of Silent Word Reading Fluency, Second Edition (TOSWRF-2; Mather, Hammill, Allen
& Roberts, 2014) and tests of the Woodcock Johnson Tests of Achievement, Fourth
Edition (WJ IVACH; Schrank, Mather, & McGrew, 2014). These measures were chosen
because they are considered highly valid measures of their respective constructs, and they
have high levels of test-retest reliability, reducing the likelihood of regression to the
91
concerning the reliability and validity of instruments, is provided in the subsequent
sections.
administration protocols. The WJ IV ACH (Schrank et al., 2014) contains tests designed
mathematics. The Woodcock-Johnson IV (WJ IV) (Schrank et al., 2014) comprises the
Tests of Cognitive Abilities (COG) and Tests of Achievement (ACH) batteries, as well as
the Tests of Oral Language. All three batteries were co-normed on a stratified random
sample of 7,416 participants ranging in age from 2 to >90 years, over 3,891 of whom
were grade school students at the time of administration (Schrank et al., 2014). Norming
and technical analyses were performed in accordance with the Standards for Educational
2014). Mean alternate form and test-retest reliabilities across all age groups for Basic
Reading Skills, Basic Writing Skills, and Phoneme-Grapheme Knowledge ranged from
0.94 to 0.95.
are required to place a line or slash between words that have no spaces between them. It
which is to draw lines between as many word boundaries as possible within the allotted
time. The test employs a graded word list, the vocabulary and complexity of which
92
progressively increase in difficulty (Mather et al., 2014). The test, therefore, measures
not only sight word recognition and reading rate but also vocabulary knowledge.
The TOSWRF-2 (Mather et al., 2014) was normed on a nation-wide sample of over 2,429
participants, ranging in age from six years and three months to 24 years and 11 months.
These participants were derived from 35 states, and participant demographics closely
resembled national demographics reported in the 2011 census (Mather et al., 2014). Test
re-test and alternate form reliabilities ranged from 0.84 to 0.91 (Allen, Morey, &
Table 3
93
Rationale for measure selection. The dependent variable measures selected for
this intervention were chosen both because of their high reliabilities and because they
real word and non-word reading and spelling were selected to ensure that the study
included measures of the identification and production of both phonetically regular and
phonetically irregular words. Identification and production of the former are indicative
indicative of sight word knowledge and familiarity with the word patterns of English. A
test of reading fluency was selected both as a broad measure of decoding ability and
because reading rate and accuracy are essential to reading comprehension, and there is a
strong correlation between reading fluency and comprehension (Pikulski & Chard, 2005),
logs and behavioral observation measures were used. Two measures of fidelity of
implementation were selected to evaluate not only whether the participants were using
the intervention for the duration specified, but also whether they were actively engaged
while using the intervention. The active engagement of participants, it was reasoned,
participants than would simple duration of use, thus serving as a metric for an important
Product usage logs. The frequency and duration of individual student's use of the
MVRC online reading intervention were measured using software usage logs generated
by the MVRC product. These logs included data concerning the duration of use of
21
Because of time constraints, comprehension measures were not conducted.
94
product components in minutes, the type and number of activities successfully
completed, and individual students’ progress toward achievement targets. Only active
use of the MVRC product was reported in the software usage logs, as the MVRC online
intervention automatically logs users out of the program after three minutes of inactivity.
biweekly throughout the study. This instrument was developed by a doctoral candidate in
operationalized as physical position (head up, eyes facing screen), headphone use,
keyboard use, and screen activity. To administer this measure, two observers
number of participants who did not appear to be engaged, per the criteria previously
outlined. Data collected with this instrument were used to document the percentage of
interval. Observers varied their observation schedules each week to reduce the likelihood
Intervention Procedure
Success for All language arts instruction in the regular classroom setting. Over 75% of
teachers in the participating schools voted to implement the Success for All program in
their schools, and school administrators expressed a high degree of support for the
95
program to examiners who visited their schools.
curriculum that includes structured lessons, cooperative group activities, and regular
assessments (Slavin & Madden, 2012). Success for All classroom activities included
instruction, guided practice, and ongoing formative evaluation. Success for All activities
are scripted and highly structured, so there is little variability in instruction among
classrooms implementing the program with a high degree of fidelity. A typical Success
for All lesson plan for a single day’s instructional activities exceeds ten pages and
phonics activity embedded in a Success for All daily lesson plan follows:
Sound words—Say each of the words below, and have the students repeat them.
• Ask: What sound can you hear in those words? [/ur/.] Say each word again,
and have the students repeat each one. Stretch the /ur/ sound in each word.
Key picture—Show the key card for “ur.” Let’s look at this picture card to
learn more about our sound for the day. This is a picture of a nurse with a
purse. “Nurse with a purse.” Let’s say that phrase together. [Nurse with a
purse.] Say each word in the phrase, and ask the students to repeat it. Stretch the
Success for All early reading instruction is conducted in small groups of children
with comparable levels of reading achievement. These groups can include children in
96
grades one through three. Texts used in the early grades are decodable, or phonetically
regular, and aligned to the student’s actual level of achievement. Early primary students
also read shared stories, in which a portion of the text is decodable and intended to be
read by the students and another portion is more complex and intended to be read by the
teacher. As the student’s achievement improves, the portion of the text read by the
student increases, while the portion of the text read by the teacher decreases. Other
reading activities include partner reading, in which a student and a partner read alternate
Success for All phonics activities begin with oral language and then move into
used to increase students’ retention and integration of the concepts learned, and
interactive activities are conducted to promote and reinforce the application of skills.
Specific components of the Success for All curriculum comprise (a) FastTrack
Phonics, in which activities include guided instruction, chants, games, and puppet shows
Shared Stories, in which reading, vocabulary, fluency, and comprehension are promoted
through partner reading and teacher-student reading; (c) Story Telling and Retelling, in
which students are taught to clarify, question, summarize, and visualize texts through
story preview, reading, review, retell, and critique activities; (d) Language Links, in
which a teacher models and guides higher order discussions involving specific reading-
97
related skills, such as the ability to identify story elements; and (e) Adventures in
received MVRC online reading instruction in addition to Success for All reading and
language arts instruction. Participants assigned to the treatment group used the software
for 30 minutes each day, Monday through Thursday, for a total of two hours per week
throughout the regular school year (mid-September through mid-April), with the
exception of holidays, school functions, and mandatory state testing days. Classroom
teachers brought students to the computer lab at the assigned time and assisted with
student log in when required; however, they did not assist students in any other way.
comparison condition received Success for All reading and language arts instruction but
did not receive the MVRC intervention. In addition to regular language arts instruction,
participants in the comparison group also received two hours of supplementary reading
instruction weekly from their classroom teachers. This instruction employed materials
and instructional techniques consistent with the Success for All curriculum and routine
classroom practice.
and a four-hour in-person seminar delivered by MindPlay personnel. The MVRC online
reading intervention requires very little participation on the part of classroom teachers, as
the content and difficulty of the intervention are adaptive in response to individual
98
student performance. Ultimate achievement targets are aligned with Common Core
standards; however, the content and instruction delivered to individual students are
theoretical structure of the program, the structure and content of the embedded lessons
and activities, facilitating and supporting student access and use of the intervention, and
accessing and understanding the data contained in student and classroom reports
generated by the software. Teachers were also provided with a teacher’s guide to
Assessment Procedure
graduate student in special education, who had over five years of experience working
with children in schools and two years of experience administering standardized tests of
achievement and cognitive abilities, as well as several retired special educators, who had
formal data concerning scoring disagreements and agreements were not collected or
99
Administration of dependent variable measures. Pre-test dependent variable
data collection was performed over a one-week period during the first 30 days of the
academic year, and post-test dependent variable data collection was performed over a
one-week period during the last 30 days of the academic year. Prior to dependent
students. Individual data collection was performed in quiet, low-distraction rooms in the
participants’ school; group data collection was performed in the participants’ classrooms.
Data collection was staggered over several days to avoid fatigue-related effects, and
The Spelling and Spelling of Sounds tests of the WJ IV ACH (Schrank et al., 2014),
students in whole-class groups, whereas the Word Attack and Letter-Word Identification
tests of the WJ IV ACH were administered individually and in a manner consistent with
beginning with the first item in each assessment and ending with items likely to be well
requirements were satisfied for all participants tested, and no participant failed to reach a
duration of students’ use of the MVRC online reading intervention were measured using
software usage logs generated by the MVRC product. Only active use of the MVRC
product was reported in the software usage logs, as the MVRC intervention automatically
100
A novel planned activity check was employed to measure student engagement
behavior, including physical position (i.e., head up, eyes facing screen), headphone use,
keyboard use, and screen activity, during treatment. Participating students assigned to
the treatment group were observed twice each month for 20 minutes throughout the
treatment period, and their engagement behaviors were recorded using the planned
Independent variable. The study’s sole independent variable was treatment with
variable was measured at the nominal level with values of 0 (not subjected to treatment)
Attack test of the WJ IV ACH (Schrank et al., 2014) were used. To operationalize real
word reading, scores from the Letter-Word Identification test of the WJ IV ACH
(Schrank et al., 2014) were used. To operationalize non-word spelling, scores from the
Spelling of Sounds test of the WJ IV ACH (Schrank et al., 2014) were used. To
operationalize real word spelling, scores from the Spelling test of the WJ IV ACH
(Schrank et al., 2014) were used. To operationalize reading fluency, scores from the
TOSWRF-2 (Mather et al., 2014) were used. All dependent variable data were reported
learner (ELL) status and special education status per an individualized educational
101
program (IEP) were recorded and reported as nominal data.
Data Analysis
strategies. The first of these was univariate analysis of variance, or ANOVA, using
participant gain scores (calculated by subtracting pre-test scores from post-test scores),
which is among the most commonly used techniques for comparing pre-test and post-test
data (Dimitrov & Rumrill, 2003). Univariate analyses of variance compare group means
for a single dependent variable (Tabatchnick & Fidell, 2013), and the use of gain scores,
as opposed to post-test scores alone, for example, serves as an informal means by which
to control for group differences (Dimitrov & Rumrill, 2003). The author rejected this
strategy for three reasons. First, there is evidence that gain scores can be an unreliable
means by which to compare group differences (Cronbach & Furby, 1970).22 Second,
scores for several dependent variable measures, indicating that participant potential for
gains might differ between groups. Univariate ANOVA, however, does not control for
such differences. Finally, the author wished to avoid the inflation in error created by the
conduct of multiple univariate tests (Tabatchnick & Fidell, 2013). Owing to all these
factors, therefore, the author determined that ANOVA was inadequate to her purpose.
The second data analytic strategy considered by the author was repeated measures
ANOVA using pre- and post-test scores, which would account for the pre-test differences
between groups (Dimitrov & Rumrill, 2003). While repeated measures ANOVA is a
22
While more recent research indicates that the degree to which gain scores are unreliable varies in relation
to the strength of correlation between pre- and post-test scores (Dimitrov & Rumrill, 2003; Zimmerman &
Williams, 1996), the author nevertheless desired to minimize the degree of unreliability of her analyses.
23
Described in greater depth in subsequent sections.
102
popular means by which to handle data in a pre-test post-test design, research has
demonstrated that the results of such analyses are often misleadingly conservative
(Dimitrov & Rumrill, 2003). Furthermore, the conduct of multiple repeated measures
ANOVAs would create the same inflation in error as the conduct of multiple ANOVAs
on gain scores. Moreover, data screening performed by the author indicated that the data
set would violate the assumption of sphericity required to perform the test.
The third data analytic strategy considered by the author was univariate analysis of
while controlling for the effects of one or more covariates (Tabatchnick & Fidell, 2013).
This statistical control increases the power and precision of analyses by reducing error
variance accounted for by the covariate(s) (Dimitrov & Rumrill, 2003). The choice of
covariate, pre-test achievement scores, would overlap with the dependent variable, gain
scores.24 Therefore, the use of ANCOVA would require the analysis of post-test scores
with pre-test scores as covariates. While this strategy would account for the differences
between groups, it would nevertheless leave the analyses subject to the inflation in error
created by the conduct of multiple univariate tests (Tabatchnick & Fidell, 2013).
The fourth method of data analysis considered by the author was repeated measures
MANCOVA on pre- and post-test scores with relevant demographic covariates. While
this multivariate method of data analysis would control for multiple comparisons,
preliminary analyses performed by the author indicated that the data would violate the
assumption of sphericity required to perform the test. Therefore, this method was also
rejected.
24
Gain scores are calculated by subtracting pre-test scores from post-test scores.
103
The fifth and final data analytic strategy considered by the author was multivariate
variables after the effects of relevant covariates have been removed from the analysis
(Tabatchnick & Fidell, 2013). There are several important advantages to the use of a
MANCOVA design over the use of multiple separate analyses of variance (ANOVA), or
used to control for pre-test differences between groups. This is particularly important in
systematic bias, potentially compromising the results (Dimitrov & Rumrill, 2003).
Second, MANCOVA accounts for the intercorrelations among the dependent variables in
the analysis, leading to a more precise overall picture of change (Tabatchnick & Fidell,
2013). Finally, the use of MANCOVA allows the researcher to more accurately identify
variables that have changed in relation to treatment, substantially reducing the likelihood
word reading, (b) real word reading, (c) non-word spelling, (d) real word spelling, and/or
(e) reading fluency post-test achievement scores of students assigned to use the MVRC
comparison condition (regular classroom reading instruction alone), once the effect of
104
demographic variables had been accounted for.
achievement scores (averaged across tests) and relevant demographic variables. The
author also elected to perform a second set of post-hoc ANCOVAs to isolate the
relationship between each dependent variable, its corresponding pre-test covariate, and
for each dependent variable, rather than using the mean pre-test score covariate employed
in the full MANCOVA, the author sought to lend more precision to the analyses and
ensure that the specific effects of pre-test achievement on each dependent variable were
105
CHAPTER FOUR: RESULTS
In this chapter, the author relates the results of data analyses, including data
screening and cleaning procedures, omnibus testing, univariate testing, and post-hoc
testing for all dependent variable measures. Measures of fidelity implementation are also
Measurement of data. All dependent variable and pretest covariate data collected
for this study were measured as obtained, raw scores using discrete, quantitative scales.
Obtained, raw scores for each post-test achievement measure comprised the study’s
dependent variables. A pre-test covariate was created by taking the mean of sample-
referenced z-scores across pre-tests.25 The conversion of scores was undertaken prior to
averaging to ensure that all data reflected equivalent scales of measurement. Such a
procedure was necessary because the number of items varied across tests, as did item
the TOSWRF-2 (Mather et al., 2014) were calculated by subtracting items incorrect from
items correct, while raw scores for all other measures were calculated by adding only
correct items.
Missing data. Owing to attrition and illness, 39 participants had incomplete data
cases, and the comparison condition comprised 81, representing 170 complete cases, or
81.34% of original cases. Because more than 10% of the original cases were missing
25
The author chose to create a single pre-test achievement covariate, rather than add the pre-test
achievement scores for each dependent variable a separate covariate to the MANCOVA model. This
decision was taken to reduce the number of variables added to the MANCOVA model, thereby increasing
statistical power.
106
data, the authors chose to exclude cases listwise from analyses. Imputation of missing
data was considered and rejected consistent with the recommendations of Tabatchnick
and Fidell (2013), who suggested imputation only when fewer than 10% of cases were
Data screening. Prior to analysis, dependent variable data were evaluated for data
entry errors using Microsoft Excel and for normality, linearity, skewness, and kurtosis
using IBM SPSS EXPLORE. Univariate outliers with respect to the dependent variables
(defined as values more than three standard deviations above the mean for the relevant
variable) were detected using IBM SPSS FREQUENCIES, and multivariate outliers were
Careful evaluation of the data revealed no values suggestive of data entry errors;
however, a total of three cases with univariate outliers were detected. Each case had only
a single outlying variable, and all three involved values between three and four standard
deviations above the mean. In order to preserve the maximum number of cases possible,
outlying values within a case were treated as missing and replaced using the highest or
lowest non-outlying value possible (i.e., a value exactly three standard deviations above
or below the mean), consistent with the recommendations of Tabatchnick and Fidell
(2013). In addition to the univariate outliers, a single multivariate outlier was detected.
The value was not extreme, however, so the author elected to retain the case, but include
Following the transformation of outlying data, the author split the data by condition
and generated scatterplot matrices to evaluate the linearity of data. Upon visual
inspection, the author determined that the data were sufficiently linear to proceed with
107
further analyses. The author then generated histograms, stem-and leaf plots, and Q-Q
plots for each of the study’s dependent variables. A visual inspection of the plots and
histograms for each of the dependent variables revealed varying degrees of (most often
negative) skewness, as well as modest amount of excess kurtosis, with the exception of a
single variable highly leptokurtic distribution. This finding was reinforced by significant
values at the p < .05 level for non-word spelling for the treatment group, real-word
spelling for the comparison group, and non-word reading for the treatment group on the
measures of skewness and kurtosis for each of the study's dependent variables by
condition, and Table 5 provides the results of the Shapiro-Wilk test of normality.
108
Table 4
Descriptive Statistics by Condition for Each Dependent Variable and Interval-Level Covariate
DV Condition Measure Statistic SE
Comparison M 42.901 1.2004
SD 10.8035
Skewness -0.255 0.267
Real Word Kurtosis -0.606 0.529
Reading Treatment M 45.944 0.8706
SD 8.2136
Skewness -0.466 0.255
Kurtosis 0.562 0.506
Comparison M 20.383 0.6654
SD 5.9887
Skewness 0.572 0.267
Real Word Kurtosis 0.162 0.529
Spelling Treatment M 23.326 0.5661
SD 5.3402
Skewness -0.136 0.255
Kurtosis 0.419 0.506
Comparison M 18.136 0.5749
SD 5.1739
Skewness -0.15 0.267
Non-Word Kurtosis -0.512 0.529
Reading Treatment M 19.629 0.4322
SD 4.077
Skewness -0.339 0.255
Kurtosis -0.279 0.506
Comparison M 14.025 0.3451
SD 3.1063
Skewness -0.229 0.267
Non-Word Kurtosis 0.341 0.529
Spelling Treatment M 16.438 0.3212
SD 3.03
Skewness -1.215 0.255
Kurtosis 2.761 0.506
Comparison M 45.593 2.2861
SD 20.5747
Skewness -0.154 0.267
Reading Kurtosis -0.337 0.529
Fluency Treatment M 63.169 2.0015
SD 18.8826
Skewness -0.581 0.255
Kurtosis 0.105 0.506
Note. DV = dependent variable; M = mean; SD = standard deviation; and SE = standard error.
Skewness refers to excess skewness.
109
Table 5
The author attempted to reduce the leptokurtosis and skewness of the relevant
normalize the data (i.e., none was sufficient to render the relevant results of the Shapiro-
Wilk test of normality non-significant). The author also considered the use of non-
identified. Therefore, the author elected to proceed with data screening and,
subsequently, the full MANCOVA. This decision was taken because multivariate
analyses of variance (MANOVA) and related tests (e.g., MANCOVA) are considered
robust to violations of normality if groups are of roughly comparable size, that is the size
of the largest group is no more than 1.5 times larger than the size of the smallest group
To screen for collinearity among dependent variables, which might indicate that
two or more variables measured the same construct, the author used IBM SPSS
110
CORELATE to perform a Pearson’s product-moment correlation analysis identifying the
relationship between each dependent variable and all others. All of the variables were
significantly correlated to all others at the p < .01 level, with correlation coefficients
ranging in strength from R^2 = .519 to R^2 = .795. Based on these results, the author
(Tabatchnick & Fidell, 2013), to determine if any latent variables might explain the
strong correlations among the dependent variables. Table 6 provides the results of the
Table 6
** Significant at the p < .01 level; * significant at the p < .05 level.
111
A PCA with Varimax (orthogonal) rotation was performed using IMB SPSS DATA
.853) indicated that the data were suitable for factor analysis per the guidelines suggested
by Tabatchnick and Fidell (2013). This finding was reinforced by the extremely high
value generated by Bartlett's test of sphericity [χ2(10) = 672.814, p < .001]. Perhaps
unsurprisingly, the results of the PCA were indicative of a single latent variable, or
component, upon which the study’s five dependent variables loaded. This component,
obviously associated with participants’ overall language arts achievement, explained over
While the results of the factor analysis pointed to a single latent variable underlying
performance across the dependent variables measured, the author felt it would be
undesirable to combine all of the dependent variables into one variable representative of
achievement across measures. First, to conflate all of the variables into one would
necessarily conflate their error, rendering subsequent analyses less precise. Second, the
high degree of correlation among the dependent variables was unsurprising, as the
intercorrelations26 reported for the WJ IV ACH (Schrank et al., 2014) tests used to
measure four of five of the dependent variables ranged from R^2 = .66 to R^2 = .82, very
closely mirroring the findings of the Pearson's product moment correlation analysis.
Furthermore, the primary reason to avoid including highly correlated variables is that
high levels of correlation among dependent variables can lead to a reduction in statistical
power (Foster, Barkus, & Yavorosky, 2006); however, in situations in which multivariate
analysis is required to reduce the possibility of a Type I error, such a loss of power can be
26
Among norm group participants aged six to eight years.
112
considered an acceptable trade-off (Foster et al., 2006). Therefore, the author chose to
analyses of variance (ANOVA) were performed to compare mean scores between groups
(i.e., treatment and comparison) for each dependent variable and demographic variable.
The author elected to determine significance at the p < .10 level to ensure that any
differences between groups that might impact dependent variable measures would be
adequately accounted for in subsequent analyses. While the author understood that the
introduction of additional covariates into the model would weaken statistical power, the
author felt that a more conservative approach was warranted, given the lack of random
covariates. Significant differences favoring the treatment group were detected at the p <
.10 level in the pre-test scores for real word reading [F(1, 169) = 2.927, p = .089], and
non-word reading [F(1, 169) = 3.314, p = .070]. Significant differences were also
detected at the p < .10 level English language learner (ELL) status [F(1,169) = 11.480, p
= .001] and special education (SPED) status [F(1,169) = 3.421, p = .066]. Therefore, a
decision was taken to add pre-test achievement, ELL status, and SPED status to the
Data Analysis27
performed to detect mean differences in (a) non-word reading, (b) real word reading, (c)
non-word spelling, (d) real word spelling, and/or (e) reading fluency post-test
27
Data analyses were performed using IBM SPSS GENERAL LINEAR MODEL unless otherwise noted.
113
achievement scores of students assigned to use the MVRC online reading intervention
achievement scores and relevant demographic variables had been accounted for.
Tests of equality of variance and covariance. Levene’s test for equality of error
variance was non-significant for each of the study’s dependent variables at the p < .05
level, as was Box’s test of equality of covariance matrices at the p < .05 level, indicating
that the data did not violate the assumption of homogeneity of variance. Therefore,
consistent with the recommendations of Tabatchnick and Fidell (2013), the author chose
to use Wilk’s lambda as the test statistic in subsequent multivariate analyses. The results
of Box’s test of equality of covariance matrices and Levene’s test for equality of variance
Table 7
Table 8
114
Multivariate tests. The results of the MANCOVA analysis revealed a significant
main effect (λ = .668, F[5, 161] = 16.014, p < .001, multivariate η2 = .332) of the
relative to those assigned to the business-as-usual comparison condition, once the effects
English language learner (ELL) status had been accounted for. The main effects of ELL
status (λ = .868, F[5, 161] = 4.903, p < .001, multivariate η2 = .132) and pre-test scores (λ
= .212, F[5, 161] = 119.487, p < .001, multivariate η2 = .788) indicated significant effects
on the combined dependent variables (DV). Special education status, however, had no
significant effect on the combined DVs. Table 9 contains the results of the multivariate
tests for the study’s independent variable and each of its covariates.
Table 9
Multivariate Tests
115
Univariate tests. Post-hoc ANCOVAs were performed to isolate the relationship
between each dependent variable, the combined pre-test covariate, and the grouping
variable (condition). The multivariate test results were confirmed by the post-hoc
four of the study’s five dependent variables: real word spelling (F[1, 165] = 16.341, p <
.001, multivariate η2 = .090), non-word reading (F[1, 165] = 4.368, p = .038, multivariate
η2 = .026), non-word spelling (F[1, 165] = 29.212, p = .001, multivariate η2 = .150), and
reading fluency (F[1, 165] = 58.348, p < .001, multivariate η2 = .261). Statistically
significant effects of the intervention, however, were not detected in real word reading
special education status, English language learner (ELL) status, and pre-test achievement
scores; however, tests of between-subjects effects revealed that only the pre-test score
covariate was significant at the p < .05 level for all dependent variables. Special
education status, by contrast, was significant at the p < .05 level relative to real word
reading alone, while ELL status was significant at the p < .05 level only in relationship to
non-word reading. Table 10 contains the parameter estimates for each dependent
variable relative to the independent variable and each covariate. Table 11 displays
116
Table 10
117
Table 11
relationship between each dependent variable, its corresponding pre-test covariate, and
the grouping variable (condition).28 The first of these ANCOVAs was conducted to
confirm the results of the previous analyses relative to the study’s real word reading
differences between groups at the p < .05 level on real word reading, once the effects of
differences in pre-test achievement scores, special education status, and English language
learner (ELL) status had been accounted for. This result was consistent with that of the
previous analyses, thus confirming the author’s original finding. Also consistent with
previous univariate analyses, the dependent variable was significantly affected by the pre-
test score covariate at the p < .05 level; however, in contrast to previous univariate
28
It was not possible to perform Bonferroni’s correction for multiple comparisons in the initial analyses.
Because a total of five comparisons were performed, only significance values of p < .01 were treated as
truly significant.
118
analyses, the SPED status covariate was not significant at the p < .05 level. Table 12
contains the parameter estimates for the dependent variable relative to the independent
variable and each covariate, while Table 13 displays pairwise comparisons between
Table 12
Table 13
spelling confirmed the results of previous analyses. The ANCOVA indicated that there
were significant differences between groups at the p < .05 level in real word reading,
once the effects of differences in pre-test achievement scores, special education status,
and English language learner (ELL) status had been accounted for. Also consistent with
previous univariate analyses, only the pre-test score covariate significantly affected the
119
dependent variable at the p < .05 level. Table 14 displays the parameter estimates for the
dependent variable relative to the independent variable and each covariate, and Table 15
Table 14
Table 15
reading were inconsistent with previous analyses. The ANCOVA failed to demonstrate
significant differences between groups at the p < .05 level in non-word reading, once the
effects of differences in pre-test achievement scores, special education status, and English
language learner (ELL) status had been accounted for. Table 16 displays the parameter
estimates for the dependent variable relative to the independent variable and each
covariate, and Table 17 shows pairwise comparisons between groups for the dependent
120
variable.
Table 16
Table 17
mirrored previous analyses. The results revealed significant effects of the intervention on
non-word spelling, once the effects of differences in pre-test achievement scores, special
education status, and English language learner (ELL) status had been accounted for.
While pre-test scores remained significant at the p < .05 level, ELL status and SPED
status were non-significant. Table 18 displays the parameter estimates for the dependent
variable relative to the independent variable and each covariate, and Table 19 shows
121
Table 18
Table 19
fluency reflected the findings of previous analyses. The ANCOVA revealed significant
effects of the intervention on reading fluency, once the effects of differences in pre-test
achievement scores, special education status, and English language learner (ELL) status
had been accounted for. Unlike previous univariate analyses, however, the ANCOVA
revealed significant effects at the p < .05 level for all the covariates (pre-test scores, ELL
status, and SPED status), thus supporting their introduction into the model. Table 20
displays the parameter estimates for the dependent variable relative to the independent
variable and each covariate, and Table 21 shows pairwise comparisons between groups
122
Table 20
Table 21
Fidelity of Implementation
Product usage logs. Product usage logs generated by the MVRC software
indicated that active participant use of the intervention exceeded 90% of the time
assigned (a mean of 44 hours of active use) among treatment group participants who
participants were observed 45 times while using the MVRC online intervention, or
approximately twice per week. Observers used a novel planned activity check instrument
to record behavioral observations. Data from this instrument were analyzed by dividing
the number of students displaying engagement behavior at each interval by the total
123
number of students observed and multiplying the dividend by 100 to determine the
for the entire twenty-minute observation period and then aggregated and averaged by
between observers per interval by the total number of agreements plus disagreements and
participants displayed engagement behavior over 90% of the intervals observed while
using the MVRC online intervention. Inter-observer agreement, however, was within the
acceptable range at just under 84%. Table 22 displays the results of the planned activity
check.
Table 22
124
CHAPTER FIVE: RESEARCH REPORT
In this chapter, the author presents the results of a formal report documenting the
present study and its findings. The report is presented at a publishable paper. Thus,
consistent with journal conventions, the format differs from that of previous chapters.
(four classrooms in each school) in two public elementary schools in the southwestern
United States. Examiners obtained reading achievement data from each participating
student. Measures included tests from the Woodcock-Johnson Tests of Achievement (WJ
were significant mean differences in (a) non-word reading, (b) real word reading, (c) non-
word spelling, (d) real word spelling, and/or (e) reading fluency scores favoring students
assigned to use the MVRC online reading intervention, once the effects of differences in
pre-test achievement scores and relevant demographic variables had been accounted for.
Analyses revealed a significant main effect (λ = .668, F [5, 161] = 16.014, p < .0001,
to the treatment condition, which was confirmed across three of the study’s independent
variables: real word spelling (F[1, 165] = 16.341, p < .0001, multivariate η2 = .090), non-
word spelling (F[1, 165] = 29.212, p < .0001, multivariate η2 = .150), and reading fluency
125
The Effects of the Use of an ICT-Based Reading Intervention on
Literacy and its component skills, the ability to read with fluency and
comprehension and write fluently and coherently, are essential to educational attainment
across domains: they “[bridge] the gap between learning to read and reading to learn”
(Duke, Bennett-Armistead, & Roberts, 2003, p. 226) and provide the key that opens the
however, has not yet achieved its potential in ensuring that as many Americans as
possible enjoy the benefits of literacy. The findings of the National Assessment of Adult
Literacy revealed that 43% of adults in the United States scored at basic or below basic
levels in prose literacy, or the ability to understand, summarize, make simple inferences,
determine cause and effect, and recognize an author’s purpose when presented with texts
of moderate density (Kutner, Greenberg, Jin, Boyle, Hsu, & Dunleavy, 2007). Results of
the National Assessment of Educational Progress (NAEP) painted an even bleaker picture
basic levels of grade-level literacy (National Center for Education Statistics, 2013).
Research suggests that once children have reached this point in their education, when the
focus of instruction has shifted from learning to read to reading to learn (Duke et al.,
2003), they are at increased risk for academic failure (Felton & Pepper, 1995; Juel, 1988),
often struggling to acquire the content knowledge necessary for academic success.
and social success. This conclusion is well supported in the literature. Kennely and
126
scores and school dropout, and researchers have consistently found that youngsters with
O’Brien, & Langhinrichsen-Rohling, 2007), placing them at increased risk for future
The vast majority of children at risk for illiteracy can be taught to read with fluency
Schuster, Yaghoub-Zadeh, & Shanahan, 2001; Snow, Griffin, & Burns, 2005). In
phonics has been shown to positively affect the reading and writing abilities of students
with reading-related challenges (Ehri et al., 2001; Hatcher, Hulme, & Snowling, 2004;
Torgerson, Brooks, & Hall, 2006). For these techniques to work, however, teachers
Despite a large body of literature showing what works, many teacher preparation
programs have failed to provide teachers with the knowledge and skills required to meet
the needs of beginning readers. In a 2006 review of course syllabi and textbooks, Walsh,
Glaser, and Dunne-Wilcox revealed that only 15% of American colleges of education
provided pre-service teachers with even minimal exposure to the science underlying
Snow et al. (2005) found that a large proportion of special and general education teachers
127
were unprepared to provide evidence-based reading interventions to their students.
elementary educators with the training required to effectively meet the needs of
beginning readers, efforts must be undertaken to identify and promote means to ensure
that all students nevertheless have access to high quality reading instruction. Systematic
reading instruction using information and communication technologies (ICT) has been
et al., 2013), and such instruction often requires little or no direct intervention on the part
of the classroom teacher (Bishop & Edwards Santoro, 2006). ICT-based instructional
programs have been widely adopted in classroom contexts, “generally with an underlying
expectation that student learning can improve … through supportive skill instruction with
practice” (Cassady & Smith, 2005, p. 362). This sentiment was mirrored in the National
and computer technologies are (a) explicit, systematic instruction in the sound-symbol
correspondences of spoken and written language (Camilli et al., 2003; Ehri et al., 2001;
Torgerson et al., 2006), (b) multimodal instruction to promote recall and retention (Low
& Sweller, 2005; Moreno & Mayer, 2007), (c) formative feedback to guide learning and
activate prior knowledge (Narciss, 2013), (d) interactivity to promote attention and
engagement (Sims, 2000, 2003), and (e) opportunities for mastery learning to enhance
128
achievement (Guskey, 2007, 2012).
unresolved (Edwards Santoro & Bishop, 2010; Grant et al., 2012), and ICT-based reading
instruction remains poorly theorized and inadequately researched (Savage et al., 2013),
particularly concerning studies involving participants aged eight years and younger
(Lankshear & Knobel, 2003). This dearth of quality research is exemplified by the
analysis of over forty studies designed to evaluate the efficacy of ICT-based reading
interventions in promoting early literacy achievement, Blok et al. (2002) found only three
studies (Barker and Torgersen [1995] and two sub-studies contained in Foster, Erickson,
language reading intervention. While the authors (Barker & Torgesen, 1995; Foster et
favoring participants in the treatment group, the software used (iterations of DaisyQuest)
is now obsolete. All of the other studies included in the meta-analysis performed by Blok
et al. (2002) were either conducted in a language other than English or involved
presentation of text, speech feedback, or virtual flashcards. Though Blok et al. (2002)
reported a modest corrected overall effect size estimate of +0.19 across studies, they
cautioned that this result should be interpreted with care, as many of the studies evaluated
were poorly designed, and several compared the effects of multiple software applications,
129
The caution urged by Blok et al. (2002) appears well founded in the light of the
results of other syntheses, whose authors applied more stringent inclusion criteria (e.g.,
requiring experimental designs or matched quasi-experimental designs with both pre- and
literature in the field, Torgerson and Zhu identified a dozen studies meeting inclusion
criteria. Of those, only one (Mitchell & Fox, 2001) included a sequential, code-based
ICT-based reading intervention delivered to participants in the early primary grades, and
the authors of that study reported no statistically significant effect of the intervention.29
Science Foundation and performed by Kulik in 2003. While the author (Kulik, 2003)
only three of those showed statistically significant positive effects favoring treatment
group participants, and all three were published in the early 1990s and evaluated the
In a more recent synthesis, Slavin, Lake, Chambers, Cheung, and Davis (2009)
criteria; however, only eight were peer-reviewed studies including a now extant
grade three. Of those eight, four were embedded in a larger study commissioned by the
Pendeleton, 2009). Of the four interventions evaluated by Campuzano et al. (2009), none
29
While Mitchell and Fox (2001) found no significant differences favoring the ICT-based reading
intervention group when compared to comparable teacher-delivered intervention group, they did find
significant differences favoring the ICT-based reading intervention group when compared to a no-treatment
control.
130
were found to have statistically significant positive effects on participants’ reading
achievement. Unlike Campuzano et al. (2009), the authors of three of the four remaining
studies reviewed by Slavin et al. (2009) reported statistically significant positive effects
of their respective interventions, with effect sizes ranging from +0.17 to +1.05. While
were not found favoring the treatment group in measures of word reading (Paterson,
Jacobs Henry, O'Quin, Ceprano, & Blue, 2003), lacked a pre-test measure for that
construct, putting its findings concerning reading achievement into question. Notably,
both of the still extant software packages for which the authors provided statistically
significant evidentiary support, Waterford Early Learning Program (Cassady & Smith,
2005; Tracey & Young, 2005) and Lexia Learning Systems software (Macaruso, Hook,
& McCabe, 2006), involved a strong emphasis on systematic phonics instruction and the
Purpose
With some notable exceptions (e.g., Macaruso, Hook, & McCabe, 2006;
McMurray, 2013; Savage et al., 2013; Savage, Abrami, Hipps, & Deault, 2009; Savage,
Erten, Abrami, Hipps, Comaskey, & van Lierop, 2010), relatively little high quality
Kulik, 2003; Slavin et al., 2009; Torgerson & Zhu, 2003). Prominent voices in the field
have suggested that teachers and education authorities remain wary of adopting any ICT-
based reading program until it has a consistent base of high quality evidentiary support
131
(Slavin et al., 2009; Torgerson, 2007; Torgerson & Zhu, 2003).30 Through the present
study, the authors wish to fill a gap in the existing ICT-based beginning reading
intervention literature, while addressing issues of research design and intervention quality
Research Question
In the present study, the following research question was addressed: Are there
significant mean differences in (a) non-word reading, (b) real word reading, (c) non-word
spelling, (d) real word spelling, and/or (e) reading fluency post-test achievement scores of
students assigned to use the MVRC online reading intervention in addition to regular
condition, once the effects of differences in pre-test achievement scores and relevant
Method
classrooms in two public elementary schools in the southwestern United States. Of those,
107 were assigned to the treatment condition, and 102 were assigned to a business-as-
comprised 89 complete cases, and the comparison condition comprised 81, representing
170 complete cases, or 81.34% of original cases. While overall data loss was below 20%,
30
Among the recommendations of Torgerson (2007) and Torgerson and Zhu (2003) was the conduct of
randomized controlled trials (RCTs). Owing to the small sample size in the present study, it was not
possible to perform random assignment to groups at the classroom level. Therefore, the authors elected to
statistically control for pre-test differences between groups.
132
data loss did not impact each condition equally, as the treatment group retained 83.12%
of original cases, while the comparison group retained 79.41% of original cases,
and within the same postal code. At the time of testing, 81.6% of students in the district
were identified as Hispanic, 12.1% as white, 4.9% as Native America, 2.7% as African
American, and 0.6% as Asian or Pacific Islander. Over nine-in-ten (93.2%) students
were eligible to receive free or reduced price school meals, and 6.2% were homeless.
(ELL), and an additional 5.3% had been reclassified as fluent English speakers.31
The percentage of students in the sample who had a documented disability tracked
varied markedly from district demographics in ELL status, as 37.4% of sample group
members were classified as ELLs, and an additional 8.8% were reclassified as fluent
English speakers. Data concerning ethnicity and markers of socioeconomic status were
not made available at the individual case level in order to protect the privacy of
participants.
routines, random assignment at the student level was impossible. Assignment to groups
31
This reclassification process occurred when students achieved a satisfactory level of English-language
proficiency per standardized testing.
32
Data were derived from district publications. A reference to the source of this data has not been
included, so as to protect the identity of participating schools.
133
was therefore performed at the school level. One school, including each of its two
participating classrooms, was assigned to the treatment condition, and the other school,
including each of its two participating classrooms, was assigned to the comparison
condition. The school whose third grade students (the youngest grade tested) performed
more poorly (39% pass rate in reading) relative to the other (53% pass rate in reading) on
the state standardized test of in the 2012-2013 school year was assigned to the treatment
condition, and the school that performed better (53% pass rate in reading) relative to the
other (39% pass rate in reading) on the state standardized test of reading in the 2012-2013
school year was assigned to the comparison condition.33 Group assignment was
performed in this manner in order to ensure that any demographic advantages would
Intervention settings. The research was conducted over the 2013-2014 school
year. During treatment, treatment group students received MVRC online reading
instruction in their schools’ computer labs during a fixed period each day, Monday
through Thursday. No other students were permitted in the lab during treatment, and the
students were provided with individual computers, monitors, and headphones. Prior to
the beginning of treatment, MindPlay Educational Software for Reading personnel visited
the school sites and confirmed the adequacy of the schools’ computer hardware and
Internet connectivity.
Materials
33
Data were derived from district publications. A reference to the source of this data has not been
included, so as to protect the identity of participating schools.
134
Software for Reading, 2015) intervention is a multi-component reading and language arts
curriculum delivered via the Internet. It contains explicit instruction in (1) phonemic
awareness, (2) phonics, (3) fluency, (4) vocabulary, and (5) comprehension in alignment
with the recommendations of the National Reading Panel (NRP) (2000), as well as
consistent with the models of alphabetic and phonological development proposed by Ehri
(2005) and Pufpaff (2009), respectively, and satisfies the large majority of the content
criteria outlined by Grant et al. (2012), as well as all of the instructional and interface
Measures
of Silent Word Reading Fluency, Second Edition (TOSWRF-2; Mather, Hammill, Allen
& Roberts, 2014) and tests from the Woodcock Johnson Tests of Achievement, Fourth
Edition (WJ IVACH; Schrank, Mather, & McGrew, 2014).34 These measures were
chosen because they are considered highly valid measures of their respective constructs,
and they have high levels of test-retest reliability, reducing the likelihood of regression to
the mean. Table 23 provides an overview of the description of each dependent variable
measure performed.
34
Copies of these instruments were provided to the researchers by the publishers prior to publication and
public release.
135
Table 23
measures selected for this intervention were chosen both because of their high reliabilities
and because they were designed to measure code-based skills foundational to proficient
reading. Tests of real word and non-word reading and spelling were selected to ensure
that the study included measures of the identification and production of both phonetically
regular and phonetically irregular words. Identification and production of the former are
latter are indicative of sight word knowledge and familiarity with the word patterns of
English. A test of reading fluency was selected both as a broad measure of decoding
ability and because reading rate and accuracy are essential to reading comprehension, and
136
there is a strong correlation between reading fluency and comprehension (Pikulski &
logs and behavioral observation measures were used. Two measures of fidelity of
implementation were selected to evaluate not only whether the participants were using
the intervention for the duration specified, but also whether they were actively engaged
while using the intervention. The active engagement of participants, it was reasoned,
participants than would simple duration of use, thus serving as a metric for an important
Product usage logs. The frequency and duration of individual student's use of the
MVRC online reading intervention were measured using software usage logs generated
by the MVRC product. These logs included data concerning the duration of use of
completed, and individual students’ progress toward achievement targets. Only active
use of the MVRC product was reported in the software usage logs, as the MVRC online
intervention automatically logs users out of the program after three minutes of inactivity.
biweekly throughout the study. This instrument was developed by a doctoral candidate in
operationalized as physical position (head up, eyes facing screen), headphone use,
keyboard use, and screen activity. To administer this measure, two observers
137
simultaneously scanned each classroom at two-minute intervals for 20 minutes and
number of participants who did not appear to be engaged, per the criteria previously
outlined. Data collected with this instrument were used to document the percentage of
interval. Observers varied their observation schedules each week to reduce the likelihood
Intervention Procedure
Success for All language arts instruction in the classroom setting. Success for All is a
structured lessons, cooperative group activities, and regular assessments (Slavin &
Madden, 2012). Success for All classroom activities included intensive daily reading
and ongoing formative evaluation. Activities are scripted and highly structured, so there
Intervention instruction. The research was conducted over the 2013-2014 school
year. Participating students assigned to the treatment group received MVRC online
reading instruction in addition to Success for All reading and language arts instruction.
Participants assigned to the treatment group used the software for 30 minutes each day,
Monday through Thursday, for a total of two hours per week throughout the regular
school year (mid-September through mid-April), with the exception of holidays, school
functions, and mandatory state testing days. Classroom teachers brought students to the
138
computer lab at the assigned time and assisted with student log in when required;
comparison condition received Success for All reading and language arts instruction but
did not receive the MVRC intervention. In addition to regular language arts instruction,
participants in the comparison group also received two hours of supplementary reading
instruction weekly from their classroom teachers. This instruction employed materials
and instructional techniques consistent with the Success for All curriculum and routine
classroom practice.
and a four-hour in-person seminar delivered by MindPlay personnel. The MVRC online
reading intervention requires very little participation on the part of classroom teachers, as
the content and difficulty of the intervention are adaptive in response to individual
student performance. Ultimate achievement targets are aligned with Common Core
standards; however, the content and instruction delivered to individual students are
theoretical structure of the program, the structure and content of the embedded lessons
and activities, facilitating and supporting student access and use of the intervention, and
accessing and understanding the data contained in student and classroom reports
generated by the software. Teachers were also provided with a teacher’s guide to
139
Assessment Procedure
graduate student in special education, who had over five years of experience working
achievement tests, as well as several retired special educators, who had extensive
supplemental training in the administration of the WJ IV ACH from one of the tests’ co-
authors.
formal data concerning scoring disagreements and agreements were not collected or
data collection was performed over a one-week period during the first 30 days of the
academic year, and post-test dependent variable data collection was performed over a
one-week period during the last 30 days of the academic year. Prior to dependent
students. Individual data collection was performed in quiet, low-distraction rooms in the
participants’ school; group data collection was performed in the participants’ classrooms.
Data collection was staggered over several days to avoid fatigue-related effects, and
140
The Spelling and Spelling of Sounds tests of the WJ IV ACH (Schrank et al., 2014),
students in whole-class groups, whereas the Word Attack and Letter-Word Identification
tests of the WJ IV ACH were administered individually and in a manner consistent with
beginning with the first item in each assessment and ending with items likely to be well
requirements were satisfied for all participants tested, and no participant failed to reach a
Data Analysis
Measurement of data. All dependent variable and pretest covariate data collected
for this study were measured as obtained, raw scores using discrete, quantitative scales.
Obtained, raw scores for each post-test achievement measure comprised the study’s
dependent variables. A pre-test covariate was created by taking the mean of sample-
referenced z-scores across pre-tests.35 The conversion of scores was undertaken prior to
averaging to ensure that all data reflected equivalent scales of measurement. Such a
procedure was necessary because the number of items varied across tests, as did item
Missing data. Owing to attrition and illness, 39 participants had incomplete data
cases, and the comparison condition comprised 81, representing 170 complete cases, or
35
The authors chose to create a single pre-test achievement covariate, rather than add the pre-test
achievement scores for each dependent variable a separate covariate to the MANCOVA model. This
decision was taken to reduce the number of variables added to the MANCOVA model, thereby increasing
statistical power.
141
81.34% of original cases. Because more than 10% of the original cases were missing
data, the authors chose to exclude cases listwise from analyses. Imputation of missing
data was considered and rejected consistent with the recommendations of Tabatchnick
and Fidell (2013), who suggested imputation only when fewer than 10% of cases were
analyses of variance (ANOVA) were performed to compare mean scores between groups
(i.e., treatment and comparison) for each dependent variable and demographic variable.
The authors elected to determine significance at the p < .10 level to ensure that any
differences between groups that might impact dependent variable measures would be
adequately accounted for in subsequent analyses. The ANOVA analyses led to the
were detected at the p < .10 level in the pre-test scores for real word reading [F(1, 169) =
2.927, p = .089], and non-word reading [F(1, 169) = 3.314, p = .070]. Significant
differences were also detected at the p < .10 level English language learner (ELL) status
[F(1,169) = 11.480, p = .001] and special education (SPED) status [F(1,169) = 3.421, p =
.066]. Therefore, a decision was taken to add pre-test achievement, ELL status, and
Test Procedure
differences in (a) non-word reading, (b) real word reading, (c) non-word spelling, (d) real
word spelling, and/or (e) reading fluency post-test achievement scores of students
142
assigned to use the MVRC online reading intervention and those of students assigned to a
once the effect of differences in pre-test achievement scores and relevant demographic
Tests of equality of variance and covariance. Levene’s test for equality of error
variance was non-significant for each of the study’s dependent variables at the p < .05
level, as was Box’s test of equality of covariance matrices at the p < .01 level, indicating
that the data did not violate the assumption of homogeneity of variance. Therefore,
consistent with the recommendations of Tabatchnick and Fidell (2013), the authors chose
main effect (λ = .668, F[5, 161] = 16.014, p < .001, multivariate η2 = .332) of the
relative to those assigned to the business-as-usual comparison condition, once the effects
English language learner (ELL) status had been accounted for. The main effects of ELL
status (λ = .868, F[5, 161] = 4.903, p < .001, multivariate η2 = .132) and pre-test scores (λ
= .212, F[5, 161] = 119.487, p < .001, multivariate η2 = .788) indicated significant effects
on the combined dependent variables (DV). Special education status, however, had no
significant effect on the combined DVs. Table 24 contains the results of the multivariate
tests for the study’s independent variable and each of its covariates.
36
Data analyses were performed using IBM SPSS GENERAL LINEAR MODEL unless otherwise noted.
143
Table 24
Multivariate Tests
between each dependent variable, the combined pre-test covariate, and the grouping
variable (condition). The multivariate test results were confirmed by the post-hoc
four of the study’s five dependent variables: real word spelling (F[1, 165] = 16.341, p <
.001, multivariate η2 = .090), non-word reading (F[1, 165] = 4.368, p = .038, multivariate
η2 = .026), non-word spelling (F[1, 165] = 29.212, p = .001, multivariate η2 = .150), and
reading fluency (F[1, 165] = 58.348, p < .001, multivariate η2 = .261). Statistically
significant effects of the intervention, however, were not detected in real word reading
(F[1, 165] = 2.328, p = .129, multivariate η2 = .014). While absolute differences were
detected favoring the treatment group, the findings lacked power, and the probability of
error was unacceptably high. Table 25 displays tests of between-subjects effects, while
144
Table 26 displays pairwise comparisons between groups for each dependent variable.
Table 25
145
Table 26
isolate the relationship between each dependent variable, its corresponding pre-test
covariate, and the grouping variable (condition). By controlling for pre-test achievement
individually for each dependent variable, rather than using the mean pre-test score
covariate employed in the full MANCOVA, the authors sought to lend more precision to
the analyses and ensure that the specific effects of pre-test achievement on each
The post-hoc ANCOVAs confirmed the results of the full MANCOVA and post-
hoc univariate tests with only one exception: the results of an ANCOVA examining
the p < .01 level (reduced from p < .05 using Bonferroni’s procedure to account for
146
multiple comparisons) in non-word reading, once the effects of differences in pre-test
achievement scores, special education status, and English language learner (ELL) status
had been accounted for. Because the ANCOVA analysis more precisely isolated the
relationship between the real word reading variable, its corresponding pre-test covariate,
and the grouping variable (condition) than did the univariate tests performed using the
combined pre-test achievement covariate, the authors chose to exclude non-word reading
from their report of significant findings. Table 27 and Table 28 provide Pairwise
ANCOVAs, respectively.
Table 27
147
Table 28
Fidelity of Implementation
Product usage logs. Product usage logs generated by the MVRC software
indicated that active participant use of the intervention exceeded 90% of the time
assigned (a mean of 44 hours of active use) among treatment group participants who
Planned activity check. Over the course of the intervention, treatment group
participants were observed 45 times while using the MVRC online intervention, or
approximately twice per week. Observers used a novel planned activity check instrument
to record behavioral observations. Data from this instrument were analyzed by dividing
the number of students displaying engagement behavior at each interval by the total
number of students observed and multiplying the dividend by 100 to determine the
for the entire twenty-minute observation period and then aggregated and averaged by
148
month. Inter-observer agreement was determined by dividing the number of agreements
between observers per interval by the total number of agreements plus disagreements and
participants displayed engagement behavior over 90% of the intervals observed while
using the MVRC online intervention. Inter-observer agreement, however, was within the
acceptable range at just under 84%. Table 29 displays the results of the planned activity
check.
Table 29
Discussion
The results of this analysis suggest a very robust effect of the MVRC intervention
on participants’ reading fluency and spelling achievement gains. The multivariate effect
size for the intervention overall was in the large range (multivariate η2 = .332), as was the
effect size for reading fluency (multivariate η2 = .261) and non-word spelling
(multivariate η2 = .150), while real word spelling (multivariate η2 = .090) was in the
149
moderate range37. While significant effects of the intervention were not detected in
isolated word reading tasks (i.e., non-word reading and real word reading), non-
possible participants in both groups had reached saturation in this aspect of decoding, as
such tasks are heavily emphasized in the Success for All curriculum, and participants in
both groups made statistically significant gains in word reading measures from pre-test to
post test. It should also be noted that unlike the Success for All curriculum, the MVRC
features that are likely responsible for the large gains in real word and non-word spelling
strong emphasis on the development of reading fluency, with sentence reading fluency,
pause-assisted fluency, passage reading fluency, and eye tracking activities of gradually
intervention may explain the particularly large gains in reading fluency favoring
broader measure of reading ability than simple decoding, and reading rate and accuracy
(i.e., fluency) are essential to reading comprehension (Pikulski & Chard, 2005).
field where most intervention research reports modest, if any, effects of the intervention.
developmentally appropriate instruction in phonics and other code-based skills, for which
there is strong evidentiary support in other research contexts. These features, similar to
37
Rules of thumb for determining the magnitude of effects were derived from Rovai, Baker, and Ponton
(2014).
150
those of the promising ABRACADABRA intervention (Savage et al., 2009; Savage et al.,
2010; Savage et al., 2013), are likely responsible for its success.
Limitations
The authors of the present study wish to acknowledge several important limitations.
Chief among these is the lack of equivalence across demographic factors between the
treatment and comparison groups. While the authors employed statistical controls to
correct for the differences, further research with random assignment to groups or a non-
participating classrooms was modest, this information would have been helpful in
focused tasks of word reading, word spelling, and reading fluency. Each of these tasks
measured aspects of basic reading and spelling skills, but none assessed the synthesis of
comprehension. While decoding ability is essential to fluent reading, the ultimate goal of
Child Left Behind and Race to the Top, fourth grade student reading achievement on the
NAEP improved only negligibly between 2003 (Grigg, Daane, Jin, & Campbell) and
2013 (National Center for Education Statistics). Because literacy is critical to informed
151
participation in the institutions and processes vital to democratic governance, systemic
change is clearly warranted. Such change, however, will require adequate time,
resources, and political will. It is therefore necessary to consider interim strategies and
Computer adaptive online reading instruction, like that provided by the MVRC
online reading intervention, holds great promise, in that it is able to tailor instruction and
classroom teacher providing large group instruction. Though computer programs lack the
insight of the experienced teacher, error analyses and responsive content presentation of
the type performed by MVRC allow for increasingly sophisticated and nuanced
how sophisticated, is unlikely ever to replace qualified teachers, but the results of the
present study suggest that it may provide a beneficial adjunct to traditional classroom
instruction.
however, the research base in ICT-based early reading interventions remains paper-thin.
further evaluate MVRC, as well as other ICT-based reading interventions commonly used
in American public schools. This research should be geared not only toward product
attention should be paid to the duration and intensity of instruction, level of instructional
152
responsiveness among students and differential levels of instructional integration and
with an established framework for high quality educational research and reporting.
153
APPENDIX A: ANCHORED MATRIX FOR
Criterion/Score 0 1 2 3
154
Criterion/Score 0 1 2 3
Quality Indicators for Implementation of the Intervention and Description of Comparison Conditions
1. The The intervention The intervention The intervention was The intervention was
intervention was was not described was inadequately adequately clearly and coherently
clearly described or specified. described or described and described and specified.
and specified. specified. specified.
2. Fidelity of Fidelity of Fidelity of Fidelity of Fidelity of
implementation implementation implementation was implementation was implementation was
was was not described inadequately adequately described clearly and coherently
described and or assessed. described or and assessed. described and assessed.
assessed.
assessed.
3. The nature of The nature of The nature of The nature of The nature of services
services provided services provided services provided in services provided in provided in comparison
in comparison in comparison comparison comparison conditions was clearly
conditions was conditions was not conditions was conditions was and coherently
described described or inadequately adequately described described and
and documented. documented. described and documented. documented.
or documented.
Criterion/Score 0 1 2 3
1. Multiple Multiple measures Multiple measures Multiple measures Multiple high quality
measures were were not used. were used but were were used, and measures were used,
used to provide an of inadequate measures were of and measures were
appropriate quality or adequate quality and carefully selected to
balance appropriateness to appropriateness to provide a balance
between measures provide a balance provide a balance between measures
closely aligned between measures between measures closely aligned with the
with the closely aligned with closely aligned with intervention and
intervention and the intervention and the intervention and measures of generalized
measures measures measures performance.
of generalized of generalized of generalized
performance. performance. performance.
2. Outcomes for No information Outcomes for Outcomes for Outcomes for capturing
capturing the was provided capturing the capturing the the intervention’s
intervention’s concerning the intervention’s intervention’s effects effects were measured
effects were timing of effects were were measured with with ample frequency
measured at measurement measured with adequate frequency and were measured at
the appropriate inadequate and were measured appropriate times.
times. frequency or at appropriate times.
measured at
inappropriate times.
155
Criterion/Score 0 1 2 3
1. The data analysis No information The data analysis The data analysis The data analysis
techniques chosen was provided techniques chosen techniques chosen techniques chosen
were appropriate and concerning data were inappropriate or were appropriate and were highly
linked in analysis were inadequately were adequately appropriate and
an integral fashion to techniques. linked to key research linked to key linked in an integral
key research questions questions and research questions fashion to key
and hypotheses. hypotheses. and hypotheses. research
questions and
hypotheses.
2. The unit of analysis No information The unit of analysis The unit of analysis The unit of analysis
chosen was clearly was provided chosen was chosen was chosen was clearly
linked to key research concerning the inadequately linked to adequately linked to and coherently
questions, hypotheses, unit of analysis. key research key research linked to key
and questions, hypotheses, questions, research questions,
statistical analyses. or hypotheses, and hypotheses, and
statistical analyses. statistical analyses. statistical analyses.
3. Did the research No information Effect size was not Effect size was Effect size was
report include not only was provided reported for all key reported for all key reported for all key
inferential statistics but concerning measures. measures, but the cell measures, and the
also effect size effect size. size was inadequate cell size was
calculations? to demonstrate adequate to
statistical power. demonstrate
statistical power.
156
Rubric for Desirable Quality Indicators
Criterion/Score 0 1 2 3
1. Data were No information Data were provided Adequate data were Data were provided
provided concerning was provided concerning attrition provided concerning concerning attrition
attrition rates. Severe concerning rates, but severe attrition rates; severe rates; severe overall
overall attrition attrition. overall attrition was overall attrition was attrition was clearly
documented and inadequately adequately explained; attrition
explained. Attrition explained; attrition explained; attrition was comparable
was comparable was not comparable was comparable across samples; and
across samples. across samples; or across samples; and overall attrition was
Overall attrition was overall attrition was overall attrition was less than 20%.
less than 30%. more than 30%. less than 30%.
4. Evidence of the No evidence of the Inadequate evidence Adequate evidence Ample evidence of
criterion-related criterion-related of the criterion- of the criterion- the criterion-related
validity and construct validity or related validity or related validity and validity and
validity of the construct validity construct validity of construct validity of construct validity of
measures was of the measures the measures was the measures was the measures was
provided. was provided. provided. provided. provided.
157
Criterion/Score 0 1 2 3
8. Results were Results were Results were Results were Results were
presented in a clear, presented in an presented in an presented in an presented in a highly
coherent fashion. unclear or inadequately clear or adequately clear and clear and coherent
incoherent fashion. coherent fashion. coherent fashion. fashion.
158
APPENDIX B: MINDPLAY VIRTUAL READING COACH
COMPONENT DESCRIPTIONS
Component Descriptions
Diagnostic assessment. All students using the MVRC intervention are subjected to
meaning, word recognition, phonetic reading, visual tracking, passage reading fluency,
generate grade equivalency scores and text Lexile measures. Vocabulary activities are
presented orally, to ensure a pure measure of vocabulary and avoid confounds arising
from poor reading ability. These assessments are computer adaptive and can take from
proficiency in any skill will reach a ceiling more quickly than a student with a greater
degree of proficiency, thus reducing the time required for testing and the likelihood of
fatigue or frustration. Figure 2, taken with permission, details the components and
RAPS360.
159
Figure 2. RAPS 360 full diagnostic assessment paths. Taken with permission
from MindPlay Educational Software for Reading.
160
Figure 3. Example RAPS360 vocabulary assessment item. Taken with permission
from MindPlay Educational Software for Reading.
the MVRC intervention targets pre-reading and emerging reading skills. Activities focus
on the identification of individual sounds: consonants and vowels, then blends and
diphthongs, and finally syllables and whole words. Place and manner of articulation are
161
segmentation activity typical of the phonemic awareness mastery component of the
MVRC intervention. Note the video window at the center left. The individual in the
video window is a reading coach, and instruction and remediation are delivered via video
Figure 4. Example phonemic awareness mastery activity. Taken with permission
from MindPlay Educational Software for Reading.
American English. Each phonics mastery lesson contains multiple interactive activities
162
feedback and remediation designed to improve areas of weakness and reinforce areas of
strength. Vowel and consonant graphemes and digraphs are taught explicitly, as are
correspondences, the place and manner of phonemic production are reinforced, in order
to further develop phonemic awareness. Orthographic patterns are taught along with
word origins and the spelling conventions associated with specific word families.
with permission, features a word identification task typical of instruction within the
phonics mastery component of the MVRC intervention. In this activity, the student is
coach articulates three words that vary in only a single phoneme. The reading coach then
directs the student to select the video in which the articulated word matches the printed
word.
163
Figure 5. Example MVRC phonics mastery activity. Taken with permission from
MindPlay Educational Software for Reading.
component of the MVRC intervention is designed to teach the patterns and underlying
order of English grammar, rather than a set of discrete rules governing English usage.
with metacognitive and mnemonic strategies to increase retention. Students first sort,
identify, and manipulate parts of speech, then phrases, sentences, and passages of
164
covered in the grammar and meaning mastery component of the MVRC intervention
include parts of speech, verb tense and agreement, sentence parts, clauses, sentence types,
features a task typical of grammar instruction within the grammar and meaning mastery
component of the MVRC intervention, and Figure 7, taken with permission, displays a
typical punctuation activity. In the first activity, the student must select the correct
conjugation in response to a question posed by the virtual reading coach, while in the
second, the student must appropriately punctuate each sentence in response to verbal and
pictorial cues.
Figure 6. Example MVRC grammar activity. Taken with permission from
MindPlay Educational Software for Reading.
165
Figure 7. Example MVRC punctuation activity. Taken with permission from
MindPlay Educational Software for Reading.
intervention is designed both to introduce and reinforce new words and to teach strategies
student’s initial level of performance, and the intervention delivers and adapts instruction
words with similar meanings, using context to derive word meanings, using word origins
to derive word meanings, using word origins to derive word spellings, and using memory
166
strategies to recall newly acquired words. Figure 8, taken with permission, features an
Figure 8. Example MVRC vocabulary activity. Taken with permission from
MindPlay Educational Software for Reading.
texts are selected from among a library of over 1,000 passages, ranging from
167
kindergarten to grade 12 in difficulty. Fluency and comprehension proficiency activities
include eye tracking exercises, in which the student tracks and identifies rapidly moving
symbols; high frequency word practice, in which the student is required to quickly
identify the most commonly used words of English; passage fluency, in which the student
fluency, in which the student must quickly read and comprehend sentences of increasing
complexity; and comprehension, in which the student reads novel passages and makes
textual inferences and answers questions relative to passage content. Figure 9, taken with
permission, features an eye tracking activity, in which the student is asked to watch
numbers as they move across the screen while counting the number of times a particular
number appears. Figure 10, taken with permission, displays a comprehension activity, in
which the student is asked to infer characteristics of the main character based upon story
content, while Figure 11 depicts a fluency activity, in which a student must rapidly read a
168
Figure 9. Example MVRC eye tracking activity. Taken with permission from
MindPlay Educational Software for Reading.
169
Figure 10. Example MVRC reading comprehension activity. Taken with
permission from MindPlay Educational Software for Reading.
170
Figure 11. Example MVRC reading fluency activity. Taken with permission from
MindPlay Educational Software for Reading.
171
APPENDIX C: PLANNED ACTIVITY CHECK
172
REFERENCES
Adams, M. J. (1990). Beginning to read: Thinking and learning about print. Cambridge,
Allen, E. A., Morey, M. E., & Hammill, D. D. (2013). Using the TOSWRF-2 to identify
students with specific learning disabilities (White paper). Council for Learning
Association.
http://www.apa.org/pubs/databases/psycarticles/index.aspx
Beck, I. L. & Juel, C. (1995). The role of decoding in learning to read. American
Bishop, M. J., & Edwards Santoro, L. E. (2006). Evaluating beginning reading software
173
Blok, H., Oostdam, R., Otter, M. E., & Overmaat, M. (2002). Computer-assisted
Browder, D. M., Wakeman, S. Y., Spooner, F., Ahlgrim-Delzell, L., & Algozzine, B.
Brunswick, N., McDougall, S., & De Mornay Davies, P. (2010). Reading and dyslexia in
Camilli, G., Vargas, S., & Yurecko, M. (2003). Teaching children to read: The fragile
link between science and federal education policy. Education Policy Analysis
Campuzano, L., Dynarski, M., Agodini, R., Rall, K., & Pendleton, A. (2009).
student cohorts (NCEE 2009–4041). Jessup, MD: National Center for Education
Cassady, J. C., & Smith, L. L. (2005). The impact of a structured integrated learning
21(4), 361-376.
Centre for the Study of Learning and Performance (n.d. a). ABRACADABRA Features.
Centre for the Study of Learning and Performance (n.d. b). Our Research. Retrieved
from: http://grover.concordia.ca/abracadabra/promo/en/research.php
174
Chambers, B., Abrami, P., Tucker, B., Slavin, R. E., Madden, N. A., Cheung, A., &
120-137.
Chambers, B., Cheung, A. C. K., Madden, N. A., Slavin, R. E., & Gifford, R. (2006).
Chambers, B., Slavin, R. E., Madden, N. A., Abrami, P. C., Tucker, B. J., Cheung, A., &
Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C.
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational
Crews, J.M. (2003) Helping poor readers: A case study of a computer assisted instruction
Information Systems.
Cronbach, L. J., & Furby, L. (1970). How we should measure change: Or should we?
175
Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational
CTB/McGraw-Hill.
Dimitrov, D. M., & Rumrill, Jr, P. D. (2003). Pretest-posttest designs and measurement
159-165.
Duke, N. K., Bennett-Armistead, V. S., & Roberts, E. M. (2003). Bridging the gap
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., & Campuzano, L.
from the first pupil cohort, (Publication No. 2007-4005), Washington, DC: US
http://www.ebscohost.com/academic/academic-search-complete
176
Edwards Santoro, L. E., & Bishop, M. J. (2010). Selecting software with caution: An
empirical evaluation of popular beginning reading software for children with early
Ehri, L.C. (2004). Teaching phonemic awareness and phonics: An explanation of the
Paul H. Brookes.
Ehri, L. C. (2005). Learning to read words: Theory, findings, and issues. Scientific
Ehri, L., Nunes, S., Willows, D., Schuster, B. V., Yaghoub-Zadeh, Z., & Shanahan, T.
36, 250-287.
Eisenberg, M. B., Lowe, C. A., & Spitzer, K. L. (2004). Information literacy: Essential
skills for the information age. Westport, CT: Greenwood Publishing Group.
Englert, C. S., Zhao, Y., Collings, N., & Romig, N. (2005). Learning to read words: The
Erickson, G. C., Foster, K. C., Foster, D. F., Torgesen, J. K., & Packer, S. (1992).
177
Fadel, C. (2008). Multimodal learning through media: What the research says. San Jose,
Foster, J. J., Barkus, E., & Yavorsky, C. (2005). Understanding and using advanced
Foster, K. C., Erickson, G. C., Foster, D. F., Brinkman, D., & Torgesen, J. K. (1994).
126–137.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S.
Girolametto, L., Weitzman, E., & Greenberg, J. (2012). Facilitating emergent literacy:
178
Grant, A., Wood, E., Gottardo, A., Evans, M. A., Phillips, L., & Savage, R. (2012).
Grigg, W., Daane M., Jin, Y., Campbell J. (2003) The Nation's report card: Reading
http://nces.ed.gov/nationsreportcard/pubs/main2002/2003521.asp
57.
Hatcher, P. J., Hulme, C., & Snowling, M. J. (2004). Explicit phoneme training combined
with phonic reading instruction helps young children at risk of reading failure.
practices in early literacy instruction (pp. 155-174). New York: The Guilford
Press.
179
Juel, C. (1998). What kind of one-on-one tutoring helps a poor reader? In C. Hulme & B.
Braun (Eds.), Reading and spelling: development and disorders (pp. 449-471).
Justice, L. M., Mashburn, A. J., Hamre, B. K, & Pianta, R. C. (2008). Quality of language
Kast, M., Meyer, M., Vögeli, C., Gross, M., & Jäncke, L. (2007). Computer-based
Kennelly, L., & Monrad, M. (2007). Approaches to dropout prevention: Heeding early
Kulik, J.A. (2003). Effects of using instructional technology in elementary and secondary
299.
Kutner, M., Greenberg, E., Jin, Y., Boyle, B., Hsu, Y. C., & Dunleavy, E. (2007).
Literacy in everyday life: Results from the 2003 National Assessment of Adult
180
Lankshear, C., & Knobel, M. (2003). New technologies in early childhood literacy
Leech, N. L., Barrett, K. C., & Morgan, G. A. (2005). SPSS for intermediate statistics:
Use and interpretation. Hove, East Sussex, United Kingdom: Psychology Press.
Lexia Learning Systems (2001). Phonics Based Reading and Strategies for Older
Macaruso, P., Hook, P. E., & McCabe, R. (2006). The efficacy of computer-based
Macaruso, P., & Rodman, A. (2011). Efficacy of computer-assisted instruction for the
172-196.
Macaruso, P., & Walker, A. (2008). The efficacy of computer-assisted instruction for
266-287.
MacGinitie, W.H., MacGinitie, R.K., Maria, K. & Dreyer, L.G. (2000). Gates-
MacGinitie Reading Tests, Level BR, Form S. Itasca, IL: Riverside Publishing.
Mather, N., Hammill, D. D., Allen, E. A., & Roberts, R. (2014). Test of Silent Word
181
Mather, N., Hammill, D. D., Allen, E. A., & Roberts, R. (2004). Test of Silent Word
McCardle, P., Scarborough, H. S., & Catts, H. W. (2001). Predicting, explaining, and
McCutchen, D., Abbott, R. D., Green, L. B., Beretvas, S. N., Cox, S., Potter, N. S.,
Quiroga, T., & Gray, A. (2002). Beginning literacy: Links among teacher
McMurray, S. (2013). An evaluation of the use of Lexia Reading software with children
MindPlay Educational Software for Reading. (2015). MindPlay Virtual Reading Coach.
Retrieved from
http://www.mindplay.com/programs/mindplay-virtual-reaching-coach/
MindPlay Educational Software for Reading. (2012). MindPlay Virtual Reading Coach
Mitchell, M. J., & Fox, B. J. (2001). The effects of computer software for developing
182
Moreno, R. (2002). Who learns best with multiple representations? Cognitive theory
Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory
32(1-2), 99-113.
214-228.
Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital
National Early Literacy Panel. 2008. Developing early literacy: Report of the National
assessment of the scientific research literature on reading and its implications for
Nelson (1998). Group Reading Test II (GRT II). London, UK: GL Assessment and the
183
Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design:
Paterson, W. A., Henry, J. J., O'Quin, K., Ceprano, M. A., & Blue, E. V. (2003).
Penuel, W., Riel, M., Krause, A., & Frank, K. (2009). Analyzing teachers' professional
Pikulski, J. J., & Chard, D. J. (2005). Fluency: Bridge between decoding and reading
Powell, D. R., Diamond, K. E., Burchinal, M. R., & Koehler, M. J. (2010). Effects of an
Reid D. K., Hresko W. P., Hammill D. D. (1989); Test of Early Reading Ability, Second
184
Rovai, A., Baker, J., & Ponton, M. (2014). Social science research design and statistics:
A practitioner’s guide to research methods and IBM SPSS analysis (2nd ed.).
Rutherford, R. B., Bullis, M., Anderson, C., & Griller-Clark, H. (2002). Youth with
71, 339-345.
Saine, N. L., Lerkkanen, M. K., Ahonen, T., Tolvanen, A., & Lyytinen, H. (2011).
Sankey, M., Birch, D., & Gardiner, M. (2012). The impact of multiple representations of
content using multimedia on learning outcomes across learning styles and modal
7(3), 18-35.
Savage, R. S., Abrami, P. C., Hipps, G., & Deault, L. (2009). A randomized controlled
Savage, R., Abrami, P. C., Piquette, N., Wood, E., Deleveaux, G., Sanghera-Sidhu, S., &
185
Savage, R. S., Erten, O., Abrami, P. C., Hipps, G., Comaskey, E., & van Lierop, D.
Savage, R. S., & Stuart, M. (2006). A developmental model of reading acquisition based
Scanlon, D.M., Gelzheiser, L.M., Vellutino, F.R., Schatschneider, C., & Sweeney, J.M.
development for classroom teachers vs. direct interventions for children. Learning
Schrank, F.A., McGrew, K.S., & Mather, N. (2014). Woodcock-Johnson IV. Rolling
Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated
21(1), 17-27.
186
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research,
78, 153-189.
with strategies for flexible and online learning. Distance Education, 24(1), 87-
103.
Slavin, R. E., Cheung, A., Groff, C., & Lake, C. (2008). Effective reading programs for
43, 290-322.
Slavin, R. E., Lake, C., Davis, S., & Madden, N. A. (2011). Effective programs for
6(1), 1-26.
Snow, C. E., Griffin, P., & Burns, M. S. (2005). Knowledge to support the teaching of
reading: preparing teachers for a changing world. San Francisco: John Wiley &
Sons.
Soe, K., Koki, S., & Chang, J. M. (2000). Effect of computer-assisted instruction (CAI)
http://www.prel.org/products/products/Effect-CAI.pdf
187
Spira, E. G., Bracken, S. S., & Fischel, J. E. (2005). Predicting improvement after first-
grade reading difficulties: The effects of oral language, emergent literacy, and
407.
Success for All Foundation. (2010). Reading Roots 4th edition teacher’s manual, level 3.
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. Boston, MA:
Pearson Education.
315.
Torgerson, C. J., Brooks, G., & Hall, J. (2006). A systematic review of the research
literature on the use of phonics in the teaching of reading and spelling. London,
http://www.dfes.gov.uk/research/data/uploadfiles/RR711_.pdf
Torgerson, C. J., & Zhu, D. (2003). A systematic review and meta-analysis of the
188
Tracey, D. H., & Young, J. W. (2007). Technology and early literacy: The impact of an
Wagner, R. K., Torgesen, J. K., & Rashotte, C. A. (1999). CTOPP: Comprehensive Test
Walsh, K., Glaser, D., & Dunne-Wilcox, D. (2006) What education schools aren’t
www.nctq.org/nctq/images/nctq_reading_study_app.pdf
Wasik, B.A. & Hindman, A. H. (2011). Improving vocabulary and pre-literacy skills of at
http://www.waterfordearlylearning.org/.
http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=19
189
Williams, R. H., & Zimmerman, D. W. (1996). Are simple gain scores obsolete? Applied
http://siteresources.worldbank.org/KFDLP/Resources/KAMBoardBriefing.pdf
Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock–Johnson III: Tests
Yoon, K. S., Duncan, T., Lee, S. W., Scarloss, B., & Shapley, K. L. (2007). Reviewing