Construct Validity Issues in The Measurement of Mo
Construct Validity Issues in The Measurement of Mo
Construct Validity Issues in The Measurement of Mo
net/publication/253574496
CITATIONS READS
2 311
2 authors, including:
Stuart A. Karabenick
University of Michigan
113 PUBLICATIONS 3,169 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Stuart A. Karabenick on 15 September 2014.
Mailing address:
Phone-734-763-1386
Fax-734-615-2164
Email: ampm@umich.edu
Presented at the biennial meeting of the Society for Research on Adolescence, San
Francisco, March 2006. Research reported herein was supported by a grant to the
Math and Science Partnership – Motivation Assessment Program (MSP-MAP) from
the National Science Foundation (EHR No. 0335369). Views expressed are the
authors’ and are not necessarily representative of the funding agency.
Validity and Motivation to Learn 2
related to the ways students think, feel, and act in schools. Evidence from research on student
learning in general (see Pintrich & Schunk, 2002; Pintrich & Maehr, 2004), and mathematics and
science in particular (e.g., Fennema, 1989; Schoenfeld, 1992), demonstrates that students’
motivation, affect, strategies, and beliefs about knowledge in these disciplines can influence their
learning and performance. Furthermore, research suggests that students’ motivation and related
outcomes are sensitive to characteristics of the learning context, including teachers’ instructional
practices as well as school and classroom climate (Ames, 1992; Anderman & Maehr, 1999;
Eccles & Midgley, 1989). It is important, therefore, for reform efforts to determine how their
programs affect student motivation, especially since such changes can precede, or even occur in
the absence of, targeted cognitive outcomes. The primary goal of the research reported on here
was to develop and make available reliable, valid, and practical tools to assess student
motivational beliefs for mathematics and science. These tools are being used with different math
and science reform projects to support evidence-based claims about the effects of their
There exist a number of different approaches to the study of motivation. For example,
consider three different approaches to the question, “What makes students want to learn in
attraction to, or liking or enjoyment of, a particular task or domain. Another perspective
conceptualizes “wanting” in terms of value, a subjective judgment of the degree to which a task
Validity and Motivation to Learn 3
or domain can fulfill needs, facilitate reaching goals, or confirm aspects of one’s self-schema. A
third approach attends to students’ goals, or their reasons for participating in achievement-related
activities. There often is considerable overlap among these constructs, with consistent, moderate
correlations among them. Though some have highlighted the importance of doing research that
considers these components simultaneously, such research has been limited to date. This paper
presents results from a large-scale study of middle and high school students that aimed to address
definition and measurement issues by considering multiple constructs deriving from different
Evidence in support of the construct validity of this set of measures to assess motivation
validity has been dealt with in different ways, with recent conceptualizations rejecting the
traditional three-part (construct, criterion, content) validity approachin favor of a more unified
validity theory (Messick, 1989; Pintrich, Wolters, & Baxter, 2000). In Messick’s (1989) unified
framework, construct validity is central and other forms of validity are subsumed under it.
validity, and described five kinds of evidence that can be used to support claims of construct
evidence concerns how well the items reflect the content of the domain. Substantive evidence
concerns the relation between data and theory, and the guiding question is whether the data
generated by the instrument are consistent with the theory. Structural evidence, on the other
hand, concerns the relation between theory and the way the data are reduced: Do the scores
obtained reflect the complexities of the theoretical model? External evidence is perhaps the most
often considered, and questions include how the instrument relates to other measures of the same
Validity and Motivation to Learn 4
construct, and whether the instrument relates to other constructs in theoretically sensible ways.
Finally, questions of generality of meaning revolve around how the findings generalize across
different populations, contexts, or subject-matter domains. In more recent work, Messick (1995)
specified an additional source of evidence, concerned with the intended and unintended
consequences of score use. This aspect of construct validity is key when tests are used for
tests. Since a complete discussion of all of Messick’s sources of evidence is beyond the scope of
this paper, the focus has been narrowed to present substantive, structural, consequential and
The set of motivation-related measures included here draws from the most often-
researched theoretical frameworks in motivation literature today. These theoretical traditions are
characterized by different approaches to the study of the basic questions most research on
motivation in education tries to answer: What makes students want to learn in school? What
makes students feel competent? How do students’ wants and beliefs in the classroom influence
whether and how they approach learning? The research described here draws heavily on
expectancy-value theory, achievement goal theory, work on personal and situational interest, and
self-efficacy theory.
Self-efficacy refers to students’ beliefs that they have the resources and confidence to do
the tasks in the classroom (Bandura, 1986; Pintrich & Schunk, 2002). It is important that self-
intervention projects make changes and improve instruction, these reforms may require students
to think differently, to do math or science differently, and to engage the material in different
ways than is usual in mathematics and science classrooms. Besides beliefs about efficacy and
Validity and Motivation to Learn 5
control, task value beliefs are another important motivational component (e.g., Eccles et al.,
1998; Pintrich & Schunk, 2002). Longitudinal research by Eccles and her colleagues (e.g.,
Eccles, et al., 1998; Fredericks, et al., 2002; Jacobs et al., 2002) has shown that student beliefs
about the importance and utility of mathematics leads them to enroll in more math courses in the
future. In addition, this research has shown that task value beliefs lead to enrollment or choices
to take more mathematics courses, but that once enrolled in the actual course, efficacy beliefs are
Personal interest refers to an individual's attraction to, or general liking and enjoyment of,
a specific activity or domain (Pintrich & Schunk, 2002). Eccles and her colleagues (Eccles, et al.,
1998) have shown that personal interest is an important component of motivation and functions
similarly to importance and utility value beliefs. In addition, other researchers have shown that
high levels of personal interest lead to more cognitive engagement, self-regulation, and
achievement (e.g., Koller, et al., 2001; Pintrich & Schunk, 2002). In many mathematics and
science reform projects, the goal is to increase student interest and positive attitudes towards
mathematics and science domains as well as interest in careers in these areas. It is an important
outcome in its own right, as well as a potentially important mediator of achievement (Koller et
al., 2001).
or students’ goals for academic learning in classroom contexts. The general distinction between
mastery and performance goals contrasts students who are mastery-oriented and focused on
learning and understanding and those students who are performance-oriented and focused on
doing better than others in terms of grades or other outcomes that invite interpersonal
comparisons (Pintrich, 2000a, b). Generally, mastery goals are positive and adaptive and lead to
Validity and Motivation to Learn 6
more interest, engagement, and learning. Performance goals, on the other hand, can be adaptive
avoid goals where students are concerned about looking dumb or trying to avoid getting the
lowest scores are clearly maladaptive and are associated with less interest, engagement, and
lower levels of performance (Pintrich, 2000a, b). As intervention projects make changes and
improve instruction, it is important to understand how these different goals may motivate
Method
Design
knowledge and tools to accurately diagnose students’ deficiencies, assess their progress,
adjust the curriculum and pedagogy, and transform the departmental culture to maximize
student learning in mathematics. Over the last two years, the partners have collaborated
to assess changes in motivation of the more than 14,000 students over the course of the
school year. Aggregated analyses of these data were disseminated to teachers and project
staff as part of professional development activities that serve as a major component of the
and in the classrooms, effecting a cultural change that creates a sustainable climate of
Validity and Motivation to Learn 7
The partnership to date has involved five waves of student motivation surveys
over two school years, as well as two waves of teacher attitude and belief surveys. Data
from the beginning and end of the first year of student surveys are presented here.
classrooms four weeks after the start of the school year and again approximately four
weeks before the end of the school year. All students in class on the day of administration
participated. Students were told that the purpose of the confidential survey was to elicit
their thoughts and feelings about the subject of math and their own math class. Students
were guided through a sample item and then completed a 110- question survey during
their math period. Items were read aloud to the middle school students; high school
students worked through the survey independently after receiving instructions from
trained research assistants. The survey took approximately 30 min. to complete. The
teacher was present in the room while the survey was being completed, but remained
Participants
Analyses presented here are based on 8,429 students (49% female) from 487
classrooms in 14 ethnically diverse, working class public middle and high schools in
Asian). Between 60 and 75% of the students in these schools were eligible to receive free
or reduced lunch. Two of the four districts have been characterized as high-need districts
Measures
math and solving math problems, task value for, and students’ personal achievement
goals. Items were rated on a 5-point Likert scale (1 = not at all true; 3 = somewhat true; 5
= very true), and all questions were worded to have students focus on the domain of
mathematics.
Task Value was measured with 18 items, which included four components
adapted from previous work. Interest (6 items, ! = .95) referred to students’ attraction to,
liking for, and enjoyment of math. (e.g., “I find math very interesting”). Utility (6 items,
! = .87) was concerned with students’ beliefs about the usefulness of math as an area of
study (e.g., “Math is useful to me for things I do outside of school”). While utility value
focused on the importance of math as a means to an end, attainment value focused on the
value of math as part of a student’s identity. Attainment value (6 items, ! = .87) referred
to students’ judgments about the importance of math for their sense of who they are (e.g.,
.81) tapped students’ judgments about the amount of effort required to be successful in
math (e.g., “Success in math requires that I give up other activities I enjoy”).
Efficacy (8 items, ! = .88) items assessed students’ judgments about their ability
and confidence to perform adequately in math (e.g., “How sure are you that you can do
even the most difficult math work”). Achievement goals (three 5-item scales items, !s =
.87, .84, .79) referred to students’ purposes when approaching, engaging in, and
focused on demonstrating ability and outperforming others (e.g., “My goal in math is to
look smarter than other students”), and performance-avoid goals focused on not looking
dumb (e.g., “My goal in math is to avoid looking like I can’t do my work”).
Substantive Evidence
Substantive evidence is concerned with the internal relations among the items in
an instrument. The guiding question is whether the data generated by the instrument are
consistent with the theory of the construct. The measures of task value and achievement
goals included in our assessment would show substantive evidence of construct validity,
according to Messick (1989), if the number and type of scores generated was consistent
with the theories from which the items were developed. In the case of task value, four
components are predicted: interest, utility, attainment, and cost. Previous studies have
sometimes had difficulty finding the predicted distinctions between utility value and
attainment value. Results from exploratory factor analyses of the task value items are
Structural Evidence
This component of construct validity asks whether the scoring of the instrument
reflects the complexities of the theory. A single total score indicates a unitary construct,
Subsumed under the structural component are issues related to scale reliability. With
older validity theories, reliability was separate from validity. This made it possible to
have scales that were valid, but not reliable, or scales that were reliable, but not valid.
With Messick’s (1989) unified approach to construct validity, issues of reliability are
factored into judgments of validity. Reliabilities for the achievement goal scales for this
Validity and Motivation to Learn 10
sample are in line with previous work, which usually shows high reliability as assessed
by indices of internal consistency (! " .85) for the mastery and performance approach
scales, but lower reliability for the avoidance scales. Table 3 shows reliabilities for both
waves, presented separately for the middle and high school students.
Consequential Evidence
A critical component of construct validity for our project concerns the intended
and unintended consequences of score use. Messick’s discussions dealt with performance
assessments or standardized tests and the use of those scores for assessment or placement
decisions. Motivation-related data are not typically associated with these same kinds of
immediate consequences for students, however the nature of our partnership is not
motivation for students. Figure 1 presents a sample report generated for a participating
school. Such reports are used in professional development workshops to help teachers
and schools find areas of strength and weakness from a motivational perspective. The
decision to present these results in their full complexity was made in response to a
would report that their students were simply “unmotivated.” Therefore, one aim of our
project was to show that there were different ways for students to be motivated (and
unmotivated), and that these different ways required different interventions from teachers
problems with Algebra 1 students was quite different from the quality of the problems
with Pre Calculus students. In response, we structured the reporting of data to support
these conversations.
The intended consequences of the use of these scores included changes in the
content of professional development and changes in the schools’ action plans. Along with
• Algebra 1A students started the year with the lowest scores on interest, mastery,
and efficacy (e.g., they saw math as less interesting than other math students, they
were less likely to focus on understanding, and were the least confident in their
math ability). However, they saw math as just as useful as other students in the
school, and had similar levels of focus on competition.
o Change –Algebra 1A students had a more adaptive pattern of change than
other students at this school; the drops were generally smaller than for
students in other courses. They saw math as less useful and were less
focused on learning but slightly more confident in their math ability.
o Goals for next year – Help students see how math is useful, and more
importantly, use TARGET TIpS to help focus students on learning and
developing (rather than just demonstrating) ability.
which the motivational data have had consequences in terms of teacher practices, and
student outcomes. In some schools, detailed yearly action plans have been revised to
include a focus on motivation. In one school the drop in student interest presented the
biggest concern for teachers, and the math department has made supporting student
interest a major focus. It is more difficult to report at this point on the unintended
generalize to other populations and contexts. Of particular interest for this set of
motivation measures are characteristics of the sample. This study included an ethnically
diverse sample of 6th through 12th students, and we found the decline in motivation across
middle and high school reported in other studies (e.g., Anderman, Maehr, & Midgley,
1999). Figures 2 and 3 show beginning and end of school year motivation profiles,
Students showed expected drops in motivation over the course of the school year.
They became less interested, saw math as less useful, and felt less confident in their
ability to understand math. In addition, they reported lower levels of achievement goals,
with lower means on all three goals. While a decreased focus on mastery goals of
The overall decrease in motivation across the school year played a smaller role in
professional development than the variability we found across schools and between
courses. Looking at variability across courses formed the basis for much of the dialogue
during professional development activities. For example, sixth graders were particularly
disadvantaged over the school year, with the greatest drop-offs over the year. They were
less interested, considered math less useful, and were less confident in their math
abilities. A positive change was the decreased focus on competition and not looking
incompetent, but this was accompanied as well by less of a focus on learning and
understanding. Targeted professional development with sixth grade teachers has focused
Validity and Motivation to Learn 13
on supporting mastery goals, interest, and value over the school year. Other questions of
populations and science intervention projects becomes available, more evidence of the
Discussion
differences are offered here as substantive, structural, and generality of meaning evidence
in support of the construct validity of this set of measures of motivation to learn. Further,
a discussion of the ways in which these data have been reported and used serves as
consequential evidence. Though these sources of evidence have been separated for clarity
of discussion, it is important to remember that there exists considerable overlap, and that
these aspects of construct validity are part of a unified validity theory that does not rely
References
Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of
Anderman, E. M., M. L. Maehr, et al. (1999). Declining motivation after the transition to
middle school: Schools can make a difference. Journal of Research and Development
Eccles, J., Wigfield, A., Harold, R.D., & Blumenfeld, P. (1993). Age and gender
differences in children’s self- and task perceptions during elementary school. Child
Eccles, J., Wigfield, A., & Schiefele, U. (1998). Motivation to succeed. In W. Damon
(Series Ed.) & N. Eisenberg (Vol. Ed.), Handbook of child psychology: Vol. 3.
Social, emotional, and personality development (5th ed., pp. 1017-1095). New York:
Wiley.
Fennema, E. (1989). The study of affect and mathematics: A proposed generic model for
research. In D. McLeod & V. Adams, (Eds.), Affect and mathematical problem solving: A
Koller, O., Baumert, J., & Schnabel, K. (2001). Does interest matter? The relationship between
Maehr, M. L., & Karabenick, S. A. (April, 2004). Tools for the evaluation of non-cognitive
outcomes of math and science instruction. Paper presented as part of a symposium entitled:
Validity and Motivation to Learn 15
Supporting evidence-based claims for impacts of science and mathematics reform on student
outcomes, at the annual meeting of the American Educational Research Association, San
Diego, CA.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-
responses and performances as scientific inquiry into score meaning. American Psychologist,
50(9), 741-749.
Pintrich, P.R. (2000a). Multiple goals, multiple pathways: The role of goal orientation in
Pintrich, P.R. (2000b). The role of goal orientation in self-regulated learning. In M. Boekaerts,
P.R. Pintrich, & M. Zeidner, (Eds.). Handbook of self-regulation (pp. 451-502). San Diego,
Pintrich, P.R., & Schunk, D.H. (2002). Motivation in education: Theory, research, and
applications (2nd ed.). Upper Saddle River, NJ: Merrill Prentice Hall.
Pintrich, P.R., & Maehr, M.L. (Eds.). (2004). Advances in motivation and achievement:
Motivating students, improving schools (Vol. 13). Oxford, England: JAI, An Imprint of
Elsevier Science.
Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). Assessing metacognition and self-
Table 1
Factor Loadings for Task Value Measures at Beginning of School Year (N = 8,429)
1 2 3 4
1. I enjoy doing math. 0.95
2. I like math. 0.93
3. I enjoy the subject of math. 0.88
4. How much do you like doing math? 0.85
5. Math is exciting to me. 0.83
6. I am fascinated by math. 0.75
7. Math will be useful for me later in
life. 0.94
8. Math concepts are valuable because
they will help me in the future. 0.83
9. How useful is learning math for what
you want to do after you graduate and
go to work? 0.71
10. In general, how useful is what you
learn in math? 0.60
11. Being good at math will be important
when I get a job or go to college. 0.52
12. Compared to most of your other
school subjects, how useful is what
you learn in math? 0.46
13. I have to give up a lot to do well in
math. 0.79
14. Success in math requires that I give
up other activities I enjoy. 0.77
15. It is important for me to be someone
who is good at solving problems that
involve math. 0.79
16. Being someone who is good at math
is important to me. 0.79
17. Being good at math is an important
part of who I am. 0.74
18. It is important to me to be a person
who reasons mathematically. 0.63
19. I feel that, to me, being good at
solving problems which involve math
or reasoning mathematically is 0.61
20. Thinking mathematically is an
important part of who I am. 0.55
Note: Factor loadings under .20 have been omitted.
Validity and Motivation to Learn 18
Table 2
Factor Loadings for Task Value Measures at End of School Year (N = 8,429)
1 2 3 4
1. I enjoy doing math. 0.96
2. I like math. 0.94
3. I enjoy the subject of math. 0.86
4. How much do you like doing math? 0.86
5. Math is exciting to me. 0.83
6. I am fascinated by math. 0.76
7. Math will be useful for me later in life. 0.96
8. Math concepts are valuable because
they will help me in the future. 0.88
9. How useful is learning math for what
you want to do after you graduate and
go to work? 0.75
10. In general, how useful is what you
learn in math? 0.65
11. Being good at math will be
important when I get a job or go to
college. 0.61
12. Compared to most of your other
school subjects, how useful is what
you learn in math? 0.51
13. I have to give up a lot to do well in
math. 0.83
14. Success in math requires that I give
up other activities I enjoy. 0.82
15. It is important for me to be someone
who is good at solving problems that
involve math. 0.81
16. Being someone who is good at math
is important to me. 0.80
17. Being good at math is an important
part of who I am. 0.80
18. It is important to me to be a person
who reasons mathematically. 0.73
19. I feel that, to me, being good at
solving problems which involve
math or reasoning mathematically is 0.65
20. Thinking mathematically is an
important part of who I am. 0.63
Note: Factor loadings under .20 have been omitted.
Validity and Motivation to Learn 19
Table 3
Reliabilities for Efficacy, Task Value, and Achievement Goal Measures for Beginning of
Efficacy
0.89 0.87 0.88
Validity and Motivation to Learn 20
Table 4
Reliabilities for Efficacy, Task Value, and Achievement Goal Measures for End of School
Year (N = 8,429)
Efficacy
0.91 0.91 0.91
Validity and Motivation to Learn 21
Figure 1
Figure 2
Students (N = 8,429)
Validity and Motivation to Learn 23
Figure 3
Cross-sections of End-of-Year Motivation Profiles for Middle and High School Students
(N = 8,429)