BOOK Research Design in Counseling 3rd Ed PDF
BOOK Research Design in Counseling 3rd Ed PDF
BOOK Research Design in Counseling 3rd Ed PDF
in Counseling
THIRD EDITION
P. Paul Heppner
University of Missouri, Columbia
Bruce E. Wampold
University of Wisconsin, Madison
To Mary Suzanne, Mary Clayton, and Martin, my real research support group.
D.M.K.
This page intentionally left blank
Contents
PREFACE xvi
Beneficence 110
Autonomy 111
Justice 111
Fidelity 112
Virtue Ethics 112
Ethical Issues Related to Scholarly Work 113
Execution of the Research Study 114
Reporting the Results 115
Duplicate and Piecemeal Publication 117
Publication Credit 118
Plagiarism 121
Ethical Issues Related to Participants 123
Risks and Benefits 125
Consent 128
Deception and Debriefing 131
Confidentiality and Privacy 133
Treatment Issues 136
Responding to Ethical Dilemmas 137
Summary and Conclusions 143
Creating Analogue Studies that More Closely Resemble Real Life 418
Evaluating Analogue Utility Within an Existing Knowledge Base 419
Summary and Conclusions 421
REFERENCES 621
The seeds of great discoveries are constantly floating around, but they only
take root in minds well prepared to receive them.
—Joseph Henry
xvi
preface xvii
the ideas of the chapters and talking to others about them, these skills of
designing rigorous research can become integrated more thoroughly into the
way students think about important issues in the field. In addition, we have added
three new chapters, one focusing exclusively on multicultural issues, another
on scale construction, and a third on research training. We have greatly
revised the other chapters as well to have a stronger emphasis on diversity
issues to portray the increasing breadth of knowledge bases across popula-
tions, as well as the increasing complexities in the counseling profession.
There are several challenges in teaching the next generation of counselors
and counseling psychologists to be inquisitive and competent researchers.
Perhaps first and foremost, many students learn about research methods in
general research methods courses in education or psychology, which utilize
what seem like very distant and even sometimes irrelevant examples from
other disciplines. Subsequently, students in the counseling profession often do
not see the utility or full applicability of research methods taught abstractly or
apart from the typical domains of counseling research. In addition, when
counseling students learn research methods when applied to, let’s say clinical
psychology or higher education, our students do not learn about the recent
counseling literature and the creative approaches being utilized by contemporary
scholars in our profession. Such a lack of knowledge and role models makes it
difficult for students to acquire strong research interests in topics related to the
counseling profession. Although the book is not comprehensive of all research
in counseling, it provides beginning graduate students an overview of some of
the studies of many of the foremost researchers in the profession.
As with the first and second editions, our goal for this new edition was to
facilitate a conceptual understanding of research design, with an emphasis on
acknowledging the strengths and weaknesses of all the designs including
quantitative and qualitative. Moreover, in this edition, we devote a chapter to
some of the unique conceptual and methodological issues related to research
addressing multicultural topics. The book does not favor one design over
another per se, but rather often emphasizes the need to consider the types of
questions being addressed, as well as the inherent strengths and weaknesses in
the previous research literature.
The third edition is divided once again into four parts. Part 1 focuses on
Philosophical, Ethical, and Training Issues. Basic philosophical and para-
digmatic issues are introduced, as are the ethical responsibilities of the
researcher. Each of the chapters has been updated with current conceptual
perspectives and methodological issues. A major change from the second edition
is in Chapter 1, where we provide a more extensive and broadened discussion
of the philosophical foundations of counseling research and qualitative ap-
proaches. Moreover, we broadened the focus on subsequent chapters to more
explicitly include qualitative approaches, such as in discussions relating to
research questions and hypotheses. A second major change in Part 1 is a
chapter totally devoted to graduate research training (Chapter 2). We not only
highlight the current research on this topic, but also discuss implications
xviii preface
effective than other interventions; these issues are discussed and various
procedures to establish validity within outcome research are presented.
We particularly want to highlight three chapters in this section. First, we
point the reader to Chapter 15 on conceptual and methodological issues related
to multicultural research, which was written by Dr. Kevin Cokley and Dr. Germine
H. Awad. Their careful thought and keen conceptual work raises many critical
questions for researchers to consider in the field of multicultural research.
Although we have attempted to infuse issues of diversity throughout the text,
this new chapter brings together the latest thinking on these issues, and we
hope it will be of great support to all researchers. Another new chapter
(Chapter 20), which discusses scale construction issues, was written by
Dr. Dong-gwi Lee and Dr. Hyun-Woo Lim. Our research can only be as strong
as the measurement of our constructs, so this is a critical chapter for researchers
who are conducting scale construction projects, as well as those who want to
understand more about the psychometric properties of existing inventories.
Drs. Lee and Lim not only take the reader through the steps of scale con-
struction, but also offer a cross-national perspective on creating and validating
scales with cross-national populations. This addition is empirically important
to our field as the counseling profession becomes increasingly global in scope.
The third chapter we particularly want to highlight is Chapter 21 on program
evaluation, which has been updated by Dr. Matrese Benkofske and Clyde C.
Heppner. The field of counseling has increasingly become one that emphasizes
intervention and social change at the program and systems level. Many coun-
selors have jobs where conducting program evaluation and assessing the
efficacy of interventions is a critical part of their position. This chapter offers
many insights about the field of program evaluation and provides a great deal of
information and support for researchers involved in assessing the effectiveness
of programs and services.
Part 4 of the book, “Professional Issues,” includes a chapter on profes-
sional writing. This chapter has been substantially revised to include as many
helpful tools for scholars as possible. A great deal of research is being con-
ducted by counselors, but good research questions and even well-designed
studies often do not get published in our journals. Certainly good research
questions and well-designed studies are necessary components, but strong
writing skills are needed to accurately depict the study and explain the results.
This chapter is aimed at helping students and young researchers with the
unique skills of scientific writing.
We have so many individuals to thank for their contributions to this
book. We especially want to thank the staff of Brooks/Cole for all of the tech-
nical support, patience, and help through this revision process; we very much
appreciate the guidance of Caroline Concilla and Marquita Flemming, as well
as Samantha Shook for her technical assistance. In addition, we are very
appreciative of Aaron Downey at Matrix Productions and our copyeditor,
Toni Ackley, for her very careful and thorough polishing of the text. Although
a number of staff have changed over the years, the early vision of Claire
xx preface
Verduin lives on in this third edition, and she deserves special attention for
her good judgment and patience in developing the first edition of this book.
We also want to thank Young-Ju Cho and M. Meghan Davidson, both
graduate students, as well as Amy Simpson, an undergraduate student, all at
the University of Missouri-Columbia, and Zachary Imel at the University of
Wisconsin, who helped to locate relevant articles, compile references, and
assisted in writing and editing; their contributions were deeply appreciated.
In addition, we thank the authors of the chapters, who added greatly to
the depth and breadth of critical topics within research design. And a special
thanks to the researchers who have been willing to share some of their jour-
ney of becoming researchers; their voices added greatly to the discussion of
the developmental process of becoming a researcher. In addition, we want to
thank so many of you for sending in your comments about the book—how it was
helpful to you and also where you would like more examples or suggestions.
We are also deeply appreciative of the scholars who provided detailed
reviews of the second and early drafts of the third edition of this book; your
feedback was very helpful in guiding the third edition: Richard Balkin, Texas
A&M University at Commerce; Janet Carlson, Texas A&M University at
Galveston; Y. Barry Cheung, Georgia State University; Stephen Cook, Texas
Tech University; David Dixon, Ball State University; Marie S. Hammond,
Tennessee State University; Steven Hoover, St. Cloud State University;
Marlene Hyde, San Diego State University; Michael LeBlanc, SUNY Oswego;
Ed Mauzey, Southeastern Oklahoma State University; Karen Multon,
University of Missouri at Columbia; Joseph Ponterotto, Fordham University
at Lincoln Center; and Christopher Stankoich, University of Dayton. In
essence, we hope many of you see your voices and suggestions reflected in this
new edition.
Thanks also to the following students who so graciously gave of their
time to contribute to the final outcome of the book, and in doing so enhanced
the quality: Chiao, Hung; Meghan Davidson; Lizette Ojeda; Amy Simpson;
and Tsai, Chia-Lin.
Finally, we would like to thank all of the authors of the articles and books
that we cite in this book; their dedication to conducting high quality
research provides the profession not only with important new knowledge, but
also with very helpful models on the “how to’s” of conducting meaningful
research.
As American folklorist Zora Neale Hurston commented “Research is for-
malized curiosity. It is poking and prying with a purpose.” We hope this text
gives you many tools to “poke and pry with a purpose” and discover important
truths for the betterment of humankind.
Philosophical, Ethical,
and Training Issues
PART
1
1
1
CHAPTER
Science and Counseling
2
chapter 1 science and counseling 3
has been fixed on new disks. We’ll send you a new set.” One wonders how many
other career centers never found this error and continued to use a program that
gave users blatantly invalid results. (Johnston, Buescher, & Heppner, 1988, p. 40)
This example involves a computer programming error that was not caught
through careful evaluation. Many other examples could be listed in which
clients receive less than desirable treatments because of outdated information,
ineffective or inappropriate counselor interventions, or erroneous knowledge
about human behavior and the change process.
Medical professionals also aid people, although they obviously focus
primarily on medical problems. The medical profession has advanced over the
centuries and has become increasingly more sophisticated. Important lessons
can be learned from the trials and tribulations of the medical profession.
Consider the historical lesson from the so-called thalidomide babies. In the
early 1960s a drug called thalidomide was prescribed in England, Canada, the
United States, and several other countries for pregnant women experiencing
morning sickness. The drug was administered before adequate empirical tests
had been completed. Some medical scientists in England argued that the effects
of the drug should have been tested scientifically, especially in comparison with
groups of women who did not receive the drug. Others, however, argued more
convincingly that it was unethical to withhold (from the comparison group) a
drug that was “known” to greatly ease women’s problems with pregnancy.
For some time after the drug was introduced, a number of medical profes-
sionals observed an increase in the number of deformed babies whose arms
and legs resembled buds that precede limb development in the human embryo.
Years later, after appropriate empirical tests had been completed, researchers
discovered that thalidomide administered to women during the critical embry-
onic period caused these major deformities in the babies. Although the drug
was quickly taken off the market, for thousands of infants the damage had
already been done. How do we know whether we are promoting the welfare of
or actually harming those who seek our services in the counseling profession?
None of us in the profession would intentionally want to harm clients.
Counseling, however, can have detrimental effects on people (e.g., Lambert,
Bergin, & Collins, 1977).
How do we know our interventions are effective? This question has led to
identifying what were referred to initially as empirically validated treatments
(Wampold, Mondin, Moody, Stich, Benson, & Ahn, 1997) and more recently
as evidence-based practice (Chwalisz, 2003). In addition, this has led to the
“great psychotherapy debate,” regarding which therapy models and methods
are most effective. (For more details, see Wampold, 2001, who not only traces
this debate over time, but also concludes that successful outcomes are more
dependent upon general therapeutic effects than any one theory or method.)
What is sufficient proof that we as a profession can afford to accept? If some-
one proclaims that a certain intervention is effective, should we believe it? If
your supervisor maintains that a certain technique is effective, is that sufficient
evidence? What kind of knowledge must a profession be based on to succeed?
4 part 1 philosophical, ethical, and training issues
The answers to these questions rest on the manner in which the profession has
developed its knowledge base.
Consider this true story of a group of faculty who were ardent believers in
Freud’s conception of psychoanalysis. These faculty members were relatively isolated
from other professionals and had a habit of hiring only their own graduates.
(They noted that because they rarely had a faculty opening and the job market
was so restricted, they would feel like traitors if they hired someone else’s student
instead of one of their own.) These people believed that clients’ paranoid fantasies
were major obstacles to uncovering unconscious psychological conflicts.
Consequently, they would not allow any live recording (audio or visual)
or direct observation of their therapy sessions. Nor would they allow any
kind of written self-report data to be collected from clients. Their primary
method of knowing seemed to be the method of authority (Freud), and it
included little opportunity to objectively confirm or disconfirm Freud’s ideas.
Moreover, they so firmly believed in their truths that they scoffed at the utility
of other therapeutic techniques such as systematic desensitization, the Gestalt
empty chair, and reflection. Consequently, this group of psychologists, in the
absence of any objective data, discovered very little beyond Freud’s early for-
mulations of the therapeutic process. The bottom line is that this group did
not advance the knowledge of their field; by today’s standards (as well as their
students’ evaluations), their therapy practices were archaic, and their training
philosophies and methods totally inadequate.
The purpose of this chapter is to examine how the counseling profession has
developed its knowledge base. This chapter focuses on the role of science in the
counseling profession, including ways of knowing, philosophical foundations of
science, and our views of four key issues for a philosophy of science in counseling.
have always known them to be true; Kerlinger and Lee (2000) noted that fre-
quent repetition of these “truths” seems to enhance their validity. A second
method of knowing is the method of authority. If noted authorities such as the
president of the United States, a state governor, a well-known psychologist, or a
clinical supervisor say it is so, then it is the truth. A third method of knowing is
the a priori method, or method of intuition (e.g., Cohen & Nagel, 1934). This
method is based on the notion that what agrees with reason, what makes sense, is
true. The fourth method of knowing is the scientific method, which involves empir-
ical tests to establish verifiable facts. We would add a fifth way of knowing—what
is learned through one’s own direct experiences in the world. Through countless
experiences, each individual construes a “reality” of the world; some of his or her
perceptions may match those of others with similar experiences, whereas other
perceptions and conclusions about the world may not match those of others.
Dangers exist if this method is used alone because biases can develop or infor-
mation can be distorted. Moreover, the events we experience can represent a
biased sample, which in turn can lead to inaccurate conclusions.
Given the overwhelming complexity of life and the vast amounts of knowl-
edge needed even in daily living, people most likely acquire “truths” through
all five of these ways of knowing. Obviously, error can be involved in any of
them. Such error, if it affects the knowledge on which counseling is based, can
be dangerous for the counseling profession and our clients. To be credible, reli-
able, and effective, a profession must be built on dependable facts or truths,
rather than on tenacity, decrees from authority figures, or subjective opinions.
A profession that aims to facilitate growth and positive change in clients
must be based as much as possible on knowledge that exists in a reality out-
side of professionals’ personal beliefs and biases. The scientific method has
been developed to create such knowledge.
Basically, the scientific method is a set of assumptions and rules about col-
lecting and evaluating data. The explicitly stated assumptions and rules enable
a standard, systematic method of investigation that is designed to reduce bias as
much as possible. Central to the scientific method is the collection of data that
allows investigators to put their ideas to an empirical test, outside of or apart
from their personal biases. In essence, the proof of the science is in the data.
“Stripped of all its glamour, scientific inquiry is nothing more than a way of lim-
iting false conclusions about natural events” (Ruch & Zimbardo, 1970, p. 31).
There are obvious costs to acquiring knowledge by using the scientific
method. Conducting empirical investigations is costly in terms of time, energy,
and resources. Putting complex and internal cognitive and affective processes
to empirical test is a difficult and elusive task. Sometimes when we try to identify
specific processes or variables we become mechanistic and lose the gestalt, or
whole picture. Sometimes the lack of sophistication of our research methods
results in conclusions that tell us little about real-life processes.
But the risks of building a profession on nonscientific evidence are far
greater. The thalidomide babies are one clear example of the risks associated
with not empirically testing one’s opinions. Conducting therapy based only on
personal hunches and opinions is risky and might well result in harming clients
6 part 1 philosophical, ethical, and training issues
(e.g., Lambert, Bergin, & Collins, 1977). It is important that the knowledge on
which the profession is built be based on objective or verifiable information that
can be put to empirical or quantifiable tests. In this way, the methods used to
establish our “truths” have a built-in self-correction process; each empirical test
is independent of previous findings and can either verify or disconfirm the pre-
vious knowledge. In contrast, subjective ways of knowing that do not involve
empirical tests run the risk of creating myths. These myths can result in ineffec-
tive or even harmful counseling, and hinder the progress of a profession.
This does not mean that the professionals’ beliefs, hunches, and even biases
are not useful in exploring ideas and perhaps extending the field’s knowledge.
We can undoubtedly learn a great deal about human behavior from the more
subjective ways of knowing; it is clear that many ideas and breakthroughs
regarding therapeutic orientations and techniques have initially sprung from
practitioners’ direct experience with people. However, it is important to note
that these ideas must be empirically tested. In fact, no major orientation has
been maintained in the profession without substantial empirical support. Par-
enthetically, even though the scientific method tends to provide data that are
prone to less bias or distortion, Howard (1982) cogently recommended that we
“periodically obtain evidence demonstrating the adequacy” of the various as-
sumptions or procedures involved in the scientific method (p. 324).
In short, the knowledge of a profession must be empirically based and
verifiable rather than subjective and untestable. Even though the scientific
method has costs and is not problem-free, building a helping profession with-
out it is too risky. Without a strong scientific foundation, the credibility of a
profession is significantly challenged.
discover the natural laws that govern each and every object in the universe, and
their relationship to each other. Physical laws that describe gravitation, magnet-
ism, and electricity are examples of positivistic statements about the universe that
are universal in terms of both time and context. A key principle is that “truth”
exists, and given enough time, brilliant scientists, and sophisticated methods, dis-
coveries will be made that illuminate the truth. In the positivistic realm, the sci-
entist is “objective”; that is, the scientist neither affects the world that is studied
nor is affected by it. Scientists are interchangeable in that a given experiment
should lead to the same outcome and conclusion, regardless of who conducts it.
Surely, some scientists are more insightful and creative than others, but in the
end, experiments yield results that are self-evident to the scientific community.
chapter 1 science and counseling 9
Confidence in results derives from the scientific method, not from the scientist;
the operations of the experiment produce the results while the scientist observes
objectively from the sidelines.
The scientific method involves well-defined steps. First, the scientist makes
a conjecture about the nature of the universe. After that, the scientist designs
an experiment such that its results will either confirm or disconfirm the con-
jecture. Knowledge, as opposed to mere opinion, is contained only in state-
ments based on or linked to direct observation. The only kinds of statements
free from personal bias (and thus distortion) are those grounded in observation.
If the data conform to the prediction, the conjecture is verified. On the other
hand, if the data do not conform to the prediction, the scientist concludes that
the phenomenon being studied does not follow the conjecture, which is then
10 part 1 philosophical, ethical, and training issues
with the participant. At the most extreme level, dialectics involve changing
constructions in the process of interpretation. This last level of dialectics (and
some would say the essence of dialectics) is more characteristic of critical
theory (see the next section) than of constructivism.
In the constructivist paradigm, there are no truths to be discovered; there-
fore, there can be no conjectures (that is, predictions based on hypothesized
truths) or tests of conjectures. Thus, data are not collected with the aim of deter-
mining whether or not observations are consistent with conjecture. Rather,
data lead to interpretations that then lead the investigator in directions that
may not have been anticipated, causing the investigator to reinterpret already
collected data or to collect additional data, often in ways unimaginable when
the investigation began. Positivistic and postpositivistic methods are linear,
whereas constructivist (and critical theory) methods are recursive (that is, the
results and method influence each other).
Critical Theory Critical theory posits that people’s social constructions are
shaped by the social, political, cultural, historical, and economic forces in the
environment, particularly forces created by powerful individuals. Over time,
the constructions take on the appearance of reality; that is, the social reality,
which has in fact grown out of the social context, is assumed to be truth.
Because the constructions are so deeply embedded in society (including in the
researchers themselves), it is extremely difficult to comprehend that these
constructions were spawned in the societal context and are not truths. For
example (and any examples chosen are necessarily controversial), the belief
that the monogamous union of one male and one female for the purpose of
reproduction (that is, heterosexual marriage) is natural is a socially derived
position.
Critical theorists would concede that it could be argued that marriage is
necessary and important for the social order (as we know it), but they would
contend that marriage, as an institution, was generated by the social system;
that there are alternatives (same-sex unions, polygamous marriages); and that
the “truth” of any “natural” propensity to marry is specious.
Ponterotto (2005b) reminded us that “there is no single critical theory”
but “there are commonalities among the variants of critical theory” (p. 130).
In critical theory, the investigator and the participant form a relationship, and
the values of the investigator are vital to the activity. Inquiry, in critical theory,
involves the level of dialectism that changes constructions. That is, the inves-
tigation involves a dialogue between investigator and other in such a way that
the other comes to realize that her or his understanding of the world is derived
from the precepts of the social order, and that these precepts can (and should)
be altered. In other words, the goal of critical theory is to have the participants
view structures for what they are—socially constructed beliefs—rather than as
unchangeable truths. Moreover, the dialectic should lead to the participants’
understanding that social action is needed to change the social order, thereby
being emancipated from oppression (e.g., oppression resulting from racism,
classism, able-bodism, heterosexism, or sexism).
14 part 1 philosophical, ethical, and training issues
For example, feminist theory falls into the critical theoretical realm in that
it contends that traditional roles for women have been socially determined, that
the power in society has been allocated to males, and that these social realities
can be altered. Feminism seeks to “raise the consciousness” of women so that
they do not consider their place in society to be fixed as truth, but instead under-
stand both the historical context that led to the current social situation and that
the first step in change is to reject the traditional roles. Many critical theorists
would contend that this worldview involves more than social action, which
tends to change society at the margins, and instead necessitates radical change
that dramatically replaces current social structures with others (e.g., Marxism).
research can be evaluated by how relevant the findings are for practitioners
(Krumboltz & Mitchell, 1979). Scientific research in counseling can thus ad-
vance our knowledge base or understanding of human behavior by providing
data that describe and help us understand a wide range of human behaviors,
and how such behaviors can be altered through counseling interventions.
It is also important to develop knowledge bases and research perspectives
that emphasize the social and historical context of the individual. A common
defining element of the counseling profession is that we typically conceptualize
a person’s behavior as a function of the environment that they experience (Fretz,
1982). People do not think, feel, or behave in isolation, but rather in the con-
text of a rich personal and social history. Research that increases understanding
of how individuals interact within a broader social and personal environmental
context is crucial to the development of knowledge about counseling. Thus, the
goal of science is to expand our knowledge not only about individuals, but also
about the interactions between individuals and a larger personal, social, cul-
tural, and historical context. In fact, it has been maintained that to ignore the
larger social, cultural, and historical context ignores critical elements in under-
standing current behavior, and in this sense can lead to ineffective and even
inappropriate interventions and unethical behavior (American Psychological
Association [APA], 2003; Toporek & Williams, 2005).
Research, however, is guided by more than practical problems and societal
needs. To achieve scientific understanding, the researcher often needs to
organize observations and facts into a logical framework that explains some
aspect of behavior. Thus, research is often guided by theoretical issues within
a line of work and seeks to establish general relations and conditional state-
ments among events that help professionals to understand phenomena. The
accumulation of facts or knowledge will not likely result in general laws or
broad scale theories of human behavior as it was earlier conceived. Human
behavior is multi-determined; that is, a single action can be determined by any
one of several preceding events. Moreover, human actions consist of complex
chains in which preceding events increase or decrease the probability that
some subsequent action will occur, but behavior is not a uniform process
across individuals or even within individuals over time. Meehl (1978) likewise
concluded that for a variety of reasons (such as individual differences, poly-
genic heredity, random events, nuisance variables, and cultural factors), hu-
man psychology is difficult to scientize and that “it may be that the nature of
the subject matter in most of personology and social psychology is inherently
incapable of permitting theories with sufficient conceptual power” (p. 829).
Thus, the range of human variability and complexity does not much lend itself to
description by general principles or broad theories, and even less to prediction.
In short, we are suggesting that it is exceedingly difficult to develop broad
scale theories aimed at predicting human behavior in general. However, skilled
therapists are able to make better predictions about individual people when they
combine research knowledge about specific relations among variables with a
host of qualifying information, namely the biographical, social, and cultural history
of the individual. In this way, therapists use “the discoveries of science, but in
chapter 1 science and counseling 17
order to bring about changes in the everyday world, also employ a great deal of
knowledge that extends beyond science” (Manicas & Secord, 1983, p. 412).
Thus, it is useful for counseling professionals to continue to organize facts and
knowledge into theoretical frameworks that can be used as ingredients within
more complex and conditional models of behavior. Theoretical frameworks that
consist of sets of conditional statements that can be qualified by specific infor-
mation about an individual may allow both the needed specificity and com-
plexity in explaining and predicting individuals’ behavior. In sum, we believe
that the second function of science is to promote understanding, as well as help
explain and predict human action, but in a much more complex and idiographic
manner than acknowledged in the traditional received view.
some specific alternative procedure, the misinterpretation might have been recog-
nized. For example, our researcher might have considered an alternative research
method: namely damming a small inlet of the sea, draining the water, and examin-
ing the bodies of the fish left behind. In finding fish smaller than two inches, the
limitations of the netting procedure would become apparent. One would not be
surprised, however, to find that the largest fish obtained via the damming approach
was substantially smaller than was obtained with the netting approach: another
potential problem. Therefore, research testing the adequacy of research methods
does not prove which technique is better but provides evidence for the potential
strengths and limitations of each. From this information, researchers can determine
when one of two approaches, or both, should be the method of choice. (p. 319)
The Need to Examine and Expand Our View of Human Nature The assumptions
one makes regarding the basic qualities of human nature (that is, cognitive,
affective, behavioral, and physiological processes) affect how one conceptualizes
human behavior. Moreover, our view of human nature affects the research prob-
lems we examine in counseling. Our views of human nature have changed dra-
matically in the past century and are still evolving. Consistent with the beliefs of
some of the previously mentioned writers (e.g., Borgen, 1984a; Howard, 1984;
Patton, 1984; Polkinghorne, 1984), we believe there is a need to expand our view
of how human beings operate, particularly within counseling. One increasingly
accepted major change of view concerning human nature pertains to human
rationality. For example, Gelatt (1989) noted that his view of human decision
making, especially within a counseling context, changed dramatically in the
previous 25 years. In 1962 he stressed rational processes in decision making,
whereas in 1989 he stressed intuitive processes and positive uncertainty (that is,
accepting uncertainty and inconsistencies). Gelatt’s 1989 perspective was con-
sistent with those of a host of other writers who at that time emphasized non-
rational and unsystematic processes (chance events or luck) within human
decision making (e.g., Heppner, 1989; Meehl, 1978; Strohmer & Blustein,
1990; Tversky & Kahneman, 1981). Meehl has cogently argued that our view
of human nature should also include chance events: “luck is one of the most
important contributors to individual differences in human suffering, satisfaction,
illness, achievement, and so forth, an embarrassingly ‘obvious’ point that social
scientists readily forget” (Meehl, 1978, p. 811).
Over the last 20 years, a number of suggestions have been made for how
we might expand our view of human beings, such as human agency (e.g.,
Howard, 1984), phenomenological perspectives within language (e.g., Patton,
1984; Pepinsky, 1984), cognitive mediational processes (e.g., Martin, 1984),
and information processing (e.g., Heppner & Krauskopf, 1987), particularly in
nonlinear causal chains (e.g., Ford, 1984; Maruyama, 1963). It is striking that
all these suggestions are process-oriented, thus suggesting that it may be fruitful
to examine more dynamic and microscopic processes at this point within coun-
seling research. Our view of human nature also pertains to our worldview, and
chapter 1 science and counseling 19
Although our science has clearly affected counseling practice today, the
essential point is that the methods of science are only the tools we use to
obtain knowledge about phenomena. A flashlight is a good analogy. A flash-
light is a useful tool, but it will shine light only where we point it. If we cannot
find the object we are looking for with a flashlight, it does not necessarily
follow that we should throw the flashlight away, but rather that we should
change the direction of the light. Similarly, our research methods will give us
information only about the content we examine. If we are dissatisfied with the
results, it does not necessarily follow that we should eliminate the research
methods, but rather that we should try new angles with our research methods.
It is possible, however, that we may need a bigger flashlight, or perhaps a
flashlight that can bend around a corner. Thus, sometimes new research
methodologies may be needed to help us acquire new or different types of
knowledge. Developing new methodologies or alternative ways of collecting
data obviously challenges the problem-solving and creative abilities of researchers.
Presently, there are many ideas that we cannot examine adequately because
we do not have the appropriate methodologies or measurement instruments.
Researchers must be creative and versatile not only in the methodologies they
use, but also in the types of data they collect in examining the phenomena that
are central to counseling and human development.
20 part 1 philosophical, ethical, and training issues
the ability of the members of the counseling pro- of all of our members to question assumptions,
fession to appropriately apply our scientific biases, and stereotypes, and to think scientifi-
knowledge to facilitate the development of a cally is of utmost importance, and will be dis-
diverse clientele is essential; to this end, the ability cussed more fully in the next chapter.
STIMULUS QUESTIONS
Reflections on Science in Counseling
The purpose of the following exercise is to see as some of the advantages and disadvan-
promote additional reflection on the role of tages of this paradigm?
science and practice in the field of counseling. 4. In what ways do you think methodological
We suggest you answer each question in writing, diversity is necessary in the counseling pro-
and then discuss your responses with a peer fession?
in class to further discuss and explore these 5. What do you see as the most important out-
issues. come of research in counseling?
6. What role does your worldview play in how
1. What do you see as the primary value of science
you might engage in research and practice?
in the counseling profession?
What do you see as the disadvantages and
2. Which philosophical paradigm is most ap-
advantages of your worldview for both re-
pealing to you personally? What do you see
search and practice?
as some of the advantages and disadvantages
7. Given all of the complexities in different
of this paradigm?
philosophical paradigms, which one do you
3. What is the second philosophical paradigm
want to learn more about? Why?
that is most appealing to you? What do you
2 CHAPTER
Research Training
It takes many years to become truly skilled at a complex set of tasks, such as
becoming a skilled potter. For example, in Japan, it is common for an apprentice
potter to make the same vase form for many months just to acquire the spe-
cific skills to consistently make 15–20 identical vases in a row. Expertise
develops over time within an environment that fosters development. In this
chapter we discuss the process of becoming a skilled researcher, and the type
of training environments that seem to be most helpful for students to acquire
the necessary skills and attitudes. For many students in counseling, graduate
school is their first introduction to research design, and it evokes many emo-
tional reactions that come with any novel experience, from joy and excitement
to anxiety and disenchantment. We seek to prepare students to approach
research with enthusiasm for the creativity involved and with a willingness to
learn the intricacies of the craft. In addition, we want to promote an aware-
ness of the anxiety that may be created by learning a technical skill that may
not be central to one’s interests but is required to accomplish a goal (that is,
obtaining a graduate degree and becoming a professional in counseling).
We begin the chapter by providing a brief overview of the scientist-practitioner
model, and discuss the value as well as some concerns about this popular
training model. Next we identify and discuss some issues related to the devel-
opmental process of acquiring research competencies, specifically the joy as
well as the challenges and fears. Then we specifically discuss ways in which
counselors and counseling psychologists can train others to become competent,
22
chapter 2 research training 23
eager, and productive researchers. In this part of the chapter, we discuss what
is known about research training, and how training environments can be
structured to create the opportunity for students both to learn about research
and to consume and produce quality research products. Finally, we discuss
ways to broaden the concept of scientific training to include scientific thinking
skills as well as research application skills. Most important, throughout this
chapter we emphasize the developmental process of acquiring skills step by
step to become a skilled researcher and scientist.
Perhaps the type of research being conducted is too distant from the reality of
the practitioner (e.g., Howard, 1984; Polkinghorne, 1984), or perhaps our
research methods reduce counseling phenomena to meaningless numbers (e.g.,
Goldman, 1976). Or maybe our professional journals are structured in ways
that make it cumbersome for practitioners to find and assimilate the infor-
mation they need (e.g., Gelso et al., 1988; Heppner, Carter, et al., 1992). Yet
another factor could be that students admitted to graduate programs have
predominantly social or interpersonal interests (e.g., Magoon & Holland,
1984) or have been selected on a very narrow range of criteria (Bernstein &
Kerr, 1993).
Another issue is how the scientist-practitioner model has been conceptu-
alized in the past. Sometimes the model has been interpreted as a 50–50 split
of performing science and practice activities. The type of model in its ideal
form (that is, implicitly 50% practitioner and 50% scientist/researcher) may
be just that—an ideal that is rarely found in reality. Gelso (1979) proposed
that it may be more realistic to train students in both domains (in varying
degrees depending on their interests) with the expectation that students will
find a suitable place for themselves in performing relevant activities on the sci-
entist-practitioner continuum. Thus, one student might prefer a 20–80 split
while another might choose a 75–25 split. Sometimes there have been implicit
values attached to either science or practice; that is, some educators might
value science more than practice, and thus feel more satisfaction when a new
graduate obtains the “best” job, which to them is an academic position com-
plete with a myriad of scientific pursuits. We strongly believe that science and
practice are both highly valued activities in the counseling profession, and that
as a profession we are stronger (and only can survive) when we train students
to be competent in both science and practice. In short, it is important for the
profession to equally value various points along this performance continuum.
We prefer to conceptualize the core of the scientist-practitioner in terms of
scientific or critical thinking. In short, we are suggesting that the role of sci-
entific or critical thinking is a central outcome of the scientist-practitioner
model and forms the core of a wide range of science and practice activities. The
choice of whether a graduate engages in science or practice activities is not the
most important outcome, but rather whether the graduate can utilize scientific
or critical thinking in whatever professional activities he or she chooses. For
example, a counseling center staff member might be engaged primarily in
direct client service and, say, one program evaluation project (see Chapter
21); this might be a 5%–95% science-practice split of professional activities. A
faculty member (three-quarters time) with a quarter-time direct service
appointment in the counseling center might have a 75%–25% split of profes-
sional activities. Regardless of the type of professional activities a person
selects along the scientist-practitioner continuum, we maintain that both of
these individuals could reflect the goals of the scientist-practitioner model
through their scientific thinking within their practice and science activities.
Hans Strupp, a noted clinical psychologist with a great deal of experience
in practice, research, training, and professional affairs, concluded in 1981 that
26 part 1 philosophical, ethical, and training issues
In the end, there is no doubt that research has enhanced the practice of
counseling. Even though a practitioner may not be able to cite a specific
reference, his or her graduate school training was likely based on a tremendous
amount of research data, all the way from personality theory to intervention
strategies. The accumulation may be slow, but the data eventually advance
our working knowledge of the field (Heppner & Anderson, 1985). The scientific
method and research have advanced the field, but they could be even more
helpful. In addition, training in both science and practice has become more
sophisticated with not only more integration, but also more emphasis on scientific
thinking, which is the core of all science and practice activities of the coun-
seling profession. For more information on a broad array of recommenda-
tions for enhancing training in the scientist-practitioner model, see Heppner,
Carter, and colleagues (1992).
researchers who provide some useful glimpses into this developmental process
of acquiring an array of research competencies. A great deal of wisdom is
communicated by these prolific and creative scholars, and we appreciate their
candid perspectives on their own developmental process.
After graduating from college with degrees in Biology and Sociology (and a
minor in Women’s Studies), I worked in an inner city battered women’s shelter.
At the shelter, I became interested in what differentiated women who returned
to the abuser from women who struggled to create nonviolent lives for them-
selves and their children. Without any knowledge of psychological research, I
developed a survey and collected information from the women in the shelter.
Around this time, I missed being in school and decided to enroll in a course of
interest to me. The professor (Nancy Murdock) encouraged me to join her
research team and pursue a doctoral degree. With her help, I completed a thesis
related to shelter workers’ perceptions of battered women. I loved how my
work in the shelter informed and was informed by my research. I also began to
see that while helping people individually was rewarding to me, I could really
make a difference through sharing my knowledge and research findings with
others. My doctoral program solidified my passion for research as I was given
the opportunity to study topics of interest to me that had real-world applications.
Helping women to achieve economic self-sufficiency developed as a focus of my
work. I sometimes struggled with a component of research (e.g., collecting data
from difficult to obtain samples), but as soon as this task became unbearable,
the next step in research brought new and exciting challenges (e.g., analyzing
those data!). As a new assistant professor at the University of Kansas, my
favorite part of my job was pursuing questions of interest with a talented (and
fun) research team. Today, mentoring students as they become involved in
research that makes a difference is a meaningful aspect of my work as an asso-
ciate professor at the University of Maryland.
—Karen O’Brien, Ph.D.
I love research! I say this in all honesty. I’ve always loved research—learning
about theories, thinking about relations among and between phenomena, find-
ing ways to better understand processes that I think are personally interesting
and socially meaningful, and empirically testing ideas. Although research has
provided me with a rich intellectual life and has been a rewarding aspect of my
career, the process of conducting research has been one of the most challenging
personal and professional journeys. I say process because developing compe-
tencies in research is not an event. Instead, it is a slow journey; as some mark-
ers are achieved, new competency goals quickly surface. Do I feel like I have
arrived? No. However, I continue to derive satisfaction as I grow and develop.
As a racial minority woman researcher, I have found the journey of devel-
oping competencies particularly challenging for a variety of reasons. On a per-
sonal level, I have often doubted whether my ideas were “good,” and make a
28 part 1 philosophical, ethical, and training issues
Performance Anxiety and Efficacy From our experience working with students,
we have found that performance anxiety is a central and crucial affective issue.
Other educators have come to similar conclusions (e.g., Claiborn, 1984; Gelso,
1979). For example, common student disclosures include: “I don’t know
enough to conduct any meaningful research on my own. How can I do a
thesis?” “Any study that I could do would be so basic and bland as to make it
worthless.” “I feel completely inadequate in even conceiving a study that comes at
counseling from a unique perspective.” “Although I made it through the statistics
and research design courses, once I start to do some research, I’ll be ‘found out’;
I am a fraud, and really don’t know how to do research.” “What if I make a
mistake and report inaccurate findings?” “I’ve had trouble writing before, and
I’m scared to death that I will run into debilitating writing blocks again in my
research.” “I am afraid that my results will be statistically nonsignificant, and
all of my research efforts and time will just be one big waste.” “I’ve seen other
students in the throes of a dissertation. The whole process seems overwhelming,
tedious, long, anxiety-producing, confusing, and complicated.”
It is important to put these and similar feelings into the context of the typical
entering graduate student. Most incoming graduate students have had some
experience (or even a lot) with helping other people, and consequently the
benefits of counseling and other practitioner activities are abundantly clear.
Moreover, for most of us, helping people is a personally rewarding experience,
and students have an understanding of those rewards. Conversely, it has been our
experience that most incoming students have had considerably less experience
with science. Typically, students do not imagine themselves making contribu-
tions to the profession by publishing in our professional journals. Since stu-
dents have had little, if any, research experience, they often have legitimate
reasons to question and doubt their skills and abilities—these feelings, con-
cerns, and fears are thus developmentally normal. In fact, it is not uncommon
for faculty to experience performance anxiety and question their efficacy. This
suggests to us that acquiring the necessary research and thinking skills occurs
over a long period of time; it does not happen all at once. Just as it is unreason-
able to expect to become an Olympic downhill skier in four years, it is also unrea-
sonable to expect to become an expert researcher in four years of graduate
education (Nancy Downing, personal communication, June 28, 1990).
In many ways, I was a late bloomer academically. I had been kicked out of
high school and somehow made my way through college on raw ability, luck,
and the kindness of others. I had little crystallized understanding of the
world. I could read and write well enough but I didn’t have clear, cogent
thoughts on many subjects. I also had few academic role models in my life.
My parents and two older brothers had never finished college and I felt
tremendous pressure to be a success. So when I began graduate school at
Virginia Commonwealth University, I had many doubts about my abilities
and talents. I had a healthy dose of ambition and motivation but a larger
dose of uncertainty and insecurity. If I had to identify a key turning point in
graduate school, it had to be when I got my first manuscript, based on my
master’s thesis, accepted for publication in Journal of Counseling Psychology.
I was overwhelmed with pride and joy. Here was validation for all my effort.
I finally started to really believe in myself as a researcher who could make a
difference. Today, I often tell my graduate students that the first publication
is like the first taste of blood. You either want more of it or you realize it isn’t
for you. I guess I liked that first taste because it definitely gave me the con-
fidence and motivation to engage in more research.
—Richard Lee, Ph.D.
courses that teach little if anything about counseling research, but seem wedded
to the scientific enterprise. These courses often deepen the student’s anxiety and
deaden motivation. The task is to get through them.
Toward the end of the first year our hypothetical student must start thinking
thesis. More anxiety. Up to that point he/she has done no research, acquired no
confidence, and at best has gotten through statistics unscathed.
The student does not know how to generate ideas that later may be translated
into scientifically answerable questions. The process goes on, through the prelim-
inary meeting with the advisor, the thesis proposal meeting, and the thesis orals.
All steps along the way are all too often filled with more threat and anxiety than
pleasure of discovery. The fortunate student gets by without massive revisions of
the thesis and can take a deep breath until the next major scientific hurdle, the dis-
sertation. When assessed against the criterion of, “Is it positively reinforcing?”
this unfortunately not very exaggerated description of the scientific regimen of
graduate school hurdles is a dreadful failure. (p. 28).
over time, and often achieves significant conclusions. For example, many of the
chapters in the Handbook of Counseling Psychology (Brown & Lent, 2000) are
based on extensive literature reviews, and nicely reflect the utility of our research
in creating knowledge as well as enhancing the sophistication of our conceptu-
alization of relevant constructs in counseling.
Clara Hill, a creative and prolific counseling researcher, was invited to
write a personal account of her evolution as a researcher. Her account illus-
trates how she made decisions, how complex the process of research can be,
and how she coped with the ups and downs along the way and, most impor-
tant, her “crisis of faith in research”:
After this series of studies I underwent a crisis in terms of research. I had achieved
tenure, had my first baby, and turned 30. I started questioning everything. Because
I could see so many imperfections in my research, I was not very proud of my work.
It had taken an incredible amount of time for little “hard information.” I could not
even remember the results of many of my studies. Further, I seemed woefully far
from even describing what happens in the counseling process let alone understanding
what counselor behaviors were useful in effecting change. I despaired that counseling
was far too complex to be studied. I had previously had lots of doubts about my
research, but it was mostly related to self-confidence in my research abilities. This
crisis seemed to have more to do with whether I felt that research was a viable means
to answer questions about the counseling process. (Hill, 1984, p. 105)
was accepted for publication. In retrospect, these initial successes were criti-
cal to my development as a researcher, as they not only propelled me into
more research projects with faculty (which greatly enhanced my learning), but
also allowed me to experience the joys of discovery as well as a sense of
accomplishment in contributing to the professional literature. These and sub-
sequent research experiences with my advisor, David Dixon, in graduate
school shifted my career aspirations toward academia.
—Puncky Heppner, Ph.D.
From our experience, it is critical not only that researchers feel efficacious
in conducting relevant and important research, but also that they begin to per-
ceive that their research is important, that it contributes useful information to
our professional knowledge bases, and that it can make a difference in the
profession as well as the lives of our clients. Such beliefs often are the result
of identifying implications of your research findings to practice, being cited in
the professional literature in meaningful ways, or seeing either that others are
using your research findings in applied settings or that your research is help-
ing other researchers subsequently ask other important questions.
Belief in the utility of research can also result from reading the professional
literature to learn the impact of others’ research. Our advice is for students to not
only read and reflect on the literature, but also to read literature reviews in par-
ticular (see Major Contributions in The Counseling Psychologist or review papers
in Psychological Bulletin). In addition, we advise students to get involved in con-
ducting research on topics that are really important to them, either topics that they
are very interested in or ones that reflect their personal values in some way.
Becoming a researcher has truly been a developmental process for me. From
coming up with a topic, to developing a research question, to determining
how to answer the research question, each aspect of becoming a researcher is
honed with practice over time. Early on in graduate school I knew that I was
interested in racial identity as a broad topic, but I did not know how to
develop a research question based on my interest. So, I began to immerse
myself in the racial identity literature. The more I read, the more I began to
understand the construct, and most importantly, the more I began to see some
gaps and unanswered questions in the literature.
I firmly believe that the best researchers usually have a connection and
passion about their research. I developed my research program, in part, to
help me understand and make sense of my own experiences as an African
American man in a predominantly White college environment struggling to do
well academically, while at the same time trying to learn more about my his-
tory and culture. This desire to better understand myself and the experiences
of other people like me led to a dissertation, my first publication, and several
other articles. The lesson to be learned is that the more you can take owner-
ship of the research topic, the greater the chances that you will feel excited
about conducting research in that area. Perhaps the most important lesson
36 part 1 philosophical, ethical, and training issues
that I have learned has been to be tenacious and resilient. Everyone has expe-
rienced challenges in writing and conducting research at some point, and
everybody has experienced rejection when trying to publish something. I have
learned that it is not always the “smartest” people who can successfully carry
out a research project and publish it, but often it is those individuals who are
the most resilient.
—Kevin Cokley, Ph.D.
Holland
Research Career
personality
interest goals
type
Research
training Research Research
environment self-efficacy productivity
Gender
Year in
program
Research satisfies, for me, two primary characteristics of my personality: (a) the
desire to investigate, and (b) the need to contribute to the well-being of members
of our society, particularly those who are not privileged. After many years of
research on various areas related to counseling psychology, including the process
of counseling and psychotherapy and interpersonal environments in scientific
laboratories, I found myself teaching a course on research on individual psy-
chological interventions. The text for the class was a handbook that reviewed
many areas of counseling and psychotherapy. A class assignment involved taking
38 part 1 philosophical, ethical, and training issues
a conclusion made by the author of each chapter, reading the original research that
was cited to support the conclusion, and deciding whether or not the research did
indeed support the conclusion. We found that authors often claimed that a par-
ticular treatment was superior to another when the research cited did not support
that conclusion. This practice was offensive to me because it tended to privilege
some treatments (mainly, cognitive and behavioral treatments) over others, with-
out scientific basis. To support my and my students’ contention that the conclu-
sion that some treatments were superior to others was not justified by research
evidence, we conducted a meta-analysis of all studies that compared two treat-
ments intended to be therapeutic and found that these treatments were uniformly
efficacious (Wampold, Mondin, Moody, Stich, Benson, & Ahn, 1997). This line
of research has continued and the results demonstrate that (a) all treatments
intended to be therapeutic produce comparable benefits, regardless of the diagno-
sis; (b) the therapist accounts for a large proportion of the variability in outcomes;
(c) clients of therapists in clinical settings attain outcomes comparable to those
attained by therapists in clinical trials with fewer sessions; and (d) the factors
common to all treatments are responsible for the benefits of counseling and psy-
chotherapy (see Wampold, 2001). This research has been the basis for arguing
that managers of mental health services should not mandate treatments nor
restrict access to services. That is, clients should have access to effective therapists
who deliver services that are consistent with the attitudes, values, and culture of
the client. I feel that my investigative spirit has yielded results that honors clients’
access to a variety of effective services; I find this immensely gratifying.
—Bruce Wampold, Ph.D.
science and practice. The second factor, labeled interactional factor because the
components involved the interaction of trainer and trainee, includes modeling,
reinforcing, involving students early, and conducting science as a social experi-
ence. In the model, Kahn and Scott (1997) hypothesized that environmental
influences affect research self-efficacy, which in turn affects research interest,
career goals related to research, and research productivity.
Although the nine components of the research environment can be reduced
to two factors statistically, the examination of the individual components that
follow reveals how research trainers can construct curricula and experiences to
foster the development of skilled and motivated consumers and producers of
research. The components have been modified since their original conceptual-
ization; nevertheless, we recommend that professionals involved in research
training read Gelso’s (1979) seminal article on research in counseling.
Teaching Relevant Statistics and the Logic of Design Many counseling students
have little fondness for statistics and are turned off to research because they
equate doing research with studying for statistics examinations. Gelso et al.
(1996) have suggested that this unfortunate situation be countered by peda-
gogy in statistics classes that “is sensitive to students’ needs” (p. 311) and by
emphasizing the logic involved in designing research. The basis of this recom-
mendation is that poorly taught or overly technical statistics classes create
unnecessarily negative attitudes toward research.
and squares, and I was thrilled that one could actually study things this way! I
also became involved in Division 17 activities, and I connected with many won-
derful people who became role models and mentors for my emerging identity as
a researcher and academic. I still wanted to help people build better lives, but
now I knew that research was an important tool in that effort.
When I re-examine my own JCP articles on methodology (on SEM [1987]
and grounded theory [2005]) that anchor the two decades of my career thus
far, I am surprised and pleased at how far I’ve traveled in developing my con-
fidence and voice as a researcher, and how comfortable I’ve become with a
variety of paradigms. I teach my students that research is a political act with
social consequences, and I have accepted the responsibility implicit in that
reality—that every single moment of my scholarly life counts for something. I
am helping people build better lives, just as I wanted to do so many years ago.
—Ruth Fassinger, Ph.D.
Teaching That All Research Is Flawed and Limited In this book we will con-
sistently suggest that any one study cannot rule out all threats to validity, and
that knowledge accumulates through repeated investigations. Students should
not feel pressure to design the perfect study; rather, they should feel motivated
to create a study that can address the research question, taking the principles
of research design as well as practical constraints into account. Requiring students
to meet standards and achieve excellence is appropriate; forcing students to
clear unrealistic hurdles is not.
Wedding Science and Practice in Training Much has been written about the inte-
gration of science and practice. However, as Gelso (1979) noted, research motiva-
tion stemming from clinical experience is often discounted. This component
recognizes that a true integration of science and practice will make research
more attractive, especially to the majority of students whose primary interest
is in practice. Courses that carefully draw the intimate connection between
science and practice, taught by instructors who feel comfortable with the con-
nection and have skills in each area, are truly motivating to students. Research
in applied settings also fosters interest.
Facilitating the Students’ Introspection for Research Ideas For many students,
looking inward is to rely on clinical experience, so this component clearly over-
laps the wedding of science and practice. When research mentors discount ideas
42 part 1 philosophical, ethical, and training issues
generated from the “person,” the person is discounted. It is not productive for stu-
dents to get the idea that legitimate research ideas should be generated only from
an objective reading of the literature. It is our belief that those students who have
a deep interest in their topic finish their dissertations faster than those who are
motivated to finish their programs but who pick a convenient topic of study.
I think it was the issue of sexual assault that made me really excited to do
research on how to better intervene in preventive ways. I can’t say that I came
into my research career with a lot of confidence or expertise, but I had been
involved in rape prevention programming on college campuses and I did want
to know more about how we can develop more effective programs to change
men’s attitudes about sexual violence. That desire led me to a rich and mean-
ingful road conducting large scale prevention studies, eventually obtaining
grant funding to support my research and really finding myself enjoying work-
ing with my research team on these important issues. I love doing research on
teams where we each contribute our own unique skills to the process and learn
from one another. My research in the areas of sexual violence prevention is
meaningful to me and the process of doing the research with students and col-
leagues has been a very stimulating and enjoyable part of my career. In a
world where sexual violence is becoming more and more prevalent, it feels
like an important issue to try to understand and prevent.
—Mary Heppner, Ph.D.
cognitive models and past research in this area (see, for example, Brown et al.,
1996; Phillips & Russell, 1994), predicted that research training and research
productivity (that is, experiences with research) would predict research self-effi-
cacy, which in turn would predict research interest and research career goals
(see Figure 2.1).
Research Interest Kahn and Scott (1997) hypothesized that research inter-
est mediates the relationship between personality type and career goals and
research productivity. Put another way, the established relationship between
personality type and research productivity (and presumably career goals) is
due to interest. Investigative types have more interest in research and there-
fore have goals related to research and produce more research products.
informed standpoint, appreciate and value the scientific method, and most
important apply critical thinking skills to a broad range of research and
applied contexts. Hill (1997) noted that even though few of her students had
taken academic positions or produced much research, she should not as a
result be judged as an inadequate research mentor.
Career Goals in Research In Kahn and Scott’s (1997) model, the second
outcome construct was goals related to research, which was defined as pref-
erences for postdoctoral positions as researchers rather than as practitioners.
Although this is an appropriate construct for their research, we agree with
Hill (1997) that training programs should not be judged by the proportion of
students going into research-related careers. Instead, programs should gradu-
ate students who are trained consistent with the mission of the program,
which we believe should include the acquisition of research skills, appropriate
attitudes toward research, and most important, critical thinking skills.
.3
.33 .32*
8
.13*
*
Research
training .35* Research .20 Research
environment self-efficacy productivity
*
.16
Gender *
.38
*
.28
Year in
program
Carl Rogers was very aware of this danger of counselor bias; in 1955 he
observed that he could “deceive himself in regard to my creatively formed
subjective hunches” about a client (p. 275). He believed that the scientific
method, as a way of thinking, led him to check “the subjective feelings or
hunch or hypothesis of a person with the objective fact” (p. 275). Rogers
would often check his hunches very directly by asking the client, “Do you
mean this?” or “Could it be this?” Rogers would sometimes go a step further
and develop a written transcript of an interview to analyze the relationships
between counselor and client statements. Years later his face would still light
up with excitement about what he would learn about a particular client, or
the counseling process, by stepping back from the immediacy of an interview
to analyze those transcripts (Rogers, personal communication with P. P.
Heppner and L. A. Lee, January 1984).
Pepinsky and Pepinsky (1954) initially articulated a prescriptive model of
counselor thinking based on the scientific method. Strohmer and Newman
(1983) succinctly summarized their model:
The counselor observes the client, makes inferences about his or her current status
and the causal inferences, and then, based on these inferences, makes a tentative
judgment about the client. The counselor then proceeds in an experimental fashion
to state the judgment as a hypothesis and to test it against independent observa-
tions of the client. Through a series of such tentative judgments and tests based
on these judgments, the counselor constructs a hypothetical model of the client.
This model then serves as the basis for making predictions (e.g., which treatment
approach is most appropriate) about the client. (p. 557)
In essence, Pepinsky and Pepinsky (1954) were suggesting that the coun-
selor incorporate a scientific or critical thinking model by (a) generating hy-
potheses based on (b) the data that the client presents, followed by (c) empirical
testing of the hypotheses, to develop (d) a model that can be used (e) to make
predictions about the client. The essence of this approach is that it is data-
based or empirical, which lessens the chance of personal biases or subjec-
tivity. Scientific thinking in this way involves a continual generation and
evaluation of hypotheses based on data. Spengler, Strohmer, Dixon, and Shivy
(1995) nicely extended the Pepinskys’ model by integrating it with the human
inference literature, which effectively operationalized scientific thinking and
identified a number of inferential errors commonly made in counseling.
Training in scientific thinking may be particularly important in evaluating
counseling outcomes. Goodyear and Benton (1986) have referred to a particular
counselor bias in assessing counseling outcomes, a bias they call a “walk-
through empiricism” mentality:
That is, as the counselor has successes, he or she will form impressions of what
techniques are proving useful. Although this walk-through empiricism can be
valuable, it is also subjective and therefore unreliable if it is used as the sole source
of data. An analogy might be drawn from medicine where not so long ago a fre-
quent remedy was to treat people by bleeding them of “bad blood.” Physicians
believed a cause-effect relationship—that the bleeding caused a cure—and contin-
ued to offer this as a common treatment for many years. Yet, the treatment was
48 part 1 philosophical, ethical, and training issues
no doubt responsible for many patient deaths that might have been avoided if the
procedure had been tested empirically. (p. 291)
Gelso (1985) has also observed that reading the research literature affects
his thinking about the counseling process:
I would offer that it is important to think about research as that which helps us
use our heads more clearly and less prejudicially. It helps us think about what we
do and organize our ever-changing personal theory of the processes in which we
are involved, be they counseling, psychotherapy, consultation, supervision, and so
on. We should not expect empirical research to capture the felt essence of those
processes. Only the practice of them will do that.
Most of the research findings that I read are relevant to my clinical practice in
the following way: they help me think in a less biased way, they help me further
refine my usually private theory of my practice, and they allow me to add small
pieces of new information to my conceptual system. (p. 553)
both scientific and practice-oriented professional “the thirst to know something” (Pepinsky, cited in
activities could be enhanced by a more complete Claiborn, 1985, p. 7).
integration of the two domains. Maintaining a pioneering attitude in profes-
We have also maintained that a basic issue in sional scholarly inquiry can also be a way of life.
scientific training is scientific thinking. Those at Enos Mills was an early pioneer in the moun-
the Atlanta conference in 1987 concluded that tainous area of Colorado that is now Rocky
“an attitude of scholarly inquiry is critical to all Mountain National Park. In the following
the activities of those educated in the scientist- quote, Mills (1924) aptly describes the qualities
professional model” (Meara et al., 1988, p. 368). of a pioneer’s life. Even though he was referring
Such an outcome requires not only scientific to pioneering in the American West, the qualities
skills, but also scientific values. The latter are of life he describes also apply to the pioneering
also important goals and need specific attention attitude involved in discovering professional
in training (perhaps by way of discussions about knowledge through scholarly inquiry:
the philosophy of science, research methods, the
slow but steady accumulation of knowledge, and Those who live pioneer lives are usually the most
students’ own research experiences). fortunate of people. They suffer from no dull exis-
An attitude of scholarly inquiry goes beyond sci- tence. Each hour is full of progressive thought, and
occasions which call for action accompanied by the
entific thinking and involves curiosity, inquisitive-
charm of exploration—action that makes their
ness, healthy skepticism, exploration, and a desire lives strong, sincere, and sweet. Their days are full
to learn. In a way, all counseling professionals are of eagerness and repose, they work with happy
pioneers in extending the boundaries of their own hands. The lives of pioneers are rich with hope and
knowledge throughout their careers, and possibly their future has all the promise of spring. (p. 9)
extending the knowledge bases of the profession as
well. In this way, scholarly inquiry involves discov- We hope that your life in the counseling pro-
ery, excitement, and even a sense of adventure. Not fession will be a good place for you to be a pio-
surprisingly, pioneers in the field of counseling and neer and that scientific thinking will enrich your
development report that they were motivated to life with discoveries that are exciting to you and
achieve, in part, by the joys of intellectual discovery, beneficial to the various clienteles we serve.
STIMULUS QUESTIONS
Reflections on Research Training
1. Why do you think some scholars are advo- the scientist role, what would be the advan-
cating a need to broaden scientific training in tages and disadvantages?
counseling? 6. What do you see as the most important ad-
2. In what ways do you think scientific thinking vantages of integrating science and practice in
might be useful for both practice and science the counseling profession?
activities? 7. What do you feel very passionate about in
3. Identify five activities you can do to incorporate your future as a counselor? Can you see any
scientific thinking into your counseling practice. way that research could help you enhance the
4. If the counseling profession more strongly work you will do within the areas that excite
emphasized the scientist role rather than the you the most?
practitioner role, what would be the advan- 8. What concerns, obstacles, and fears do you
tages and disadvantages? have about scientific training?
5. If the counseling profession more strongly 9. Examine the reactions you listed in question 8;
emphasized the practitioner role rather than why do you think you have these reactions?
Identifying
and Operationalizing
CHAPTER
3
Research Topics
51
52 part 1 philosophical, ethical, and training issues
to think that no one has thought of this idea before (at least for a short period
of time!). Various pioneers in the field of counseling have commented on the
joys of discovery and learning they have experienced by engaging in research.
Interested students might read some of the interviews with early counseling
pioneers that appeared in the “Pioneers in Guidance” and “Lifelines” series in
the Journal of Counseling and Development, as well as in the “Legacies and
Traditions” forum of The Counseling Psychologist. For example, Anne Roe,
in reflecting on her career, stated simply that “nothing is as much fun as
research. . . . I miss that (in retirement)” (Wrenn, 1985, p. 275).
Sometimes developing research ideas may seem difficult for inexperienced
researchers; they cringe at the thought of developing an original research idea
that no one has ever had before. Typically, the experienced researcher has little
difficulty in developing research ideas. In fact, often the veteran researcher has
too many research ideas, and a more difficult problem is deciding which ideas
to pursue.
Several ingredients differentiate the experienced researcher from the inex-
perienced. The experienced researcher usually has a very large knowledge
base, not only about a given topic, but also about several other topics. Such a
researcher can most likely process information about these topics in sophisti-
cated ways, identifying the most important findings, combining research find-
ings, dovetailing an idea from one topic with another, and elaborating on
or extending ideas in novel ways. In addition, the experienced counseling
researcher often has a considerable wealth of information from his or her
applied counseling experiences that can be the source of many ideas and
hypotheses. The veteran researcher also typically knows a great deal about the
skills needed to conduct research, such as knowledge of research design,
methodology, assessment, statistics, data collection, data analysis, and techni-
cal writing. All of these knowledge bases are important tools in facilitating the
processing of large amounts of information about specific research projects in
sophisticated and often novel ways. Perhaps equally important, the experi-
enced researcher typically has confidence that he or she can effectively con-
duct research, and thus has the needed level of self-efficacy.
In contrast, the inexperienced researcher has far less knowledge about
specific research topics and often has trouble identifying the most important
or relevant information. The novice often has less applied experience in coun-
seling, and most likely has less-developed conceptualizations of the counseling
process. The inexperienced researcher has perhaps only a vague sense of
the various research activities (such as recruitment of participants and data
collection) and harbors doubts about his or her ability to do research well. In
fact, in many counseling programs, well-articulated guidelines for conducting
research are not available to students, leaving the impression that the research
process is rather mystical. Drew (1980) noted that a logical place to turn for
information is a college or departmental catalog that describes a thesis or dis-
sertation project. Typically such documents say something to the effect that
the thesis “must be an original contribution to the field.” When viewed liter-
ally, the word original can engender considerable anxiety as the trainee tries
chapter 3 identifying and operationalizing research topics 53
to develop a completely novel research idea. In addition, the idea must represent
“a contribution to the field,” which makes the task seem even more formidable.
It is the rare trainee who believes at the outset that he or she can make a “real
contribution” to the field; after all, we are talking about science! Sometimes the
inexperienced researcher interprets “original contribution” much too broadly
and tries to develop a new topic area by creating a new assessment instrument
to measure new constructs, a new research methodology to collect data previ-
ously not collected, and new statistical procedures to handle old problems. In
reality, most experienced researchers would feel quite a sense of accomplishment
if they did all of these things during an entire career. In short, the inexperienced
researcher often takes on too much in trying to develop an original contribu-
tion. Not surprisingly, our experience has been that graduate students in be-
ginning research design courses ask questions about how to identify “good”
research topics and “good” research ideas. Also not surprisingly, they often ask
whether a particular idea is “enough” for a thesis.
We will make some suggestions in this chapter to help beginning re-
searchers to identify a research topic. Essentially, the task is to identify some
general topic that may (1) contribute to the profession’s knowledge bases in
meaningful ways, and (2) simultaneously stimulate and motivate students to
explore and learn more about the topic. Most inexperienced researchers in
counseling need to learn more about the body of knowledge and the directions
of current research efforts. Thus, for such students we do not simply recom-
mend sitting and thinking hard about research topics and hoping that the
ideal research question will present itself, but rather taking a more active,
information-collecting approach.
A good first step in identifying possible research topics is to start collecting
information about previous research, both within and outside of counseling.
Thus, read widely in the professional journals and books. Reading widely not
only will provide you with information about what is being published, but also
may help you to clarify what topics are of most interest to you. Sometimes it
is useful to start with a general review of the counseling literature (see Betz &
Fitzgerald, 1993; Borgen, 1984a; Gelso & Fassinger, 1990; Heppner, Casas,
Carter, & Stone, 2001; Mackay, Barkham, Rees, & Stiles, 2003, or peruse the
various handbooks related to the counseling profession). Another strategy is to
begin by looking for more focused literature reviews on specific topics. (The
Psychological Bulletin is a journal devoted to evaluative and integrative re-
views; see Dolliver [1969] for an excellent example of an evaluative review of
measured versus expressed vocational interest, as well as another excellent
example by Oyserman, Coon, and Kemmelmeier [2002] evaluating theoretical
assumptions of individualism and collectivism.) Books on specific topics are
often useful resources. For example, a student interested in conducting research
with racial/ethnic minorities would be wise to examine Ponterotto and Casas
(1991). In short, there is no substitute for this time-consuming process of read-
ing the relevant literature; at first you might peruse and read abstracts to
develop a broad overview. A good beginning is to spend at least five hours a
week reading and exploring the journals for six weeks or more.
54 part 1 philosophical, ethical, and training issues
views about counseling when they were told that counseling involves reveal-
ing highly personal information. Difference questions are often examined in
between-group and within-group designs (see Chapter 7).
Relationship questions explore the degree to which two or more con-
structs are related or vary together. Such questions tend to use correlational
statistics or more complex regression analyses. For example, a study by Marx
and Gelso (1987) examined a relationship question by studying the relation-
ship between client satisfaction with termination and five client variables sug-
gesting loss in the termination process. Similarly, Smith and Ingram (2004)
studied the relationships among workplace heterosexism, unsupportive social
interaction, and adjustment of lesbian/gay/bisexual individuals. Likewise, Good
et al. (1995) examined the correlations between gender role conflict and sev-
eral other inventories (for example, attitudes about masculinity, fear of inti-
macy, social desirability) to provide estimates of construct validity for the
Gender Role Conflict Scale (O’Neil, Helms, Gable, David, & Wrightsman,
1986). Cournoyer and Mahalik (1995) extended this line of research by using
canonical analysis, a multivariate approach to examining relations among
multiple variables. Relationship research questions are discussed more fully in
Chapters 10 and 11.
What constitutes a testable research question? According to Kerlinger and
Lee (2000), a research question (a) asks a question about (b) the relationships
between two or more constructs that can be (c) measured in some way. First,
the question should be worded clearly and unambiguously in question form.
Second, the research question should inquire into a relationship between two or
more constructs, asking whether construct A is related to construct B. (If a par-
ticular relationship is stated, the research question becomes a hypothesis.) This
second criterion pertains mostly to difference and relationship questions, whereas
descriptive questions often seek to collect or categorize information. Finally, not
only is a relationship between constructs examined, but somehow this relation-
ship also must be measurable.
For example, consider a research question like, “Is supervision effective?”
One might immediately ask, “Effective at what?” Is the researcher interested
in the effectiveness of supervision to lower a trainee’s stress level to conceptu-
alize clients, or to intervene with clients? In short, such a question lacks speci-
ficity. Now consider the research questions developed by Wiley and Ray
(1986), who were interested in the topic of the changing nature of supervision
over the course of training. In particular, they were interested in testing
Stoltenberg’s (1981) counselor complexity model concerning developmental
levels of counselor trainees. (Stoltenberg proposed that counselor trainees
develop in a predictable way over the course of graduate training, and that
counseling supervision environments should be adapted in ways that match
the needs of the trainee.) Wiley and Ray developed three specific research
questions, each of which inquired about relationships between two or more
constructs and was amenable to being measured or tested in some way. For
example, one of their questions was: To what extent do supervision dyads
with a more congruent person–environment match on developmental level
60 part 1 philosophical, ethical, and training issues
report higher satisfaction and learning than those with a less congruent match?
The construct of the congruent person–environment match was concretely
operationalized via the use of an assessment instrument called the Supervision
Level Scale. Likewise, satisfaction and learning were operationalized in terms
of a brief outcome instrument. Parenthetically, although Wiley and Ray
obtained results that provided some support for conceptualizing supervisees
and supervision environments developmentally, mean satisfaction and learning
ratings did not differ by person–environment congruency.
In short, the function of a testable research question or hypothesis is to
provide direction for experimental inquiry. The testable research question or
hypothesis not only identifies the topic, but also identifies specific constructs of
interest within that topic. For more examples about writing research questions,
see Heppner and Heppner (2004). After a researcher has developed a specific
research question or hypothesis, he or she can then proceed to determine what
instruments to use, how data can be collected, what participants to use, and so
on. Many of these methodological decisions are directly dependent on the in-
vestigator’s specific question or hypothesis.
Although the research question provides direction for designing a study,
it is important to note that in the formative stages—as an investigator contin-
ues to develop the design and methodology of a particular study—it is not
uncommon to revise or change the original research question. The investiga-
tor may encounter measurement or participant availability problems, which
may dictate a slightly different research question. Or the researcher may find
new data that suggest additional complexity in the topic and thus require revi-
sion of the research question.
In short, any of a number of events may lead the researcher to process
more information and subsequently to revise or sharpen the research ques-
tions in the formative stages of study design.
Sometimes graduate students get discouraged about their false starts and
begin to feel a sense of inadequacy because they “couldn’t get it right the first
time.” There often is an assumption that a “real researcher” proceeds through
a logical series of ordered steps, starting first of all with a brilliant, incisive
research question. Our experience has been that effective researchers generally
examine a wide range of design issues when developing a research question,
such as choosing instruments and participants or examining external validity
issues. In short, revision of research questions is typical during the formative
design stages, and often it is desirable. Of course, once data collection has
begun, revision of the research question is no longer functional or appropriate.
specific terms throughout this book to describe the various types or classes of
variables found in research designs. Specifically, the terms independent
variable and dependent variable have been used in both experimental and
descriptive research to define different types of variables. In true experimental
designs (see Chapter 7), the researcher attempts to examine causality by sys-
tematically varying or altering one variable or set of variables and examining
the resultant changes in or consequences for another variable or set of vari-
ables. In such experiments, the variable that is varied or altered is called the
independent variable. More specifically, the independent variable is the vari-
able that is manipulated or controlled in a study. Usually an experiment
involves two or more levels of the independent variable (for example, treat-
ment and no treatment), sometimes referred to as conditions. For example, in
a study that compares cognitive versus interpersonal treatments for depres-
sion, the type of treatment (cognitive vs. interpersonal) would be the inde-
pendent variable.
To examine the effects of the manipulation of the independent variable,
concomitant changes in another variable, the dependent variable, are observed.
In an experimental study, changes in the dependent variable are supposed
to depend on or be influenced by changes or variations in the independent
variable. In the previous example, the comparison between cognitive versus
interpersonal treatments is made by measuring some dependent variable.
One example of a dependent variable is the depression scores on a stan-
dardized test (for example, the MMPI-Depression scale). In a true experimen-
tal design we infer that a change (if one exists) in the dependent variable was
caused by the manipulation of the independent variable. Thus, the terms in-
dependent variable and dependent variable have causal implications. These
terms are sometimes used to describe the variables in nonexperimental stud-
ies as well. For instance, the predictor variables in a regression equation are
sometimes referred to as independent variables, and the criterion variable is
sometimes referred to as a dependent variable. This can be confusing because
of the notions of causality implied in the terms. To alleviate this type of con-
fusion, we will primarily utilize the terms independent variable and depen-
dent variable to describe variables in experimental studies, although some
exceptions will have to be made. Independent and dependent variables are
discussed further in Chapters 12 and 13.
STIMULUS QUESTIONS
Initial Reflections on Possible
Research Topics
The purpose of this exercise is to promote reflec- 4. Who in particular could help you to learn
tion on possible research topics. You might want more about these topics?
to respond to these questions now, or spend several 5. Consider your responses to the questions
hours examining recent articles in some counseling above, and rank order the two to three topics
journals and then consider these questions. that might be most appropriate to further
examine.
1. Brainstorm a list of topics that excite and
6. Finally, spend a few days beginning to ex-
motivate you. Are there any themes in this
plore the top ranked topic, finding a few
list? Rate the degree to which each topic
recent articles on this topic, talking to rele-
addresses an important and pressing societal
vant peers and faculty to collect additional
need (1 = low, 10 = high).
information about the topic, and seeing how
2. Can you narrow the list and identify two to
you now evaluate this topic. Has your interest
three topics on which you might like to conduct
increased or decreased? Why?
a study? Evaluate each of the topics (1 = low,
10 = high) in terms of how passionate you are If this topic continues to interest and
about this topic as well as how proud you would intrigue you, continue to learn more about it
feel about contributing your scholarly efforts. by searching the literature and talking to
Indicate why you rated each topic as you did. knowledgeable faculty. Conversely, if the first
3. Now list as many resources as possible in topic loses appeal, you might want to explore
your environment to help you find relevant your second ranked topic to obtain additional
literature on each of these topics to expand information to determine if this topic might
your knowledge. better fit your interests.
Choosing Research
Designs
CHAPTER
4
65
66 part 1 philosophical, ethical, and training issues
In this chapter, as well as the next, we will focus more on what we mean by
systematic and controlled.
The basic task of the experimenter is to design research in such a way as
to describe a phenomenon or identify relationships between constructs while
ruling out as many plausible rival hypotheses or explanations as possible. The
goal, put simply, is to more fully understand a phenomenon or construct, even
though many sources of bias and confounding variables might distort that
understanding. Perhaps an analogy might help. Ever since human beings be-
gan harvesting grains, there has been a need to separate the grain itself (the
wheat) from its protective shield (the chaff), a dry, coarse, inedible material.
In a way, the chaff gets in the way of digesting the wheat. In a similar way, the
researcher wants to isolate the constructs of interest to his or her research
question (the wheat) and remove as much as possible of any other constructs
(the chaff) that might contaminate, confound, bias, or distort the constructs
of interest. Although the analogy of separating the wheat from the chaff is an
oversimplification, it does highlight the essential task of the scientific method:
isolating the constructs of interest and trying to draw useful conclusions about
those constructs. Any particular study can never completely eliminate all the
explanations; some types of explanations will be left untested. This is a very
crucial point to understand about research design; we will elaborate on this
point throughout the chapter.
How does the researcher separate the “wheat” from the “chaff”? The
basic tool of the researcher is what we call research design. Research design
involves developing a plan or structure for an investigation, a way of conduct-
ing or executing the study that reduces bias, distortion, and random error.
Different research designs have different strengths and weaknesses, and each
will minimize different types of bias. Sometimes bias is also referred to as error,
error variance, or noise. One of the most critical decisions in research is select-
ing a research design whose strengths and weaknesses help the researcher to
examine specific research questions in a valid and systematic manner by reduc-
ing as many rival hypotheses or explanations as possible while isolating the
relevant variables of interest. Research design, then, is a set of plans and pro-
cedures that researchers use within scientific inquiry to obtain empirical evidence
(data) about isolated variables of interest. From the evidence, or data, the
researcher then draws inferences about the constructs in his or her research
question. We say “inferences” because the researcher can never rule out all of
the rival hypotheses between, say, two constructs, A and B.
Internal validity
High Low
Experimental Descriptive
High
field field
External validity
This study is high on external validity because of its large number of par-
ticipants and because the sample was obtained from across the United States
and Canada. Conversely, it is not known how many people were asked but
declined to participate, or whether the methods of recruiting participants
resulted in any systematic biases in the sample. The study is low on internal
validity, because no variables were manipulated. Thus, it is not possible to
make causal statements concerning the effects of therapist practices on thera-
peutic outcomes. In addition, any of a number of other counselor behaviors
may have affected the counseling process and outcome. The study used an open-
ended (free recall) format with an unspecified time (for example, one or five
years ago), which might have resulted in some distortion. For other examples of
descriptive field studies, see Erdur, Rude, and Baron (2003); Lichtenberg and
Tracey (2003); and Woodhouse, Schlosser, Crook, Ligiero, and Gelso (2003).
For researchers interested in conducting descriptive field studies, Chapter 9
(on single-subject designs) and Chapter 19 (on process research) provide
guidance, with an emphasis on therapeutic counseling. Chapters 10 and 11
provide an overview of some common quantitative descriptive designs and
qualitative designs in particular, whereas Chapter 8 covers quasi-experimental
designs.
(including mixed methods) and programmatic research are also basic con-
siderations in selecting a design.
Typically, research on a particular question is conducted within an existing
body of knowledge. Thus, it is imperative for the researcher to ascertain both
what the previous research suggests about a particular topic area and the kinds
of questions that remain unanswered. As a researcher forms a particular re-
search question, it is important to ask what kind of knowledge will add to the
existing literature. At the same time, the researcher must evaluate what type of
research design will provide the kind of knowledge that is needed. Perhaps a
qualitative study would add the most useful knowledge or basic normative
information about a topic. Or perhaps an experimental study that isolates the
interactive effects of two independent variables would help explain previous
contradictory findings. Thus, the utility of a research design needs to be evalu-
ated in the context of the existing research knowledge in a given area.
Equally important is the type of research design used and the inferences
drawn to develop the existing knowledge bases. The types of research designs
used affect the types of inferences made in developing a knowledge base.
Thus, if a particular topic has been predominantly researched in laboratory
settings, then perhaps research focused on field settings will now add the most
useful knowledge that would address questions about external validity in that
area. Or if a topic has been investigated through tightly controlled experi-
mental studies, then perhaps descriptive studies might now add some useful
information. Any type of design can be overused in a particular area, a con-
dition that can produce an unbalanced and subsequently weak knowledge
base. (In Chapter 17 we present a brief overview of the social influence liter-
ature in counseling, and we provide details of an example of this problem of
overusing any particular design.)
Many times inexperienced researchers do not read the method sections of
research reports. One of this book’s authors admits (sheepishly) to commit-
ting this sin during much of his early graduate studies. Instead, students usu-
ally read the introduction and then skip to the discussion. Although this might
suffice for obtaining content knowledge in an area, it misses the important
aspect of learning about how the studies were conducted. We suggest a simple
technique that inexperienced researchers can use in examining a body of lit-
erature. Make a copy of Figure 4.1, and as you read the method sections of
various studies within a topic area, place the study into the appropriate cell.
It should quickly become apparent which designs have been used and perhaps
overused in a particular topic area.
It is also important to note that different designs require different re-
sources and have different costs. For instance, a researcher might decide that
a descriptive field study was needed to examine the relationship between
counselor techniques and a client’s perception of the working alliance. But
should she do a correlational study or use an intensive single-subject design?
The answer to this question should be obtained, in part, by examining the re-
sources available. To do the correlational study, the researcher would proba-
bly need to find 30 to 50 client–counselor dyads. It may take a great deal of
76 part 1 philosophical, ethical, and training issues
work to find these dyads, but the data analyses may be fairly easy and pain-
less. On the other hand, for an intensive single-subject study the researcher
may have an easy time finding one dyad willing to participate. However, a
rather involved, intensive process of analyzing the data will likely ensue. Thus,
a researcher must not only examine the resources available, but also look at
the costs of choosing a particular design.
In choosing a research design, it is also of utmost importance to remember
that each experiment has strengths and weaknesses, and moreover that each
experiment is typically flawed in some way. Gelso (1979) understood this idea
when he offered the Bubble Hypothesis, which suggests that doing research is
similar to trying to apply a sticker to a car windshield. When an air bubble
forms between the sticker and the windshield, the owner presses the bubble in
an attempt to eliminate it. No matter how hard he or she tries, however, the
bubble reappears somewhere else. The only way to get rid of the bubble is to
throw the sticker away, but then the owner is left without a sticker. In a similar
manner, every piece of research and every research design is flawed (has a
bubble). The different research designs will have different limitations and
strengths (the different designs may change the location of the bubble), but no
single design can entirely eliminate the bubble. The researcher can either stop
doing research (throw the sticker away) or be cognizant of the size and location
of the bubble for any given design.
The Bubble Hypothesis clearly points out that if only one type of research
design is advocated by a discipline, then the bubble will always be in a similar
place on the sticker—all the research will contain similar flaws or blind spots.
On the other hand, if multiple research designs are advocated, each with dif-
ferent bubble locations, then the cumulative effect will be a clearer, more accu-
rate picture of the topic under examination. Viewed in this manner, the
usefulness of a particular design at a particular time is determined by the loca-
tions of the bubbles in the studies that have previously addressed this ques-
tion. This type of reasoning led Gelso (1979) to suggest that all types of
research designs are useful and that knowledge can be advanced only when
the same problem is examined using multiple design strategies. He thus ar-
gued for paradigmatic diversity. Over the years there has been a growing con-
sensus within (and outside) the field of counseling that the discipline is
strengthened when alternative designs are used (see, for example, Creswell, 1994;
Hanson, et al., 2005; Haverkamp, Morrow, & Ponterotto, 2005; Polkinghorne,
1984; Tashakkori & Teddlie, 2001).
The Bubble Hypothesis and the need for paradigmatic diversity under-
score the importance of programmatic research on a particular topic. Put
another way, a series of investigations, conducted by the same or different
researchers, that successively extends our knowledge bases along a particular
line of research on a particular topic is highly desirable for the profession. The
reason is that a series of related investigations that build on each other tends
chapter 4 choosing research designs 77
STIMULUS QUESTIONS
Following are two exercises that are designed to Chapter 4 to further explore the advantages and
help students to apply some of the material from disadvantages of different types of research designs.
chapter 4 choosing research designs 79
e. What are the advantages or limitations of c. Would this study be high or low on exper-
the design on the conclusions you could imental control and external validity?
draw from the study? d. What outcomes would you predict would
4. In your topical area of interest, conceptualize be found from the study?
a study that utilizes an experimental field design. e. What are the advantages or limitations of
a. What would be the intended purpose of the design on the conclusions you could
this study? draw from the study?
b. Describe the methods you would utilize to
conduct the study.
Validity Issues
in Research Design
CHAPTER
5
81
82 part 1 philosophical, ethical, and training issues
will result in clinical activities and policies that ultimately result in benefits to
those people with whom counseling psychologists work. Although there are
many ways to look at the validity of research, the framework presented by
Shadish, Cook, and Campbell (2002) represents the current state of this evolu-
tion. Shadish et al. have created a taxonomy that classifies validity into four
types: statistical conclusion validity, internal validity, construct validity, and ex-
ternal validity. This typology was derived from Campbell and Stanley’s (1963)
original conception of internal and external validity, as further refined by Cook
and Campbell (1979). Other discussions of validity are presented by Bracht and
Glass (1968) and by Wampold, Davis, and Good (1990).
A second issue is that no study will be able to rule out every threat to the
validity of the conclusions reached. A study for which the threats to validity
are not severe enough to discredit the conclusions completely will remain
useful scientifically because the conclusions reached can be tentatively ac-
cepted. Additional studies should be designed that will rule out the threats
that plagued the original study. Through the accumulation of studies, threats
to a conclusion can be ruled out and a strong statement can be made. For
example, no single study of smoking and health has unequivocally established
a causal relationship between smoking and disease; however, the accumula-
tion of many studies (and there have been thousands) rules out, with near
absolute certainty, any threats to this conclusion. (At one time, the Tobacco
Institute claimed that no one study had ever scientifically established an un-
equivocal causal relationship between smoking and disease—in isolation, this
is a true statement, but it ignores the accumulation of evidence.)
A third issue is that, more or less, most threats discussed here are possibly
present in any study. However, more important is the determination of the
plausibility of a threat in a particular study and its implication for the conclu-
sion. Thus, the validity of a conclusion is suspect if a threat is plausible and the
threat created conditions that could have produced the evidence supporting the
conclusion. For example, suppose that it is concluded that a treatment was
effective based on superior outcomes obtained when the treatment was admin-
istered to volunteers vis-à-vis the outcomes of nonvolunteers who did not
receive the treatment. Given evidence that volunteers respond differently than
nonvolunteers (e.g., Rosenthal & Rosnow, 1969), it is plausible that the non-
random assignment affected the outcomes; moreover, it would be expected that
volunteers would perform better given their enthusiasm for the treatment, and
thus the nonrandom assignment is not only a plausible threat, but also could
well have explained the superiority of the treatment found in this study. Thus,
the validity of the conclusion is suspect because a threat was plausible and
provided an alternative explanation. On the other hand, had the same study
produced a result that supported the conclusion that the treatment was no
more effective than no treatment (i.e., no significant differences between the
two groups), the threat remains plausible but would not likely have accounted
for the result because even with volunteers, the treatment did not produce
superior outcomes compared to no treatment. In the latter case, the conclusion
retains some degree of validity despite the plausibility of the threat.
84 part 1 philosophical, ethical, and training issues
On the other hand, some threats, although logically possible, are not
plausible. Suppose that a treatment is found to be effective in reducing depres-
sion related to career-ending knee injuries to athletes, and the conclusion is
made that the treatment is useful for the treatment of career-ending orthopedic
injuries. Technically, the experiment did not include participants with non-
knee orthopedic injuries, but there is little reason to believe that depression or
its treatment would differ for patients with knee injuries versus those with
shoulder injuries. So although a threat exists, the plausibility that it renders
the conclusion invalid is low.
A fourth issue is that there are often tradeoffs to be made in the design and
implementation of research. Designs that increase the certainty of both causal
inferences (internal validity) and statistical conclusion validity may decrease
the certainty of generalizing inferences from samples to populations (external
validity) or the meaning of the operations (construct validity). Likewise, de-
signs that increase the certainty of inferences from samples to populations or
about constructs may do so at the expense of decreasing the certainty of infer-
ences about the extent of relationships or causality. The point is that there
may be trade-offs with different types of research designs, not only with
regard to these four types of inferences, but also with respect to other factors.
Gelso (1979) used the metaphor of trying to eliminate a bubble underneath a
sticker to describe these trade-offs—removing the bubble in one place creates
a new bubble somewhere else. Nevertheless, a bubble that casts grave doubts
on the conclusions made from a study cannot be discounted simply because
the presence of bubbles is inevitable.
A final issue is that considerations of validity are important for those who
design and conduct research and those who consume research. A researcher
who designs a study should consider each of the possible threats vis-à-vis the
possible outcomes of the study to determine whether when the study is fin-
ished relatively valid conclusions may result—if not, the study should not
be undertaken. The design of a study should be modified to reduce threats
to validity, and sometimes various ancillary aspects can be incorporated to
address various threats—such aspects will be discussed throughout this book.
Of course, there are no guarantees that the study will produce valid conclu-
sions, because despite the researcher’s best efforts unexpected events occur.
Furthermore, publication or dissemination of results does not ensure that
conclusions are valid, and consumers of research should examine threats inde-
pendently. For example, generally accepted conclusions that interventions are
effective sometimes are incorrect, as has been the case with D.A.R.E. (Drug
Abuse Resistance Education, a program of classroom lessons delivered by
police officers) to reduce drug use (e.g., Thombs, 2000).
used in this study and scores on the ABC Test. In our example, there was a sta-
tistically significant relationship between the independent and dependent
variables. Often, one of the major inferences made in interpreting research con-
cerns the existence of a relationship between (or among) the variables in the
study. The researcher may conclude that there is a relationship or that there is
no relationship. Statistical conclusion validity refers to the degree to which the
researcher has come to the correct conclusion about this relationship.
The second major inference to be made in interpreting research is an
answer to the following question: Given that there is a relationship between the
variables, is it a causal relationship? In our anxiety example, the researcher
concluded that the statistically significant differences between the anxiety levels
for the two groups was due to (i.e., caused by) the addition of the exercises.
Internal validity refers to the degree of certainty with which one can make
statements about the existence of a causal relationship between variables.
The third major type of inference is construct validity. Construct validity
concerns how well the variables chosen to represent a hypothetical construct
actually capture the essence of the hypothetical construct. One of the major
issues with construct validity involves confounding—the possibility that what
one researcher interprets as a causal relationship between constructs A and B,
another researcher might interpret as a causal relationship between A and C,
or between D and B. In our example it was presumed that the ABC Anxiety
Test used in a contrived situation with a confederate was a suitable measure
of the social anxiety of the participant, and that the particular exercises used
in this study were truly in vivo exercises appropriate for social interactions. If
the operationalizations of the constructs of this study were adequate, then the
causality attributed to the independent and dependent variables justifies state-
ments about the causality of the constructs used in the research hypotheses.
Thus, construct validity refers to the degree to which the measured variables
used in the study represent the hypothesized constructs. In our example, in
vivo exercises (in conjunction with cognitive therapy) were the putative cause,
and social anxiety was the putative effect.
To be of any value to researchers and practitioners, the causal relationship
between the hypothesized constructs must be generalizable to units (typically
persons, but not always), treatments, outcomes, and settings other than those
in the particular study. In the context of our fictitious example, to what extent
can we generalize the use of in vivo behavioral exercises to other socially
anxious people? External validity refers to the degree to which the causal
relationship is generalizable across units, treatments, outcomes, and settings.
though for any of a variety of reasons the researcher did not find it. This type
of error is called a Type II error. One of the major reasons for Type II errors
is that variability in the participants’ responses tends to obscure true relation-
ships. This variability, often called error variance, can be thought of as static
on a radio receiver that obscures the true signal. Even if the true signal is
strong, an electrical storm can generate sufficient static that one cannot hear
a favorite program. Conditions that create error variance lead to threats to
statistical conclusion validity (more on this later).
It is important to realize that one is never totally certain that a statistically
significant result indicates that a true relationship exists. Similarly, a nonsignif-
icant result does not absolutely indicate that no relationship exists. Nevertheless,
various factors or threats can decrease the confidence with which we conclude
that there either is or is not a true relationship between variables.
The statistical testing of null hypotheses is deeply embedded in probability
theory and the philosophy of science and remains a source of controversy. In-
deed, criticism of null hypothesis statistical testing (NHST) began shortly
after Sir Ronald Fisher proposed the procedure, and has continued ever since
(Nickerson, 2000; Tryon, 2001). The problem became so acute that the American
Psychological Association Task Force on Statistical Inference was convened
(Wilkinson & the Task Force on Statistical Inference, 1999) to make recommen-
dations on appropriate procedures. Despite the controversy, the NHST is widely
used, and when applied correctly provides information critical to making
research inferences (Nickerson; Wilkinson et al.). Nevertheless, it is recom-
mended that other statistical information, such as effect size and confidence inter-
vals, be considered and reported (American Psychological Association, 2001;
Wilkinson et al.). Further issues with regard to NHST are beyond the scope of
this book, and likely will be discussed in basic statistics courses.
The remainder of this section discusses the various threats to statistical
conclusions validity.
are violated, the researcher and consumer may be misled about the probabil-
ities of making Type I and Type II errors. For example, if the p level of a sta-
tistical test is set at 0.05 and the test is statistically significant (that is, p < .05),
one commonly believes that the likelihood of incorrectly concluding that there
is a true relationship is less than 5%. However, if the assumptions of the test
are violated, this probability may be much higher. Thus, the statistical con-
clusion validity is reduced because of the increased chance of making a Type
I or II error. The pernicious aspect of violated assumptions is that it is diffi-
cult to determine whether or not there are violations, and, if so, the degree to
which the violations affect the results. Although violation of some assump-
tions carries with it little risk, other violations often create conditions that
lead to incorrect conclusions. One of the assumptions of most parametric sta-
tistical tests is the independence of observations. If this assumption is violated
by ignoring the dependence among clients seeing the same therapist, for exam-
ple, the probability of falsely concluding that a particular treatment is effec-
tive can be dramatically increased (Wampold & Serlin, 2000). We advise you
to be aware of the assumptions of statistical tests and the consequences of vio-
lating those assumptions.
Inaccurate Effect Size Estimation There are instances when effects detected
in studies will be inaccurately estimated. For example, correlation coefficients,
particularly in small samples, can be dramatically affected by outliers, so one
unit with extreme scores on both measures will result in a sizable correlation
when no true correlation exists. Some statistics are biased in the sense that
they consistently overestimate population effects, which is the case for R2 (the
sample value of the proportion of variance accounted for in multiple regression).
Reporting R2, particularly in small samples, provides an inflated sense of the
relationship between the independent variables and the outcome variable.
X O1
Design 2: Nonequivalent group posttest-only
design O2
R X O1
Design 3: Randomized posttest-only design
R O2
Even if the threats to the internal validity of the studies can be ruled out, it
would appear that the manipulation of the independent variable caused the
concomitant change in the dependent variable, and not vice versa. However,
the direction is not as clear in designs in which the independent variable is not
manipulated. Consider studies in counseling that examine counselor empathy
and therapeutic gains in clients; several studies have found a positive relation
between these two variables (Mitchell, Bozarth, & Kraft, 1977). Does the
empathy of the counselor cause client progress, or does client progress cause
the counselor to be more empathic? In other cases, the hypothesized cause
clearly precedes the effect in time, but the measurement of the cause is as-
sessed retrospectively. For example, it may be hypothesized that aspects of
the parental attachment cause transference in therapy (e.g., Woodhouse,
Schlosser, Crook, Ligiero, & Gelso, 2003). Clearly, the temporal precedence
is clear; however, if parental attachment is measured concomitantly with the
transference, it is not clear that what is measured (i.e., recollections of attach-
ment) precedes the transference in time.
History History refers to an event that transpires during the time when the
treatment is administered and may affect the observations. Thus, history
refers to any events in the participants’ school, work, or home life (for in-
stance, a television program, a newspaper article, a term paper, or the death
of a family member). In our example, history is a threat in Design 1 because
a television special on sexual abuse may have been aired while the interven-
tion was being administered. There is no way to determine whether it was the
television special or the psychoeducational program that resulted in the in-
crease in knowledge.
The primary way to control history is to use two groups (as in Designs 2
and 3) so that the event affects both groups equally (or nearly equally). In our
example, the participants in the treatment and control groups would have
equal access to the television special, equalizing this threat. (Note that in
Design 2, students in one class might stay up later, possibly due to increased
homework or some other reason unique to that group, making late night specials
more accessible to them than to the other class.) Still, try as the researcher
might, it is possible that an event could occur that would affect only one of
94 part 1 philosophical, ethical, and training issues
the groups. The threat that occurs from an event that affects only one of the
groups is called local history.
Threats due to history can be reduced in a number of other ways. First,
observations on the groups should be made at the same time. For example, in
Design 3, O1 and O2 should occur at the same time. Delaying observations for
one group leaves open the possibility that some important event may occur
after one group is tested but before the other is tested, creating a threat due to
local history. Second, the shorter the treatment, the less opportunity there is
that an event will occur. Third, the participants can be isolated during the treat-
ment, thereby reducing the likelihood that an extraneous event will affect
them. This is similar to sequestering a jury; however, this is extremely difficult
to accomplish with human participants in naturalistic settings.
Regression Regression refers to changes in scores due to the fact that gen-
erally, participants who score low on the pretest will score higher on the
posttest, and participants who score high on the pretest will score lower on
the posttest. (For this reason, regression often is referred to as regression
toward the mean.) As an example, consider a batting champion in baseball.
Obviously, he obtained this title because he is a good hitter. Still, his batting
average for a given year is also due in part to serendipity. Perhaps there was
a warm spell in his home city in the spring, the player next in the lineup had
a good year (and so the opposing team could not pitch around him), he was
injury-free, he had more than his share of luck as several balls just eluded the
outstretched gloves of fielders, his personal life was stable, and so on. It is
unlikely that all these factors will be favorable the next year, and so it is log-
ical to predict that although he likely will have another good year, he will not
be the batting champion again. (Indeed, batting champions rarely repeat.)
Similarly, someone who scores low initially is likely to score higher the
next time around. Consider the example of a state bar examination. The ex-
aminee who scores the lowest during an exam administration is obviously
deficient in his or her knowledge. However, because the examination does not
perfectly measure knowledge of the law, this score is also due to other factors.
The examinee may have been late to the examination (and therefore is more
chapter 5 validity issues in research design 95
anxious), may have misunderstood some questions, may have missed all ques-
tions on which he or she guessed, and so forth. On re-examination, all of these
factors are unlikely to be in operation and he or she will do better, although
still below average. (Interestingly, the examinee might attribute the gain to
better study habits!)
Regression is a problem, especially when an experimenter chooses research
participants because of their extreme standing on some variables (such as high
levels of depression). If participants are selected based on their extremely low
scores, then as a group they can be expected to score higher on the posttest,
regardless of whether or not they have received any treatment. Again, con-
sider Design 1 in Figure 5.1. Suppose that participants were selected for a
study because they fell above a certain cutoff score on a paper-and-pencil test
of depression (higher scores indicate greater depression). Upon subsequent
testing (that is, at posttest), these participants generally will score lower (less
depressed) than they did previously. Therefore, a statistically significant dif-
ference from pretest to posttest (that is, O1 > O2) may be due entirely to sta-
tistical regression. Design 3 controls for regression because the participants
are randomly assigned (that is, have comparable scores), and thus the regres-
sion toward the mean for both groups will be about the same.
in their own right, it may well be that the most anxious participants will drop
out of the cognitive therapy plus exercises group (treatment group) rather
than complete the exercises (a not-uncommon avoidance reaction). Because
the participants who drop out of the first group are the most anxious, their
attrition will tend to decrease the anxiety scores in this group (that is, decrease O1)
and could be responsible for a significant difference between the groups (that
is, O1 < O2) in favor of the treatment group.
A third application of Design 3 will demonstrate how differential attrition
can act against ascertaining that a treatment is effective. Consider a treatment
for depression that is effective and provides an immediate palliative effect.
Again suppose that the participants are randomly assigned to the treatment
condition and to a waiting-list control group (the participants in this group
will receive the treatment after the study ends, if it is found to be effective).
Because depression is a particularly distressing disorder, the most depressed
participants in the control group might be most inclined to drop out of the
study and seek treatment elsewhere. If this does in fact occur, the control
group scores will reflect a drop in depression because the highest scores have
been removed. If this drop due to differential attrition is about the same as the
effect of the treatment, then the posttest scores will be about equal (that is,
O1 = O2), even though the treatment was effective!
In short, attrition can often serve as a threat to internal validity. However,
to some extent, the effects of differential attrition often can be assessed by
administering a pretest. This topic will be discussed in Chapter 7.
Testing Testing refers to changes in scores on a test due to taking the test
more than once. Participants’ scores often improve due to familiarization with
the test, recall of items and previous responses, and so forth. For example,
participants might be asked to perform anagram tasks both before and after
a problem-solving intervention. However, the practice performed on the first
anagram task might account for improved performance on the posttest, apart
from the effect due to treatment. Testing is a threat in Design 1 because
improvement in scores from the pretest to the posttest could be due to taking
the test a second time. Effects of testing should always be considered when
pretests are given. Testing is not a threat in Designs 2 and 3 because the par-
ticipants are tested only once in these designs.
relationship among the variables is generalizable if participants are strangers and do not have the
beyond the study to other units (usually people), opportunity to meet.
settings, treatments, and outcomes. Although It is also possible to reduce threats to validity
one can never establish each of the four types of by building into the study some aspects that con-
validity with total certainty, researchers establish trol for the threat. Consider the example of the
estimates of validity by ruling out as many study in which photographs of counselors of
threats to the validity as possible. Most impor- both genders were presented to the participants.
tant, different types of designs typically repre- Recall that personal attractiveness was a poten-
sent trade-offs with regard to the four types of tial confound in that study. To control for the
validity. confound, the researchers could have judges rate
A wide range of different threats to each of the personal attractiveness of various photo-
the four types of validity exist. How does one graphs and then match them so that the personal
assess the severity of a threat? In some instances, attractiveness of the photographs was constant
statistical tests can be used to determine whether across the groups.
a threat is problematic. For example, if pretests In sum, our discussion of validity provides a
are administered, differences among participants framework for assessing the types of inferences
who drop out of a study can be compared statis- made in research and the subsequent validity of
tically to those among remaining participants. a study. The fact that many threats were pre-
Or external validity can be assessed by examin- sented indicates that many things can weaken or
ing the statistical interaction between the inde- strengthen any particular study. Keep in mind
pendent variable and some person, setting, or that no research can be designed that is not sub-
time variable. ject to threats to validity to some degree, which
A second way to assess validity is to logically is in essence the thesis of Gelso’s (1979) Bubble
examine the likelihood of a threat’s occurrence. Hypothesis. The objective is to design and con-
In some instances, it is very unlikely that a par- duct research in such a way as to minimize the
ticular threat is problematic, even in the absence threats and maintain the possibility of obtaining
of direct evidence. For example, although matu- interpretable findings. In this respect, program-
ration may be a threat in some designs, if the matic research is needed because studies can
treatment lasts only one hour, participants are build on each other, and a threat to one study
very unlikely to mature much during that time. can be ruled out in a future study. In fact, pro-
Or if the pretest is a commonly used test and the grammatic research on a given topic that exam-
treatment is lengthy, interaction of testing and ines similar variables over time is essential to
treatment probably will have little effect on the creating useful knowledge bases within scientific
results. Diffusion of treatments will be impossible inquiry in the counseling profession.
STIMULUS QUESTIONS
Validity Issues
1. Select two studies in your area of interest. 2. A researcher is interested in studying racial
a. Identify the major conclusions made by and ethnic identity. The researcher uses ethnic
the authors. minorities attending the researcher’s university
b. Identify three major threats to the validity as a sample. What would be some important
of these conclusions. validity considerations of using this sample?
c. Discuss how these studies could have been 3. Experimental studies are often criticized on
designed to better rule out these threats. the grounds that, due to the desire to control
chapter 5 validity issues in research design 107
extraneous variables and exert experimental with high internal validity and low external
control, the results are not applicable to real- validity and another with low internal validity
world situations. Discuss both sides of this and high external validity. Discuss how each
argument (i.e., why experimental studies are has handled the tradeoff between internal and
useful and why the results are not useful). external validity. Which study is more inform-
4. Search the literature to find several studies ative? Finally, how do the studies compliment
bearing on the same question. Identify one study each other in terms of building knowledge?
6 CHAPTER Ethical Issues
in Counseling
Research
Ethics are not simply proper etiquette, but rather “they are expressions of our
values and a guide for achieving them” (Diener & Crandall, 1978, p. 14). In
essence, “ethics is a generic term for various ways of understanding and exam-
ining the moral life” (Beauchamp & Childress, 2001, p. 1). Ethical principles
help researchers achieve their goals while avoiding strategies that compromise
their values, and ethics help them make decisions when their values are in con-
flict (Diener & Crandall). We believe that ethics are nothing less than central
to the conduct of research. Just as morality is a part of everyday living, ethics
are a way of living that permeates the research enterprise. Because of the cen-
trality of ethics in research, this chapter is placed toward the beginning of the
book so that ethical reasoning can be integrated into basic design considera-
tions not as an afterthought, but as an intrinsic feature of the research endeavor.
Indeed, research suggests that students in counseling training programs regard
ethical training as essential for their professional roles (Wilson & Ranft, 1993).
To assume that being a counseling researcher involves technical research design
skills alone is to have a very incomplete picture. We maintain that it is essen-
tial for researchers to be aware of their ethical responsibilities to research par-
ticipants, co-workers, the profession, and society as a whole.
In this chapter we focus on the investigator’s responsibility in two general
categories: ethical issues related to scholarly work and ethical issues related to
participants. In today’s multifaceted world these topics take on a complexity
not even imagined several decades ago. It is important to underscore at the
108
chapter 6 ethical issues in counseling research 109
outset that ethical issues in research are rarely cut-and-dried, and sometimes
one ethical principle will conflict with another to create a tangled and murky
dilemma. Our main goals in this chapter are to (1) introduce and sensitize the
reader to the ethical issues involved in counseling research; (2) underscore the
complexity of real-life ethical dilemmas, which sometimes do not have clear
answers; and (3) discuss common strategies and the reasoning process for
designing research with ethical rigor.
The first section of this chapter discusses fundamental ethical principles as
well as virtue ethics that form the core of our professional values. Specifically,
we will discuss the fundamental ethical principles of nonmaleficence, benefi-
cence, autonomy, justice, and fidelity. In addition, we will introduce ethical
guidelines that have been provided by the American Psychological Association
and the American Counseling Association. Next we introduce the notion of
virtue ethics and specifically highlight prudence, integrity, respectfulness, and
benevolence. In the second section of the chapter we discuss ethical issues
related to scholarly work, specifically (1) execution of the research study, (2)
reporting the results, (3) duplicate and piecemeal publication, (4) publication
credit, and (5) plagiarism. In the final section we discuss ethical issues per-
taining to participants: (1) risks and benefits, (2) informed consent, (3) deception
and debriefing, (4) confidentiality and privacy, and (5) special considerations
for treatment issues. We end with a brief section on responding to ethical
dilemmas.
have promoted the teaching and application of the foundational ethical prin-
ciples (see Kitchener & Anderson, 2000). In this chapter we focus on five fun-
damental ethical principles: nonmaleficence, beneficence, autonomy, justice,
and fidelity. In essence, these fundamental ethical principles are central but
implied building blocks for the professional codes of the ACA and APA. We
briefly discuss these general principles to clarify the essence of ethical issues
and facilitate an understanding of professional ethical codes. Readers inter-
ested in a fuller discussion of such fundamental ethical principles are directed
to Beauchamp and Childress (2001), Diener and Crandall (1978), Drane
(1982), Kitchener (1984), and Lindsey (1984).
Nonmaleficence
Diener and Crandall, in their book Ethics in Social and Behavioral Research
(1978), succinctly concluded that “the most basic guideline for social scientists
is that subjects not be harmed by participating in research” (p. 17). This cen-
tral principle has been referred to as the principle of nonmaleficence (above all
do no harm; Beauchamp & Childress, 2001). This includes not inflicting inten-
tional harm and avoiding the risk of harming others. Thus, it is the responsi-
bility of the investigator to plan and act thoughtfully and carefully in designing
and executing research projects, because harm can occur intentionally or unin-
tentionally. Kitchener (1984) noted that a number of ethicists and psycholo-
gists have argued that nonmaleficence should be the paramount ethical principle
in applied psychology (Beauchamp & Childress; Brown, 1982; Frankena,
1963; Rosenbaum, 1982; Ross, 1930). Thus, these professionals have argued
that if we must choose between harming someone and perhaps helping another
person, the strongest obligation would be to avoid harm. Diener and Crandall
have argued that nonmaleficence can be superseded only if volunteers know-
ingly participate and the benefits are of great import.
Beneficence
Beauchamp and Childress (2001) concluded that acting ethically not only
involves preventing harm, but also contributes to the health and welfare of
others. Doing good for others is beneficence. This central ethical principle is
the essence of the goal of counseling—to help people resolve problems that
they have been unable to resolve on their own. Moreover, beneficence consti-
tutes the core of the ethical principles advocated by the APA and ACA. In the
Preamble to the EPP (APA, 2002, p. 4), the Ethics Code “has as its goals the
welfare and protection of . . . individuals and groups.” Likewise, the first sen-
tence in the Preamble to Ethical Standards (ACA, 2005, p. 3) proclaims that
“members are dedicated to the enhancement of human development through-
out the lifespan.”
Inherent in beneficence is competence. If our value is to help others, par-
ticularly those in need who come to rely on our services, then we have an obli-
gation to help others as competently as possible. Such reasoning has a number
chapter 6 ethical issues in counseling research 111
Autonomy
The principle of autonomy centers around the liberty to choose one’s own
course of action, including freedom of action and freedom of choice (Kitchener,
1984). The principle of autonomy is woven into American political institutions,
law, and culture. Not surprisingly, Rokeach (1973) found that Americans
ranked individual freedom as one of their most esteemed values. In many ways,
autonomy is the cornerstone of subjects’ rights to voluntarily participate in psy-
chological research, or conversely to decline to participate. Since the Nuremberg
trials after World War II, the principle of autonomy has received increased atten-
tion in research. At the center of this attention is the notion of informed consent,
or educating potential subjects about a particular research project so that they
can make informed decisions about participation.
Justice
The principle of justice implies fairness (Benn, 1967). Because the quantity of
services and goods in any society is limited, there are conflicts between people.
Thus, a vast array of laws has developed as part of our judicial system for
deciding what is fair. In essence, the principle of justice is based on the
assumption that people are equals. Thus, as initially suggested by Aristotle,
equals should be treated as equals, and unequals should be treated unequally
only in proportion to their relevant differences (Beauchamp & Childress,
2001; Benn). Gender and race are not relevant characteristics for deciding
access to mental health services, but this is not to suggest that gender and race
112 part 1 philosophical, ethical, and training issues
Fidelity
The principle of fidelity implies faithfulness, keeping promises or agreements,
and loyalty (Ramsey, 1970). This principle applies directly to voluntary in-
terpersonal relationships, including counselor–client, student–teacher, and
researcher–participant. Not fulfilling a contract (by, for example, engaging in
deception or breaching confidentiality) is a violation that infringes upon the
other individual’s choice to enter into a mutually agreed-upon relationship.
Issues of fidelity and trustworthiness are central to the helping professions
such as counseling. The principle of fidelity is important for the reputation of
the profession as well as for individual professionals in their work as coun-
selors, supervisors, consultants, educators, and researchers.
VIRTUE ETHICS
Within the past decade virtue ethics have received increased attention (see,
for example, Beauchamp & Childress, 2001; Kitchener & Anderson, 2000;
Meara & Day, 2003; Meara, Schmidt, & Day, 1996). The previous section on
ethical principles focused on fundamental principles, or what some may refer
to as obligations (Meara et al., 1996). By contrast, virtue ethics focuses on
ethical ideals to which professionals aspire. But to which ethical ideals should
our profession aspire?
Four major virtues have been identified (Meara & Day, 2003; Meara et al.,
1996): prudence, integrity, respectfulness, and benevolence. Prudence reflects
foresight, appropriate caution, and good judgment. Meara and her colleagues
suggested that prudence is the cornerstone virtue for professional psychology
and is required for fulfilling the spirit of many of the basic ethical principles.
Moreover, Meara et al. maintained that particularly in a multicultural milieu,
prudence is especially relevant because “a prudent individual is aware that
another’s definition of the situation is not necessarily one’s own” (p. 40). Integrity
refers to a “coherent integration of reasonably stable, justifiable moral values,
together with active fidelity to those values in judgment and in action”
(Beauchamp & Childress, 1994, p. 473). Thus, integrity is a virtue that involves
adherence to a set of values over time. Respectfulness refers to holding another
in high esteem, not interfering with another, and in essence regarding individ-
uals or communities in terms they themselves define (Meara & Day; Meara et
al.). Such a virtue is, again, especially important in a multicultural society.
Benevolence refers to “wanting to do good” and “contributing to the common
good” (Meara et al., p. 45).
Clearly, virtue ethics pertain to a wide range of ethical issues involving
scholarly research and its participants. Virtue ethics are at the heart of our
chapter 6 ethical issues in counseling research 113
In short, the essential point is that more information does not necessarily pro-
mote human welfare. At a minimum, expanding our knowledge bases raises
deeper moral issues about right and wrong (K. S. Kitchener, personal commu-
nication, January 2, 1990). Thus, it is imperative to note the complexity, and
sometimes the contradictions, in the seemingly straightforward goals of
extending knowledge bases and promoting human welfare.
Next we will discuss the implications of the researcher’s responsibility
to provide accurate information concerning five matters: execution of the
research study, reporting the results, duplicate and piecemeal publication,
publication credit, and plagiarism.
does more than create confusion; it can also reduce the effectiveness of the
counseling profession, which affects the lives of real people.
The U.S. federal government has promulgated rules that define fabrication,
falsification, or plagiarism as misconduct. Furthermore, the rules require insti-
tutions to have procedures for investigating and sanctioning the misconduct of
scientists they employ (see Department of Health and Human Services, 1989).
Publication Credit
Researchers have a responsibility to adequately and accurately assign credit for
contributions to a project (EPP, 8.12; CE, G.5). The issues involved with pub-
lication credit primarily relate to the fundamental ethical principle of justice. On
the one hand, assigning publication credit seems like a straightforward and
simple process. People who made minor contributions are acknowledged in a
footnote, while those making major contributions are given authorship and
listed in order of how much they contributed. In reality, these decisions can be
complicated and emotional, primarily because of ambiguity surrounding the
term contribution. What constitutes minor and major contributions? Some con-
tend that the person who contributed the most time to a project deserves to be
first author, while others argue that expertise, or even seniority, should deter-
mine author order. At other times it is reasoned that the one who conceived the
idea for the study should be the principal or first author. Determining the author
order often becomes difficult when the authors were primarily engaged in sep-
arate activities, such as writing the manuscript, analyzing the results, collect-
ing the data, designing the study, and supervising the conduct of the study.
Assigning publication credit becomes complicated as researchers debate whether
all of these contributions are equally important, or whether some contribu-
tions should be assigned greater weight than others.
Accurately assigning publication credit is important for several reasons.
First and foremost, it is important to publicly acknowledge the contributions
of all the people involved in the study—to give credit where credit is due (EPP,
8.12; CE, G.5). In addition, publication credit is often important in one’s pro-
fessional career, helping one gain entrance into graduate school, obtain pro-
fessional employment, and earn professional promotion. Moreover, public
acknowledgment of one’s professional contributions can serve as a “psychic
reward” to compensate for the low monetary rewards associated with writing
for scholarly outlets (Koocher & Keith-Spiegel, 1998). Sometimes the order of
authorship on a publication is important, because the first author is accorded
more credit (and responsibility) for the scholarly work than are the other
authors. For example, only the first author will receive recognition in citation
indices such as the Social Science Citation Index. Clearly, then, determining
the order of authorship is relevant to career-related issues.
Ethical Principles of Psychologists and Ethical Standards are ambiguous
in addressing most of these issues. The Publication Manual of the American
Psychological Association (2001) provides more direction; major contributions
typically include writing the manuscript, formulating the research question or
hypotheses, designing the study, organizing and conducting the statistical
analysis, and interpreting or writing the results. It is often suggested that
“minor contributions” to publications be credited in footnotes. Typically,
“minor” professional contributions include such activities as giving editorial
feedback, consulting on design or statistical questions, serving as raters or
judges, administering an intervention, collecting or entering data, providing
extensive clerical services, and generating conceptual ideas relevant to the
chapter 6 ethical issues in counseling research 119
study (for example, directions for future research). Paid research assistants are
remunerated for their contribution. Thus, a common introductory footnote
(usually found at the bottom of the first page of a journal article) reads some-
thing like: “The authors would like to thank Josephine Computer for statis-
tical assistance and Helen Grammar and Chris Critical for helpful editorial
comments.” Usually, these contributors went out of their way to help the
authors in minor but significant ways. Thus, it is important to publicly rec-
ognize these minor contributions. However, the author should receive per-
mission from contributors before thanking them in a footnote. Another type
of footnote is a public acknowledgment of a funding source that sponsored
the research. A footnote might acknowledge, “This research was supported by
a grant received by the first author from the National Institute of Mental
Health” (complete with reference to the grant number).
How to distinguish between a minor contributor and a major contributor
(an author) and how to determine the order of multiple authors are not clearly
specified in EPP and CE. EPP does state that “principal authorship and other
publication credits accurately reflect the relative scientific or professional con-
tributions of the individuals involved, regardless of their relative status” (8.12).
Spiegel and Keith-Spiegel (1970) surveyed over 700 professionals to examine
their opinions about determining authorship. They found modal trends, but not
a firm consensus, for the following criteria, in their respective order: (1) gener-
ation of hypotheses and design, (2) writing the manuscript, (3) establishing the
procedure and collecting the data, and (4) analyzing the data. Contributions
tend to be valued according to their scholarly importance, as opposed to the
amount of time they required. Moreover, respondents in two studies did not
rate professional status as a determining variable (Bridgewater, Bornstein, &
Walkenbach, 1981; Spiegel & Keith-Spiegel), which suggests that merit rather
than degrees or status is typically a stronger determinant of professional contri-
bution. In short, the list of authors should include those individuals who made
a major scholarly contribution to the study in the ways just listed.
The order of authorship (in the case of multiple authorship) typically re-
flects differential amounts of scholarly contributions; that is, the person who
made the greatest scholarly contribution to a project should be the principal or
first author, with the others listed in order of their relative contributions. The
process of determining authors and order of authorship is very important, per-
haps as important as the outcome. Because a great deal of ambiguity enters into
deciding authorship, authors may have different opinions about author order.
The potential for authors to feel slighted or cheated is greater when author order
is autocratically decided by one person, such as the first author. Thus, from our
experience, a mutual decision-making process is most desirable, and preferably
a consensus model in which those involved discuss these issues.
Sometimes the order of authorship is decided at the conclusion of a study
(a post hoc strategy) and just prior to the submission of a manuscript for edi-
torial review to a journal or to a professional convention. The advantage of
assigning authorship at this time is that it is possible to assess how much each
person actually contributed. The disadvantage is after-the-fact disappointments:
120 part 1 philosophical, ethical, and training issues
A person might have wanted or expected to be first author but was unaware
of either how the order was to be decided or other members’ contributions.
Or a worse scenario: A person might have thought that his scholarly contri-
bution was sufficient to qualify him as an author, but then learned after the
study that his contribution was deemed minor and that he would be acknowl-
edged only in a footnote.
Another strategy (the a priori strategy) is to assign authorship before
implementing a study. The advantage here is that as a result of the opportu-
nity to discuss and clarify the relevant issues beforehand, informed decisions
and agreements can be made by all participants. The disadvantages to this
strategy are that a person might contribute considerably more or less than he
or she initially agreed to, or an inexperienced researcher might want to be first
or second author without clearly understanding the implications of such an
assignment in terms of the tasks and skills needed.
A third strategy is to combine both the post hoc and a priori strategies, dis-
cussing author-related issues before the study is conducted, perhaps developing
a tentative author order, and then evaluating the accuracy of that initial order
after all the tasks have been completed. This strategy offers the benefits of both
of the other strategies and minimizes the disadvantages and disappointments.
A final strategy is to assign the order of authorship by chance (for example,
by drawing straws). This strategy is sometimes used when it truly seems that
each author contributed equally, and it is literally impossible to differentiate
among their contributions. It may also seem that any author order would mis-
represent the contributions of both the first and last authors. In these situations,
authors may use some arbitrary method of assigning the order of authorship. If
this strategy is used, an introductory footnote should acknowledge that the
author order was determined by chance. Parenthetically, assigning author order
by alphabetizing names is not a random process (ask people with names like
Zimmer or Zytowski). If chance is to be used as the method of assignment, then
drawing straws, pulling numbers from a hat, drawing cards from a deck, or
some other random means of assigning order is more desirable.
Winston (1985) has developed a system for analyzing contributions to
data-based articles that can facilitate decisions about author order. The system
delineates 11 activities common to a research project: (1) conceptualizing and
refining research ideas, (2) searching the literature, (3) developing a research
design, (4) selecting the instrument, (5) constructing the instrument or design-
ing a questionnaire, (6) selecting statistical analyses, (7) collecting and pre-
paring data, (8) performing statistical analyses, (9) interpreting statistical
analyses, (10) drafting manuscripts (first draft, second draft, and so on), and
(11) editing the manuscript. Winston suggested that the people involved in the
project, as a group and through consensus, assign points for each task. Be-
cause some tasks require more or less skill and time than others and vary in
importance, the assigned points will be different for different activities; the
interested reader might compare his or her assignment of points to those ini-
tially specified by Winston. The next step, again through group consensus, is
to assign points to each person for each of the 11 activities. Points are then
chapter 6 ethical issues in counseling research 121
totaled, and the order of authorship is based on the point distribution. This
system appears promising not only because it provides more specific criteria
for determining author order, but also because it could facilitate communica-
tion and make explicit the decision-making process within the research group.
One final note: A very complicated issue pertaining to publication credit
involves graduate students’ theses and dissertations. Often graduate students
feel that because they have contributed a great deal of time, effort, and some-
times money, they have contributed the most to their project. Faculty advisor
input might include providing encouragement and technical assistance in
designing the study, developing major interpretative contributions, providing
funding and other support, and writing major parts of the manuscript.
However, it is unclear how much of the faculty advisor’s contribution is a part
of her or his teaching and training role within the university. There is a real
potential for exploiting graduate students if a faculty member has them per-
form most (if not all) of the research tasks and then claims publication credit,
particularly first authorship. EPP indicates that “except under exceptional cir-
cumstances, a student is listed as principal author on any multiple-authored
article that is substantially based on the student’s doctoral dissertation”
(8.12). Similarly CE (G.5) indicates that “For articles that are substantially
based on students’ course papers, projects, dissertations or theses, and on
which students have been the primary contributors, they are listed as princi-
ple authors.” Keith-Spiegel and Koocher (1985) noted that the number of eth-
ical complaints about this issue has resulted in a policy statement issued by the
APA Ethics Committee (1983), which provides detailed guidelines. Whereas
the Ethics Committee wrote specifically about dissertations, their guidelines
pertain equally well to theses. The guidelines are as follows:
1. Only second authorship is acceptable for the dissertation supervisor.
2. Second authorship may be considered obligatory if the supervisor desig-
nates the primary variables, makes major interpretative contributions, or
provides the database.
3. Second authorship is a courtesy if the supervisor designates the general
area of concern or is substantially involved in the development of the
design and measurement procedures or substantially contributes to the
write-up of the published report.
4. Second authorship is not acceptable if the supervisor provides only en-
couragement, physical facilities, financial support, critiques, or editorial
contributions.
5. In all instances, agreements should be reviewed before the writing for pub-
lication is undertaken and at the time of submission. If disagreements arise,
they should be resolved by a third party using these guidelines.
Plagiarism
Researchers have a responsibility to acknowledge the original contributions of
other writers and to clearly distinguish their own original scholarly insights
122 part 1 philosophical, ethical, and training issues
from the work of others (EPP, 8.11; CE, G.5). Again, these issues revolve pri-
marily around the fundamental ethical principle of justice. Plagiarism can
occur in the direct, verbatim copying of another’s work, or less explicitly, as
in duplicating ideas from others’ work without proper citation. Quotation
marks and proper citation form should be used when quoting a passage ver-
batim from another article; paraphrasing sentences from other articles should
include a citation to the original work. In both cases of plagiarism, the origi-
nal author does not receive proper acknowledgment or credit for his or her
work. Keith-Spiegel and Koocher (1985) nicely depicted this issue:
Copying the original work of others without proper permission or citation attri-
bution is often experienced as “psychic robbery” by the victims, producing the
same kind of rage expressed by those who arrive home to find the TV set and
stereo missing. When plagiarizers reap financial rewards or recognition from pass-
ing someone else’s words off as their own, the insult is still greater. Readers are
also misled and, in a sense, defrauded. Plagiarism and unfair use of previously
published material are among the more serious ethical infractions a psychologist
can commit. (p. 356)
Not citing the original author may seem like a rather small issue, but it is
really quite important. Imagine that you have worked very hard for two or
three years, creating and building a new conceptual model. Naturally, you are
very proud of this accomplishment. Then someone publishes a similar model
and this person receives the credit for developing this innovative model. Being
cited is often meaningful to authors and serves as an important psychic
reward. In addition to fairness and integrity, there is also the matter of salut-
ing (in a small way) previous researchers for their accomplishments. From a
historical perspective, it is important not only to recognize the authors whose
work preceded one’s own, but also to recognize where one’s work fits into the
bigger picture. It is a sad comment on the profession when petty jealousies
prevent mutual recognition or even collaboration that might result in impor-
tant contributions. Keep in mind as well that the federal government consid-
ers plagiarism to be misconduct in science, and that employing institutions are
required to investigate all cases of suspected plagiarism and provide sanctions
where appropriate.
Although not appropriately recognizing an author’s contributions is prob-
lematic, overcrediting an author can also be an issue, most notably when
citing one’s own research. For example, overly laudatory reference to one’s own
“seminal, ground-breaking research that will revolutionize the entire field in
the next several years” is inappropriate. It is best to leave these types of judg-
ments to others.
research participants. In fact, the Nuremberg Code has been the basis for sub-
sequent ethical principles regarding human subjects in research (Koocher &
Keith-Spiegel, 1998). Unfortunately, yet other research tragedies stimulated
additional concern and the need for additional regulation.
In 1962, the thalidomide scandal (see Chapter 1) came to public attention
because of innumerable gross neonatal deformities. Consequently, the U.S.
Food and Drug Administration introduced much stricter regulations for tightly
controlled experimentation on drugs and other products (Stricker, 1982). Not
long afterward, public attention became focused on a program conducted by
a hospital in Brooklyn, where 22 chronically ill patients were injected with
cancer cells as part of a study to examine the body’s capacity to reject foreign
cells. The patients were not informed of their participation. This, and other
studies involving excessive shock treatment conditions and subjecting partici-
pants to diseases such as syphilis, served to raise public awareness of ethical
issues related to informed consent of research participants. Subsequently the
issue of informed consent crystallized, and obligations to research participants
were made clearer (Stricker).
Diener and Crandall (1978) discussed a number of research studies that
have raised concern about ethical issues in research (for example, inducing
extreme levels of fear in participants, suppressing disconfirming data, jeop-
ardizing continued employment of participants). As awareness of and sensi-
tivity to the rights of both human and animal participants in psychological
and educational research have increased, there have been major changes in the
regulation of research.
One of the major changes has been development of the Code of Federal
Regulations (rev. March 3, 1983), which implemented Public Law 93-348
(July 12, 1974) establishing institutional review boards (IRBs) and an ethics
guidance program to protect human participants in biomedical and behav-
ioral research.
All research projects are subject to federal regulations governing research.
Initially, IRBs were established as five-person panels to preview research
proposals and weigh potential risks and benefits for all research that sought
funding from the Department of Health, Education, and Welfare (now the
Department of Health and Human Services, hereafter referred to as DHHS).
Although IRBs still serve this function, most institutions now routinely have
all research proposals reviewed by a campus IRB (C-IRB) committee of peers
at their institution. In essence, the typical C-IRB certifies that projects comply
with the regulations and policies set forth by the DHHS regarding the health,
welfare, safety, rights, and privileges of human participants. The general pro-
cedure is for the investigator to complete and submit to the C-IRB a form that
summarizes basic information about the research, including purpose of the
study, types of participants needed, informed consent procedures, method of
collecting data, and funding sources. Please contact your particularly campus
IRB for specific requirements and procedures.
Key issues in evaluating the ethicality of any research project are the risks
and benefits involved to the participants, and whether participants have been
chapter 6 ethical issues in counseling research 125
fully informed about the study so they can make an informed decision to vol-
untarily participate (informed consent). Given the increased sensitivity to in-
formed consent, documentation of the participant’s consent is now required
unless specifically waived.
In this section of the chapter we discuss a number of complexities and
complications related to using human participants that often arise for the psy-
chological researcher in general, and for the counseling researcher in particu-
lar. Assessing potential harm or risk is a difficult and sometimes imprecise
process. Because there is some level of risk (even if it is minuscule) in every
experiment, how much risk or harm is too much? In some research, deception
is needed to adequately investigate a particular construct, and if the full truth
were known to a participant, the validity of the experiment might be signifi-
cantly reduced. Thus, without deception, knowledge of some aspects of human
behavior may be inaccessible. But deception conflicts with informed consent
and the fundamental principles of autonomy and fidelity. This section focuses
on these issues in greater detail as we discuss the issues involved in protecting
the dignity and welfare of the people who participate in research investigations.
Specifically, we will discuss issues pertaining to risks and benefits, consent,
deception and debriefing, confidentiality and privacy, and treatment issues.
The professional codes of ethics suggest that the researcher should carefully
assess the potential risks of involvement for participants and take precautions
to protect participants from physical and mental discomfort, harm, and
danger that might occur in a study (EPP, 3.04; CE, G.1). Implied in these
statements is that it is the responsibility of the investigator to reduce risk and
prevent harm by detecting and removing any negative consequences associ-
ated with a study, to the extent possible.
One of the problems inherent in assessing risk potential is that the task
is often subjective, ambiguous, and involves an estimation of probabilities.
Typically one does not have prior, empirical, objective data about whether the
experimental condition is stressful (and to collect such data would require
administering the experiment to participants). Moreover, the type and level of
stress that would be harmful is ambiguous and likely varies across cultures
and individuals; that is, what is perceived as harmful in one culture may not
be perceived as such in another culture. Thus, assessing harm may also involve
cross-cultural sensitivity. In short, assessing risk is difficult, if not impossible,
to quantify.
Acknowledging the difficulty, ambiguity, and imperfectness of the task,
there are at least two main strategies for obtaining approval to conduct a
study, typically from an institutional review board. The first strategy involves
making a best estimate of the risk/benefit ratio of the study; that is, a com-
parison should be made of the potential benefits that might accrue from the
study relative to the potential risks to participants. This involves a three-step
process: (1) assessing risks, (2) assessing benefits, and (3) comparing risks and
benefits. For example, a study might be considered ethically acceptable if the
potential benefits greatly outweighed the potential risks, or if failure to use the
experimental procedures might expose participants to greater harm. Assessing
the benefits derived from a particular study, however, is also a difficult and
ambiguous task. This assessment is complicated by the question of “benefit
for whom?” That is, should participants be the ones to receive the benefit
directly, or could it be a larger group, as when a profession’s knowledge base
is increased. Some may argue that benefits from any single study may be min-
imal, but that over time programmatic research does increase the profession’s
knowledge base. Still, balancing individual costs against societal benefits is a
difficult task. Moreover, it can be argued that the investigator is at a disad-
vantage to judge the cost/benefit ratio accurately because he or she may be
overly biased regarding the benefit of the study (Diener & Crandall, 1978). In
short, in principle the risk/benefit ratio is appealing, but in practice it is diffi-
cult to apply. Nonetheless, the risk/benefit ratio is one useful strategy for
assessing the ethical issues associated with a particular study.
The second strategy involves several procedures to minimize risk or
reduce the probability of harm. Whenever the potential for substantial risk
is present, the investigator should search for other possible designs or pro-
cedures. The researcher needs to exhaust other possibilities for obtaining
the same or similar knowledge by using a slightly different design. A com-
mon practice is to consult with colleagues not only to obtain ideas regarding
chapter 6 ethical issues in counseling research 127
Consent
A critical issue in conducting studies involving risk pertains to informed consent.
After a researcher has carefully evaluated potential harm and developed the
best design to answer his or her question while preserving the participant’s
dignity and welfare, the researcher is then ready to approach participants with
a fair, clear, and explicit agreement about the experiment in question (in-
formed consent). The issue of informed consent revolves around the funda-
mental ethical principles of autonomy and fidelity. Consent refers to the
process of giving participants the opportunity to decide whether or not to par-
ticipate in a particular research study. This might appear to be a rather simple
matter: Simply ask the participant if he or she would like to participate. But
a number of factors make obtaining consent a rather complicated process, and
a considerable amount of attention has been given to this topic in the past 20
years (see Koocher & Keith-Spiegel, 1998; Schmidt & Meara, 1984).
The professional codes of ethics clearly indicate that the investigator has
a responsibility to obtain informed consent from participants (EPP, 6.11;
CE, G.2). The investigator seeks to develop a specific type of relationship
with potential participants and thus is ethically bound to establish a clear
and fair agreement that clarifies obligations, risks, and responsibilities prior
to the study.
Turnbull (1977) discussed consent in this special relationship in terms of
three key elements: capacity, information, and voluntariness. Capacity refers
chapter 6 ethical issues in counseling research 129
safeguard the welfare of the client. The professional codes of ethics clearly indi-
cate that care should be taken to protect the privacy of participants (EPP, 6.02;
CE, G.2).
Maintaining the anonymity or confidentiality of participants is now stan-
dard in counseling research. Anonymity exists when there are no identifiers
whatsoever on project materials that can link data with individual participants;
often researchers assign participants coded designations that appear on their
respective questionnaires in lieu of their names. Thus, the participants’ re-
sponses are anonymous, and even the investigator cannot identify the partici-
pants. At other times, investigators collect data and ask for participants’ names.
Researchers have an obligation to maintain the confidentiality of information
obtained in such research. If names are used, typically code numbers will be
assigned to participants when the data are transferred to coding sheets, and the
original questionnaires containing participants’ names will be destroyed. If some-
one other than the experimenter will have access to data (for example, a research
assistant), this should be explained to the participants (and usually is stated in
the consent form) along with plans for maintaining confidentiality. In field
settings, researchers also should be alert to minimizing the invasiveness of data
collection so as to protect the participants’ privacy within a social milieu.
Schmidt and Meara (1984) indicated that because confidentiality is cen-
tral to the counseling relationship, counseling researchers often need to be
especially sensitive to maintaining confidentiality, particularly with regard to
research conducted in an agency such as a university counseling center. For
example, researchers must be sensitive to releasing demographic information
that might identify participants in a small therapy group (for example, an
eating disorder group or a consciousness-raising group for men over 30); on
small college campuses, demographic information about the group composi-
tion can easily identify clients. Likewise, research using an intensive single-
subject design or an intrasubject design with only a few participants also
demands sensitivity to identifying characteristics; typically it is advisable to
provide fictitious descriptive information that is similar to the truth if it is nec-
essary to describe a particular client in detail, and to explicitly indicate this in
whatever written or oral report is made. Sometimes investigators also com-
municate to participants how the data will be used. They may also obtain
feedback and written approval from clients on written descriptions of the
study’s results to further reduce any breaches of confidentiality.
Another confidentiality issue arises if researchers want to investigate some
aspect of a particular treatment procedure after clients have already begun
treatment at an agency. Suppose there is an active relaxation training program
at a university counseling center, in which students, staff, and faculty are
encouraged to participate. Clients enter this program with the usual assurance
of confidentiality. Let’s say a researcher from the counseling department (outside
the agency) is interested in evaluating some aspect of the treatment program
and examining the effects of certain individual difference variables (such as
coping style) on the relaxation training. Given that the clients have been
assured of confidentiality at the outset, it would be a breach of confidentiality
chapter 6 ethical issues in counseling research 135
at this point to reveal client names to a researcher outside the agency. Some
clients might also feel their privacy has been invaded if they were identified to
a researcher in the agency, because they did not consent to having that person
know of their seeking such services. Likewise, if an investigator should con-
duct a study with a sample of counseling center clients that gave their consent
to participate in a study with certain specified procedures, the investigator is
limited to accessing only the information or data that the clients consented to
disclosing or providing, as opposed to any information in the clients’ agency files
(see Keith-Spiegel & Koocher, 1985). In short, counseling researchers some-
times have dual responsibilities; they need to be sensitive to confidentiality
issues pertaining to research endeavors as well as to confidentiality issues
inherent in therapeutic relationships in general.
Confidentiality and privacy issues in research settings can also intersect
with a psychologist’s duty to protect the welfare of participants and other
individuals, thereby creating an ethical dilemma for the researcher. The most
notable examples involve participants with homicidal and suicidal intentions
that become evident during the course of a research investigation. For exam-
ple, counseling researchers routinely assess the psychological adjustment of
participants in various ways, such as by using the Beck Depression Inventory
(BDI) (Beck et al., 1961). In this case, the difficult question concerns what to
do when the investigator finds that a participant scored very high on the BDI.
Or suppose the researcher is investigating suicidal intentions and in adminis-
tering the Scale for Suicide Ideation (SSI; Schotte & Clum, 1982) learns that
one or more participants scored very high on the SSI. Another ethical dilemma
can arise when a participant reveals some information such that the partici-
pant or others will be liable for a violation of the law. In short, sometimes the
counseling researcher obtains information about participants that either cre-
ates considerable concern for the general well-being of particular participants
or other individuals, or brings up criminal or civil liabilities.
Concern for the well-being of a particular participant must also be con-
sidered in light of the individual’s right to privacy. In approaching one partic-
ipant who had a very high BDI score after a particular investigation, the
investigator was curtly informed that the participant “consented to participate
in a psychological experiment, not psychotherapy.” Some participants may
feel embarrassed if attention is called to them within a group setting; obvi-
ously, care must be taken to avoid breaching confidentiality to other partici-
pants and to being sensitive to the effects of isolating particular individuals.
Concern for the well-being of a particular participant must also be con-
sidered relative to the amount of information an investigator has about the
participant. Whereas one researcher might have only one data point (a BDI
score) that is causing some concern, another researcher might have a much
broader array of information (for example, questionnaire data, interview
data, information about environmental stressors, knowledge about past sui-
cide attempts) that more strongly suggests considerable reason for concern.
Clearly, the counseling researcher has ethical obligations beyond the
research. Moreover, each situation presents a slightly different context and calls
136 part 1 philosophical, ethical, and training issues
Treatment Issues
In the past, a common strategy among researchers was to use a between-groups
design to compare two or more groups of participants; one group received a
particular treatment, and instead of the treatment one of the other groups
received a placebo or had treatment delayed. Although such designs offer
methodological rigor, they can present ethical problems related to withholding
treatment from people in need. Clients in a waiting-list or placebo group could
be at risk as they continue to struggle under duress. Thus, researchers who are
interested in examining questions about comparative treatments often must
examine additional ethical issues.
One of the essential issues pertains to the necessity of withholding treat-
ment. In general, if there are treatments that are known to be effective, then
withholding them from participants raises serious ethical concerns. However,
there is less concern about withholding an intervention of unknown effectiveness.
The researcher might examine the need to compare a particular treatment
against a no treatment group. The researcher might also consider alternatives,
such as comparing the treatment of interest against a well-known treatment.
chapter 6 ethical issues in counseling research 137
other in our endeavors to conduct our research ethically and to uphold our
professional responsibilities. Thus, it is important to talk with each other about
ethical issues and dilemmas, particularly when we witness events that make us
uncomfortable. From our experience, inexperienced professionals (for example,
students, assistant professors) may become paralyzed when they witness what
appears to be an ethical infraction and withdraw from the situation. Similarly,
people are typically reluctant to raise ethical issues with people in authority
positions. Such reservations can be very valid in some situations, but sometimes
they are not justified at all. Informal peer monitoring of ethical issues is a pow-
erful mechanism not only to monitor the appropriateness of our behavior, but
also to increase our sensitivity and knowledge about ethical issues.
We strongly encourage readers to confront ethical dilemmas, and when-
ever possible, engage in informal peer monitoring. Typically, it is recom-
mended that concerned individuals review ethical codes, evaluate the evidence
(facts versus hearsay), examine their motivations, consult with knowledgeable
colleagues, and then discuss the issue with the person or people involved (see
Koocher & Keith-Spiegel, 1998, for more details). Another strategy for deal-
ing with people in authority positions is to involve other professionals in
authority positions through consultation and problem solving. For more seri-
ous or repeated ethical violations, sanctions may be sought from local, state,
or national ethics committees (such as state licensing boards or professional
organizations such as the ACA or APA), or even the judicial system. The inter-
ested reader should consult Koocher & Keith-Spiegel (1998), who provide
excellent guidelines for a range of options (and sanctions) in responding to
ethical situations. We encourage readers to examine not only the EPP and ES
codes in the appendixes, but also the Publication Manual of the American
Psychological Association (2001). When in doubt, consult with trusted, knowl-
edgeable colleagues and faculty.
EXHIBIT A
Consent to Serve as a Participant in Research1
1. I hereby consent to take part in research directed by Dr. Mary Heppner and
Meghan Davidson, and sponsored by the University of Missouri. Dr.
Heppner is an Associate Professor and Meghan Davidson is a doctoral student
in the Department of Educational and Counseling Psychology. I understand
that other persons will assist Dr. Heppner and Meghan Davidson in con-
ducting this research.
2. Further, I understand that:
a. Purpose. The purpose is to study the validity and reliability of a new
measure of empathy.
b. Requirements. My part of this research will be to complete a paper and
pencil survey designed to measure the way I empathize with others. Surveys
will be completed by groups of students in university classrooms.
chapter 6 ethical issues in counseling research 139
(Continued)
c. Time needed. The total time required will be approximately 25 minutes.
d. Voluntary participation. My participation is completely voluntary. Even
after I begin participating, I will be free to stop at any time. I have the
right to stop after I have started participating, or I have the right to
decide not to participate at all in this study. Although the researchers
ask that I try to answer every item, I understand that I can skip any item
that I simply do not wish to answer. (I do not need to give a reason for
skipping any item.) In no case will there be a negative effect for my non-
participation or non-completion.
e. New developments. I will be told of any new information that develops
during the course of this research that might affect my willingness to
participate in this study.
f. Benefits. I will receive a debriefing sheet that explains more about
empathy. General benefits will come for myself and others in the form of
an increased scientific understanding of how people relate to and under-
stand other people.
g. Protections. I understand that the following precautions have been taken
for my protection: (1) no part of the surveys will ask for my name or other
identifying information, my responses will remain completely anonymous;
(2) no questionnaire asks me to describe specific incidents; (3) I am free to
discontinue my participation at any time for any reason; (4) although the
researchers would like me to answer every item, I am free to skip any
question or item that I find too sensitive or stressful; (5) when the results of
this study are published, only aggregate data (for example, group aver-
ages) will be reported.
3. My questions about this research have been answered. If I have further ques-
tions, I should contact:
Signature ___________________________________________________
Date _______________________________________________________
140 part 1 philosophical, ethical, and training issues
EXHIBIT B
Sample Youth Assent Form (Children)2
You are invited to participate in a research study conducted by a graduate student at the
University of Missouri-Columbia. As a participant, you should read and understand the following
statements. Ask any questions before you agree to participate.
1. Goal of the Project: The goal of this research project is to develop a new survey that measures high
school students’ attitudes about healthy relationships and sexual coercion in dating relationships. It is
hoped that this survey will be used in evaluating educational programs about sexual coercion in dating
relationships. Please note that you will be asked about information pertaining to your attitudes, beliefs
and own personal experiences.
2. Participation Procedure and Guidelines:
a. You will receive an assent form (this page), get any questions that you might have answered, and
then complete the surveys.
b. The information you provide will be kept completely anonymous. That is, your name will not be
on any of the forms.
c. It will take about 30 minutes to complete the surveys.
3. Participation Benefits and Risks:
a. Your participation in this study does not involve risks that are greater than those you experience
in your daily life. You might feel some mild discomfort from reading and responding to some items
on the questionnaires. But again, the risk of discomfort is not greater than you might have in class
or in other normal activities.
b. You also might experience some benefits from participating in this project. These benefits might
be positive feelings from helping with an important research study.
c. By filling out the survey, you will be entered into a raffle to win one of fifteen gift certificates to any
store in the Columbia Mall.
4. Rights to Refuse or Withdraw: Your participation is VOLUNTARY, and there is no penalty for you
not wanting to participate. This means that you are free to stop at any point or to choose not to answer
any particular question.
5. Rights as a Participant: You have a right to have any questions about this research project
answered. Please direct any questions to the following individuals:
M. Meghan Davidson, M.A. Mary J. Heppner, Ph.D.
Department of Educational Department of Educational
& Counseling Psychology & Counseling Psychology
16 Hill Hall 16 Hill Hall
University of Missouri-Columbia University of Missouri-Columbia
Columbia, MO 65211 Columbia, MO 65211
(573) 884-4328 (573) 882-8574
For additional information regarding participation in research, please feel free to contact the University
of Missouri-Columbia Campus Institutional Review Board office at 882-9585.
6. Agreement to Participate:
Signature___________________________________________________Date________
chapter 6 ethical issues in counseling research 141
EXHIBIT C
Oral Debriefing3
That concludes your participation in the study. Thanks so much for your help.
Now that you have finished giving your opinion, I can explain more to you about
the whole purpose of the study. I could not do so before now without biasing your
responses. First, the study is concerned with more than “interviewing styles.” We
are more interested in impressions of college students about bereaved and
depressed persons. Specifically, we wanted to find out both about your personal
reactions to someone who is bereaved and also your attitudes about what is
normal or pathological grief. We didn’t want people to know exactly what we
were looking for in advance, because it could have influenced who signed up for
the experiment or the answers they gave. We regret that we could not more fully
inform you before you participated. We strongly hope you will respect our need
to withhold this information and will not discuss this experiment with your fellow
classmates.
Some of you received instructions that you were listening to a tape of a
middle-aged widow; some were told that she had become widowed three weeks
ago and some were told she became widowed two years ago. If you received these
instructions, you were in one of the experimental groups. Others received instruc-
tions that you were listening to a tape of someone who had lost a job. You were in
a control group. In addition, some subjects hear a depressed woman on tape and
others hear a nondepressed woman. We will be looking for differences in the
answers of these various conditions depending on whether the subjects are male
or female.
I want to tell you now that none of you will come back to participate in a fur-
ther part of the experiment. When you leave today, your participation will end. It was
important that you think you might come back, so we could get your reaction about
whether you were willing to meet the woman you heard on the tape.
Next, let me explain that this is an analogue experiment. That means that the
people you heard on tape were playing parts that were written for them in advance.
The purpose of this is so each time a new group hears a particular conversation, it
is done exactly the same as the last time a group heard that conversation. This
allows for better control of the experiment and helps to eliminate unknown influ-
ences on the answers you gave.
I want to thank you for your participation today. Again, it is very important that
you do not talk about this experiment with anyone once you leave this room. If
people who participate later in the study are aware of its purpose or procedures,
their answers may be biased. This would cause us to report misleading results. As
we hope our research may some day assist actual bereaved persons, this is a seri-
ous problem. Please give others the chance to fairly contribute as you have today.
Does anyone have any questions? [Pause for questions.] I will sign your
research cards and you are free to leave. I will stay for a moment in case you have
other questions.
If you’re having any difficulty dealing with either bereavement or depres-
sion, I have the telephone numbers of our University Psychology Clinic and of the
Counseling Service, and I will be glad to give them to you when you have finished.
142 part 1 philosophical, ethical, and training issues
EXHIBIT D
Written Debriefing4
This sheet will further explain the purpose of this research project beyond the
oral explanation you have already heard. It will outline the independent and
dependent variables and research hypotheses. It is crucial that you do not dis-
cuss the information on this sheet with any of your friends (who might inadver-
tently communicate with future participants) or with the experimenter who is
present today. She must remain blind (uninformed) concerning the hypotheses
in order to avoid influencing the experiment. You may direct any questions to
Carol Atwood at 484-1676 (leave a message if no answer). Please sign this
sheet as soon as you finish reading it, place it back in the envelope provided,
and seal the gummed flap. Thank you very much for your help.
1. Nature of the project: This project would best relate to the major research
area of social psychology—attitudes and social perception.
2. Findings of related studies: There is little previous research concerning the
layperson’s views of what is a healthy versus an unhealthy grief reaction, and
whether or not laypersons reject or avoid the bereaved. Vernon (1970) asked
participants how they would respond to a recently bereaved person that they
knew. Only one-fourth of participants indicated they would spontaneously
mention the death; another one-fourth preferred that neither side mention the
death at all. Other researchers, such as Lopata (1973) and Glick, Weiss, and
Parkes (1974), have indirectly addressed the question by interviewing widows
themselves, who frequently reported experiencing strained relationships or
the breakup of friendships after the deaths of their husbands.
3. Independent variables: These are the variables in the experiment that the
investigator manipulates or controls. There are three independent variables in
this project. The first is gender of the participants. We will look for differences
in the responses of male and female subjects. Second is the depression con-
dition (whether the woman heard on the tape is depressed or nondepressed).
Third is the “bereavement (or widowhood) status”; that is, the woman on the
tape is either recently widowed, long-term widowed, or not widowed (loss of
a job is mentioned), depending on which written instructions you received.
4. Dependent variables: Used to measure the effects of manipulation of the
independent variables. In this project, the dependent variables consisted of
the written questionnaire you completed. We want to find out how much you
would reject the woman heard on the tape, what your social perceptions of
her were, and how pathological you found her to be.
5. Hypotheses: The research questions to be examined in the project. Please
do not share this information with today’s experimenter.
A. How do college students’ judgments of emotional disturbance compare,
based on whether the woman on the tape is recently bereaved, long-term
bereaved, or nonbereaved? How do ratings of disturbance differ, depending
on whether the woman on the tape sounded depressed or not depressed?
B. Do college students reject a bereaved person or a nonbereaved person
more, and is this rejection affected by whether the woman sounds de-
pressed or not depressed?
chapter 6 ethical issues in counseling research 143
(Continued)
C. How does the gender of the participant (male or female) affect partici-
pants’ responses?
6. Control procedures: Procedures to reduce error or unwanted variance. In this
study, random assignment of participants to experimental conditions was
used, except that it was not possible to randomly assign participants based on
participant gender. Other control procedures used include keeping the exper-
imenter blind to the study hypotheses, not informing participants before the
experiment about the true purpose, use of an analogue procedure in which
actors were used on the tapes, and use of a control group of participants who
listen to the tape of a woman who is neither a widow nor depressed.
I have read the above information concerning the nature of the study, Reactions
to Stressful Life Experiences. I agree not to disclose this information either to
potential future participants or to the experimenter present today.
Signature ___________________________________________________
Date _______________________________________________________
Notes
1. An earlier version of this consent form was written 2. An earlier version of this consent form was written
by Meghan Davidson. At the time this chapter was by Meghan Davidson. At the time this chapter was
written for the third edition of the book, Meghan written for the third edition of the book, Meghan
was a doctoral candidate in counseling psychology was a doctoral candidate in counseling psychology
at the University of Missouri-Columbia. at the University of Missouri-Columbia.
144 part 1 philosophical, ethical, and training issues
3. An earlier version of this oral debriefing was writ- 4. An earlier version of this written debriefing was
ten by Carol Atwood. At the time this chapter was written by Carol Atwood. At the time this chapter
written for the first edition of the book, Carol was was written for the first edition of the book, Carol
a doctoral student in clinical psychology at the was a doctoral student in clinical psychology at the
University of Missouri-Columbia. University of Missouri-Columbia.
STIMULUS QUESTIONS
Ethical Issues in the Research Process
In this chapter, we have maintained that it is without the appropriate reference to the cul-
essential for researchers to be aware of their eth- tural context?
ical responsibilities to research participants, co- 3. What, in your view, is the most serious con-
workers, the counseling profession, and society sequence of publishing fraudulent data?
as a whole. The purpose of this exercise is to 4. What do you think Carl Rogers meant when
promote additional thinking about ethical issues he said “the data are always friendly”?
related to participants and scholarly work. We 5. What do you think is the best strategy for
suggest you reflect on the questions below, write determining the author order of a manu-
your responses, and then discuss your responses script? Why?
with a peer in the class to further your under- 6. In what instances, if any, do you think deception
standing of ethical issues and responsibilities re- is acceptable in counseling research? In what
lated to the research process. instances would you definitely not allow de-
ception?
1. Why might it be functional to focus on learning
7. Identify two situations where investigators’ bias
the foundational ethical principles rather than
could result in insensitive cross-cultural research.
just relying on a rule-bound conception of ethics?
8. What is the difference between confidential-
2. Explain why it is essential to interpret the
ity and anonymity?
findings of any study within the cultural con-
9. Why do you think it is often recommended to
text in which the data were collected. Can
talk directly to the relevant people when eth-
you list at least three problems that occur
ical issues arise?
when researchers misinterpret their findings
Major Research
Designs
PART
2
145
7 CHAPTER Between-Groups and
Within-Subjects Designs
146
chapter 7 between-groups and within-subjects designs 147
In this chapter we will discuss two designs that are often referred to as
true experimental designs because of their emphasis on experimental control,
minimizing extraneous variables, and internal validity. Even though students
sometimes feel intimidated about true experimental designs because of the
heavy, ominous meaning that the words sometimes convey, the designs are
actually quite straightforward; the label is more ominous than the actual
design. The two true experimental designs are commonly labeled between-
groups design and within-subjects design.
The between-groups design often adheres to the MAXMINCON principle.
Differences between treatments can be maximized by making the treat-
ment (independent variable) stronger or even exaggerated. Thus, researchers
will often examine the effects of extreme treatments, such as five counselor
disclosures in 50 minutes, or three counselor influence attempts in 15 minutes.
Moreover, the between-groups design can be arranged to control extraneous
variables and minimize error variance through randomization and experi-
mental control.
The essential feature of between-groups design is the comparison of vari-
ables across two or more groups under tightly controlled experimental condi-
tions. In early counseling research, a common comparison group was some
type of control group, a group that did not receive one of the active treatments
in the study. Over time, differences between or among experimental treat-
ments have been compared. To adequately make comparisons across groups
necessitates that the groups do not differ in important ways before the exper-
iment. Thus, initial differences between groups in terms of individual difference
variables, demographics, and situational variables must be minimized prior to
experimental manipulations to reduce threats to internal validity. Because of
the emphasis on comparison and equivalent groups, assignment of participants
to groups is a critical consideration in between-groups design. In fact, one of the
major identifying features of between-groups design is the random assignment
of participants. In short, the between-groups design is a powerful investigative
tool, and often the most strongly favored design (Kazdin, 2003, Kerlinger,
1986; Shadish et al., 2002).
The hallmark of the within-subjects design is that it attempts to minimize
error variance due to individual variation by having each participant serve as
his or her own control because all participants are exposed to all of the treat-
ment conditions. This design, like the between-groups design, is often called a
true experimental design because of the random assignment of treatments and
manipulation of the independent variable. The random assignment that occurs
in the within-subjects design is assignment to a time period in which the treat-
ments are delivered.
For example, perhaps a researcher has located two videos that may be
useful for increasing participants’ empathy related to issues of poverty and
social injustice. In a within-subjects design, all participants would view both
videos, but not at the same time. For example, one group of participants
would get treatment X1 (video #1) before X2 (video #2), whereas the other
group would receive the opposite sequence, X2 before X1. In within-subjects
148 part 2 major research designs
BETWEEN-GROUPS DESIGNS
In this section we first discuss some of the historical events affecting the emer-
gence and development of the between-groups design. We then proceed to a
discussion of the strengths and weaknesses of three specific between-groups
designs. Because the central focus of between-groups designs is to compare
treatment groups and control groups, the third section explicitly discusses
issues pertaining to control groups. The fourth section discusses more complex
designs that contain two or more independent variables, which are called fac-
torial designs. Fifth, we examine a central issue of between-groups design—
randomization and participant assignment. The strength of the between-groups
design is based on group equivalence before the experimental manipulation,
which underscores the importance of participant assignment. As a result of
our discussion of participant assignment and especially group equivalence, in
the last section we discuss related issues of matching and dependent samples
designs.
R X O1
R O2
150 part 2 major research designs
In its most basic form, this design involves the random assignment of partici-
pants to two groups; one of the groups receives exposure to a treatment while
the other group serves as a control group, and thus receives no treatment.
Both groups receive a posttest, but neither group receives a pretest. The basic
purpose of the design is to test the effect of X, the independent variable, on
observations of the dependent variable, vis-à-vis O1 and O2.
Strengths The posttest-only control group design controls for most of the
threats to internal validity, and thus is a powerful experimental design. For
example, history would have affected each group equally because O1 and O2
occurred at the same time. Likewise, maturation, instrumentation, testing
effects, and regression are controlled in that they are expected to be equally
manifested in both the experimental and control groups. For example, if
extreme scores were used, the control group would be expected to regress as
much as the experimental group.
Both selection and selection-maturation effects are controlled for in that
randomization would most likely make the groups comparable on these
dimensions before the study. Attrition rates can be examined to determine if
differential losses may have occurred across groups, although again random-
ization would decrease the probability of differential attrition due to preex-
isting differences in participants.
In many ways the posttest-only control group design is the prototypical
experimental design and most closely reflects the characteristics needed to
attribute a causal relationship from the independent variable to the dependent
variable (Shadish et al., 2002). The difference between O1 and O2 reflects the
degree to which treated participants are different from untreated participants
at the end of the treatment period. Of course, the observed difference needs
to be statistically significant (have statistical conclusion validity) to justify a
claim that the treatment indeed is effective.
In spite of the simplicity of the posttest-only design, there are some con-
cerns regarding it. The primary concern is that because the dependent variable
is examined only at the end of treatment, statements about actual change
cannot be made; put another way, there is no evidence to show that the treat-
ment group improved vis-à-vis their level of functioning prior to treatment.
However, in our view the level of functioning of treated individuals (at O1)
versus their level of functioning had they not been treated (O2) is the most
important comparison, not the change from before treatment to after treat-
ment. The logic of experimentation does not require that pretreatment levels
of functioning be assessed; thus, a pretest is not used.
One of the strengths of the posttest-only control group design, therefore,
is that a pretest is unnecessary. Practically speaking, sometimes the repeated
testing of participants is bothersome to the participants and expensive to the
researcher in terms of time and effort. Furthermore, the absence of pretests
removes the need to collect both the pretest and posttest scores, and hence it
may be possible to have participants respond anonymously, thereby protect-
ing the confidentiality of responses. Another advantage of the posttest-only
chapter 7 between-groups and within-subjects designs 151
smoking and cancer; previous research found that many women do not make
a connection between smoking and cervical cancer (Marteau, Rana, & Kubba,
2002). Hall et al. used a between-groups posttest-only design to compare the
effectiveness of providing women with a detailed versus minimal explanation
of the association between smoking and cervical cancer. Women from England,
ages 20–64 years, were randomly assigned to either a minimal explanation
leaflet group, a detailed explanation leaflet group, or a no leaflet (control)
group. Although the results did not find differential impact across the two
types of leaflets, the women who received leaflets (versus no leaflet) reported
higher ratings of vulnerability to cervical cancer, had a more coherent expla-
nation of the link between smoking and cervical cancer, and had stronger inten-
tions to stop smoking in the next month. The authors interpreted the results
as suggesting that having a coherent or meaningful explanation of the link
between an increased vulnerability to a health threat and smoking is a critical
component to having a strong intention to change behavior (Hall et al.).
R O1 X O2
R O3 O4
This design involves the random assignment of participants to two (or more)
groups, with one group receiving treatment while the other group receives no
treatment and thus serves as a control group. Both groups receive a pretest
and a posttest. The purpose of the design is to test the effect of the independent
variable, X, which is reflected in the differences on the dependent variable,
specifically between O2 and O4.
Strengths This design controls for most of the threats to internal validity dis-
cussed by Shadish et al. (2002), and in that way it is similar to the posttest-only
control group design. The unique strength of this design pertains to the use of
the pretest, which allows the researcher to perform various analyses that may be
helpful in making valid inferences about the effects of the independent variable.
One of the most important reasons for giving a pretest is that pretest scores
can be used to reduce variability in the dependent variable, thereby creating a
more powerful statistical test. In essence, such a strategy attempts to minimize
error variance in line with the MAXMINCON principle. Much of the variance
in any dependent variable is due to individual differences among the participants.
Knowledge of the pretest level of functioning allows the researcher to use sta-
tistical methods, such as the analysis of covariance, to remove the variance
found in the pretest from the variance in the posttests. Such procedures can
reduce drastically the number of participants needed to achieve a desired level
of statistical power (Porter & Raudenbush, 1987). Of course, the pretest in this
case need not be the same measure as the posttest; however, it must be corre-
lated with the posttest to allow a covariance analysis.
chapter 7 between-groups and within-subjects designs 153
R O1 X O2
R O3 O4
R X O5
R O6
Strengths In terms of internal validity, this design controls for most threats
to internal validity. The Solomon four-group design contains the strengths
of both the pretest-posttest control group and the posttest-only control
group designs. The unique strength of the Solomon four-group design is that
it explicitly examines the potential sensitizing effect of the pretest as it might
interact with the treatment (X). The researcher can easily test for pretest
sensitization by comparing O2 to O5, the only difference between these
observations being the pretest prior to the treatment. Thus, this design is
useful if the investigator not only wants to test the effect of some treatment,
X, but also wants to examine whether the pretest sensitizes participants to
the treatment.
Another strength of this design is that it inherently includes a replication
test of the treatment intervention. That is, if a significant treatment effect is
found when comparing O2 to O4, this design allows the researcher to examine
whether this treatment effect can be replicated when comparing O5 versus O6.
If the researcher finds a treatment effect in both cases, the results of the study
will be considerably stronger because the treatment has been found to be effective
across two trials. In this way, the replication results will enhance the generaliz-
ability of the treatment, and thus the external validity of the study.
Weaknesses The major drawback to using this design is that it costs the
investigator a great deal in terms of time, energy, and resources. In essence,
the Solomon four-group design contains the two previous designs (pretest-
posttest control group and posttest-only control group), which means that the
experimenter is almost conducting two investigations. The Solomon four-
group design is especially costly if the treatment is a long and rather complex
intervention. Such treatments often require considerable effort, such as train-
ing therapists (perhaps before each treatment session), preparing training
materials, and obtaining financial compensation for therapists. Many times
ethical considerations are involved in the intervention, particularly with regard
to the control group; an example is withholding treatment from the control
group during part or all of the research study. A common option is to offer
treatment to control participants after the experiment is over; it is important
to plan on this additional cost (that is, additional service delivery for two more
groups of participants).
Often it is practically and ethically difficult to have a group that does not
receive any treatment. However, a viable control condition can be obtained by
using a waiting-list control group. Typically, participants are randomly assigned
to either the treatment condition or the waiting-list control group; at the end
of the treatment phase and the posttests, the treatment is made available to the
participants in the waiting-list control group. (If more than two treatments were
studied, the participants might be given the choice of the available treatments,
or the one that proved most effective.) In either the pretest-posttest control group
design or the posttest-only control group design, the treatment given to the
waiting-list participants is not part of the design but can be analyzed anyway
to test the reliability of the results (Kazdin, 2003) or to rule out threats to the
validity of quasi-experimental designs (Shadish et al., 2002). One disadvan-
tage of the waiting-list control group is that long-term follow-up of the con-
trol participants is lost (because they have by then received treatment).
Another disadvantage is that although ultimately the participants in the waiting-
list control group receive treatment, the treatment is withheld for some time.
(For more details on this topic, see Chapter 18.)
Another type of control group is the placebo control group. Participants
in a placebo control group are led to believe that they are receiving a viable
treatment, even though the services rendered them are nonspecific and sup-
posedly ineffective. For example, in a group counseling outcome study, par-
ticipants in the placebo condition may be in a discussion group with no active
group counseling. The rationale for including a placebo control is that it
enables the researcher to separate the specific effects of a treatment from
effects due to client expectations, attention, and other nonspecific aspects.
Some investigators contend that the major effects of the counseling process
are due to nonspecific factors (Frank, 1961; Wampold, 2001); inclusion of a
placebo control group allows determination of whether the effects of a treat-
ment are greater than those obtained under conditions that appear to clients
to be viable but do not contain the major aspects of the active treatments.
An interesting placebo is the subconscious reconditioning placebo (SRP).
Participants are told that the SRP reconditions the subconscious by using sub-
liminal messages, when actually nonsense syllables are presented. Participants
believe they are receiving a viable treatment; in fact, SRP has been found to be
as effective as other treatments in the areas of smoking cessation (Sipich,
Russell, & Tobias, 1974), test anxiety (Russell & Lent, 1982), and speech anxiety
(Lent, Russell, & Zamostny, 1981). Of course, inclusion of a no-treatment or
a waiting-list control group with a placebo control group strengthens the design
considerably because differences between the placebo and no-treatment control
groups give an estimate of the strength of effect of nonspecific factors.
A final type of control group is the matched control group. Participants
in a matched control group are paired in some way with participants in the
treatment group. Although the primary purpose of this type of design is to
reduce variance due to a matching factor (which will be discussed later in this
chapter under dependent group designs), one application of this design is worth
mentioning here.
chapter 7 between-groups and within-subjects designs 159
Factorial Designs
Factorial designs are used when two or more independent variables are em-
ployed simultaneously to study their independent and interactive effects on a
dependent variable. It is sometimes cumbersome to relate factorial designs to
the three between-groups designs previously discussed; nonetheless, factorial
designs are extensions of these earlier designs, namely by the addition of inde-
pendent variables. Whereas previously we used the notation of Shadish et al.
(2002), with factorial designs it is more useful to visualize the design by dia-
gramming the levels of the independent variables into cells. For example, if a
study were examining three levels of an independent variable, there would be
three cells, diagrammed like this:
Treatments
X1 X2 Control
Males
Gender
Females
Alternately, if there were two independent variables that each had three
levels or conditions, this would be considered a 3 × 3 design and have nine cells.
In a way, our fictitious example indicates how the factorial design can
provide important qualifications about relationships between variables. The
factorial design provides some answers about the conditions under which a
treatment may operate, such as the gender of participants, the type of inter-
vention, the age of clients, or the problem-solving style of clients. Whereas the
single-variable study most often investigates whether a variable (most notably
some treatment) has any effect, the factorial design examines more complex
questions that approximate the complexity of real life.
Weaknesses Although at first one might think the more information, the
better, it is important to realize the costs involved as more variables are added
to designs. With the addition of variables, the results of the study become
more complex and sometimes too complex. In a 2 × 2 design, the researcher
would typically examine the main effects of two levels of variable A, the main
effects of two levels of variable B, and the interaction of A with B. In a 2 (A) ×
2 (B) × 2 (C) design, the investigator would typically examine the main ef-
fects of variables for two levels of A, B, and C; the two-way interactions of A
with B and B with C; and the three-way interaction among A, B, and C.
Complex interactions between three, four, or more independent variables typ-
ically are difficult to interpret, and the results of the study may be unclear.
Researchers should not add independent variables just to have more than one
independent variable; instead, independent variables need to be carefully se-
lected on theoretical and empirical grounds after thought is given to the re-
search questions of interest.
Another disadvantage of the factorial design is the flip side of an advan-
tage: If additional independent variables are added to the design and these
variables turn out to be unrelated to the dependent variable, then the power
of some statistical test may be reduced. There are also complications regard-
ing the conclusions that can be drawn when the independent variable is a
status variable (for example, counselor gender) and is not manipulated. (For
more details, see the discussion in Chapter 14.)
alone did not result in these significant changes, and the control group did not
see significant changes either. Moreover, combining the two interventions did
not result in a more effective intervention than the print/video intervention
alone. The authors were encouraged by the utility of the professionally devel-
oped materials delivered to each person, as opposed to a more diffuse inter-
vention delivered by a nonprofessional.
Participant Assignment
Because the basic purpose of the between-groups design is to make compar-
isons between participants from different groups, it is critical that the people
in the groups do not differ in important ways before the experiment or mea-
surement begins. Participant assignment is also predicated on participant se-
lection, which pertains to external validity. That is, a researcher might
satisfactorily assign undergraduate students to one of four group conditions
(participant assignment), but the results might not be generalizable to coun-
seling center clients (a participant selection issue).
The intended outcome of assigning people to groups is to eliminate sys-
tematic differences across groups before the experiment, so that if any changes
are detected in one or more of the groups after the experiment, the change can
be attributed to the independent variable. Participants therefore need to be
assigned to groups in an unbiased fashion and free from extraneous variables.
The most effective way of ensuring comparable groups is to assign par-
ticipants to groups randomly, or in such a way that each participant has the
same probability of being assigned to each group. Such assignment tends to
equalize both known and unknown sources of participant variation across
groups, so that extraneous variables will not bias the study.
A number of procedures exist for randomly assigning participants to
groups. The most common method is to use a table of random numbers, or a
computer program that generates random numbers to determine the order of
assigning participants to groups. For example, if a researcher had four groups,
participants could be randomly assigned by the order that the numbers 1, 2,
3, and 4 appear consecutively in the random list of numbers (for example, 4,
1, 4, 2, 2, 3, 1, and so on). Numbers other than 1 through 4 would be ignored.
In this instance, the first participant would be assigned to group 4, the next to
group 1, the third to group 4, and so on. Such a procedure reduces the prob-
ability of systematic biases or variations in participants between groups.
Note that random assignment would most likely result in unequal num-
bers of participants in each of the four groups. For statistical purposes it is
better to have equal numbers across groups. To deal with this issue, Kazdin
(2003) suggested assigning participants in blocks, or as in the preceding exam-
ple, in blocks of four participants. Within each block of four participants,
the experimenter would simply randomly assign one participant to each of
the four groups. This procedure is particularly useful when participants begin the
experiment periodically, or at different times.
chapter 7 between-groups and within-subjects designs 163
more powerful test. The analysis of covariance does not require that partici-
pants be matched, and the reduction in unexplained variance is accomplished
statistically, by the design of the experiment. The dependent samples design
reduces uncertainty by matching comparable participants. Two participants
who have high pretest scores are also likely to have high posttest scores; dif-
ferences in posttest scores for these two matched participants are due pre-
sumably to the treatment (and other uncontrolled factors).
Dependent samples can be accomplished in other ways as well. Often nat-
ural pairs, such as monozygotic twins, are used. Because monozygotic twins
have identical genetic material, using such pairs holds all hereditary factors
constant. Other natural pairs include litter mates (not often applicable to
counseling researchers), marital partners, siblings, and so forth.
Another means of creating dependent samples is to measure the participant
in more than one condition, even though this is difficult to accomplish in the
treatment and control conditions. Consider two treatments—say, treatments
for increasing the degree to which participants use a computerized career
exploration program. Each participant would be exposed to one of the treat-
ments (for example, token reinforcement for use of the system), and then use
of the system would be assessed; then the participant would receive the other
treatment (for example, social reinforcement for use of the system), and use
of the system is again assessed. Of course, the order of the treatments would
need to be randomized. In this way, all nuisance factors are controlled because
each participant serves as his or her own control. Such a design, called a re-
peated measures design, contains many of the aspects of the within-subjects
design discussed in the next section.
The idea of two dependent samples can be expanded to include more than
two groups (for example, two treatment groups and a control group). Typ-
ically, the dependency is created by matching or by repeated measures (it is a
bit difficult to find enough monozygotic triplets for such a study!). For example,
Haupt (1990) was interested in counseling students on religious issues; he had
reason to believe that the type of counselor responses to a client’s religious
statements would affect how the client perceived the counselor’s expertness,
attractiveness, and trustworthiness. Haupt believed that participants’ beliefs
about Christianity would most likely affect how participants would react to the
counselor responses. In considering assigning participants to one of three
group conditions, Haupt decided to ensure group equivalency on this attitudi-
nal variable. Consequently, as he assigned participants to one of the three
groups, he first identified three participants who had identical scores on a scale
measuring beliefs about Christianity, and then randomly assigned each of these
participants to a different group. As a result, participants had identical scores
on their beliefs about Christianity across each of the three groups. When more
than two participants are matched and assigned to conditions, the design is
called a randomized block design. Each group of matched participants is called
a block, and the participants within blocks are randomly assigned to conditions.
The randomized block design is typically analyzed with a mixed model analysis
of variance (see Wampold & Drew, 1990).
166 part 2 major research designs
WITHIN-SUBJECTS DESIGNS
The remainder of this chapter examines within-subjects designs. Remember
that the hallmark of the within-subjects design is that it attempts to minimize
error variance due to individual variation by having each participant serve as
his or her own control. Similar to the between-groups design, participants are
randomly assigned to groups or treatments, and independent variables are
manipulated. The unique feature of the within-subjects design is that all par-
ticipants are exposed to all of the treatment conditions; random assignment
involves assigning people to different sequences of treatment.
In this section we first provide an overview of two types of within-
subjects designs: crossovers and Latin Square designs. We then discuss the
strengths and limitations of the traditional within-subjects design.
Crossover Designs
Suppose a researcher wanted to compare the effects of two treatments (inde-
pendent variables)—test interpretation of the Strong Interest Inventory (SII)
and work genograms—on a dependent variable, vocational clients’ career
maturity. The researcher could use the following within-participants design:
O1 X1 O2 X2 O3
chapter 7 between-groups and within-subjects designs 167
R O1 X1 O2 X2 O3
R O4 X2 O5 X1 O6
The R’s in the design indicate that participants are randomly assigned to two
groups. Again, X1 and X2 in the diagram represent the two treatments, and
the O’s represent the different observation periods. O1 and O4 designate a
pretesting assessment; O2 and O5 represent an assessment at the crossover
point, and O3 and O6 indicate testing at the end of the experiment. Thus, the
groups differ only in the order in which they receive the treatments. In this
case, counterbalancing also controls for sequence effects: X1 precedes X2 once,
and X2 precedes X1 once.
It is important to be aware of two issues with regard to counterbalancing.
First, the researcher can now use some simple statistical procedures to deter-
mine whether the order of the treatment conditions made any difference vis-
à-vis the dependent variables. For example, a simple t test can be conducted
168 part 2 major research designs
The one problem with the Latin Square design is that the particular
sequence of treatments cannot be controlled (or assessed statistically). In the
preceding design, for instance, treatment X1 never directly follows treatment
X2. Because all possible sequences are not represented in the Latin Square, it
is not possible to entirely rule out this type of sequence effect as a rival
hypothesis. This problem is usually considered minor, however, compared to
order effects.
Hermansson, Webster, and McFarland (1988) used a Latin Square design
to examine the effects of deliberate counselor postural lean on levels of com-
municated empathy, respect, and intensity. They manipulated three levels of
the independent variable, postural lean (forward, backward, and counselor’s
choice). Each counselor conducted three consecutive sessions with a different
client. For the first 9 minutes of an 18-minute session, the counselor was
seated upright, then for the next 9 minutes each counselor leaned forward,
backward, or counselor’s choice. The particular order of forward, backward,
or choice was determined by the Latin Square. Each counselor was randomly
assigned to a particular sequence.
The results of this study suggested a compensatory process between delib-
erate counselor lean and verbal communication. Specifically, a required
forward lean was associated with decreased levels of intensity and empathy,
whereas a required backward lean showed a significant increase in the
levels of intensity and empathy. There were no significant effects for the
choice condition.
170 part 2 major research designs
Statistical Power Because each participant receives all levels of the inde-
pendent variable, there are typically some advantages from a statistical per-
spective. In general, a researcher can use half the number of participants in a
counterbalanced crossover design and still retain the same statistical power as
in the between-subjects design (see Kerlinger, 1986, for a more complete sta-
tistical discussion of this matter).
are threats to internal validity. Even when order effects are controlled, as in
the counterbalance crossover and Latin Square designs, it is still important to
check whether the order of the treatments affected the dependent variable.
Sometimes it is assumed that because counterbalancing equalizes any effects
due to order, the researcher can ignore such order effects. This strategy, how-
ever, does not provide any information about the basic question: Were there
any order effects in a particular study? Such information can be useful to
future researchers as they design their investigations on a similar topic. Likewise,
practitioners may be interested in knowing if the order of treatments makes
any difference as they plan to maximize their interventions.
Sequence effects can also create threats to internal validity. In a counter-
balanced crossover design, this sequence, or carryover effect, can be statisti-
cally examined because all possible sequences are represented in the study.
Recall that Stiles and colleagues (1988) examined for and found no sequence
effects. However, in the Latin Square design all possible sequences are not rep-
resented. Thus, the Latin Square design typically has more threats to internal
validity because sequence effects are more difficult to eliminate.
R O1 X1 O2 X2 O3
R O4 X2 O5 X1 O6
Suppose tshat in that study the effect of the two treatments (interpersonal psy-
chodynamic treatment and cognitive-behavioral treatment) on client depres-
sion was also assessed. Further suppose that at pretest (O1 and O4) all clients
had a pretest score of 21 on the Beck Depression Inventory (BDI), and that
after the first treatment for both groups of participants, the crossover O2 and
O5 revealed that all clients had a BDI mean score of 2. Because the BDI score
cannot be lower than 0, there is little or no room for the second treatment to
show improvement.
There is a related measurement problem in within-subjects designs that
involves the equivalency of scale points. For example, is a change in the mean
BDI score from 15 to 10 equivalent to a change from 10 to 5? Again, this
problem is not atypical of between-groups designs, but these problems are
172 part 2 major research designs
pretest-posttest control group design, and the of traditional within-subjects designs. In short,
Solomon four-group design. These experimental within-subjects designs offer a powerful means of
designs are clearly powerful designs, because they identifying causal relationships. The advantage of
can rule out many rival hypotheses. Each design these designs is their ability to reduce both error
controls for all the common threats to internal variance (by using each participant as his or her
validity. A key feature of these designs is the own control) and the number of participants
random assignment of participants; randomly needed in a particular study. The within-subjects
assigning participants to groups is a major source design has a number of strengths, most notably
of control with regard to internal validity. the reduction of participant variability, that ide-
Kerlinger (1986) concluded that between-groups ally suit it for research in counseling. Nonetheless,
designs “are the best all-around designs, perhaps even a perusal of journals reveals that within-sub-
the first to be considered when planning the jects designs have been underused in counseling
design of a research study” (p. 327). Because con- research. The strengths of the within-subjects
trol groups are commonly used in these designs, designs merit more consideration in our research.
we discussed issues pertaining to different types Clearly, the between-groups and within-sub-
of control groups, such as no-treatment groups, jects designs are useful designs for examining
waiting-list control groups, placebo groups, and research questions of interest to those in the
matched control groups. Because randomization counseling profession. These designs are flexi-
of participants is a defining characteristic of ble and can be made applicable to a wide vari-
between-groups design, we also discussed partic- ety of research problems; in fact, the factorial
ipant assignment, group equivalence, and design is widely used in counseling research.
dependent samples designs. However, it is important for the researcher in
We also described two traditional within- counseling to evaluate the strengths and limita-
subjects designs, the crossover and Latin Square tions of these designs relative to the type of
designs. Both of these designs make comparisons research question being asked and type of par-
between two or more groups of participants, but ticipants needed. Given the applied nature of
in a different way from the between-groups many of our research questions in counseling,
design. In the crossover design, all participants the researcher needs to consider carefully a
are switched to another experimental condition, broad range of issues pertaining to external
usually halfway through the study. Counterbal- validity to evaluate the utility of the true exper-
ancing was introduced within this design as a imental designs in providing the most-needed
way of reducing bias due to order effects. The information. In addition, many times the
Latin Square design was introduced as a design random assignment of participants to groups
suitable for research questions that examine cannot be done because of ethical constraints,
more than two levels of the independent vari- such as in a study of the effects of different
able. We suggested that at least six issues specific levels of sexual harassment. We think it is er-
to the traditional within-subjects design can roneous for students to be taught that the
affect its utility for examining a particular between-groups design is simply “the best”;
research question, namely (1) experimental con- instead, students should be encouraged to con-
trol (particularly with regard to individual par- sider the strengths and weaknesses of various
ticipant variation), (2) statistical power, (3) time, designs in relation to different research ques-
(4) order and sequence effects, (5) measurement tions. In other words, the utility of the design
considerations, and (6) restriction of certain needs to be evaluated in the context of the
independent variables. In particular, we encour- research question, the existing knowledge
aged researchers to be creative in the application bases, and internal and external validity issues.
174 part 2 major research designs
STIMULUS QUESTIONS
Between-Groups
and Within-Subjects Designs
This exercise is designed to promote reflection 3. Identify two topics you could examine with
on between-groups and within-subjects experi- both a between-groups and a within-subjects
mental designs. After reading this chapter, write design. Write four research questions, one
your responses to the following questions. Then between-groups question and one within-
discuss your responses with a peer in your class. subjects question for each topic. After you
have written all four research questions, com-
1. Talk to faculty and peers about their percep-
pare and contrast the research questions.
tions of the usefulness of the between-groups
What similarities and differences do you see?
and within-subjects design. What advantages
4. Make a list of the pros and cons of using a
and disadvantages first come to mind for
control group. Of all the issues you list, can
them? Is there a pattern in the responses that
you pick only one that you believe is the most
is reflected when others speak of the dis-
important methodological issue?
advantages of between-groups and within-
5. In the early days of counseling, more between-
subjects designs?
groups and within-subjects designs were used.
2. Compare the primary strengths and weak-
Now it is difficult to find good examples of
nesses of between-groups and within-subjects
them in our major journals. Why do you think
designs. Could you argue that one of these
this trend exists, and what do you think it
designs is better than the other?
means for the field of counseling?
Quasi-Experimental
and Time-Series Designs
CHAPTER
8
175
176 part 2 major research designs
framework for the study was social norms theory (Berkowitz, 2003), which
maintains that the social influence of one’s peers is based more on what one
thinks his or her peers do and believe (i.e., perceived norms) than on actual
behaviors and real beliefs (i.e., actual norms).
The experimental group participated in a three-session intervention on
acquaintance rape that incorporated local social norms. The control group at-
tended their classes as usual. As hypothesized, the participants’ ratings of their
peers were significantly different (worse) from the peers’ ratings of themselves at
pretest regarding attitudes toward sexual violence. Furthermore, the experimental
group participants’ ratings of their peers were significantly more accurate follow-
ing the intervention, indicating they viewed their peers as less supportive of sexual
assault. At posttest, the experimental group demonstrated a significant decrease in
rape-supportive attitudes, as compared to the control group, and this decrease was
maintained at follow-up 4 weeks later. Although additional research is needed,
this quasi-experimental study was able to provide very useful information about
sexual assault prevention in the high schools, and the generalizability of the find-
ings to other Midwestern urban high school students would be considered high.
In short, it is often difficult to achieve the conditions of the true experimental
design in field settings, especially the requirement of random assignment into
groups. Because of this difficulty in much applied research, the quasi-experimental
design offers more flexibility. Quasi-experimental designs, like true experimental
designs, involve the manipulation of one or more in-dependent variables, but not
the random assignment of participants to con-ditions. With this flexibility, however,
quasi-experimental designs also bring some limitations. For example, Kazdin
(2003) noted that some researchers refer to these designs as “queasy-experimental
designs” (p. 169) due to their lack of randomization. Nonetheless, with creativity
in the use of the designs and appropriate use of controls and statistical procedures,
quasi-experimental designs can be very useful for applied researchers.
In this chapter we will first illustrate an early quasi-experimental design
that will provide a historical context for this type of design. Then we will dis-
cuss some considerations for determining when quasi-experimental designs
might be appropriate. We will then focus on the two major classes of quasi-
experimental designs, the nonequivalent groups designs and the time-series
designs. In nonequivalent groups designs, comparisons are made between par-
ticipants in nonrandomly formed groups. Specifically, we will discuss three
types of uninterruptible designs, four types of interpretable nonequivalent
groups designs, and cohort designs, which are a special case within the broad
category of nonequivalent groups designs. Then we discuss the time-series
designs, which have as their defining feature multiple observations over time.
Cost
One of the most functional reasons for not conducting a true experimental
design is often that of cost. Conducting a true experiment can be quite expen-
sive in terms of time and resources. In true experimental designs, researchers
often must pay participants to be part of often two or three treatment groups,
along with a control group. Conversely, it is a lot less expensive to evaluate
naturally occurring differences in treatment settings, classrooms, or other
places where people naturally come together. For example, even in the study
of sexual assault prevention provided earlier, if the researchers would have
decided to utilize a true experimental design, they would have likely had to
obtain a group of 30–40 participants who, after random assignment, would
have been willing on their own time to attend a pretest, three intervention ses-
sions, a posttest, and a follow-up one month later. This is a substantial time
commitment, and most likely the participants would need to be compensated
for their time and involvement. In short, although an experimental design pro-
vides a more rigorous design, sometimes the cost is prohibitive. In essence, the
researchers may be willing to compromise experimental control in order to
conduct the study in the field.
Selection of Participants
The hallmark of experimental designs is randomly assigning participants
to various treatment conditions. Ideally, the investigator recruits a pool of
participants, and randomly assigns them into the treatment and control groups,
which meet at prearranged times. However, a number of issues that arise in field
research can make the selection of participants a difficult and complex process.
For example, some of the participants might be available to come to a treat-
ment group in the early afternoon, but not in the evening. Such issues of time
availability are an important logistical concern in applied research, and often
make it more feasible to conduct quasi-experimental investigations within ex-
isting group settings such as high school classes, as in the previous example.
In some field settings it would be difficult or even inappropriate to ran-
domly assign participants to an experimental or a control group. For example,
an investigator may want to examine the effect of a group session summary
(a therapist’s written summary of a group session that is mailed to each group
chapter 8 quasi-experimental and time-series designs 179
Ethical Considerations
Some studies focus on participants who are in need of immediate services (like
counseling or medical assistance). For example, a researcher might be study-
ing a phenomenon that happens infrequently, such as breast cancer. It may
take some time to identify a sufficient number of patients that seek help from
a particular agency and to randomly assign them to a group. Consequently, it
may raise ethical issues to withhold treatment while waiting for more patients
to be randomly assigned into groups. The quasi-experimental designs allow
for the use of intact groups who may have already come together in a partic-
ular setting. Although randomization may not then be possible, other controls
can be designed into the study using the methods we will describe in this chapter.
this prior group formation, they may differ on several characteristics before
the intervention (Kazdin, 2003). For example, a researcher may want to examine
the effects of a videotape that provides precounseling information on subse-
quent counseling dropout rates. He or she may be able to find a counseling agency
that uses such a tape and compare the agency’s dropout rate with the dropout
rate for an agency that does not use this type of tape. Obviously because the
clients at the two agencies may be different on a number of variables that may
relate to dropout rate (for example, ethnicity or social class status), the clients
in the two agencies represent nonequivalent groups. The usefulness of a
nonequivalent groups design is related in part to how much the researcher
knows about possible pretreatment differences among participants in the
nonequivalent groups.
These types of quasi-experimental designs have also proved beneficial in
studying the impact of various training models on counselors in training. For
example, in a recent study Crews and her colleagues (2005) used a nonrandom,
pretest-posttest design to study the role of counselor personality traits on
counseling performance. Counselors in training were given pretesting to deter-
mine their level of self-monitoring (on the Skilled Counseling Scale; SCS) and
then self-selected into one of two training conditions, an Interpersonal Process
Recall (IPR) condition and a Skilled Counseling Training Model (SCTM) con-
dition. The purpose of the study was to determine the impact of two different
kinds of training on counselors with differing levels of self-monitoring. The
results revealed that there were no statistically significant differences in pretest
or posttest scores on the SCS. In addition, both the IPR and SCTM groups
improved their scores on the SCS; however, the SCTM group improved signif-
icantly more than those counselors in the IPR group.
To examine this design further, we will diagram the nonequivalent groups
quasi-experimental designs. The symbol Non R represents the nonrandom
assignment of participants to groups. As in the previous chapters, X indicates
the independent variable or treatment, and O indicates observations of the
dependent variable.
X1 O1
182 part 2 major research designs
In this design, observations are made of the dependent variable only after par-
ticipants have undergone some type of treatment. This design is impossible to
interpret because there is no way to infer that any type of change has taken
place. In addition, the lack of a control group makes it impossible to investi-
gate the presence of maturational or historical processes.
A posttest-only nonequivalent design can be diagrammed as follows:
Non R X O1
Non R O2
In this design, the two groups are formed in a nonrandom manner. The par-
ticipants in the first group receive the experimental treatment (X) while the
participants in the second group do not receive any treatment. Change is
measured by comparing the posttests (O1 and O2).
It is important to note that the posttest-only nonequivalent design need
not compare a treatment with a control group. Two or more active treatments
can be compared using this type of design. The following is a diagram of a
posttest-only nonequivalent design comparing three active treatments:
Non R X1 O1
Non R X2 O2
Non R X3 O3
Once again, the groups are formed on a nonrandom basis. Treatments (X1,
X2, and X3) are administered to the participants in the three groups, and then
posttests (O1, O2, and O3) are used to assess changes.
In essence, the posttest-only nonequivalent designs are especially weak
because of the difficulty in attributing results to the intervention. The lack of
random assignment of participants to groups allows the possibility that the
groups may have differed along any of a number of important dimensions prior
to treatment. Typically, students are assigned to classes, patients to wards, cli-
ents to groups, and residents to living groups based on some rationale, which
suggests that the natural groupings we encounter will differ prior to treatment
on a few, or in some cases many, dimensions. Thus, one of the problems with
the posttest-only nonequivalent designs is the lack of information about any of
the possible differences in the groups that exist before treatment.
Consider the following example. Suppose an investigator wanted to ex-
amine the usefulness of an in-class program in alleviating depression in chil-
dren. He or she might select two classes of sixth graders in a school and then
provide one class with the intervention. After one month he or she assesses the
students’ level of depression. Suppose further that after treatment, the students
who received the intervention show less depression. This result may indicate
an effect of the treatment, or it may reflect differences between the two classes
in their levels of depression before the intervention. Perhaps the principal
decided to assign students to classes on the basis of their social skills levels.
chapter 8 quasi-experimental and time-series designs 183
Research has documented the relationship between social skills and depression
(see, for example, Lewinsohn, Mischel, Chapel, & Barton, 1980). Because
there was no pretest, the possible differences in the initial levels of depression
were unknown. It is quite possible that the control group actually consisted
of students with lower levels of social skills, and subsequently a significantly
higher level of depression in the beginning could not be assessed.
The third type of uninterpretable design that we will consider is the one-
group pretest-posttest design. This design is diagrammed as follows:
O1 X O2
Non R O1 X O2
Non R O3 O4
The A and B in this design represent two forms of a test or tests designed to
measure similar constructs. In this design, groups are formed nonrandomly
and a proxy pretest (OA1) is administered to both groups. Later, one group
gets the experimental treatment (X), and then both groups are retested with a
different posttest (OB2). The viability of this design depends on the ability of
the researcher to find a pretest measure (OA1) that relates conceptually and
empirically to the posttest (OB2).
For example, researchers may want to examine a new method of coun-
selor training. They find two training programs willing to participate and
institute the new method in one program. At the end of the first year, the
researcher administers a paper-and-pencil counseling skills test to all students
in the two programs and finds that the students in the treatment program
scored higher on this test. However, the researcher is worried about possible
pretreatment differences in counseling skill level. Suppose the researcher finds
that Graduate Record Exam (GRE) scores (which conveniently all students
took before starting graduate school) are correlated (r = 0.80) with scores on
the paper-and-pencil counseling skills test. (In actuality, the GRE does not pre-
dict counseling skills, but suppose it to be the case for this illustration.) In this
case, the researcher can use the pretreatment GRE score (OA1) to examine pos-
sible pretreatment differences between students in the two programs.
The pretest-posttest nonequivalent groups design can be strengthened by
the use of an additional pretest. This design is diagrammed as follows:
Non R O1 O2 X O3
Non R O1 O2 O3
This design is similar to the pretest-posttest nonequivalent groups design
except for the addition of a second pretesting to enhance the interpretability
of the design. A major threat to the internal validity of a pretest-posttest non-
equivalent groups design involves a selection-by-maturation interaction. In
other words, the participants in the two groups may be maturing at different
rates because of some selection characteristic. The addition of a second pretest
allows the researcher to examine this possibility; the difference between O1
and O2 for the treatment and control groups can be examined to see if the
groups are maturing at different rates and enhances the interpretability of a
nonequivalent groups design. A review of the counseling literature, however,
suggests that two pretests are rarely, if ever, used. We strongly recommend
that researchers contemplating the use of a nonequivalent groups design con-
sider the addition of a second pretest.
We will next discuss the reversed-treatment pretest-posttest nonequivalent
groups design, which is also rarely used in counseling research. We include a dis-
cussion of this design here because it is one of the stronger nonequivalent groups
designs. We hope that an understanding of the strengths of this design will
encourage its use in counseling research. The design is diagrammed as follows:
Non R O1 X+ O2
Non R O1 X– O2
186 part 2 major research designs
Cohort Designs
Cohort designs are a special case of nonequivalent groups designs that utilize
adjacent cohort groups that share similar environments. For example, the sixth
grade class at a particular school one year is likely to be similar to the sixth grade
class the following year. In essence, cohort designs allow researchers to make
causal inferences because comparability can often be assumed between adjacent
cohorts that do or do not receive a treatment (Shadish et al., 2002). However,
the compatibility in a cohort design will never be as high as in an experiment
with random assignment. Nonetheless, cohort designs have a relative advantage
188 part 2 major research designs
over other types of nonequivalent groups designs because cohorts are more
likely to be similar to each other than in typical nonequivalent groups designs.
It is important for the researcher to have as much knowledge as possible
about conditions that could affect the cohorts. Cohort designs are strength-
ened when the researcher can argue conceptually and empirically that the two
cohorts did in fact share similar environments, except of course for the treat-
ment. For example, two successive sixth grade classes at a particular school
will likely be similar across two years. However, this would not be the case if,
for example, school district lines were redrawn between the two years, or if a
new private school opened in the community and attracted many of the
wealthier children away from the public school.
Three types of cohort designs have been used in counseling research. The
first design, a posttest-only cohort design, is diagrammed as follows:
O1
––––––
X O2
In this design, the broken line indicates that the two groups are successive
cohorts and not nonequivalent groups. The O1 represents a posttest adminis-
tered to one cohort, whereas the O2 represents the same posttest administered
to the second cohort. It is important to note that the testing occurs at differ-
ent times because the cohorts follow each other through the system; however,
the posttesting occurs at a similar point in each cohort’s progression through
the institution.
Slate and Jones (1989) used a posttest-only cohort design to test the effect
of a new training method for teaching students to score the Wechsler Intelli-
gence Scale for Children-Revised (WISC-R). One cohort of students took the
intelligence testing course during the fall semester, the other cohort during the
spring semester. The fall cohort received a standard scoring training proce-
dure, whereas the spring cohort received the new training method. The results
indicated that students in the spring as opposed to the fall made fewer scor-
ing errors on the WISC-R. Slate and Jones concluded that the new training
method was effective. These authors assumed that the students in the fall and
spring cohorts were similar prior to training, and buttressed this assumption
by examining several possible sources of pretreatment differences. For example,
they found that the gender composition was similar across the two cohorts
and that the students in the two cohorts had similar GRE scores and grade
point averages.
A second type of cohort design, a posttest-only cohort design with parti-
tioned treatments, is diagrammed as follows:
O1
–––––
X1 O2a
X2 O2b
chapter 8 quasi-experimental and time-series designs 189
O1 is the posttest given to the first cohort, X1 represents the first level of treat-
ment, X2 represents the second level of treatment, and O2b is a posttest meas-
ure given to all members of the second cohort regardless of level of treatment
administered. In essence, the posttest-only cohort designs are strengthened by
partitioning the treatment, which involves giving different amounts of the
treatment to different groups of participants within a cohort.
In the Slate and Jones (1989) study, suppose that some of the students in
the second cohort practiced the new scoring procedure for two hours and that
the other students in the cohort practiced it for four hours. Slate and Jones
could have analyzed the results separately for these two groups of students. If
the students who had practiced for four hours (O3) committed significantly
fewer scoring errors than the students who practiced for two hours (O2), and
if the treatment cohort committed fewer errors than the no-treatment cohort,
then the assertion of treatment efficacy would be strengthened. Moreover, the
results would provide additional information about the amount of training
needed. In short, the posttest-only cohort designs can be useful, particularly
relative to the posttest-only nonequivalent groups design. Because clients ex-
perience various aspects of counseling treatments in different amounts, we
urge researchers to use partitioning as a way of strengthening the internal
validity of the posttest-only cohort design in counseling research.
The third cohort design that we will discuss is the pretreatment-posttreatment
cohort design, diagrammed as follows:
O1 O2
–––––––––
O3 X O4
The first cohort is pretested (O1) and posttested (O2), and then the second
cohort is pretested (O3), treated (X), and posttested (O4). The main advantage
of the pretest-posttest cohort design over the posttest-only cohort design is the
increased assurance the pretest provides for asserting that the two cohorts were
similar prior to the treatment. In addition, the use of the pretest as a covariate
in an analysis of covariance provides a stronger statistical test in general. The
main disadvantage of this design is that the pretest can constitute a threat to
external validity because of pretest sensitization; that is, taking the pretest itself
in some way sensitized the participants and caused their scores at posttest to
differ. In most cases the advantages of a pretest to examine pretreatment com-
patibility across groups outweigh the threat to construct validity.
BDI scores greater than 14 and who met the criteria for nonpsychotic unipolar
depression. These participants could have been tested during the same time
frame as the treatment participants. This type of control could have served to
test the temporal (Christmas holiday) confound in examining the results.
TIME-SERIES DESIGNS
The defining characteristic of a time-series design is multiple observations
over time. These observations can involve the same participant (for instance,
the client’s ratings of counselor trustworthiness after each counseling session)
or similar participants (for example, monthly totals of clients requesting ser-
vices at a counseling center). In an interrupted time-series design, a treatment
is administered at some point in the series of observations. The point at which
the treatment takes place is called an interruption of the series. The logic of
the interrupted time-series design involves comparing the observations before
and after the treatment or interruption. If the treatment has an effect, there
should be a difference in the observations before and after the interruption.
Although the logic of comparing pre- and postinterruption observations for
evidence of difference is simple and straightforward, the statistical analysis
can be complex; see Shadish et al. (2002) for more details.
In this section we concentrate on the logical analysis of interrupted time-
series designs. (Chapter 9 discusses time series as applied in single-participant
designs.) In the next section we describe two time-series designs with the hope
of stimulating counseling researchers to consider these designs in planning
their research.
Multiple observations occur both before (O1–O6) and after (O7–O12) the treat-
ment (X) is initiated. The diagram shows an equal number of observations
before and after the treatment, but this is not a requirement for the design.
The interrupted time-series design has two advantages over the quasi-
experimental designs previously described. First, the time-series design allows
the researcher to detect maturational changes that may occur prior to treat-
ment initiation. The researcher does this by looking for changes in the pre-
treatment observations. If found, these maturational changes can be controlled
for in a statistical analysis, allowing a more powerful test of the effect of the
treatment. The second advantage of the time-series design is that it also allows
for the analysis of seasonal trends. Often, data examined by counseling re-
searchers vary systematically over time. For example, more clients seek coun-
seling around holiday periods. It is obviously important to account for this type
192 part 2 major research designs
4
Cognitive-affective
Immediate
Complementarity
2
Supervisor enters
0
0 20 40 60 80
Counselor statements
(OA) shows an interruption at the time that the time-limited model was intro-
duced, but the number of clients requesting services (OB) does not show a sim-
ilar interruption, then the director can conclude that the initiation of the
time-limited model caused a reduction in the waiting list. It is unlikely that
history could cause this effect because history would likely also affect the
number of clients requesting services.
Kivlighan (1990) used the interrupted time-series analysis with nonequiv-
alent dependent variables to study the effects of live supervision in counselor
training. Beginning counselor trainees saw a recruited client for four 50-minute
counseling interviews. Advanced counseling doctoral students provided live
supervision for the counselor trainees. This supervision involved viewing the
counseling interview from behind a one-way mirror, entering the session at
some point, commenting on the counseling process, and providing direction for
the counselor. The observations in this study consisted of ratings of each of the
counselor statements. Trained judges rated each counselor statement on both a
cognitive-affective dimension and an immediacy dimension (statements about
the client–counselor relationship versus statements outside of the counseling
experience). Based on the interpersonal training model used, Kivlighan pre-
dicted that after the interruption (the supervisor entering the room), the coun-
selor’s statements would be less cognitive and more immediate.
Figure 8.1 shows ratings of statements taken from one counselor–client–
supervisor triad during the interview. The supervisor intervened between the
60th and 61st counselor statements. A visual inspection of these graphs suggests
that the counselor’s statements became more immediate and less cognitive after the
194 part 2 major research designs
170
160
150
140
COSE
120 Phase 6 Phase 7
case
110
100
[|]–case
90
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
Practicum sessions
y-axis and practicum sessions are plotted along the x-axis. Trainee 3 had her
first case presentation during the tenth practicum session. There were six
COSE observations (practicum sessions 1, 2, 3, 6, 8, and 9) before this case
presentation (interruption) and seven observations (practicum sessions 10, 11,
19, 20, 21, 23, and 24) after the case presentation. The overall omnibus F test
for this data was significant (F (2, 10) = 8.30, p < .01), indicating that there
was an overall change in COSE scores from pre- to postinterruption. Other
analyses indicated that the average level of counselor self-efficacy was higher
after the case presentation than before it. The significant analysis indicated
that the rate of change for counselor self-efficacy was different after the case
presentation than before it. Specifically, before the case presentation there was
a downward trend in self-efficacy scores, but after the case presentation the
scores leveled off. These results suggested that, at least for this one trainee, the
case presentation resulted in an overall increase in self-efficacy and also halted
a pattern of declining self-efficacy. For a more extended discussion of coun-
selor self-efficacy, see Larson (1998) and Larson and Daniels (1998).
In essence, the researcher observes two dependent variables over time. In this
example, the researcher would check to see whether the counselor’s level of
self-disclosure adds predictability to the client’s level of self-disclosure over
and above the predictability obtained from patterns in the client’s level of self-
disclosure. An introduction to the statistical analysis of concomitance in time
series can be found in Cook and Campbell (1979).
Working Through Resistance scales of the MARS. The second series of obser-
vations (series B) consisted of weekly ratings, by trained observers, of the level
of client resistance using the Resistance Scale (Schuller, Crits-Christoph, &
Connoly, 1991). The time-series analysis revealed that for Exploring and
Working Through Resistance there was a significant reduction of resistance in
later sessions. Specifically, there were significant negative relationships between
the Exploring and Working Through Resistance scores and the composite level
of client resistance. These results indicated that increases in the therapists’ at-
tempts to explore and work through the client’s resistance in one session result
in fewer manifestations of client resistance in the two subsequent sessions.
Carefully designed quasi-experiments have the Although there are threats to validity inherent in
potential for offering critical knowledge of psycho- these designs, many of these threats can be con-
logical phenomena in naturally occurring settings. trolled by design and statistical techniques.
STIMULUS QUESTIONS
Quasi-Experimental Designs
This exercise is designed to promote reflection on be done to increase the validity of using intact
quasi-experimental designs. After reading this chap- groups?
ter, think about your responses to the following 4. Identify two research questions you could
questions: examine with a quasi-experimental design.
What type of quasi-experimental design would
1. Talk to faculty and peers about their percep-
you use to examine these questions? Diagram
tions of the usefulness of the quasi-experimental
the design using the notation from the chapter.
design. What advantages and disadvantages
5. What safeguards would you use to reduce
first come to mind for them? Is there a pattern
threats to the internal validity of your study?
in the type of disadvantages others speak of
6. Early in the history of counseling, quasi-
when they discuss quasi-experimental designs?
experimental designs were used frequently.
2. Why are some quasi-experimental designs un-
Now it is more difficult to find good examples
interpretable? What design elements could make
of them in the major counseling journals.
them interpretable?
What factors do you believe have brought
3. One of the concerns with quasi-experimental
about this change?
designs pertains to using intact groups. What can
9 CHAPTER Single-Subject
Designs
Studying the individual to understand more about humanity has been a major
interest driving students to study counseling and psychology in general.
Understanding ourselves and those close to us can be a fascinating endeavor.
For example, what makes one individual experience despair while another
remains resilient in similar situations? In the past, applied psychologists have
sought such answers about an array of individuals by carefully examining
single cases, studying a rare psychological phenomenon, or studying an inten-
sive treatment effect with an individual client. However, methodological issues
have plagued the development of a scientific analysis of individuals to the
point where the study of just one person has sometimes been regarded as
unscientific. Error variance, extraneous variables, and numerous sources of
bias—major threats to Kerlinger’s MAXMINCON principle—have beset re-
searchers using a single subject.
Consequently, many more studies in our scientific journals rely on study-
ing groups rather than single individuals. The field has conducted many more
investigations that produce group averages or means as a way to promote
understanding of individual behavior. Single-subject designs are not currently
utilized extensively in counseling research. For example, a content analysis of
the articles published from 1990 to 2001 in the Journal of Counseling and
Development (Bangert & Baumberger, 2005) revealed only 1% of the articles
(two studies) used a single-subject design. Yet as Kazdin (2003) maintained when
making valid inferences about human behavior, “there is nothing inherent to
198
chapter 9 single-subject designs 199
the approach that requires groups . . . findings obtained with groups are not
necessarily more generalizable than those obtained with the individual case”
(p. 265). Moreover, with the increased availability of different kinds of sophis-
ticated statistical software to analyze single-subject data (e.g., Dennin & Ellis,
2003; Levin & Wampold, 1999), there are more opportunities to examine a
variety of issues within the counselor and client domains that have promise to
increase understanding of topics of importance to counseling.
A broad range of scholarship has been very loosely labeled under the term
single subject designs. These studies have ranged from loosely controlled anecdotal
observations to highly stringent and rigorous experimental designs. Although
anecdotal, observational studies can be important for heuristic purposes, at
times producing useful observations for further hypothesis testing. This chapter
will focus on ways to increase the scientific rigor and validity of investigations that
study just one individual such that they can be of maximum benefit to the field.
The central thesis of this chapter is that intensive single-subject designs,
like other experimental designs, can play an important role in the pursuit of
knowledge within the counseling profession. However, important distinctions
must be made concerning different types of studies that use a single subject,
and concerning the varying amounts of experimental control that are used.
This chapter will present a variety of approaches that can be considered single-
subject designs. In all cases, single-subject research examines variables that
vary within the subject over time. Therefore, single-subject designs involve a
longitudinal perspective achieved by repeated observations or measurements
of the variable(s). In terms of the type of observations, the data can be quan-
titative or qualitative. Quantitative data can be categorical, ordinal, interval,
or ratio. Qualitative data, in contrast, take the form of text. A critical distin-
guishing feature among the designs pertains to the operation of the indepen-
dent variable; it can be either manipulated or simply observed. Single-subject
designs can also differ in the goal or focus of the research. The goal of the research
can focus on either testing or generating hypotheses; in other words, testing a
theory or discovering new information.
We will discuss three categories of single-subject designs in this chapter:
(1) uncontrolled case studies, (2) intensive single-subject quantitative designs, and
(3) single-subject experimental designs. Briefly, the uncontrolled case study
typically does not contain systematic observations, but rather contains many
uncontrolled variables. A defining feature of intensive single-subject designs
are systematic, repeated, and multiple observations of a client, dyad, or group
to identify and compare relationships among variables. Even though both the
single-subject experimental and the single-subject intensive quantitative designs
share a number of similarities and both use quantitative data, the hallmark
and distinguishing feature of the single-subject experimental design is the
manipulation of the independent variable. Conversely, in the intensive single-
subject quantitative designs the independent variable is observed. Single-
subject experimental designs have been conducted almost exclusively within
the behavioral and cognitive behavioral tradition, although the theoretical
orientation need not be restricted by the type of single-subject design.
200 part 2 major research designs
The chapter is divided into four major sections. The first section provides
a brief historical perspective on the use of single-subject designs within psy-
chology, and particularly within applied psychology. By understanding the
historical context, students can better appreciate how this particular design
was born and how it has changed as the research methods have become more
sophisticated within the field of applied psychology. Second, we differentiate
the uncontrolled traditional “case study” from the intensive single-subject design.
Unfortunately, there are often misconceptions about single-subject designs, in
part based on uncontrolled methodologies, and we want to dispel the myth
that all single-subject designs are unscientific and cannot add important infor-
mation to the counseling literature. It is hoped that this discussion and illus-
trative examples will not only help students understand important methodological
issues in studying individuals, but also promote an understanding of the value
and richness of the more rigorous single-subject designs. In the third section,
we will focus on single-subject experimental designs (where the independent
variable is manipulated). We discuss several defining features of these time-
series designs, and then describe and illustrate two different types of designs:
AB time series and multiple baselines. In the final section we discuss advan-
tages and limitations of single-subject designs.
that some clients improved but other clients actually became worse (Truax &
Carkhuff, 1967; Truax & Wargo, 1966). In short, although group comparison
methods and inferential statistics substantially facilitated research on the effects
of psychotherapy, researchers quickly encountered a number of confounding
variables that underscored the complexity of the therapeutic experience.
In response to the confusion about and the complexity within the thera-
peutic process, other methodologies have been subsequently proposed and
explored, such as naturalistic studies (Kiesler, 1971), process research (Hoch
& Zubin, 1964), and a more intensive, experimental single-subject design
(Bergin & Strupp, 1970). On the surface it might seem that the applied pro-
fessions have come full circle, returning to the study of the individual. This is
true only in part; there is considerably more methodological sophistication in
scientifically studying an individual subject today than there was during the
early part of the twentieth century.
In this section, we will discuss two major types of designs used in single-
subject quantitative designs: AB time-series designs and multiple baseline
designs. We will start with the AB time-series designs, which have many vari-
ations; to illustrate the AB time-series design we will discuss three specific
designs: AB, ABAB, and randomized AB.
5 A B
4.5
4
3.5
Group cohesion
3
2.5
2
1.5
1
0.5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Session number
Figure 9.1 AB Design Examining Group Cohesion (Measured on
a Five-point Likert Scale, with 1 = Low Cohesion,
5 = High Cohesion) by Session Number (the Agenda-GO-
Round Exercise was Instituted at Session 11)
The ABAB Design In contrast to the AB design, which as we have said is basi-
cally a two-phase experiment, the ABAB design is a four-phase experiment,
and is also frequently referred to as a reversal design. In essence, the ABAB design
examines the effect of a treatment (or independent variable) by either pre-
senting or withdrawing the variable during different phases in an attempt to
provide unambiguous evidence of the causal effect of the independent variable.
The ABAB design starts with a period of baseline data gathering (A1) and a
treatment phase (B1), and then it returns to a baseline period (A2) where the
intervention is withdrawn, and then finally a second treatment phase (B2).
The assumption underlying this reversal is that if the independent variable
caused the change in the dependent variable in the B1 phase, then a removal
of the independent variable should return the subject to a level similar to the
baseline phase. Moreover, if the reversal in fact results in a return to the baseline,
then readministering the independent variable at B2 will serve as a replication,
further strengthening the inferred causal relationship. If the behavior at A2
does not revert to the baseline levels, then a causal relationship between the
independent and dependent variables cannot be inferred because other (un-
known) variables may have been the causal influence. Thus, in the Yalom
agenda-setting example given earlier when describing the AB design, an ABAB
design would be collected in the following manner: Ten group sessions in which
the researcher collected the baseline cohesion and member involvement data
(A1), followed by 10 sessions in which the group leader implemented the agenda-
go-round intervention (with continued data collection) (B1), followed by another
10 group sessions of data collection without the agenda-go-round intervention
(A2), and then a final 10 sessions with the agenda-go-round intervention rein-
stituted (B2). Figure 9.2 presents a graph of data collected from this ABAB
design. Because group cohesion increased in both B phases, we can infer that
the agenda-go-round exercise caused an improvement in group cohesion.
Let’s look at another example of the ABAB design to examine the rela-
tionship between therapist behavior and client resistance. This study is also a
good illustration of programmatic research that uses both descriptive and exper-
imental designs to study a phenomenon. First, using a descriptive research
strategy, Patterson and Forgatch (1985) examined the likelihood of client
resistance following various types of counselor behavior (for example, sup-
portive or teaching responses). In essence, they found that client resistance fol-
lowing therapist teaching responses was significantly higher than the clients’
baseline resistance. One might guess that this was because the client was feel-
ing talked at or lectured to, and they may have felt resistance as a result. But
because this first study was only descriptive, the authors could not infer that
therapist teaching caused client resistance. But this question did provide the
210 part 2 major research designs
5 A B A B
4.5
4
Group cohesion
3.5
3
2.5
2
1.5
0.5
0
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Session number
researchers with a wonderful question with which to use the ABAB design. Six
counselors met with their clients for videotaped sessions. The counselors
started with a baseline phase (A1) in which they interacted with their clients
without using teaching interventions; this was followed by a treatment phase
(B1) in which the counselors increased their use of teaching responses. A
second baseline phase followed in which the counselors returned to interacting
without the teaching responses (A2). There was then a final treatment phase (B2)
in which the counselors once again increased their use of teaching responses.
The results of this ABAB study showed that the counselors did give signifi-
cantly more teaching responses during the two treatment phases, indicating
that the experimental manipulation was successful (such a procedural check
is referred to as a manipulation check). More important, the clients were more
resistant during the treatment phases (B1 and B2). Thus, these data more strongly
suggest that therapist teaching behavior caused client resistance.
Although there are many strengths to the ABAB design, it does have three
problems that we would like to note. The first problem is a statistical one.
Most authors and editors want to report some statistical test that describes the
amount of difference between the two phases (A and B; see previous refer-
ences regarding statistical issues for these designs). When reported, this test is
usually a t test or an F test. Both t and F tests, however, assume independent
observations, and this is not the case in the ABAB design (Wampold & Freund,
1991). A second problem with the ABAB design is the possible presence of
carryover effects (that is, effects from the manipulation at phase B1 that affect
A2 or B2). It could be that the effects of the B1 intervention are irreversible and
cannot be withdrawn. For example, if the dependent variable involves learning
chapter 9 single-subject designs 211
A A B A B A B B B A B A A B B A B A B A
Session 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Data are assigned to cells based on whether they come from phase A or B
and on which phase preceded the phase in question. For example, the first cell
in the table contains the data from all phase-A sessions (all sessions without
the agenda-go-round) that were preceded by a phase-A session (session without
212 part 2 major research designs
the agenda-go-round). In a like manner, the second cell of the table contains
all data from phase-A sessions (without agenda-go-round) that were preceded
by a phase-B session (with agenda-go-round). A factorial analysis of variance
(ANOVA) using this setup would provide a main effect for phase A versus B,
a main effect for preceding phase (preceding A versus preceding B), and an
interaction effect for phase × preceding phase. The interaction analysis di-
rectly tests for carryover effects.
Some final comments about the AB time-series designs seem warranted.
First, although we have discussed and illustrated AB designs as having only
two (A and B) phases, this need not be the case. A design could contain three
(A, B, and C) or more phases. A three-phase design could compare a baseline
phase to two different treatments, or it could compare three different levels of
an independent variable. The basic logic of the AB design remains the same
no matter how many phases are involved—the researcher is attempting to iso-
late the effects of the independent variable by examining different levels (or
trends) across different phases.
Many times AB designs are not considered with particular research proj-
ects because of possible ethical concerns. For instance, one may question
whether it is appropriate to withdraw a treatment that seems to be having a
positive effect on a client. Although this is an excellent question, it is impor-
tant to note this is the same dilemma faced by members of the medical pro-
fession in the early 1960s in prescribing thalidomide for morning sickness;
unfortunately, they decided not to withhold treatment because of the per-
ceived effectiveness of the drug and the results were terrible birth defects. In
short, it is important to adequately test the efficacy of our counseling inter-
ventions empirically before assuming their effectiveness.
Another concern with AB designs is related to carryover effects of an
intervention and the perceived unsuitability of some variables for some theo-
retical orientations. For example, Gelso (1979) reasoned that one cannot
reverse a client’s insight (the theoretical effect of an appropriate interpreta-
tion), and thus the AB designs would not be well-suited for such content. We
believe that the AB designs can be successfully applied to a wider range of
topics than is typically acknowledged, although the researcher may need to be
creative in adapting the AB designs. Doing so can bring the researcher a
wealth of data that are highly useful to the practitioner, in addition to gener-
ating important research questions in an area.
Multiple-Baseline Designs
The second type of single-subject experimental quantitative design that we
will discuss is the multiple-baseline design. The essential feature of this design
is that data are recorded on more than one dependent variable, or target
behavior, simultaneously. Like the AB designs, there is continual collection of
data, but in multiple-baseline designs there are two or more data collection
baselines on different dependent measures. The basic assumption is that with
several dependent measures, one or more of these variables can serve as controls
chapter 9 single-subject designs 213
the group are realistic, interpersonal, and here-and-now oriented. The researcher
could begin by making a baseline assessment of each group member’s interac-
tions over the first five group sessions. He or she could then begin to train
individual group members to set realistic, interpersonal, and here-and-now
agendas. Only one group member would be trained at a time, and the initia-
tion of the training with the individual group members would be staggered;
the researcher might train a different member every fifth group session. A
change in any individual’s quality of group participation would be expected
to occur contingent on his or her receiving the agenda training, whereas no
change would be expected in those not trained. The basic assumption in this
example is that individuals can serve as controls when they have not received
the intervention. Data collection is continuous, and only those individuals who
have received the intervention are expected to change on the dependent measure.
A possible source of contamination in this example is that subjects who have
not received training might learn how to set effective agendas by observing
their peers who have received training.
The third variation of the multiple-baseline design is to identify a given
response for one subject but across different situations. As in all illustrations
of the multiple-baseline design, a subject can refer to an individual, a group,
a classroom, or a larger unit. Suppose a researcher wants to examine the
effects of token reinforcement (the independent variables) on a child interact-
ing with peers (the dependent variable). The amount of interaction with peers
could be observed both before and after school on the playground, during a
baseline period. The researcher could then begin token reinforcement of
prosocial interactions only before school. The researcher would then expect
to find a change in the amount of interaction with peers in the morning but
not in the afternoon. Later the token reinforcement could be done in the after-
noon also, and a consequent change in this baseline would then be expected.
Because the logic of the multiple-baseline design requires that the researcher
show changes in one assessed behavior while another assessed behavior remains
constant, one of the main concerns in using this type of design is the inde-
pendence of the dependent variables. If there is a relationship between two or
more of the behaviors, then a change in one behavior may well lead to a change
in the other. If two behaviors show simultaneous change but only one received
the independent variable, then it is impossible to rule out threats to internal
validity such as history and maturation. Such unintended changes in baselines
seriously jeopardize the strength of the multiple-baseline design and typically
produce a study with uninterpretable results.
One way to guard against this complication is to carefully assess the inde-
pendence of the behaviors, perhaps by correlating the baseline behaviors
(Christensen, 1980). Another possible solution is the use of several dependent
variables; Kazdin and Kopel (1975) recommended using four or more vari-
ables. By increasing the number of baselines, the researcher can guard against
the possibility that two of the baselines are dependent on each other.
A recent example of using a multiple baseline design was the study men-
tioned earlier by Dennin and Ellis (2003). Briefly, in this study Dennin and
chapter 9 single-subject designs 215
setting” (p. 92). In this section we discuss some of the strengths and advantages
of the single-subject design, specifically as a means of: (1) collecting information
and ideas and generating hypotheses about the therapeutic process, (2) testing
therapeutic techniques, (3) testing new methodologies, (4) studying individuals
and rare phenomena, and (5) providing exemplars and counterinstances.
of techniques might happen at several levels: (1) the discovery of a new technique,
(2) an intentional examination of a relatively new technique, (3) the applica-
tion of an established technique to a new population or treatment situation,
and (4) an in-depth examination of a well-established technique. Occasionally
therapists will discover new techniques through a trial-and-error process with
individual clients. Breuer’s work with Anna O. resulted in the discovery of a
“talking cure” or catharsis. In essence, Breuer found through trial and error
that some of Anna’s symptoms were relieved or disappeared simply by talking
about them (Breuer & Freud, 1955). George Kelley, in part out of boredom
with the Freudian interpretations, began fabricating “insights” and “prepos-
terous interpretations,” and discovered that often clients could change their
lives in important ways if they believed these “alternative constructions of the
world” (Monte, 1980, p. 434). Likewise, Rogers abandoned the traditional
directive and diagnostically oriented therapeutic style after a dismal thera-
peutic failure in which a client reentered and redirected therapy by initiating
a discussion of her troubled marriage. Rogers discovered that “it is the client
who knows what hurts, what direction to go, what problems are crucial, and
what experiences have been deeply buried” (Rogers, 1961, pp. 11–12). In
short, therapists often stumble onto or create new techniques through their
therapeutic work with individual clients. The intentional and consistent exam-
ination of new techniques within the informal case study, as well as the more
formal single-subject design, can yield useful information about the counseling
process. Thus, new observations by the therapist might be informally or for-
mally tested with additional clients to further determine the effectiveness and
generalizability of the technique.
The single-subject design can also be used effectively to test the application
of an established technique with a new problem or a new population. Kazdin
(2003) noted that the extension of a given technique to a new problem is really
quite common in the literature. For example, systematic desensitization has been
found to be applicable to a wide range of phobias or fears, such as fear of heart
attacks (Furst & Cooper, 1970) and fear of sharks (Krop & Krause, 1976).
Finally, the single-subject design also lends itself well to an in-depth exam-
ination of the effectiveness of a certain technique. For example, a great deal
of information could be collected about how a client is processing or reacting to
repeated use of a certain technique, such as the Gestalt empty-chair technique,
counselor touch, or counselor self-disclosure. Thus, in-depth information could
be collected on the use of a specific technique over time.
In this way, the single-subject design can be used to collect data about the
“thinking frameworks” of individuals (Edgington, 1987; Neimeyer & Resnikoff)
or higher mental processes (Wundt, 1916). Polkinghorne aptly suggested that
human action and decision making, which are central to the therapeutic and
counseling processes, appear to be related to a “means-end rationality” or mode of
processing information. Heppner and Krauskopf (1987) noted similar thinking
processes by using an information-processing model to describe client problem
solving in a counseling context. The single-subject design may be particularly
well suited for gathering information about (1) how clients process informa-
tion, (2) the thinking steps or means involved in reaching some end or goal (log-
ically or illogically), (3) an individual’s plans and intentions (Heppner &
Krauskopf; Howard, 1985), and (4) how such plans affect the processing of
information and subsequent behavior. The complexity of such information
processing, at least initially, may be examined more feasibly via a single-subject
design. In short, single-subject designs are useful to study complex phenomena,
particularly as counselors use some of the more qualitative approaches in
examining higher mental processes.
The single-subject design is particularly useful in studying rare phenomena,
such as multiple personality. In addition to rare phenomena, the single-subject
design can also be used to study relatively low-frequency occurrences, such as
male anorexics and college students under unique situational stress (e.g.,
Middle Eastern students after 9/11, or aborted grief reactions; see Corazzini,
1980). Although a typical university counseling center usually has a very broad
range of clients, it is often difficult to study infrequent occurrences in a group
comparison format. It is not usually feasible, or ethical, to withhold treat-
ment until enough of these low-occurrence clients seek counseling to enable
random assignment of clients to groups, and so forth. Thus, the single-subject
design is useful to investigate various idiosyncrasies or commonalities across
clients with similar presenting problems, such as clients with aborted grief
reactions.
and to work within the therapist’s framework. Strupp then used these results
to propose that patient variables were more powerful events in assessing thera-
peutic outcomes, even to the point of overshadowing therapists’ attitudes and
technical skill.
Kazdin (1980) addressed the problem of symptom substitution stemming
primarily from psychoanalytic thinking. The general assumption was that
treatment of overt, problematic behavior, rather than treatment of the under-
lying cause, may result in symptom substitution. However, very early research
on specific problematic behaviors such as phobias or bedwetting did not result
in the appearance of substitute symptoms (see, for example, Jones, 1956; Meyer,
1957; Yates, 1958). Thus, this early research using primarily uncontrolled
case studies cast doubt on the unqualified notion of symptom substitution, sug-
gesting that this phenomenon may be more complex than previously assumed.
In sum, there are many advantages to a well-designed and carefully con-
trolled single-subject design. In essence, the single-subject designs can study
important phenomena that other designs are less suited to examine in depth.
of an individual, dyad, or group under experi- we strongly believe in the potential of this method-
mental conditions designed to identify and com- ology, our first recommendation is to use the
pare relationships among variables. Thus, the more rigorous single-subject designs more fre-
comparisons and control in the intensive single- quently in counseling research. These designs
subject quantitative design provide stronger em- can be especially rich in providing more holistic
pirical support, and thus give it more utility than data of both a content and process nature.
the case study. A number of the examples cited in this chap-
The single-subject experimental designs also ter were intentionally drawn from group coun-
examine the relationship between two or more seling. One of the main obstacles in group
variables, typically within one or a few subjects, counseling research is sample size. Many times in
and involve considerable levels of experimental group counseling research, the variable of interest
control. These designs are characterized by spe- is a group-level variable such as cohesion. Often,
cific treatment goals, numerous and repeated however, researchers use individual scores, such
measurements of the dependent variable, and the as each subject’s attraction to the group, so they
inclusion of different phases or periods of time can increase the size of the groups in a between-
(each representing a different experimental con- groups design. Researchers could circumvent this
dition such as a baseline phase or an intervention sample-size problem by using more single-subject
phase). The basic purpose of changing from one experimental time-series designs. Accordingly, our
phase to another is to demonstrate change, pre- second recommendation is an increased use of
sumably due to the onset of the independent vari- single-subject experimental designs in group (and
able or intervention. family) research. This recommendation could
Two major types of single-subject experimental also help address a second problem in group
designs are the AB time-series designs (AB, ABAB, research—that most group counseling studies use
randomized AB) and multiple-baseline designs. analogue groups of very short duration. Time-
We maintain that the time-series designs are not series designs allow the researcher to study the
only appropriate for behavioral researchers, but group as an entity and to use real client groups
also can be fruitfully employed within other theo- without having to line up a prohibitively large
retical orientations. number of subjects.
We firmly believe that both the intensive Our third recommendation is the increased
single-subject quantitative design and the single- use of randomized designs in AB time-series
subject experimental designs can be powerful research. Randomized designs offer additional
methodologies to increase understanding of complexity and flexibility, and enable the re-
counseling and the counseling process. These searcher to examine for the presence of carry-
designs can be used for (1) collecting informa- over effects between treatments. In addition,
tion and ideas, and generating hypotheses about randomized designs permit the use of powerful
the therapeutic process; (2) testing therapeutic parametric statistical analyses. Editors are often
techniques; (3) testing new methodologies; reluctant to publish studies that do not have sta-
(4) studying individuals and rare phenomena; tistical comparisons, and readers rely on these
and (5) providing exemplars and counterin- statistical comparisons as a way of comparing
stances. However, limitations imposed by threats studies for AB time-series designs; parametric
to both internal and external validity must be statistical tests can only be performed appro-
clearly examined and considered. Bergin and priately when A and B phases are randomly
Strupp (1970) initially predicted that the single- assigned.
subject design would be one of the primary strate- A final methodological consideration in
gies used to clarify the process of counseling; this discussing the use of the single-subject method-
remains to be demonstrated. Nonetheless, because ology is the type of data or knowledge currently
chapter 9 single-subject designs 223
available on a particular research question. stopping was based largely on data collected
For example, a considerable number of studies from a single methodological perspective, a phe-
have investigated the therapeutic technique of nomenon that has been called paradigm fixation
thought stopping. In 1978, Heppner reviewed (Gelso, 1979). The point is that merely ascer-
the published investigations to determine the taining the advantages and limitations of the
clinical efficacy of thought stopping. This body single-subject design is not enough; researchers
of literature consisted of 22 studies, 20 of which must also weigh these advantages and limita-
were case studies that lacked experimental control tions relative to the applicable existing scientific
(Heppner, 1978b). Thus, this body of “knowl- knowledge base and the types of designs used in
edge” concerning the clinical efficacy of thought past research.
STIMULUS QUESTIONS
Single-Subject Designs
This exercise is designed to help promote reflec- in thinking and the reduced use of the uncon-
tion on single-subject designs. After reading this trolled case studies?
chapter, think about your responses to the fol- 3. Think about the topics you feel passionate
lowing questions. about in your own scholarly work and practice.
Identify three research questions that would
1. Some students have told us that their training
lend themselves well to some sort of a single-
programs “do not allow” single-subject designs
subject quantitative design. How would you
for a thesis or dissertation. Why do you think
design a study using an intensive single-
that is the case? If you were a person who
subject design for this topic?
wanted to conduct an intensive single-subject
4. One of the concerns with single-subject de-
design for your own thesis or dissertation, what
sign is the issue of generalizability. What can
arguments would you use in order to con-
be done to increase the level of external valid-
vince your training program of their validity?
ity or generalizability with such designs?
2. In the history of counseling, there have been
5. Talk to faculty and peers about their percep-
a number of uncontrolled case studies. Now
tions of the usefulness of the single-subject
it is more difficult to find examples of these
designs. What advantages and disadvantages
designs in our major journals. What factors
first come to mind for them?
do you believe have brought about this change
10 CHAPTER Quantitative Descriptive
Designs
Descriptive designs are research strategies that help to define the existence and
delineate characteristics of a particular phenomenon. For example, in the
counseling profession, we are often interested in the frequency of counseling-
related phenomena, like the frequency of eating disorders in college students
(see Mintz, O’Halloran, Mulholland, & Schneider, 1997). Such information
is very useful for developing both remedial and preventive interventions.
Thus, a very important role of our research is to describe the occurrence of
phenomena of interest. Similarly, we often want to describe characteristics of
particular phenomena, like different dimensions of heterosexual knowledge
and attitudes regarding lesbian, gay, and bisexual individuals (see Worthington,
Dillon, & Becker-Schutte, 2005). Likewise, we often find it useful to describe
the relationship between two or more variables, such as the relationship
among ethnic identity, discrimination, and depression in Korean Americans
(see Lee, 2005). In short, such descriptive information promotes greater
understanding of various phenomena, and this knowledge can then be used to
increase the effectiveness of a host of counseling interventions. Thus, descriptive
designs are very important and useful research strategies of the counseling
profession and consequently are commonly found across counseling journals.
Historically, texts on research design have devoted little attention to
descriptive designs. This lack of attention stemmed partially from what we
call the “pure science myth,” which held that the experimental design para-
digm was the “correct” or “best” mode of scientific investigation. Moreover,
224
chapter 10 quantitative descriptive designs 225
the received view of science tended to emphasize the testing and verification of
theories, as well as comparisons among competing theories. Twenty years ago,
Greenberg (1986a) argued that counseling psychology should place greater em-
phasis on discovery paradigms. He maintained that the empiricist tradition, with
its emphasis on the controlled between-groups experiment, was overvalued by
counseling researchers. Consequently, too often researchers attempted to manip-
ulate and control variables before enough was known about the phenomenon of
interest. Inadequately described phenomena were often subjected to rigid verifi-
cation, with disappointing results. Before one can test the adequacy of a theory
in explaining a phenomenon, one needs a reliable and detailed description of the
phenomenon. Clearly, descriptive designs have an important role in the scien-
tific process, which is reflected today in their frequent utilization.
The utility of a descriptive study is directly dependent on the quality of the
instruments or assessments used to describe the phenomenon. Moreover, it is
impossible to observe all instances of a phenomenon, so instead the researcher
carefully studies the phenomenon in a sample drawn from the population of
interest. Thus, the reliability and validity of the observations and the generaliz-
ability (or external validity) of the sample are two critical issues in descriptive
research. At first glance, the MAXMINCON principle that we have emphasized
seems antithetical to descriptive research. For example, by exercising experimen-
tal control, it might seem that the experimenter would interfere with or change
the natural phenomenon he or she was attempting to describe. Indeed, many
variables are left uncontrolled in descriptive research, and thus cause-and-effect
statements are inappropriate. On the other hand, the investigator can exercise
considerable care in making observations so as to minimize measurement errors,
which can reduce considerable error variance. Thus, the quality of the observa-
tions is the heart of descriptive design. Likewise, the descriptive researcher can
exercise experimental control to reduce bias and extraneous variables by using
random sampling. Accordingly, in this chapter we emphasize measurement, obser-
vation, and sampling as we review the different types of descriptive designs.
The purpose of this chapter is to discuss three major types of quantitative
descriptive research: (1) survey or epidemiological research, (2) classification
or data reduction research, and (3) correlational research designs. Briefly,
research strategies designed to characterize the occurrence of behaviors are
commonly known as epidemiological or survey designs. Classification research
uses factor and cluster analysis as strategies for categorizing and reducing
data, primarily by identifying underlying dimensions or groupings in a set of
variables. Research strategies that examine the relationship among or between
variables are referred to as correlational designs. This type of design includes
studies that use correlational or regression analyses.
information has helped the staffs of college counseling centers to design pro-
grams and deliver services appropriate to the needs of students. Parenthetically,
this line of research has led to a number of studies that have examined not
only the typical problems experienced by students, but also the appropriateness
of services delivered by counseling centers (as viewed by students, faculty, and
counseling center staff), and ultimately larger issues such as the role and functions
of university counseling centers (see the review by Heppner & Neal, 1983).
Other examples of survey research include studies examining the level of
distress in psychologists (see, for example, Thoreson, Miller, & Krauskopf,
1989), alcoholism among psychologists (see, for example, Thoreson, Budd, &
Krauskopf, 1986), job analysis of psychology internships in counseling center
settings (Ross & Altmaier, 1990), career development of college women
(Harmon, 1981, 1989), and a survey of professional advocacy efforts of 71
leaders in national, regional, and state credentialing associations in counseling
(Myers & Sweeney, 2004).
A common strategy when using survey designs is to compare information
across two or more groups. For instance, Koplik and DeVito (1986) compared
the problems identified by members of the college classes of 1976 and 1986;
this is known as a survey cross-sectional design. Other times a researcher
might use a survey longitudinal design and compare, say, students’ responses
when they were freshmen versus their responses when they were sophomores.
Several authors have also examined gender differences in, for example, students’
perceptions and reporting of problems (see, for example, Koplik & DeVito;
Nagelberg, Pillsbury, & Balzor, 1983) and in academic careers (Thoreson,
Kardash, Leuthold, & Morrow, 1990). Comparisons have also been made
based on race, year in school, educational level, time of year, type of academic
program, and type of service requested. The accumulation of this type of
research, particularly for a college population, enables counselors to describe
with a high degree of confidence both the types of problems encountered on
college campuses and their differential distribution across segments of the student
population. In addition, the accumulation of these descriptive findings is
instrumental in the process of theory building. For instance, one consistent
finding in this body of research is the difference between males and females in
the number and types of problems recorded. This finding has led to theoretical
speculation about gender differences in problem perception.
Survey research is not limited to documenting the existence of problems
within a population. In fact, a wide range of other types of behavior or vari-
ables can be described using this type of approach. For example, Hill and
O’Grady (1985) were interested in counselor intentions, which they described
as the “why” of a counseling intervention. They analyzed the responses of 42
experienced therapists and documented the relative frequency of use of 19
theoretically derived intentions. In addition, they examined the differential use
rates of the various intentions as a function of the therapist’s theoretical ori-
entation. From these descriptive analyses, Hill and O’Grady were able to con-
struct a profile of intention use for experienced counselors and to identify
intentions that characterized various theoretical positions.
228 part 2 major research designs
Perhaps the most frequent problem with survey research is not with
survey methods per se, but rather with the lack of knowledge about conduct-
ing sound survey research. A common misperception is that survey research is
easy to conduct in general, and that the survey items can be quickly generated
in a few hours. For example, Fong (1992, p. 194), in an editorial in Counselor
Education and Supervision, said, “. . . I have become aware of the large
number of survey studies being conducted that are so casually designed that
they are invalid and unpublishable.” She went on to identify two critical prob-
lems with many survey studies:
Too often, I have reviewed manuscripts that report surveys of a sample that is con-
venient, not representative, or that represents only a subgroup of the intended
population under study. Likewise, I regularly find that the survey instrument was
not developed by careful selection of relevant variables or was solely developed by
the researcher without any pilot or reliability studies. (Fong, 1992, p. 195)
the researcher’s question in Chapters 3 and 4, and will discuss to some extent
deriving the sample in Chapter 14, selecting inventories in Chapter 13, con-
structing inventories in Chapter 20, and selecting and developing the data
collection method in Chapter 13. These chapters should all be consulted for
relevant activities in developing a survey study.
In terms of the actual design of a survey study, unlike true or quasi-exper-
imental designs, the researcher does not form the actual groups a priori.
Although groups may be formed for comparison’s sake (for example, males
versus females), the survey is often given to the entire identified sample. Even
when the researcher decides ahead of time to compare, for example, males and
females, he or she often finds these participants as they occur in the sample
population. However, some surveys purposefully sample from two groups—for
example, interns versus practicum students, or clinical psychology programs
versus counseling psychology programs. Even when this type of purposeful
sampling can be done, the researcher cannot assign participants to be in one
group or another as can occur in true experimental or quasi-experimental
research. Likewise, in survey designs the researcher does not manipulate an
independent variable, and so is not concerned with manipulation checks.
In terms of developing and/or selecting inventories for the survey, we
want to emphasize the need to carefully consider the psychometric properties
of existing inventories (see Chapter 13). Perhaps most important are the pro-
cedures for developing new survey items. It is a major misconception that
developing new survey items is quick and easy. Readers are encouraged to
consult Chapter 20 on scale construction, because many of those myths and
procedures apply to survey construction, most notably myths about scale con-
struction, convenience samples, and cultural issues, as well as steps in devel-
oping scales or surveys such as writing items, performing content analysis,
and piloting items.
Perhaps the most frequent way of collecting data in survey research is
through self-report questionnaires, particularly mailed or online question-
naires. The primary advantage of such surveys is the ease of data collection,
particularly when the sample covers a wide geographic area (in which case it
would be difficult to collect on-site data). One of the biggest potential disad-
vantages is the difficulty of getting participants to respond and return the
completed questionnaires. For example, often the return rate from a first mail-
ing is only 30%, which raises questions about the external validity of the
results. Was there some reason why the majority of the participants did not
respond? Would their responses be different from those of the 30% who
responded? It is doubtful that one could safely generalize from a 30% return
rate of a sample to the target population. Because the return rate is such a crit-
ical issue in mail surveys, researchers usually try to make it easy both to com-
plete the questionnaire (by keeping it short) and to return it (by including a
stamped, addressed return envelope). Two sets of reminder letters or post-
cards are also usually sent. Typically a follow-up letter is sent about two or
three weeks after the initial mailing, and subsequently two or three weeks
later as a final follow-up. Researchers commonly report obtaining around a
230 part 2 major research designs
30% to 40% return rate from an initial mailing, and approximately 20% and
10% returns from the two successive follow-ups. Although some published
survey research is based on less than a 40% return rate, some researchers rec-
ommend at least a 50% return rate as an “adequate” basis for findings (Baddie,
2001); others recommend at least 80% to 90% (Kerlinger, 1986).
A more recent method of data collection is through an online website. The
advantage of such a data collection method is that a broad range of individuals
in the target population, from whatever location within the United States or
beyond, can complete the questionnaire. Consequently, a researcher can often
obtain larger data sets, especially with restricted populations, than through
more traditional data collection methods. Conversely, it is difficult to ascertain
the representativeness of those participants who complete the survey.
Whatever method the researcher uses to collect data, the final step entails
data analysis. A critical starting point for data analysis is checking the ade-
quacy of the sample. This involves checking how closely the sample resembles
the general population along a number of important dimensions. For instance,
is the proportion of male and female (or racial/ethnic minorities and whites,
or young and old) respondents in the sample similar to the proportions in the
general population? Another especially important type of check when using
mail or online questionnaires is a comparison of respondents and nonrespondents.
For example, when using a college population, do respondents and nonre-
spondents differ in sex, year in school, major, grade point average, or the like?
Only after this type of sample adequacy checking has been done should the
data be analyzed and interpreted.
Survey research with diverse populations can present some unique chal-
lenges. Ponterotto and Casas (1991) discussed three issues that counseling
researchers should consider when using a survey design with culturally, lin-
guistically, and/or economically diverse groups. The first issue concerns how
to tailor the survey to the particular group under study: Do the respondents
understand the questions in the way that the researchers intended? Are any
items offensive to people in the group the researcher wants to survey? Do
potential respondents understand how to respond to the format? According
to Ponterotto and Casas, these questions can be addressed by examining the
language and format of potential questions in a pretest conducted on a pilot
sample that is representative of the target population. These authors suggest
that the survey be pilot-tested on a sample that represents about 5% of the
target sample. When the pilot test is administered, the respondents should be
interviewed to make sure that the questions asked were meaningful, clear, and
appropriate for the purpose of the survey.
The second issue raised by Ponterotto and Casas (1991) involves research
with participants who do not speak standard English. In this case, the
researcher must translate potential questions into the participants’ native lan-
guage. Unfortunately, this translation process is neither straightforward nor
simple (see Chapter 20, as well as Mallinckrodt & Wang, 2004). Briefly, we
will highlight two issues here: translation/back translation and decentering. In
back translation, a bilingual person first translates the original questions into
chapter 10 quantitative descriptive designs 231
the new language; once this translation is complete, another bilingual person
translates the translated questions back into English. Any discrepancies be-
tween the original and the back-translated versions of the questions are identi-
fied and corrected. The process of translation and back translation is then
repeated until the back-translated version matches the original version.
However, it is important to note that even though the words of a question can
be correctly translated and back translated, this does not mean that the meaning
of the words are the same in both languages. Decentering attempts to address
the issue of comparable functional and cultural meaning. In decentering, the
researcher attempts to ensure that no specific language is the “center” of
attention. This decentering is accomplished by having bilingual judges exam-
ine both versions of the survey and compare the functional and cultural equiv-
alence of the questions.
The third issue to be addressed involves the distribution and collection of
the survey. As noted by Ponterotto and Casas (1991), some socioeconomic
and/or cultural groups may not trust researchers who are connected with the
white middle-class establishment. This suggests that obtaining an adequate
response rate may be a larger problem with these groups of participants.
According to Ponterotto and Casas, it is important to be able to ascertain the
reason for a low response rate. These authors recommend that the researcher
attempt to interview a random subsample of respondents and nonrespondents.
In particular, the researcher should attempt to determine the characteristics
that distinguish the respondent and nonrespondent groups.
The first two questions called for an examination of the occurrence or fre-
quency of these specified events and were measured with two author-
constructed scales. To address the third research question, the authors chose
three constructs based on their review of the current literature. The three con-
structs were measured with validated inventories, including the Satisfaction
with Life Scale (Diener, Emmons, Larsen, & Griffen, 1985), the Work Related
Strain Inventory (Revicki, May, & Whitley, 1990), and the Brief Symptom
Inventory (Derogatis, 1992).
The authors wanted to generalize their results to the population of coun-
seling psychologists. Accordingly, a sample of counseling psychologists was
232 part 2 major research designs
obtained by randomly selecting 1,000 members (500 men and 500 women) of
Division 17 (the Society of Counseling Psychology) of the American Psycho-
logical Association. These 1,000 potential participants were sent the Division
17 Health Practices Survey via the U.S. Postal Service. In an attempt to
increase the response rate to the survey, Good, Thoreson, & Shaughnessy
(1995) sent a second copy of the survey to potential participants two weeks
after the initial mailing, and then sent a reminder postcard to each potential
participant one week after the second mailing.
Of the 405 surveys returned, 12 were incomplete or unusable. This rep-
resented a 39% return rate. The final sample, based on returned surveys, was
55% female and 45% male, with an average age of 48.2 years. The sample
consisted of predominantly married respondents (71%), with single (14%),
divorced (13%), and widows/widowers (3%) making up the rest of the sample.
In addition, the sample was predominantly white (94%); the remainder of the
sample was black/African American, 2%; Asian/Asian American, 2%; His-
panic/Hispanic American, 1%; and other, 1%.
One particularly strong aspect of Good, Thoreson, and Shaughnessy’s
(1995) study was their attempt to check the representativeness of their sample.
They accomplished this check by comparing selected demographic character-
istics of their sample’s participants to those same demographic characteristics
for members of Division 17 as a whole. For example, there was a similar per-
centage of degree type in the sample (Ph.D., 82%; Ed.D., 14%; and master’s
degree, 3%) and in the membership of Division 17 (Ph.D., 78%; Ed.D., 17%;
and master’s degree, 5%). Good and colleagues concluded that their sample
was representative of Division 17 because the demographic characteristics of
the sample closely matched those of Division 17.
As to the first research question—What is the prevalence of past and cur-
rent substance use reported by counseling psychologists?—the authors’ results
revealed that 19.7% of the sample reported “almost daily/daily” previous use
of alcohol, and 16.4% of the counseling psychologists surveyed indicated
“almost daily/daily” current use of alcohol. Usage rates were also reported for
cigarettes, marijuana, tranquilizers, hallucinogens, stimulants, opiates, cocaine,
and sedatives.
In response to the second research question—Are counseling psycholo-
gists aware of colleagues with substance abuse problems?—the researchers
found that 40% of the sample knew at least one female psychologist and 62%
knew at least one male psychologist who had substance abuse problems in the
past. However, only 19% of the respondents reported that they had con-
fronted a colleague about his or her substance problem. The most frequently
cited reasons for not confronting the colleague about her or his substance
abuse problem were: (1) knew of subtle changes but lacked tangible evidence
of negative impact (53%), (2) did not see it as detrimental to their job per-
formance (42%), and (3) thought it would do no good (39%).
To determine the relationship between substance use and psychological
functioning (the third research question), the authors initially created an index
of substance use by summing the reported levels of use. This substance use
chapter 10 quantitative descriptive designs 233
Factor Analysis
Factor analysis is a class of multivariate statistical methods whose primary pur-
pose is data reduction and summarization (Hair, Anderson, & Tatham, 1987;
Hair & Black, 2000; Fabrigar, Wegener, MacCallum, & Strahan, 1999). This
statistical technique is often used in developing and validating assessment in-
ventories. For example, a researcher might carefully develop a 50-item ques-
tionnaire to measure beliefs about social justice and social advocacy. Factor
analysis could be used to identify how the 50 items could be grouped together
in theoretically meaningful categories, with the end product being five distinct
factors consisting of 30 items. In essence, factor analysis examines the interre-
lationships among a large number of items (or variables) and condenses (sum-
marizes) that information into a smaller set of common underlying dimensions
or factors. These dimensions or factors presumably correspond to underlying
psychological constructs. Thus, the fundamental aim of factor analysis is to
search for underlying psychological constructs seen in the common dimensions
that underlie the original items or variables (Hair et al., 1987).
There are two major types of factor analysis: exploratory and confirmatory.
In exploratory factor analysis, the researcher examines a set of data to determine
underlying dimensions, without any a priori specification of the number or con-
tent of these constructs. In confirmatory factor analysis, the researcher first
identifies (either theoretically or through previous research) the number of di-
mensions he or she expects to find along with the items in the data set that will
correlate with (or “load on”) each dimension. In essence, the researcher has an
existing model of the factor structure. The researcher then examines how well the
model fits the actual relationships observed in a new set of data. The analysis can
either confirm (hence the name) or disconfirm the researcher’s model.
As an example of exploratory factor analysis, suppose a researcher has
observed over a 10-year period that for some adult clients, changing careers
is a difficult process. The researcher begins to wonder what distinguishes the
people who have difficulty from those who do not. After considerable reflection,
more observation, and a review of the professional literature, she develops a
100-item questionnaire that assesses barriers associated with changing careers.
chapter 10 quantitative descriptive designs 235
Such an instrument has not existed before because at this time little is known
about barriers to changing careers. But answering a 100-item questionnaire is
time consuming, and there is a lack of empirical support for putting the items
into subgroups or scales; consequently, the researcher can only compute a
total score of barriers, or look at individual barrier items. Thus, factor analysis
can be a useful statistical tool to condense these 100 items by putting them
into factors that group items that correlate with each other. In this way, the
researcher can use empirical methods to group items. For example, let’s say
the researcher would find through exploratory factor analysis that there are
three main types of career barriers, reflecting (1) career myths, (2) self-efficacy,
and (3) attributional styles. The researcher could then summarize a client’s
career barriers into these three types and provide scores for each of these factors.
For example, a client might score exceptionally high on career myths, which
would suggest that interventions are especially needed on this dimension.
Moreover, the three factors provide theoretical extensions of the topic of
career barriers or of the career planning process in general. In sum, factor
analysis analyzes a large number of items (or variables) and condenses or sum-
marizes them into common or underlying dimensions. Parenthetically, inter-
ested readers could follow up on this not-so-hypothetical example of career
transition barriers by reading Heppner, Multon, and Johnston (1994).
Factor analysis has been used frequently in the counseling literature in
scale construction (see Chapter 20). For example, this technique has been used
to describe dimensions of vocational interest (Fouad, Cudeck, & Hansen,
1984), supervisory styles (Worthington & Stern, 1985), expectancies about
counseling (Tinsley, Workman, & Kass, 1980), perceptions of counselors
(Barak & LaCrosse, 1975), religious commitment (Worthington, 2004), and
counselor self-efficacy in working with gay, lesbian, and bisexual clients
(Dillon & Worthington, 2003). Thus, for instance, in examining client expec-
tations about counseling, Tinsley et al. (1980) found four factors that account
for a large portion of the variance in clients’ expectations: personal commitment,
facilitative conditions, counselor expertise, and nurturance. Likewise, Phillips,
Friedlander, Pazienza, and Kost (1985) described three dimensions of deci-
sion-making styles: rational, intuitive, and dependent. These examples nicely
illustrate how factor analysis can identify dimensions to further understand-
ing of relevant constructs for the counseling profession.
Although factor analysis is frequently used, many readers of the counsel-
ing literature may not be aware of the many complexities and decision points
in conducting a factor analysis. In fact, factor analysis is a multivariate statis-
tical method that involves a number of decision points for the researcher,
which sometimes entails rather complex procedures with specialized vocabulary.
Many of these decision points are beyond the scope of this text; a wide variety
of more detailed resources are available for further consultation to aid deci-
sions about which variables will serve as the basic data (see, for example,
Dawis, 1987; Kline, 2005; Patten, 2001; Pett, Lackey, & Sullivan, 2003;
Thompson, 2004). We will, however, provide a brief overview of some of the
major decision points.
236 part 2 major research designs
Typically, the researcher will develop his or her own questionnaire, the
items of which constitute the variables. A great deal of time, planning, and
reflection are needed to develop quality items (see Chapter 20). For example,
first the researcher must decide what type of factor analysis to use in analyz-
ing the data. The choice of factor analysis type involves assumptions about the
treatment of item variance; in Chapter 20 we discuss the different options,
and follow Kline’s (2005) recommendation of utilizing common factor analysis.
Second, the researcher must decide how many factors exist in a set of data.
Although specific criteria can be used to suggest how many factors may exist
(e.g., a scree test), many times deciding on the number of factors involves
examination of several different factor solutions to identify the most mean-
ingful factor structure. It is important to understand that this is not a clear-
cut process, and two researchers looking at the same data might decide that a
different number of factors exist. A third decision point involves the method
of rotation. The solution from a factor analysis must be rotated to distribute
the variance across the factors; the researcher has a choice of orthogonal or
oblique rotations, which typically has important implications for the inter-
pretation of the underlying constructs. After the researcher decides upon a
factor solution, the researcher needs to name the factors. To do this, he or she
examines all of the items that make up a factor and attempts to choose a name
that captures the conceptual meaning inherent in the items. Obviously, this
naming process is subjective, and researchers examining the same set of items
may propose different names for the same factor. In short, you should also be
aware of the subjective nature of the factor naming process when examining
the report of a factor analysis.
Examples of Factor Analysis Tinsley, Roth, and Lease (1989) used both
confirmatory and exploratory factor analysis to examine the dimensions
that define group leadership. The main purpose of this study was to attempt
to confirm a four-factor model of group leadership originally described by
Lieberman, Yalom, and Miles (1973). In a classic study of group leadership,
Lieberman and colleagues found that leader, member, and observer ratings of
leadership converged on a four-factor structure of leadership. They labeled
these factors emotional stimulation, caring, meaning attribution, and execu-
tive functioning. The researchers wanted to determine whether group leaders’
self-ratings, obtained from a large number of these leaders, would confirm
this four-factor structure.
The study used both survey and factor analysis methods. These authors
randomly selected 500 members of the Association for Specialists in Group
Work and mailed them survey questionnaires. Usable responses were returned
by 200 of the 500 identified group leaders. The survey instrument contained
130 items, 5 items measuring each of the 26 leader characteristics studied by
Lieberman et al. (1973). Twenty-six scale scores (one score for each of the 26
leader characteristics) were calculated for each participant. These scale scores
were used as the data for the factor analysis. Tinsley et al. (1989) first used
confirmatory factor analysis to ascertain whether the 26 scales would result
chapter 10 quantitative descriptive designs 237
in the four factors described by Lieberman et al. They found that only 8 of the
26 scales loaded on (or scored highest on) the factor that they theoretically
should have loaded on. Tinsley et al. concluded that their analysis failed to
confirm the factor model proposed by Lieberman et al.
Because of this failure to confirm, Tinsley et al. (1989) then used exploratory
factor analysis to determine how many and what dimensions accounted for
self-rating of leadership. The authors examined 8-, 9-, and 10-factor solu-
tions before adopting an 8-factor solution that they believed best explained
the data. Next, Tinsley et al. examined the scales that correlated with each
factor and determined a name for the underlying construct that these items
represented. For example, the scales managing/limit setting, mirroring com-
mand stimulation, cognizing, charismatic leader, and model all loaded on
(were a part of) the first factor. Tinsley et al. named the underlying construct
“cognitive direction.” In a similar manner they examined the scales that
loaded on the other factors and derived names for the underlying constructs.
The constructs they named were affective direction, nurturant attractiveness,
group functioning, verbal stimulation, charismatic expert, individual func-
tioning, and nonverbal exercises.
Tinsley et al. (1989) concluded that the four factors identified by Lieberman
et al. (1973) could not adequately account for the self-rating of leadership
obtained in their sample. In fact, eight factors were used to account for the obtained
self-ratings. Tinsley et al. also concluded that group leadership is more com-
plicated and multifaceted than the Lieberman et al. model would suggest.
The Tinsley et al. (1989) study is an excellent example of the use of factor
analysis. It is exemplary because the researchers explicitly delineated for readers
the basis for their choice of the number of factors and for naming the factors.
In this way, readers are invited to make their own judgments about the critical
choices made by the researchers.
Exploratory factor analysis was also used to identify underlying dimen-
sions of a new inventory, an Ethnocultural Empathy Scale. Wang et al. (2003)
noted that as the United States is increasing in racial/ethnic diversity, there are
also “reoccurring conflicts among racial and ethnic groups, from the Rodney
King beating to hate crimes that resulted from the tragedies of September 11,
2001” (p. 221). They suggested that more than tolerance is needed, specifi-
cally “more understanding, awareness, and acceptance of individuals from
different racial and ethnic backgrounds” (p. 221). The main purpose of the
authors of this study (who were six graduate students in counseling psychol-
ogy at the time) was to develop a measure of cultural empathy. To generate
scale items they first conducted a comprehensive literature search on empathy
and ethnocultural theory; from this literature, they delineated three domains
of empathy: intellectual empathy, empathetic emotions, and communicative
empathy. After operationalizing these constructs, they then generated 71 items,
with each item needing to have a primary focus on one of the three domains
of empathy. The authors then had three judges match each item to one of the
three categories; this process led to the authors deleting nine items and revis-
ing six others. It is important to note that the authors spent a great deal of
238 part 2 major research designs
time not only generating items, but also obtaining some face validity for the
items from independent judges.
Data were obtained from 323 undergraduate students, who completed
not only the Scale of Ethnocultural Empathy (SEE), but also a demographic
form and a measure of social desirability. For the exploratory factor analysis,
they made the following decisions: (1) They began with a principle-components
analysis. (2) Based on a scree test and eigenvalues above 1.0, they selected a
four-factor solution with oblique rotation; the four-factor solution was chosen
because it yielded strong psychometric properties as well as being conceptu-
ally meaningful. (3) The four factors were named empathetic feeling and
expression, empathetic perspective taking, acceptance of cultural differences,
and empathetic awareness. The authors also found that the SEE total and four
factors were not strongly associated with social desirability, which provided
an important estimate of discriminate validity.
The authors conducted a second study to examine the stability of the SEE
factor structure, as well as to examine other validity estimates of the SEE. In
this study, Wang et al. (2003) used data from an additional 364 undergraduate
students and conducted a confirmatory factor analysis (CFA). The results of
the CFA provided empirical support for the initial four-factor model, but also
found support for a second-order hierarchical model, which suggested not only
evidence for the four distinct factors, but also a general ethnocultural empathy
factor as reflected in the total SEE score. Moreover, the results of this study also
provided estimates of concurrent validity with other empathy scales, which
is a very useful research strategy in the initial phases of scale construction.
Finally, a third study was conducted with another 51 undergraduate students
who completed the SEE on two administrations two weeks apart; the results
indicated that the SEE has acceptable test-retest reliability (rs = 0.64–0.86).
In sum, Wang et al. (2003) conducted an excellent study to develop a new
scale. It provides a very good example of using exploratory factor analysis to
first identify four underlying dimensions for a new scale on ethnocultural
empathy, and then to use confirmatory factor analysis to test the stability of
that four-factor model in another sample.
Cluster Analysis
Often in counseling we would like to be able to identify natural groupings or
subtypes of people, such as clients or counselors. Cluster analysis is frequently
used to put people into subgroups, which is quite functional for examining
individual differences in counseling research. For example, students who are
undecided about their choice of career might be undecided for very different
reasons. Some students might lack the necessary career information whereas
others may experience a great deal of anxiety in making career decisions.
Indeed, when researchers have utilized cluster analysis to address this issue,
they have found that there are multiple subtypes of career indecision at both
the high school (Larson & Majors, 1998; Multon, Heppner, & Lapan, 1995)
and college levels (Chartrand et al., 1994; Kelly & Pulver, 2003; Larson,
chapter 10 quantitative descriptive designs 239
Heppner, Ham, & Dugan, 1988; Lucas & Epperson, 1990). Brown and Krane
(2000) analyzed these cluster analytic studies and concluded that at least three
types of individuals have been identified consistently on career indecision
subtypes:
(1) those who seem just to need additional occupation information and help with
occupational exploration, (2) those whose career-choice problems are primarily
anxiety-related, and (3) those who are more severely impaired clients who present
with a constellation of problems revolving around anxiety, poor vocational identity
development, high perceived needs for occupational information, and low feelings of
efficacy around their abilities to solve problems and make career decisions. (p. 757)
A recent cluster analytic study utilizing 278 actual career clients found
support for Brown and Krane’s (2000) observation of these three clusters
(Multon, Wood, Heppner, & Gysbers, in press). In addition, the researchers also
found a cluster of psychologically healthy adults who did not seem to be expe-
riencing many career problems, but were perhaps the type of proactive individ-
uals who availed themselves of resources to affirm choices they had already
made. In terms of practice, the results of these cluster analytic studies suggest
that it may be very useful to design different career planning interventions to
address the specific needs of these distinctly different subgroups of clients.
In essence, cluster analysis is a multivariate statistical method that reduces
data by identifying and then classifying similar entities into subgroups (Hair
et al., 1987; Hair & Black, 2000). Borgen and Barnett (1987) noted three pri-
mary purposes for cluster analysis: exploration (to find a certain structure or
set of groupings), confirmation (to test an existing classification, perhaps
based on theory), and simplification (to reduce a complex data set into a sim-
pler structure). Cluster analysis can be used to categorize objects (e.g., coun-
selor statements), people (e.g., counseling center clients), or variables (e.g.,
items on a test) (Borgen & Barnett). In terms of identifying types of people,
Megargee and Bohn (1979) used cluster analysis to develop a typology of
criminal offenders based on Minnesota Multiphasic Personality Inventory
(MMPI) profiles, whereas Rojewski (1994) used the Career Decision Scale
(Osipow, Carney, Winer, Yanico, & Koschier, 1976) to identify three types of
career-indecisive adolescents. Hill and O’Grady (1985) used cluster analysis
of variables to develop a classification of counselor intentions. Their 19 ther-
apist intentions were collapsed into four categories: assessment, therapeutic
work, nonspecific factors, and problems. Using a similar analysis, Hill, Helms,
Spiegel, and Tichenor (1988) found five clusters of client reactions. As an
example of clustering other objects, Wampold and White (1985) used cluster
analysis to analyze research themes in counseling psychology. Examining
common citations among 27 articles published during 1982, they concluded
that the social influence model was a common underlying theme in the 27
articles. In essence, cluster analysis classifies objects, variables, or persons so
that each object is very similar to others in its cluster. Thus, within each clus-
ter, the objects are homogeneous; however, there is considerable heterogeneity
among clusters (Hair et al., 1987; Hair & Black, 2000).
240 part 2 major research designs
variable (x) increases, so do the scores on the second variable (y); then x and
y vary together, or covary, and have a “strong positive relationship.” If x
scores do not vary with y scores, we typically say there is not a relationship
between x and y. The correlation coefficient between two scores can range
from +1.00 (a very strong positive relationship) to –1.00 (a very strong nega-
tive relationship). The amount of variance that is shared between two vari-
ables is the square of the correlation. Thus, the correlation between x and y
might be +.5, which means that the amount of variance shared between these
two variables is 25% (.52). Sometimes in the past (see Cook & Campbell,
1979) these designs were also referred to as passive because the researcher nei-
ther actively forms groups or conditions through random or nonrandom as-
signment, nor actively manipulates an independent variable.
For example, Hoffman and Weiss (1987) were interested in the relationship
between individual psychological separation from parents and healthy adjust-
ment. They developed an inventory of common problems to reflect healthy
adjustment, and correlated this measure with scores from another inventory
that measured four aspects of psychological separation. Two hundred sixty-
seven white college students completed the two self-report inventories. Hoffman
and Weiss found that students with greater conflictual dependence with their
mothers or fathers reported more problems (r = .43 and .42, respectively).
Because the results are correlational, however, the direction of the relationship
between conflictual dependence and reported problems cannot be determined;
that is, conflictual dependence may cause more emotional problems, or having
more problems may lead to greater conflictual dependence on parents. Or it is
also possible that a third variable may cause both conflictual dependence and
emotional problems. In short, the correlational design Hoffman and Weiss used
allowed them to describe the degree of relationship between two variables—
individual psychological separation and psychological adjustment.
Studies employing a correlational design have been used to describe rela-
tionships among a wide variety of variables of interest to counseling re-
searchers. A study by Nocita and Stiles (1986) is a particularly noteworthy
example of the use of a simple correlational design. These authors wanted to
assess the relationship between a client’s level of introversion and his or her
perception of counseling sessions. The client’s level of introversion was assessed
by the Social Introversion scale of the MMPI. At the conclusion of each coun-
seling session, clients also filled out a Session Evaluation Questionnaire (Stiles,
1980), which assessed the client’s perception of session depth and smoothness
and his or her feelings of positivity and arousal. Nocita and Stiles correlated
the client’s social introversion score with each of his or her scores on the four
Session Evaluation Questionnaire scales. Introverted clients saw the counsel-
ing session as less smooth and felt less positive after the session than the more
extraverted clients. The noteworthy and unusual aspect of this study is that
the correlational results were replicated with two different client samples. As
with all correlational designs, however, this type of analysis does not allow for
causal explanation. This type of replication provides useful information and
suggests that the results may generalize to other client samples.
246 part 2 major research designs
Multiple Regression
Whereas a correlation identifies the relationship between two variables, most
often researchers are interested in describing the relationships among more
than two variables. For example, one might ask: If we know the correlation
between x and y, would it not be more powerful to include variables a, b, and
c (along with x) to study y? Indeed, in many cases it is, and thus multiple
regression has become increasingly popular in the counseling literature.
We will briefly focus on multiple regression here as a way of increasing our
chapter 10 quantitative descriptive designs 247
ability to describe the relationships among multiple variables. (For more de-
tails, see Cohen & Cohen, 1983; Hair et al., 1987; Hair & Black, 2000; Ped-
hazur, 1982; and Wampold & Freund, 1987.)
Multiple regression is a statistical method for studying the separate and
collective contributions of one or more predictor variables to the variation of
a dependent variable (Wampold & Freund, 1987). In essence, multiple regres-
sion can be used to describe how multiple predictor variables are related to a
single “dependent” (criterion) variable. Thus, researchers frequently refer to
predicting the criterion variable and discuss the extent to which they can accu-
rately predict the criterion. The relationship between a “dependent” variable and
a set of multiple “independent” variables is expressed as the multiple correlation
coefficient R, which is a measure of how well the predictor scores correspond to
the actual scores of dependent variables. The square of the multiple correlation
coefficient (R2) is the proportion of variance in the dependent variable explained
by the independent variables. The word explained here does not necessarily
imply a causal relationship, but rather an association of the dependent variable
with variability in the predictor variables (Wampold & Freund).
There are three basic methods for entering predictor variables in regres-
sion equations: simultaneous, stepwise, and hierarchical regression. Because
each method serves slightly different purposes and outcomes, it is important
for the researcher to be familiar with the strengths and weaknesses of each method
(see Wampold & Freund, 1987, for an overview). In simultaneous regression,
all of the predictor variables are entered concurrently (simultaneously) into
the regression equation. Simultaneous regression is most often used when
there is no basis for entering any particular predictor variable before any other
predictor variable, and the researcher wants to determine the amount of variance
each predictor variable uniquely contributes to the prediction of the criterion vari-
able (after the common varience among the predictor variables has been removed).
For instance, Parham and Helms (1985b) examined the relationship between
African-American racial identity attitudes and self-esteem. Four racial identity
attitudes were simultaneously entered in a regression equation to predict self-
esteem; the authors found a multiple R of .36 for the analysis. Thus, racial
identity accounted for about 13% of the variance (.362) in the African-
American participant’s self-esteem. Examination of the specific racial identity
attitudes revealed that African-American students in the pre-encounter stage of
racial identity formation had lower self-esteem than students in the immersion
stage of identity formation.
Asner-Self and Marotta (2005) also used simultaneous regressions to ex-
amine the predictors of psychological distress in 68 Central American immi-
grants who had been exposed to war-related trauma. The authors noted that
one of the fastest growing immigrant groups in the United States was people
from wartorn Central America. (Marotto & Garcia, 2003). The authors were
particularly interested in examining whether developmental indices related to
mistrust, identity confusion, and isolation would predict symptoms such as
depression, anxiety, and posttraumatic stress (PTS). Although a randomized
sampling procedure is preferable, it is difficult to recruit people from this and
248 part 2 major research designs
The residuals (the variance in the posttreatment scores that could not be
accounted for by the pretreatment scores) from this regression were used as the
dependent variable in the stepwise regression. In this manner, the researchers
were able to examine how much the Holland scores could accurately predict
change in vocational identity from a pretest period to a posttest period.
In hierarchical regression, the researcher specifies the order of entry of the
predictor variables based on some rationale (for example, research relevance,
causal priority, or theoretical grounds). Lent, Brown, and Lankin (1987) were
interested in examining the relationships among self-efficacy, interest congru-
ence, consequential thinking, and various career and academic behaviors. One
behavior of interest was academic grade point average. In the first step of the
regression analysis, a measure of composite ability (high school rank, Preliminary
Scholastic Aptitude Test) was entered first to control for the effects of previ-
ous academic performance. Next, the three theoretically derived variables
(self-efficacy, interest congruence, and consequential thinking) were entered in
a sequential manner. In predicting career indecision, the multiple R for com-
posite ability was .34. When interest congruence was added, the multiple R
increased to .44 (F = 9.62, p < .01). This result indicated that interest con-
gruence added additional predictive variance beyond that accounted for by
composite ability. By using a hierarchical model in which composite ability
was entered first, Lent and colleagues were able to perform a more stringent
test of the relationship between interest congruence and career indecision.
In evaluating the usefulness of multiple regression, it is always important
to remember that the results of these types of analyses are based on correla-
tional data. Multiple regression designs use terminology from experimental
designs (dependent variable and independent variables); we prefer the terms
criterion and predictor. Moreover, even though the variables may be referred
to as independent and dependent variables, it is important to note that the
results obtained are relational, not causal. Likewise, it is important to note
that the choice of a criterion variable and of predictor variables is always arbi-
trary. In other words, prediction is not causality, and thus causal statements
are not appropriate with these designs. Multiple regression is suited to describing
and predicting the relationship between two or more variables, and is espe-
cially useful in examining the incremental as well as total explanatory power
of many variables (Hair et al., 1987). Perhaps the main caveats for researchers
pertain to inadequate sample sizes (see Wampold & Freund, 1987) and spurious
results due to methodological procedures.
Path Path
a b
Predictor Criterion
variable Path c variable
the mediator is added to the model, the mediating variable is called a partial
mediator. To test for mediation, Baron and Kenny recommend that a series of
regression models be conducted to test the four conditions listed. (For more
statistical and design details, see Frazier et al., 2004.)
We will examine several research examples to illustrate mediating vari-
ables in the counseling literature. For example, there is growing empirical
evidence of an association between prejudice and discrimination and greater
levels of psychological distress with several previously marginalized groups
such as African Americans, white women, and lesbian, gay, and bisexual indi-
viduals (e.g., Corning, 2002; Moradi & Subich, 2003; Waldo, 1999). However,
Moradi and Hasan (2004) noted that there has been a dearth of evidence
examining the prejudice stress–distress link with Arab Americans, a group
that has been experiencing increasing discrimination. Based on theory and
some previous findings, they hypothesized that personal control mediates the
relationship between perceived discrimination and both self-esteem and psycho-
logical distress in Arab Americans. Their results indicated a relationship between
perceived discrimination and psychological distress (r = .32). Moreover, they
found that a sense of personal control partially mediated perceived discrimi-
nation and psychological distress, and fully mediated the relationship with
self-esteem. Thus, the loss of control seems to play an important role in under-
lying the perceived discrimination events leading to decreased self-esteem and
increased psychological distress for Arab Americans.
In another example, Wei, Vogel, Ku, and Zakalik (2005) were interested in
whether different types of affect regulation might mediate the relationships be-
tween attachment styles and psychological distress. More specifically, based on
theory and previous research, they hypothesized that: (1) the association between
attachment anxiety and negative mood or interpersonal problems would be medi-
ated by emotional reactivity (i.e., overreacting to negative feelings) and (2) the
association between attachment avoidance and negative mood or interpersonal
problems would be mediated by emotional cut-off (i.e., suppressing their negative
feelings). Their results supported both hypotheses, and also supported growing
evidence suggesting that the link between attachment and distress is not simply a
direct relationship, but one involving mediating psychological processes (e.g., Wei,
Heppner, & Mallinckrodt, 2003). The results of this study suggest that the anxi-
ety and avoidant attachment styles prefer to use different affect regulation strate-
gies. This study suggests that practitioners can help individuals with anxiety and
avoidant attachment styles by recognizing the temporary and long-term positive
benefits of using their specific maladaptive affect regulation strategies.
Parenthetically, Moradi and Hasan (2004) used path analysis whereas
Wei et al. (2005) used a statistical analysis called structural equation model-
ing, which allowed them to simultaneously examine the direct and indirect
effects required for mediational models. Note that in both studies, the authors
provide figures depicting the relationships among the hypothesized variables.
(See both articles for more details, as well as Heppner & Heppner, 2004, for
more detailed explanations of how to write a results section when using struc-
tural equation modeling.)
254 part 2 major research designs
of multiple predictor variables on a criterion that researchers consider examining multiple rela-
variable. Moreover, regression analyses can be tionships, especially ones that are predicted to
used to examine more complex relations con- show results in opposite directions. Our third
cerning moderation and mediation among vari- recommendation is that researchers pay particular
ables. Increasingly, counseling researchers are attention to sample characteristics. For instance,
using moderation and mediation analyses to Gade et al. (1988) purposefully selected a varied
examine complex theoretical models. The same sample in an attempt to eliminate any chance
cautions that apply to simple passive designs results based on sample characteristics. Researchers
also apply to the more complex designs. may want to select samples in which various
We would like to offer three suggestions for demographic or psychological characteristics are
counseling researchers considering a descriptive held constant. For example, researchers wanting
design. First, descriptive research should be un- to compare “good” versus “poor” counseling
dertaken from a strong theoretical grounding. sessions in terms of type of therapist response
Researchers would do well to avoid questions used may want to select only sessions that are
such as, How do these sets of variables relate? temporally close. This type of selection would
How do these groups differ? Rather, theory lessen the chance that the results were influenced
should be used to inform the research in the by the stage of therapy.
determination of the variables examined. Second, The designs described in this chapter can be
descriptive designs are strengthened when they important building blocks in the scientific
contain differential predictions. For example, process. Based on a careful analysis of the cur-
theory may indicate that a variable (realistic rent state of knowledge in a given area, the re-
Holland code) might be positively related to one searcher can choose a design that will lead to a
variable (lack of accidents), not related to progressively better understanding of the content
another (weight), and negatively related to a third area. When chosen wisely, the descriptive de-
(school satisfaction). A study is strengthened signs can serve the important function of de-
when these patterns of relationships are pre- scribing phenomena of interest in counseling
dicted and assessed. Specifically, we recommend research.
STIMULUS QUESTIONS
Questions to Test Your Knowledge about
Quantitative Descriptive Designs
The purpose of this exercise is to test your under- 4. In which situations do you use an exploratory
standing of some key issues regarding descriptive factor analysis and in which do you use a
designs. Write down your answers to the following confirmatory factor analysis?
questions, and then discuss your responses with 5. List three advantages of survey research.
a peer in your class. 6. The strength of descriptive research is that it
allows us to describe phenomena. When might
1. What is the difference between a mediating
it be possible to use theory to guide the descrip-
variable and a moderating variable?
tive study, and when is it more helpful not to
2. There are three main types of regression;
use theory?
what are they and how do they differ?
3. Cluster analyses and factor analyses have some
similarities and some differences; please describe.
11 CHAPTER Qualitative Research
Yu-Wei Wang
256
chapter 11 qualitative research 257
One day, and this just brings tears to my eyes to say, one day I remember I was
a young teenager, about thirteen. My dad was working all the time, doing drugs,
was an alcoholic, and my mom went through the same abuse I went through [with
my husband], with my dad. She had a loaded .357 Magnum lying on the table and
I walked in and she says, “See this gun?” I was scared to death of what she was
going to do. She says, “I’m going to pick it up as soon as you walk out the door,
and I’m going to shoot myself, unless you can give me ten reasons not to.” And
me being only thirteen, I had to give her ten reasons why. And that was a hard
thing for me to do. (Lawless, 2001, p. 97)
PHILOSOPHICAL FOUNDATIONS
In Chapter 1, we contrasted four paradigms that bear on the research process.
Readers are encouraged to review that chapter again to be familiar with the
philosophical foundations of scientific inquiry.
Overall, quantitative research is aligned with two of the paradigms—
constructivism and critical theory. In general, all qualitative paradigms assume
relativist ontology (there are multiple realities that are socially and individually
constructed) and transactional epistemology (the knower and the known
are inextricably intertwined), as well as dialogic/interpretive methodology
(Guba & Lincoln, 1998). It is crucial to understand these underlying para-
digms so that they match the researcher’s own personal values, beliefs, and
personality, as well as personal and mental models (Morrow et al., 2001).
Also, the research approach adopted should be appropriate for answering
research questions within the context of existing knowledge. As introduced
chapter 11 qualitative research 261
later in this chapter, there are many different strategies of inquiry in qualita-
tive research; each inquiry has somewhat different philosophical underpin-
nings (see Ponterotto, 2005b, for an excellent discussion about locating the
strategies of inquiry within research paradigms). Therefore, understanding the
basic tenets and philosophical foundations of qualitative research will help
students select a particular paradigm and strategy of inquiry that may best
address a particular area of inquiry.
Readers who are interested in advancing their knowledge about the philo-
sophical underpinnings of these paradigms can find excellent introductions
and discussions in the Handbook of Qualitative Research (e.g., Lincoln &
Guba, 2000; Schwandt, 2000), the Handbook of Counseling Psychology (Morrow
& Smith, 2000), the Handbook of Multicultural Counseling (Morrow et al.,
2001), and the special issue on qualitative research in the Journal of Counseling
Psychology (e.g., Ponterotto, 2005b). Next, we will shift our focus to qualita-
tive research methods and design issues.
STRATEGIES OF INQUIRY
Each qualitative strategy of inquiry is connected to specific paradigms and
research designs. Researchers should be clear about the differences among
various strategies of inquiry in order to make informed decisions regarding
what qualitative approaches to use and when to use them (Creswell, 1998),
and subsequently to design their studies according to the guidelines of a par-
ticular chosen strategy. The rationale of using a specific paradigm and strategy
of inquiry should be presented and the relationship between the paradigms/strate-
gies of inquiry and purposes/research questions of the studies should be expli-
cated in the final write-up.
It is worth mentioning that great confusion may arise at this stage because
a variety of strategies of inquiry exist across various disciplines. For example,
Strauss and Corbin (1990), who are in the field of sociology and nursing,
grouped qualitative strategies of inquiry into five categories (grounded theory,
phenomenology, life history, ethnography, and conversational analysis);
conversely, Moustakas (1994), who is in the field of psychology, listed six quali-
tative traditions (grounded theory, hermeneutics, empirical phenomenological
research, ethnography, heuristic research, and transcendental phenomenology).
Narrative studies provides another example of this diversity: According to
Hoshmand (2005), “narratology is a term historically used to refer to the study
of narratives in the literary field, though other disciplines in the humanities
(such as history) and the social sciences (such as cultural studies) also are
associated with the study of narratives”(p. 178). Hoshmand further clarified
that she used the term narratology “as a way of distinguishing a mode of qual-
itative inquiry and data analysis that is informed by narrative theory,” differing
from “other qualitative research that involves narrative data but not a narra-
tive perspective per se” (p. 178).
262 part 2 major research designs
as communication as culture
content symbolic
discourse ethnoscience
analysis structural interactionism,
analysis
ethnography ethnomethodology
ethnology
of communication
discerning of themes
(commonalities and uniquenesses) interpretation
phenomenology
case study hermeneutics
reflection life history
educational reflective
heuristic research
connoisseurship phenomenology
Grounded Theory
This section first provides the definition and purpose of the grounded theory
approach. Then, its primary features—(1) constant comparative method, (2) memo
writing, (3) theoretical sampling, and (4) the emerging theory that is grounded
in data—will be discussed and elucidated with exemplar studies.
The grounded theory approach was developed by Glaser and Strauss
(1967) in an attempt to challenge the dominant trend of “excessive reliance
on the quantitative testing of hypotheses derived from a small number of
grant (totalizing) theories, typically through numerical survey and other
statistical approaches” (Henwood & Pidgeon, 2003, p. 132). According to
Charmaz (2000),
Grounded theory methods consist of systematic inductive guidelines for collecting
and analyzing data to build middle-range theoretical frameworks that explain the
collected data. Throughout the research process, grounded theorists develop ana-
lytic interpretations of their data to focus further data collection, which they use
in turn to inform and refine their developing theoretical analyses. (p. 509)
Data preparation
Data collection
Data storage
Initial analysis
Coding
Key concepts
Definitions Outcomes
Memos
Relationships and models
Pidgeon and Henwood (1996) also warned that researchers should “write a
memo as soon as the thought has occurred, for, if left unrecorded, it is likely
to be forgotten” (p. 95). In sum, memo writing is a vital technique for
grounded theorists. It not only serves as an instrumental mechanism for the
constant comparative practice, but also facilitates the theoretical sampling and
theory development processes that will be described in the following sections.
tactic that will be further explained later in this chapter) provide the readers an
understanding of the context of the phenomenon. The authors also did a nice
job of describing the participant recruitment process: Letters that detailed the re-
search’s purpose and process were sent to therapists with expertise in working
with sexual abuse survivors; these therapists were asked to give the research
announcement to those clients “who might benefit from or be interested in
participating in the study” (p. 25).
Morrow and Smith (1995) clearly identified their sampling criterion: par-
ticipants’ “self-identification as an abuse survivor” based on the constructivist
approach (i.e., “[accepting] the stories of participants at face value as their
phenomenological realities”) (p. 25). The authors utilized a variety of data
sources, including: (1) semi-structured, 60- to 90-minute, in-depth interviews
with individual participants; (2) a 10-week focus group with 7 of the 11 research
participants that emphasized the survivors’ coping experiences and thoughts on
the emerging categories from the initial data analyses; (3) documentary evidence
such as participants’ journals and artistic productions; and (4) Morrow’s self-
reflective and analytic memos, which consisted of the researcher’s reflections
and assumptions about the data. The interviews were transcribed verbatim,
and “the data corpus consisted of over 2,000 pages of transcriptions, field
notes, and documents shared by participants” (p. 25).
Following Strauss and Corbin’s (1990) suggestions, the authors con-
ducted open coding, axial coding, and selective coding on the data, and “the
language of the participants guided the development of code and category
labels, which were identified with short descriptors, known as in vivo codes,
for survival and coping strategies” (Morrow & Smith, 1995, p. 26). Subsequently,
the authors generated a theoretical framework for the surviving and coping
process experienced by those childhood sexual abuse survivors. Figure 11.3
shows the theoretical model derived from the data analysis.
The authors provided a rich description of the model, which serves as an
overall illustration of the phenomenon under study. They also elucidated the
categories in the model with quotes from the participants. For example, the
following is the description of a “Phenomena” category:
[P]articipants experienced what was termed helplessness, powerlessness, and lack
of control. Lauren provided an exemplar of the second category, illustrating the
pervasiveness of her perpetrator’s power:
He stands there. A silhouette at first and then his face and body come into
view. He is small, but the backlighting intensifies his figure and he seems
huge, like a prison guard. He is not always there but it feels like he might as
well be. When he’s not there, I search the distance for him and he appears.
He seems to be standing there for hours. As if he’s saying, you are weak, I
am in control.
Not only did Lauren experience powerlessness during her abuse, but her lack of
control invaded her dreams and her moments alone. (pp. 27–28)
Context
• Sensation
• Frequency
• Intensity
• Duration
• Perpetrator
characteristics
Strategies Consequences
Phenomena
• Threatening or • Keeping from • Paradoxes
Causal • Surviving
dangerous being overwhelmed
Conditions • Coping
feelings by threatening &
• Cultural norms dangerous feelings • Living
Forms of sexual Helplessness, • Healing
abuse powerlessness, • Managing • Wholeness
lack of control helplessness, • Empowerment
powerlessness, • Hope
& lack of control
Intervening
Conditions
• Cultural values
• Family dynamics
• Other abuses
• Resources
• Victim age
• Rewards
method and theoretical sampling. They described how the researcher’s memos
facilitated this process:
Meghan foreshadowed one of these phenomena the first night of the group, when
she said, “To keep from feeling my feelings, I have become a very skilled helper of
other people.” Throughout the data, others echoed her words. The analytic
moment in which this category emerged is illustrated in the following analytic
memo written by Morrow (in vivo codes are in italics):
I’m reading a higher level of abstraction. Is the overarching category protec-
tion from feelings? Many categories are subsumed under it: One talks to get
out the stories; the feelings are less intense. Fake orgasm (sex) because you
don’t have any physical feelings. Art was used to deal with feelings, express
anger, release the pressure of the feelings, use chemicals to deal with feelings
(and a whole complex interaction here) . . .
Existing and emergent codes and categories were compared and contrasted with
this category; the category was modified to accommodate the data, producing the
chapter 11 qualitative research 269
Morrow and Smith (1995) also explicitly explained the rigor of their
method: triangulating multiple data sources (i.e., interviews, focus groups,
documents, memos), immersing in the data (the entire data collection period
lasted over 16 months), member check (participants were invited to verify
data analysis results and revise the emergent theoretical model), and peer
debriefing (Morrow invited other qualitative researchers to review the analysis,
her writing, and the “audit trail that outlined the research process and evolu-
tion of codes, categories and theory” [p. 26]).
Other exemplar grounded theory studies include Rennie and his col-
leagues’ research on psychotherapy (Rennie, 1994a, 1994b, 1994c; Watson &
Rennie, 1994) and Fassinger and her colleagues’ studies on career develop-
ment of highly achieving Black and White American women (Richie et al.,
1997) and highly achieving women with disabilities (Noonan et al., 2004).
Interested readers may also find the researchers’ own thoughts about their
investigation process in their articles on the grounded theory methods (e.g.,
Fassinger, 2005; Rennie, 1996).
Phenomenology
The purpose of phenomenology is “to produce an exhaustive description of the
phenomena of everyday experience, thus arriving at an understanding of the essential
structures of the ‘thing itself’, the phenomenon” (McLeod, 2001, p. 38). In the
following paragraphs, the historical development of the phenomenological approach
and its major principles will be introduced along with exemplar studies.
Phenomenology has its roots in the work of Edmund Husserl on phenom-
enological philosophy. Since then, many individuals (e.g., Giorgi, Moustakas,
Polkinghorne) in the social science field have followed its tenets and trans-
ferred Husserl’s work from philosophy to psychological or sociological
research. Over the past decades, different approaches to phenomenology were
developed (e.g., empirical/psychological phenomenology, hermeneutic phe-
nomenology, existential phenomenology, social phenomenology, reflective/
transcendental phenomenology, and dialogical phenomenology). (For a review,
see Creswell, 1998.) McLeod (2001) provided an overview of the three
branches of “new” phenomenological traditions that evolved from Husserlian
phenomenological principles: (1) “the Duquesne school of empirical phenom-
enology” developed by Giorgi and his colleagues in North America; (2) “the
method of ‘conceptual encounter’ developed by Rivera” who was influenced
by the Duquesne school and “the German social psychologist Lewin”; and
(3) “the existential-phenomenological investigations” of “the Scottish psychia-
trist Laing and his colleagues” at the Tavistock Institute for Human Relations in
England (pp. 40–48). Interested readers are referred to McLeod for a compari-
son between the original phenomenological philosophy developed by Husserl and
the “new” phenomenological traditions advocated by psychological researchers.
270 part 2 major research designs
Research Questions and Lived Experiences The key subject matter studied
by phenomenological researchers is “the lived world” of human beings (i.e.,
“the life-world manifests itself as a structural whole that is socially shared and
yet apprehended by individuals through their own perspectives”) (Wertz,
2005, p. 169). A phenomenological study aims to “fill in the gap” between
“knowledge and reality that requires qualitative knowledge, that is, an under-
standing of what occurs” (Wertz, 2005, p. 170). The research questions for a
phenomenological study are developed in order to understand the everyday
lived experiences of individuals and to explore what those experiences mean
to them (Creswell, 1998). Data are typically generated through written or
verbal responses of the research participants to “a descriptive task with in-
structions” or an open-ended interview (Wertz, 2005, p. 171). The research
questions guide the development of the descriptive task or specific interview
questions. Long interviews are often utilized to generate in-depth dialogues
between the researcher and the participant about a specific phenomenon.
Wertz suggested that the best type of data for phenomenological researchers
is “concretely described psychology life,” which helps to further our under-
standing about a given phenomenon beyond “any previous knowledge or
preconceptions” (p. 171).
chapter 11 qualitative research 271
careers at least once in the past. The authors indicated that past research has
not focused on the process of self-initiated career change, and most studies
used quantitative methodology. The phenomenological approach was used for
this study because it allowed the researchers to understand the meanings that
participants ascribed to their autonomous career change in a specific context.
Teixeira and Gomes (2000) explicitly described the criterion they used to
select participants: All of the participants would have experienced at least one
self-initiated career change. Specifically, the participants were chosen “who
had left one career for which they had received a university training for
another that also required advanced training, such that the change had been
voluntary (criteria for time spent in the careers were not established)” (p. 81).
A “thick” (detailed) description of the participants (e.g., their first and current
careers) was provided in the article.
The authors utilized interviews as the data collection method for this
study. They started the interviews with a broad question, “I should like you
to tell me about your career choices,” and left it open-ended for the partici-
pants to freely discuss their career trajectories. Interviews were conducted
with seven participants (four women and three men; age range from 32 to 42
years), audio taped, and subsequently transcribed. Locations (i.e., partici-
pants’ home or work settings) and the actual lengths (i.e., one to two and a
half hours) of the interviews were described in their manuscript.
Teixeira and Gomes (2000) followed the Duquesne school of empirical
phenomenology tradition and explicitly delineated the phenomenological analytic
approach utilized in their study:
The analytical procedures observed the methodological recommendations of
Giorgi (1997) and Lanigan (1988) in the following way: transcriptions and suc-
cessive readings of the material, search and demarcation of significant units
within each interview, verification of cross units among interviews, and definition
of the main themes. The relations between parts and whole were worked out by
the preparation of a synthesis for each narrative. The confront [sic] between
these syntheses and their constitutive themes were the base for the phenomeno-
logical description. Throughout this process, the researchers’ prior knowledge
of the subject was set to one side, thereby avoiding interferences from anti-
cipated qualitative judgments (phenomenological epoché). The analysis rig-
orously observed the three steps of phenomenological method (description,
reduction, interpretation) as indicated below. Excerpts taken from the inter-
views were used as illustrative evidence of the researcher’ phenomenological
understanding. (p. 82)
The authors concluded that their analysis yielded five essential themes
that “comprise a general structure of the experiences described by the inter-
viewees, although the limits, contents and scope of each theme may vary from
case to case according to personal circumstances” (p. 82). Following the
methodological recommendations of prominent phenomenological researchers,
Teixeira and Gomes (2000) defined each theme (which they called a “moment”)
and subsequently identified the interviewees’ accounts that “specify the affective,
cognitive and conative limits of the experience that participants lived through
chapter 11 qualitative research 273
in relation to each of the selected themes” (p. 82). The authors then provided
a rich description of the themes with the participants’ narratives. Following is
the description of one of the five themes (“moments”).
Moment 5: Evaluating the change and present situation
The participants evaluated their professional careers, including their present situ-
ations, in a positive way. The evaluation was accompanied by a feeling of having
matured and of personal transformation. Career change was generally associated
with personal changes, for example the feeling of being less tied down and capa-
ble of making changes in other spheres of life (S2: “. . . after I took this decision
I feel much more given to change, just as I had changed”; S4: “. . . I think that all
that I had gone through made me succeed in not being so tied up with things . . .
I am much more . . . much lighter”) or yet as a transformation in the personal out-
look, ceasing to like working to a “prepared formula” (S5).
Career change was also seen as a personal search for interpersonal develop-
ment, self-determination and, most importantly, increased self-esteem (S7; “. . . it
is something that makes me feel good, I like to hear people say that I am a doctor,
I feel proud of what I am”). Yet others viewed their professional trajectory as an
opportunity for steady development of their personality and of their capacities
and interests, integrating all the professional experience they had (S3, S1).
S1: “So I made it . . . made use of knowledge, right, built up through a piece
of my life in all the areas that I worked in, and I’m still reaping the benefit today. . . .”
The contrast with these positive evaluations on career changes, in which
those interviewed considered themselves more mature and in the process of re-
engagement in a profession, is given by S6. She still claims to have adolescent
doubts about future career, just as she is on the point of finishing a course in
Visual Arts (her professional interest abandoned in adolescence).
S6: “I think that basically I didn’t change . . . I think I still . . . I’m 36 now,
and I still feel I have those adolescent doubts. . . . ‘What am I going to be in
the future?’ . . . I have the feeling that I’m still looking for something . . .
[Can you see how you will be, I don’t know, ten, fifteen years from now?]
No . . . no, I can’t, I can’t, for me it’s all far too hazy.” (p. 87)
After summarizing and illustrating each theme that emerged from the
data analyses, Teixeira and Gomes (2000) presented the essential structure (phe-
nomenological reduction) of the phenomenon under study. They used the reduc-
tion to illustrate the characteristics of the phenomenon of career change. The
complete description is approximately three pages long; below is its summary:
In summary, career change can be defined as a broad process of change in the rela-
tionship between the participants and their work. The change begins with the
awareness of feelings of dissatisfaction with their career, leading the participants to
ask questions of themselves and what they do. The questioning of career emerges
in the consciousness as a figure whose background is the other roles lived by the
participants in their daily lives. Personal plans are redefined and new priorities
established, enabling the career to acquire a different meaning in this new context.
Thus the search for a new career is also a search for a new way for the participants
to express themselves in the world, which can lead them—although not necessar-
ily—to feelings of greater self-determination and personal satisfaction. (p. 90)
274 part 2 major research designs
The rich and exhaustive descriptions about the seven Brazilians’ experi-
ences with voluntary career change in Teixeira and Gomes (2000) extended
previous career theories and provided new knowledge about a phenomenon
that was understudied. The authors’ detailed illustrations of the method,
analysis results, and their reasoning help the reader understand how their
interpretations were derived from the data. More exemplars of phenomeno-
logical studies can be found in Wertz (2005), McLeod (2001), and the Journal
of Phenomenological Psychology.
Researchers, Researcher Biases, and Training of the Research Team Hill and
colleagues (1997) stress the importance of clearly describing the researchers
because the data analyses of the CQR rely on the consensual process of the
research team members. Although the composition and the dynamics of the
team are essential, there is no absolute correct type of team composition
(e.g., faculty members, senior practitioners, graduate or undergraduate stu-
dents). Researchers need to clearly consider the various possibilities of the
team composition and be very careful in selecting the team members, consid-
ering possible power differences among the group members, types of expertise,
and the level of commitment needed from each team member. Before the data
276 part 2 major research designs
Data Analytic Procedure Hill and colleagues (1997) delineated the step-by-
step data analytic procedure and divided it into the following sequential
stages: (1) identify domains, (2) summarize core ideas, (3) construct categories
from the cross-analysis, (4) audit the analysis results, (5) check the stability of
the results, and (6) chart the findings. Specifically, these procedures were
summarized in the update article about the CQR method as follows:
Domains (i.e., topics used to group or cluster data) are used to segment interview
data. Core ideas (i.e., summaries of the data that capture the essence of what was
said in fewer words and with greater clarity) are used to abstract the interview
chapter 11 qualitative research 277
Consensus process 1. Researchers should openly discuss their feelings and disagreements.
2. When there are disagreements among the researchers about the
interviews, everyone should listen to the tape of the interview.
Biases 1. Report demographics and feelings/reactions to topic in the Methods
section.
2. Discuss the influence of biases in the Limitations section.
3. Openly discuss biases among the research team throughout the
process.
4. Journal reviewers need to be aware that biases are a natural part of any
research, including CQR.
The research team 1. Either set or rotating primary teams are acceptable.
2. All team members must become deeply immersed in all of the data.
3. At least 3 people should comprise the primary team.
4. The educational level of team members should match the abstractness of
the topic.
5. Team members with more designated power should not claim “expert
status.”
6. Power issues should be addressed openly.
7. Rotate the order of who talks first to reduce undue influence.
Training team members 1. Prior to training, read Hill et al., (1997), the present article, and exemplar
studies.
2. Consult with an expert if having difficulty learning the method.
3. Describe training procedures in the Methods section.
Sample 1. Randomly select participants from a carefully identified homogeneous
population.
2. Choose participants who are very knowledgeable about the phenomenon.
3. Recruit 8 to 15 participants if 1 to 2 interviews are used.
Interviews 1. Review the literature and talk to experts to develop the interview
protocol.
2. Include about 8–10 scripted open-ended questions per hour.
3. Allow for follow-up probes to learn more about the individual’s
experience.
4. Conduct several pilot interviews to aid in revising the interview protocol.
5. Train new interviewers.
6. Ideally, each interviewee should be interviewed at least twice.
Data collection 1. Match the data collection format to the data desired and the needs of
the study.
2. Record reactions to interviews; review tape before subsequent interviews.
Domains 1. Develop the domains from the transcripts or a “start list.”
2. The entire primary team codes the data into domains in the first
several cases; the remaining coding can be done by 1 researcher
and reviewed by the team.
chapter 11 qualitative research 279
Table 11.3 Categories within the Domains of Supportive Factors, Obstacles, and
Living in Two Worlds, by Educational Level (Reprinted from Juntunen et
al., 2001, by Permission)
Secondary Education Participants Postsecondary Education Participants
Six of the seven participants with a high school education viewed the Native and
White cultures as quite distinct. In the words of one woman, “Those [White]
people are way different from us . . . they don’t understand a lot of our ways”
(Participant 3). Another participant felt the need to adjust her own behavior to be
effective outside of the reservation. She stated, “We have to live like Whites when
we’re out there” (Participant 6). This woman also noted that during the times she
had lived off the reservation, she frequently needed “a powwow fix” as a way to
stay connected to the Indian community. . . .
The experience of living in two worlds was dramatically different for the college
group. Rather than seeing the two worlds as distinct and distant, the respondents
who had attended college described the concept in two ways: moving between
two worlds (a typical category) and finding a holistic, third world (a variant
category). . . .
The process of moving between two worlds was often described as difficult,
both emotionally and cognitively. A tribal elder remembers her experience in the
college classroom, several hundred miles from her home reservation:
You always have to do twice as much . . . because you have to process every-
thing you think, reprocess it back into a thinking that you know [doesn’t] fit
with your thinking to make it right for the school. . . . Back when I applied to
law school . . . they said ‘You got to stop being Indian and you got to get
through these 3 years of school’ . . . anyway, you can’t do that. You can’t take
that piece of you and set it aside. (Participant 13)
However, not all of the participants who moved between two worlds found
the experience difficult. . . . People who sought a holistic, third world spoke of the
282 part 2 major research designs
desire to appreciate their own culture yet integrate those pieces of the majority
culture that they accepted. Some respondents viewed this as a personal internal
choice, which can be made whether one lived on or off the reservation.
Another woman spoke of her experience of two worlds as an evolution into a
third world, which she has created from those aspects that are most important to
her from each world.
I’ve lived in those two worlds and there’s also even a middle world. I mean, you
have the Native, the non-Native and you have a middle world that I have expe-
rienced and it’s a very personal, personal world where you gather, it seems like
you gather from both worlds, and create, you know, a happy medium for
yourself. I have two children that have lived with me on the reservation and off
the reservation and I like it that way. I like them to be able to know both worlds.
Because it just opens so much more range of experiences. (Participant 11)
However, creating that third world does not mean that the original experience
of her Native world is diminished in any way. “There’s no leaving the Native
American world. That comes along. When it’s your core being, you can’t leave it.
I wouldn’t want to leave it . . . I’m Indian to the core” (Participant 11). (Juntunen
et al., 2001, pp. 281–282)
CQR researchers often use quotes from the interviews or core ideas to
illustrate each category. As Hill and colleagues (2005) suggested, visual rep-
resentations of the results in tables, diagrams, or figures also help the readers
understand the relationships among categories. Finally, narrative case exam-
ples may provide a holistic picture of the findings. Examples of case descrip-
tions can be found in the articles written by Williams and colleagues (1998)
and Ladany and colleagues (1997).
Gathering Data
Although there are many ways to collect data in qualitative research, we dis-
cuss three primary sources here: observations, interviews, and existing mate-
rials. Wolcott (1992) refers to these sources in the active voice as experiencing,
enquiring, and examining, respectively, which provides a glimpse of the qual-
itative researcher’s state of mind in these three activities.
Making Observations
The Nature of Observations Observations are obtained by a trained observer
who is present and involved in the phenomenon of interest and who makes
chapter 11 qualitative research 283
The fifth and final step is to collect the data. Most frequently observa-
tional data are the researcher’s field notes (memos) taken during or immedi-
ately after the observations. Field notes are descriptions of everything relevant
to understanding the phenomenon. Because relevance is not always clear in the
beginning, initially field notes are likely to contain everything that happened.
The novice observer will feel overwhelmed, but it should be recognized that
almost anything missed will be repeated over and over again. As the obser-
vations become more focused, so will the field notes. For example, if it is
observed that one group is superior in multidisciplinary work, the observa-
tions may be focused on how this is established as new members join the
group; the field notes would similarly be focused on the process of transmitting
power to incoming persons.
Field notes should contain basic descriptions of the setting—time, physical
setting, persons present, the purpose of activity, and so forth—as well as com-
plete descriptions of the interactions among the participants. Patton (1987)
provides a good contrast between vague, generalized notes and detailed,
concrete notes:
Vague: The new client was uneasy waiting for her intake interview. Detailed: At
first the client sat very stiffly on the chair next to the receptionist’s desk. She
picked up a magazine and let the pages flutter through her fingers very quickly
without really looking at any of the pages. She set the magazine down, looked at
her watch, pulled her skirt down, and picked up the magazine again. This time
she didn’t look at the magazine. She set it back down, took out a cigarette and
began smoking. She would watch the receptionist out of the corner of her eye, and
then look down at the magazine, and back up at the two or three other people
waiting in the room. Her eyes moved from people to the magazine to the cigarette
to the people to the magazine in rapid succession. She avoided eye contact. When
her name was finally called she jumped like she was startled. (p. 93)
The latter description is more complete and involves little inference on the part
of the observer, whereas the vague description both involves the inference that
the client was uneasy and lacks data to enable confirmation of this inference at
a later time. People’s conversations should be recorded as close to verbatim as
possible. Audiotapes can be used to supplement field notes if possible. Field notes
should contain the observer’s interpretations of events, but these interpretations
should be so labeled to distinguish them from descriptions. Field notes might
also contain working hypotheses, suggestions for interviewers, and so forth.
Undoubtedly, observations and field notes are influenced by the personal
constructions of the observers. Multiple observers cross-checking their descriptions
and their interpretations are vital for the integrity of observations, because
trained observers may see a situation very differently. Acknowledging and honoring
these perspectives is part and parcel of qualitative research.
Conducting Interviews
Types of Interviews Qualitative researchers are interested in using interviews
to make sense of people’s actions in naturalistic settings. The language used
by respondents is a powerful means to accomplish this goal. Basically, the
chapter 11 qualitative research 287
the semi-structured format lies in the decision of how much structure should
be imposed on the interview process. For an example of a relatively unstruc-
tured interview, consider Rennie’s (1994b) study of clients’ deference in psy-
chotherapy, in which clients viewed a videotape of a recent counseling session
(a variation of Interpersonal Process Recall; Kagan, 1975) and stopped the
tape when they were experiencing something interesting or significant. The
interviewer then “conducted a nondirective inquiry into the recalled experience”
(Rennie, 1994b, p. 429), creating a description of the client’s perceptions of the
therapy in the client’s own language without imposing predetermined categories
for understanding these descriptions. As an example of a more structured
approach, Frontman and Kunkel (1994), who also wanted to access clients’
reactions to therapy, asked clients immediately after a session (rather than
being stimulated by a videotape) to respond to the following probe: “As if you
were making a personal journal entry, write what you felt was successful
about the session” (p. 493). Because this probe was responded to in writing,
no follow-up questions were used. Although the probe for this study was
standardized, no preconceived categories were used to make sense of the
responses, so that the conclusions were grounded in the data and not structured
by the investigators’ prior understanding of the phenomenon.
A third type of interview is the group interview, in which more than one
person is interviewed simultaneously. Madriz (2000) describes a focus group
as “a collective rather than an individualistic research method that focuses on
the multivocality of participants’ attitudes, experiences, and beliefs” (p. 836).
Particularly exciting possibilities exist for focus groups, a modality that evolved
from marketing research related to reactions to products, advertisements, and
services. (For a brief history of the development of the focus group method,
see Morgan, 1988, or Madriz, 2000.) In marketing research, the members of a
focus group are strangers, but applications to qualitative research suggest pos-
sibilities for using intact groups, such as the staff of a mental health clinic. The
goal of a focus group is to obtain the participants’ opinions, not to reach a
consensus or reconcile opposing views; the expression of different opinions is
informative. Typically, the participants in a focus group are relatively homo-
geneous and are asked to reflect on a particular issue. The economy of using
group interviews is self-evident, but a less obvious advantage of the group
format is the interaction among the respondents, which can provide richer
information as various members of the group provide more details, disagree
on points, and reconcile differences of opinions. Having participants respond
in a social context is thought to provide honest and responsible comments.
However, the interviewer needs to ensure that minority opinions are allowed
expression, given the natural tendency for such opinions to be suppressed
(Madriz, 2000). A good illustration of this method has been provided by
O’Neill, Small, and Strachan (1999) who used focus groups to explore the
employment issues facing people with HIV/AIDS.
and the qualitative research interview. However, note that “research inter-
viewing has different goals and requires different skills” (Polkinghorne, 2005,
p. 143). Although the goals of qualitative interviewing will depend on the type
of qualitative research conducted, a generic goal is to understand the experi-
ence of the participant; therapeutic actions should be avoided. Table 11.4, a
typology of questions suitable for qualitative research, provides some exam-
ples of questions aimed at gaining understanding at various levels.
We now briefly examine the various steps in conducting qualitative inter-
views, adapted from Fontana and Frey (2000). The first step is to gain entrée
to the setting in a way similar to gaining entrée for observations. Wampold
et al. (1995) were able to gain access to two chemistry groups by first obtaining
the support of the laboratory leader. Although all the members subsequently
agreed to be observed, several declined to be interviewed because of the time
it took away from their work.
The second step involves preparing to understand the language and the
culture of the interviewees. A qualitative investigator strives to have participants
express their experiences in their own language. To achieve that goal, the
290 part 2 major research designs
Using Existing Materials Existing materials are written text and artifacts, which
can often inform qualitative research in ways that observations and interviews
cannot; such materials are essential to any historical study for which obser-
vations are impossible. As always, the goal is to use this material to provide a
richer understanding of the phenomena being examined.
Written documents are of two types, official records and personal docu-
ments (Lincoln & Guba, 1985). Official documents include government reports,
licenses, contracts, diplomas, and so forth. Personal documents include diaries,
letters, email, literature, field notes, and so forth. Artifacts include material
and electronic traces, such as buildings, art, posters, nontextual computer
files—essentially any disturbance of the natural environment created by
people.
One type of existing material has assumed primacy in qualitative research
in counseling: the counseling interview (see, for example, Elliott et al., 1994;
Thompson & Jenal, 1994), either represented as a verbal record (that is, a
tape recording) or as text (that is, a transcript). In such studies, interpretations
are made of the records of counseling. For example, Friedlander, Heatherington,
Johnson, and Showron (1994) examined family therapy sessions to under-
stand how sustaining engagement was important to change.
Of the three sources of qualitative data discussed in this chapter (observations,
interviews, and existing materials), existing materials present the most issues,
even though such materials can be particularly valuable. The difficulty with
existing materials involves the interpretation of the text or artifacts, particu-
larly when the analyses of existing materials and use of other methods
(e.g., observations or interviews) yield different or even opposing results.
Hodder (1994) gives several examples of discrepancies between material
traces and people’s reports of their activities (e.g., report of alcohol use versus
number of beer bottles found in garbage).
excerpts from interviews, field notes, and existing materials (text and/or de-
scriptions of artifacts).
The consumers of thick descriptions have the raw data, which for the most
part have not been altered by the investigator. In this way, “the data speak for
themselves” and provide a rich account of what happened. Of course, the
process of observing, interviewing, collecting materials, and deciding what to
describe filters what is available to consumers. Still, even though the under-
standing achieved by readers of a thick description is affected by the research
process, thick descriptions are closer to the phenomenon being studied than
are any other means, qualitative or quantitative.
Despite the primary advantage that the data are minimally filtered, there
are numerous disadvantages to using thick descriptions. True descriptions are
too lengthy for journal articles, and many important thick descriptions lan-
guish as lengthy dissertations or unpublished reports. Another disadvantage is
that the thick descriptions may be unfocused and uninteresting: “Readers are
likely to get the idea that the researcher has been unable to sort out (or unwilling
to throw away) data and has simply passed the task along. . . . [D]ata that do
not ‘speak’ to the person who gathered and reported them are not likely to
strike up a conversation with subsequent readers either” (Wolcott, 1994,
pp. 13–14). It is hoped that the story, as told by the participants or as observed
by the investigator, will be fascinating enough to hold the interest of readers.
Rarely are qualitative investigators able to present all data verbatim in a
thick description, and thus they must use various strategies to present a con-
densed but still relatively complete description. As discussed earlier in this
chapter, the methods of condensing and presenting the data vary in accor-
dance with the strategies of inquiry adopted. Regardless, the investigator has
the critical task of deciding what must go, and there are few guidelines.
Crucial decisions about data should be made through a process that acknowl-
edges the investigators’ suppositions. Moreover, the process involved in exam-
ining these suppositions and making sense of the data in the context of these
suppositions should be a prominent part of the presentation so that readers
can understand how the descriptions were distilled to a reasonable length.
Themes and Relationships Themes are recurrent patterns in data that repre-
sent a concept, whereas relationships are the interconnections among the
themes. Through development of themes and relationships, the essence of a
phenomenon is revealed. Extracting themes and establishing relationships are
the essence of the three strategies of inquiry discussed earlier (i.e., grounded
theory, phenomenology, and CQR). Essentially, the relationships of themes
constitute the theory or the structure of the phenomenon, which is grounded
in (is inductively developed from) the data.
The first step in this process is to generate codes for various themes that are
intrinsic to the phenomenon being studied. Often tentative codes are first
generated from initial observations or field notes on a word-by-word, sentence-
by-sentence, or paragraph-by-paragraph reading of the text, and are then written
directly on the transcripts. These codes name various patterns. For example,
chapter 11 qualitative research 293
STIMULUS QUESTIONS
Analyzing Three Qualitative Studies
The following exercises are designed to help d. How did the authors provide a “thick
readers become familiar with the qualitative method- description” of the participants, the con-
ology and various strategies of inquiry. text, and their own perspectives about the
phenomenon under investigation?
1. Identify three published articles that utilized
e. How did the authors present their findings
grounded theory, phenomenology, or consen-
and their interpretations of the results?
sual qualitative research approach in the fol-
f. What methods did the authors utilize to
lowing journals: Grounded Theory Review,
establish the trustworthiness of their
Journal of Phenomenological Psychology,
research?
and Journal of Counseling Psychology. Review
2. After responding to the previous questions
these three articles and respond to the follow-
for all three articles, compare and contrast
ing questions for each of the three articles:
your answers. Then, select a topic of interest
a. How did the authors situate their study in a
and a strategy of inquiry that may help to
particular paradigm (philosophical tradition)?
answer your research questions. Design a
b. How did the paradigm selected by the authors
study using this particular strategy. Locate
fit the conclusion of the literature review, the
and read a number of publications that
research question(s), the intended purpose(s),
would help you gain more knowledge about
and the strategy of inquiry?
this type of qualitative approach. Write down
c. Which strategy of inquiry was adopted?
the appropriate paradigm, research method
Write down the corresponding data collec-
and design, analytic procedure, and expected
tion method and analytic procedure utilized
outcomes.
by the authors.
This page intentionally left blank
Methodological Issues PART
3
297
12 CHAPTER Designing and Evaluating
the Independent Variable
One of the primary goals of the research endeavor is to establish a causal rela-
tionship between the independent and dependent variables. After a researcher
has identified a research question, he or she takes the critical step of selecting
or designing these variables. This chapter focuses on issues pertaining to
independent variables, while Chapter 13 focuses on dependent variables.
Selection, design, and evaluation of the independent variable are crucial in
establishing and interpreting causal relations in a study. If the independent variable
is poorly designed, the researcher’s effort will be unrewarding—either the
expected effect will not be found or the results will be ambiguous or meaningless.
Poorly designed independent variables create unwanted bias and extraneous
variables, which are clear threats to Kerlinger’s MAXMINCON principle.
This chapter discusses four issues related to the development and selection
of independent variables. The first section discusses operationalizing the
independent variable. But even when a researcher has carefully designed an
independent variable, there is no assurance that the experimental manipulation
will achieve its purpose, and thus the second section describes methods to
check or verify the manipulation of the independent variable, often called
manipulation checks. The third section focuses on interpreting the results of a
study, whether the manipulation of the independent variable was successful or
unsuccessful.
Thus, the first section focuses on issues pertaining to the independent variable
that are relevant before an experiment begins, the second section on issues during
298
chapter 12 designing and evaluating the independent variable 299
an experiment, and the third section on issues after an experiment has been
conducted. In the final section of the chapter we discuss independent variables
that are not amenable to manipulation, which we define as status variables.
Determining Conditions
In counseling research, the typical independent variable consists of several
conditions. In the between-groups designs discussed in Chapter 7, the emphasis
is on independent variables with two conditions: treatment and no treatment
(that is, a control group). However, an independent variable can contain any
number of conditions.
A treatment study can examine three treatments as well as a no-treatment
condition (that is, four conditions in all). Or a treatment group may be con-
trasted with a placebo control group and a no-treatment control group (that
is, a total of three conditions). Of course, the independent variable is not
restricted to psychological treatments; in a study of preferences for counselors,
European-American or Mexican-American counselors (two conditions) would
constitute an appropriate independent variable (see, for example, Ponce &
Atkinson, 1989). In this chapter, independent variables are discussed generally;
Chapter 18 contains a presentation of treatment outcome designs in which the
independent variable is used to establish the efficacy of treatments (that is, at
least one of the conditions is a treatment).
Two notes need to be made about our discussion of conditions. First, we
use the term conditions to indicate the groups that constitute the independent
variable. However, levels of the independent variable, groups, categories, and
treatments are other interchangeable terms that are used in the discussion of
research design. In this context, treatments refers generically to conditions,
and not to psychological interventions. Second, we have conceptualized the
independent variable as a categorical variable—that is, each discrete category
(level, condition, group, or treatment) is different. The independent variable
need not be restricted to categories. Classical regression designs involve quan-
titative independent variables; in that case, the conditions reflect different
amounts of something (see Wampold & Drew, 1990). For instance, in a drug
study, the independent variable could consist of different dosage levels (for
example, no drug, 2 cc, 4 cc, and 6 cc) or, in a psychotherapy treatment
study, the independent variable might be the amount of homework assigned.
300 part 3 methodological issues
MANIPULATION CHECKS
Even when great care has been taken to define and operationalize the inde-
pendent variable, there is no assurance that the experimental manipulation
will achieve its purpose. It is possible for the researcher to misjudge the
salience of the independent variable. To verify that a manipulation has been
adequately designed, it is often advisable to check the characteristics of the
manipulation. The goal of manipulation checks is to show one or more of the
following: (1) that conditions vary on the intended dimension, (2) that condi-
tions do not vary on other dimensions, and (3) that treatments are imple-
mented in the intended fashion.
To determine whether the conditions vary on the intended dimension,
judgments of characteristics related to the dimension should differ across con-
ditions. This determination can be made in a number of ways. First, inquiries
can be made of the participants themselves. For example, Jones and Gelso
(1988), in a study of the effects of the style of interpretation, manipulated
style by having participants listen to audiotapes of a counseling session. In one
condition, the counselor’s interpretations were tentatively phrased and ended
with a question, and in the other condition the counselor’s interpretations
were decisively phrased. The manipulation check was accomplished by having
participants rate on a seven-point scale whether the counselor’s comments
were phrased tentatively or decisively, and whether they were in the form of
questions or statements. As anticipated, there were significant differences
between the conditions on both of the seven-point scales, providing evidence
that the manipulation was indeed salient to the participants.
Another means to assess differences on the intended dimension is to have
independent raters (persons other than the participants or the experimenters)
judge the experimental materials. These independent raters could be either
naïve individuals (those untrained in counseling) or experts. In the ethnicity-
of-counselor study discussed previously, Ponce and Atkinson (1989) also
varied counselor style (directive versus nondirective). Graduate students in
counseling psychology rated the dialogue of the sessions, and the intended dif-
ferences in directiveness were found, lending support for the adequacy of the
independent variable.
Independent raters and the participants can also be used to establish that
the conditions do not vary on dimensions other than the intended one. Recall
that Ponce and Atkinson’s (1980) use of photographs of the counselors intro-
duced a possible confound related to the counselors’ personal attractiveness.
To control for this threat, undergraduates rated the attractiveness of several
chapter 12 designing and evaluating the independent variable 305
Hazaleus & Deffenbacher, 1986); the researchers found that these ratings
were comparable across the three treatments.
In treatment studies it is important that the treatments be delivered to par-
ticipants in the intended fashion. When the saliency of a manipulation is in
doubt, checks provide a means of verifying the researcher’s claim that the con-
ditions differ on the intended dimension only. Whether or not manipulation
checks are worth the extra time and effort required to implement them is a
determination that can be made only in the context of a particular research
study. Further issues related to manipulation checks are discussed in Chapter 18.
INTERPRETING RESULTS
The purpose of an experimental design is to establish a causal relationship
between the independent and dependent variables. Thus far we have discussed
topics related to design of the independent variable and to checking on the
manipulation. Equally important is interpreting the results of an experiment,
which provide much of the information upon which inferences are based. In
this section we discuss various problems in interpreting statistically significant
and statistically nonsignificant results with regard to the independent variable.
the presence of a confound: The manipulation check was accurate (that is, the
conditions did not vary on the intended dimension), but the conditions varied
on some other dimension. Even if the researcher checked other dimensions
and found no differences, it is not possible to check all confounds. A third
possibility is that the statistically significant results were in error (that is,
Type I error). Clearly, statistically significant results in the presence of failed
manipulation checks are difficult to interpret.
It would seem that the best situation is when the results are statistically
significant and the manipulation check shows that the conditions differed on
the desired dimension. But even here ambiguities may exist. A manipulation
check can be reactive, and thus significant results may be due to the demand
characteristics of checking the manipulation and not to the independent vari-
able. More specifically, asking the participants about the experimental manip-
ulation may have sensitized them to many aspects of the study, and their
responses on the dependent measures may have been due to this sensitization.
For example, asking participants whether the counselor’s comments were
phrased tentatively or decisively (see, for example, Jones & Gelso, 1988)
might cause the participants to review the preceding session critically and ret-
rospectively change their opinion of the counselor.
To minimize reactivity, the researcher should consider administering the
check after the dependent measure (of course, then the check may be influ-
enced by the dependent measure), making the check indirect rather than trans-
parent, and using unobtrusive measures (see Chapter 13). It is worth repeating
that successfully checking the manipulation to determine that the conditions
varied on the intended dimension does not rule out confounds, for it is entirely
possible that the conditions varied on other dimensions as well. Nevertheless,
interpretation is least ambiguous when the check was successful and the
expected differences among conditions were found.
been easy to attribute the null results to lack of salience of differences in con-
ditions (different interpretation styles). Although Jones and Gelso discussed
many possible explanations for their nonsignificant findings, the manipulation
check strengthened the possibility that counseling outcomes are not dependent
on the interaction of client type and interpretation style, as had been thought.
Nonsignificant findings can also accompany unsuccessful manipulation
checks, as occurs when the check indicates that the conditions did not differ
on the intended dimension, and the expected differences on the dependent
variable are not found. This circumstance suggests the distinct possibility that
poor design of the independent variable was responsible for the nonsignificant
findings; consequently, the importance of the null findings for the field of
counseling is mitigated.
STATUS VARIABLES
In this chapter we have emphasized the fact that the nature of the independent
variable is determined by the researcher. By designing the independent vari-
able in some particular way, the researcher attempts to examine its effect on
the dependent variable. We have used the word manipulation to characterize this
deliberate process. As mentioned previously, a study may contain more than
one independent variable, in which case the effects of independent variables
are typically examined in a factorial design, as discussed in Chapter 7. For
example, Ponce and Atkinson (1989) manipulated both counselor ethnicity
(European American or Mexican American) and counselor style (directive and
nondirective) in a 2 × 2 factorial design.
Counseling researchers are often interested in variables that are not
amenable to manipulation, due either to ethical constraints or to logical im-
possibilities. It is not ethically permissible to assign participants to a spouse
abuse condition, nor is it possible to assign participants to a gender condition.
We define all participant-related variables that cannot be assigned as status
variables. Examples include personality variables (for example, locus of control),
socioeconomic variables (such as education), gender, and ethnicity. Although
many researchers label these variables as independent variables, the distinction
between status variables and independent variables is critical to understanding
the types of conclusions that can be drawn from these two types of variables.
Independent variables are manipulated and the effect on the dependent
variable is subsequently assessed; if everything goes well, a causal relationship
is established. In contrast, status variables cannot be manipulated, and sta-
tistical tests involving them detect associations. For example, Vredenburg,
O’Brien, and Krames (1988) classified college students as either depressed or
nondepressed; depressed students vis-à-vis nondepressed students were less
assertive, had less control over their depressions, had lower degrees of instru-
mentality and persistence, and had a higher degree of dysfunctional attitudes.
Because Vredenburg et al. were not able to randomly assign participants to
levels of depression (that is, manipulate the independent variable), it would
chapter 12 designing and evaluating the independent variable 309
not be proper to assert that depression caused any of the personality variables.
The causal relation could be in the opposite direction; for example, dysfunc-
tional attitudes may be the cause of depression for college students. Or a third
variable (for example, some biochemical dysfunction) could be the cause of
both depression and the personality variables.
An important point must be made about the statistical analysis of status
variables: Even though the analysis of status variables is often identical to that
of independent variables, it is more difficult to make causal inferences because
status variables are not manipulated (Shadish et al., 2002). It is the design, not
the analysis, that determines the inferential status of the study (Cohen, 1968;
Wampold & Freund, 1987). For example, Vredenburg et al. (1988) conducted
analyses of variance with two groups (depressed and nondepressed); because
depression was not manipulated, it cannot be said that depression was the
cause of differences in the dependent variables.
It is not unusual to include both independent variables and status vari-
ables in the same study. Kim, Hill, et al. (2003) included adherence to Asian
cultural values as a status variable as well as the manipulated self-disclosure
independent variable. Frequently, research hypotheses are directed toward an
interaction of an independent variable with a status variable; studies that
address the question of which treatments work best with which clients are of
this type. Two studies discussed in this chapter are examples: Malkiewich and
Merluzzi (1980) examined the interaction of the conceptual level of the client
(a status variable) with the structure of the counseling (an independent vari-
able), and Jones and Gelso (1988) predicted an interaction between client type
(a status variable) and counselor style (an independent variable). When inter-
preting the results of studies with multiple variables, one needs to keep clearly
in mind the distinction between independent and status variables, particularly
with reference to causal inferences.
We do not make the distinction between independent variables and status
variables so that one type can be considered first class and the other inferior.
The important point is that independent variables are manipulated so that
causal inferences can be made directly. This is not to say that causality can
never be attributed to some status variables. However, inferences in this case
are made in a much different (and more difficult) manner. Consider the research
on smoking and health. Smoking behavior cannot be ethically manipulated;
for example, participants cannot be assigned to smoking and nonsmoking
conditions. Even though there is little ambiguity about the fact that smoking
is the cause of a number of diseases (Holland, 1986), this causal relationship
was established by animal studies, epidemiological surveys, cross-cultural
studies, retrospective comparisons, and the like. Because smoking cannot be
an independent variable, the American Tobacco Institute is correct when it
states that there has not been one study that has established scientifically that
smoking is the cause of any disease; however, the causal relation has been
firmly established over many studies. The first step in this process was to
establish that a relationship exists between smoking and disease; then, alter-
native explanations were ruled out. Cook and Campbell (1979) provide a
310 part 3 methodological issues
manipulation check is successful and there independent variables, the inferences that can be
are statistically significant differences on the de- made are much different. When status variables
pendent variable, causal attributions are most are used, statistical tests detect associations rather
plausible, although the researcher needs to make than causal relations.
sure that the manipulation check was not reactive. Clearly, design of the independent variable is
In many counseling studies, status variables a critical step in research. It is not unusual for
are included in the design and analysis. Status researchers to have confidence in their manipula-
variables are variables that cannot be manipu- tions only to discover after the data have been col-
lated by the researcher, such as personality vari- lected that a threatening confound was present. It
ables, socioeconomic variables, gender, and is best always to be one’s own greatest critic, and
ethnicity. Although the analysis of status vari- attempt to think of every possible problem with the
ables may be identical to the analysis of true independent variable before a study is conducted.
STIMULUS QUESTIONS
Methodological Issues
1. Find three studies in your area of interest. For how you would design a study to opera-
each study, identify the independent variable. tionalize the independent variable.
For each independent variable, state the various 3. Find a study in which a manipulation check was
levels of the independent variable and indicate successfully used and one in which the failure to
whether they are experimentally manipulated use a manipulation check led to a major threat to
or are status variables. validity of the conclusions. Describe how the use
2. Suppose a researcher is interested in editorial of the manipulation check improved the validity
reactions to manuscripts on oppositional in the first study. For the second study, design a
defiant children, where the research manu- manipulation check that could have been used.
scripts differ on the basis of the ethnicity of 4. Suppose a researcher compared the efficacy
the children (European American versus of two treatments for depression and found
African American) and the attribution for the that one treatment was superior to another.
cause of the children’s difficulties (external, Discuss possible threats to this conclusion
such as poor family conditions, versus inter- that could result from the manner in which
nal, such as poor impulse control). Describe the independent variable was designed.
13 CHAPTER Designing or Choosing
the Dependent Variable
The basic purpose of the dependent variable (sometimes called the dependent
measure) is to measure the construct that is hypothesized to be the effect
(referred to as the effect construct; see Chapter 5). Thus, selecting or designing
dependent variables and the methods of data collection vis-à-vis the dependent
variables are critical activities for the researcher. Typically, one subsection of
the methods section of a journal article is entitled “Dependent Variables” or
“Dependent Measures” and contains a brief description of and some psycho-
metric information about the dependent variables used in the study. Only infre-
quently, however, is a rationale given for the choice of dependent variables:
Why were these particular variables included and others excluded?
Extreme caution must be exercised in this process because the choice of
dependent variables can be critical to the merits of the research. For example,
a mother’s report may be used to assess her children’s behavior, but her ratings
of her children’s behavior may be affected more by her own psychopathology
than by the children’s actual behavior (Webster-Stratton, 1988). Likewise, the
reported outcome of psychotherapy and counseling differs depending on
whether the effects are judged by clients, therapists, or independent raters
(Orlinsky, Grawe, & Parks, 1994). Investigations with poorly chosen or poorly
designed dependent variables will at best be uninformative or uninterpretable
and at worst be erroneous and misleading. Conversely, creatively designing a
set of dependent variables might reveal new information that adds greatly to
the knowledge base in a particular area.
312
chapter 13 designing or choosing the dependent variable 313
Psychometric Issues
One important question about the operationalization of a construct involves
the psychometric properties of the dependent variable. Researchers need to
know to what extent the dependent variables they select to operationalize a
314 part 3 methodological issues
construct are reliable and valid. If the estimates of reliability and validity are
poor, then the operationalization of the construct is likely to be inadequate.
Although entire volumes have been devoted to psychometrics, we will review
the rudiments nontechnically here because they are critical to understanding
the degree to which a construct is properly operationalized, and how this
affects the validity of research. The skilled researcher needs to have a strong
background in psychometrics and broad knowledge of the psychometric properties
of the dependent variables used in a study.
Random Response Error There is often some error in any response that a
participant makes. The most obvious example of these errors occurs in response
to written items in a paper-and-pencil instrument, but random response error
chapter 13 designing or choosing the dependent variable 315
occurs in measurements of all kinds. One participant may read the word
“ever” as “never” and respond accordingly; another participant may be dis-
tracted by a noise during testing and mark a response to the wrong item; a
third participant might forget which end of the scale is “disagree” and which
is “agree”; and so forth.
Although later in the text we discuss ways to calculate error due to random
responses, a few important points need to be made here. First, the assessment
of almost all meaningful characteristics of individuals and situations contains
random response error. Simply asking participants “Are you male or female?”
produces a random response error rate of about 5% (that is, it has a reliability
of coefficient of 0.95; Campbell, Converse, Miller, & Stokes, 1990). With regard
to measuring more ambiguous characteristics than gender or age, performance
tests (such as intelligence tests) typically have the lowest random response
error rates (namely, reliabilities in the neighborhood of 0.90). Measurements
of other characteristics, such as personality traits or therapist skill level, generally
have larger random response error rates.
A second point is that instruments typically contain many items measur-
ing the same trait so that a single random response will not unduly affect the
total score. Given items of the same quality, instruments with more items will
be more reliable than instruments with fewer items. Consider a 15-item scale
with reliability of 0.84. It can be shown mathematically (with the Spearman-
Brown formula) that randomly selecting seven items to compose the scale would
produce a reliability of 0.70. Extrapolating further, a one-item scale would have
a reliability of 0.25, a value that is typical of a single-item scale (see Schmidt
& Hunter, 1996, scenario 1, for an elaboration of this example). The point is
that whenever anything is measured using a single item, one must assume that the
reliability of this measurement is catastrophically low (with some exceptions,
such as gender and age).
Although researchers are aware of this problem, occasionally they believe
some phenomenon to be so straightforward that a single item is sufficient. For
example, it is all too common to see global evaluations of satisfaction, such as
“On a scale of 1 to 100, how satisfied were you with this experience?” In such
cases, researchers pay dearly in terms of low reliabilities.
a way that the conditions at that time or with the particular stimulus affect
the measurement of the trait. Consider the measurement of depression, the
manifestation of which can be affected by transient mood states: A depressed
college student’s responses to a depression inventory, for example, would be
affected by receiving a failing grade on an examination in the hour preceding
the assessment; other participants’ responses would be similarly affected by
moods created by recent events.
Transient errors can be induced by administering tests in a particular
order. For example, an instrument used to assess anxiety may in fact create in
participants an anxious mood, which in turn would affect scores on a subse-
quent instrument, producing artifactual scores on the second instrument.
These transient effects create error that is unrelated to true scores.
A related problem is particularly problematic in counseling research.
Suppose that one were interested in the assessment of the cultural competence
of a beginning counselor, and such a determination is made by the novice
counselor’s responses to only one client. Idiosyncrasies of the client and of the
novice counselor’s relationship with this client will affect the assessed level
of skill, creating unreliability in the assessment of cultural competence.
into another student intentionally), but as they are exposed to more egregious
behavior (students striking other students or threatening others with weapons)
they may become desensitized to and consequently ignore less objectionable
behavior, all the while maintaining rater agreement. This is called observer drift.
quite different than what was postulated; for example, a personality measure
may be measuring social desirability rather than the targeted construct. De-
velopers of scales often attach names to them that indicate some construct (for
example, the ABC Scale of Social Skills), but adequate reliability does not
establish that the instrument actually measures that construct (in this instance,
social skills). It is validity, which will be discussed later in this chapter, that is
concerned with whether the construct being measured is the construct of
interest. A third point is that reliability is based on the scores and not on the
instrument from which they were derived. The scores have certain properties
that are derived from the characteristics of the instrument. A vital consequence
of this distinction is that reliability estimates are restricted to the types of par-
ticipants on whom, and the conditions under which, the psychometric study
was conducted.
An instrument may perform adequately for one type of participant but
not for another type, or under one set of conditions but not under others. For
example, an anxiety measure that yields adequate reliability estimates with
undergraduates when administered in a classroom may be completely useless
for measuring the anxiety of agoraphobics in a laboratory setting. Put another
way, the instrument may be very sensitive to midrange differences in anxiety
but insensitive at the upper range. This is called a ceiling effect; all the agora-
phobics may have scored at or near the maximum, and thus their scores were
not reflective of true differences in anxiety. Of course, this problem may also
be manifest at the bottom of the range, creating a floor effect. Reliability is
also dependent on characteristics of the participants, such as reading ability
and age. An instrument may yield adequate reliability for college students but
not for high school dropouts because of random error created by the latter’s
difficulty in reading the items. Moreover, instruments may contain items that
have different meanings to different cultures, and it should not be assumed
that reliability estimates are transferable. Ponterotto and Casas (1991) found
that only 25% of counseling instruments used in multicultural counseling
research were developed psychometrically for ethnic minority populations.
The implication of this discussion is that researchers should choose instru-
ments that are sensitive in the range of scores anticipated and the type of par-
ticipants used in the study. Such a choice requires a careful reading of the
psychometric studies conducted on various instruments. Alternatively, the reli-
ability of the scores actually obtained in a study could be estimated; typically
this is impractical because large numbers of participants are needed for such
studies (typically in excess of 300; Nunnally, 1978) and because reliability
estimates are affected by mean differences obtained for the various conditions
of the independent variable. How high should reliability be? Some sources
indicate that reliability estimates in excess of 0.80 are sufficient. Certainly, all
things being equal, the instrument that yielded the highest reliability in the
desired range should be chosen over other instruments. However, all things
are rarely equal, and choices must be made.
Other factors need to be considered, including validity, time required to
complete the instrument, and costs (topics to be discussed later in this chapter).
chapter 13 designing or choosing the dependent variable 319
Put into words, the correlation between the constructs is equal to the
obtained correlation between the measures divided by the square root of the
product of the reliabilities of the measures. In our present example, the corre-
lation between the constructs would be
That is, the correlation between the constructs is perfect, and the only differ-
ences between the scores on X and Y are due to random error—any interpre-
tations involving two distinct constructs would be in error.
The point of this example is that even if two constructs are perfectly cor-
related, the obtained correlation will be dramatically attenuated by unrelia-
bility. (Note that we have not discussed the effect of sampling error, which
could also dramatically affect the obtained correlation.) This is not an artifi-
cial example; many subscales of instruments are correlated in the neighbor-
hood of 0.70, with reliabilities in the neighborhood of 0.70, suggesting that
these subscales are measuring the same construct, rather than distinct con-
structs (see, for example, Atkinson & Wampold, 1982).
Now consider the following example, which illustrates how unreliability
can make it almost impossible to obtain expected results in a study. Suppose
that a researcher is interested in the relation between the skills of counselors
in training and counseling outcome. The generic skill of the beginning coun-
selors is rated by the practicum instructor on a single-item scale anchored
by “Very skilled—top 5% of all practicum students” and “Very unskilled—
lowest 5% of all practicum students.” Suppose that a single measure of out-
come was used—for example, a measure of depression. Now also suppose
that the researcher is very fortunate to sample all of the students in a large
program, say n = 30. What are the chances that the researcher will detect a
true relationship between skill and outcome? As we will see, the probability is
low. As we discussed earlier, the reliability of skill ratings, especially on a
single-item instrument, is probably extremely low; for this example, suppose
that the reliability of such ratings is generously assigned a value of 0.50.
Suppose also that the measure of depression is fairly reliable—say, ryy = 0.80.
Furthermore, suppose that about 20% of the variance in outcome is due to
the skill of the counselor (a reasonable estimate, given that variance in out-
come is also due to initial severity, treatment administered, motivation for
therapy, social support of client, and so forth). If 20% of the variance in out-
come is due to skill, then the population correlation between the constructs of
skill and outcome would be 0.45 (that is, variance accounted for is the square
of the correlation coefficient). However, this correlation is attenuated by the
unreliability of the measures of the constructs; using the attenuation formula,
the correlation is reduced to 0.28. The power to detect a population correla-
tion of 0.28 with 30 participants is about 0.35; that is, the probability of
rejecting the null hypothesis of no relationship between skills and outcome is
0.35 when the true correlation is 0.28. Said another way, about 65% of the
times this study would be executed, the researcher would conclude that there
was no relationship between therapist skill and outcome, despite the fact that
the true relationship between skill and outcome is strong! This is obviously
a disturbing result, because it will likely be concluded that the skills of
practicum students are unrelated to outcome, when this is not the case. In
Chapter 5 we discussed threats to the statistical conclusion validity of a study
due to unreliability of measures and low power; this is a graphic illustration
of these effects. The central point here is that the obtained relation between
322 part 3 methodological issues
measures of constructs may be very different than the true relationship be-
tween constructs, due to the unreliability of measures. When any statistical
relation is represented, one must be very clear about whether one is discussing
variables (measures of constructs) or constructs.
Although the preceding example involved correlations, the same princi-
ples apply to experimental designs. The reliability of the dependent variable,
and the degree to which the independent variable faithfully and saliently rep-
resents the intended differences between conditions, attenuates the size of the
effect and reduces the power of the statistical test of differences among groups.
Any conclusion that a treatment resulted in no differences in outcome may be
due to the low power resulting from unreliability.
Validity Of the many types of validity, the most important type for research
purposes is construct validity—the degree to which the scores reflect the de-
sired construct rather than some other construct. Clearly, unreliable scores
cannot have construct validity because they are due mostly to random error.
Nevertheless, as mentioned previously, reliable scores may reflect one or more
constructs other than the one specified. Specifically, scores may be quite re-
liable but lack construct validity. Determining construct validity, although
complicated and indirect, is vital to the integrity of a study.
One way to establish construct validity is to examine the relation between
scores on the instrument and scores on other instruments intended to measure
the same and other constructs. Clearly, there should be a high correlation
between instruments that measure the same construct. If these expected
correlations are found, then convergent validity is said to exist. Measures of
different constructs should not be highly correlated, although a moderate cor-
relation can be tolerated and may even be expected. Nevertheless, the corre-
lation of measures of different constructs should be smaller than correlations
of measures of the same construct; if this pattern is found, discriminant valid-
ity is said to exist.
Generally, this is not a procedure that produces satisfactory results. There are
three problems: (1) The method is atheoretical and may lead to factors that
have little psychological basis and are driven by the data; (2) even if the factor
analysis uses a method that produces independent factors, the subscale scores
likely will be highly correlated, because items load to some degree on all fac-
tors; and (3) the reliability of single items is low, and thus the results of factor
analyses are often unstable, in which case cross-validations are necessary.
An improvement on the exploratory factor analytic strategy is to develop
items specifically to measure factors of a construct. This strategy was used to
develop one of the most widely used counseling instruments, the Counselor
Rating Form (CRF; Barak & LaCrosse, 1975). The CRF is a 36-item scale
designed to measure three characteristics of the counselor related to the social
influence the counselor possesses vis-à-vis the client: trustworthiness, attrac-
tiveness, and expertness. Each item contains an adjective and its opposite (for
example, logical/illogical), and respondents rate their perception of the coun-
selor on a seven-point scale with regard to these adjectives (1 = logical, 7 =
illogical). Subscale scores for trustworthiness, attractiveness, and expertness
are determined by summing the scores for the items within each subscale.
There are 12 items for trustworthiness, 12 items for attractiveness, and 12 items
for expertness.
Although factor analyses of the CRF (see Heppner & Claiborn, 1989)
have verified the existence of three factors (that is, the 12 items loaded on the
expected factors), the correlation among the factors was high (generally in the
range of 0.60 to 0.80), suggesting that one general factor may be operating.
This general factor, labeled the “good guy” factor (Ponterotto & Furlong,
1985), suggests that responses to the CRF are due primarily to a general opin-
ion about the counselor. Based on a revision of the CRF called the Counselor
Rating Form—Short (CRFS; Corrigan & Schmidt, 1983), Tracey, Glidden,
and Kokotovic (1988) showed the pervasiveness of the general evaluation
factor using a variation of confirmatory factor analysis. Moreover, in a study
of premature termination, Kokotovic and Tracey (1987) found that contin-
uers and dropouts differed in the degree to which they rated their counselors
as trustworthy and expert, but when the effects of general satisfaction (mea-
sured on a different instrument) were controlled, the three CRF scales poorly
discriminated continuers from dropouts.
The preceding discussion of the CRF and its subscales raises an issue:
whether one should use the total score of an instrument or its subscale scores.
The choice is exclusive; that is, one should never use both the total score and
one or more of the subscale scores in the same analysis, because they are lin-
early dependent and will result in nonexistent or meaningless solutions in sta-
tistical analyses. The decision to use subscale scores or total scores is primarily
related to the hypotheses of the study, but it is partially related to psychomet-
rics as well. If the hypotheses of the study reference the general construct (for
example, global evaluation of the counselor), then one should either use the
total score or combine the subscale scores, rather than performing analyses on
each separate subscale. Trying to interpret results for the subscales in the
324 part 3 methodological issues
Figure 13.1
Use of
Multiple
Variables to Variable 1
Operationalize
a Construct
Variable
2
Portion of construct
not assessed by any
of the variables CONSTRUCT
Variable 3
absence of hypotheses about the subscales will result in ambiguity when some
of the subscales lead to statistically significant results but others do not (see
Wampold, Davis, & Good, 1990). However, if the hypotheses specify rela-
tionships for the constructs of the various subscales, then one should analyze
the subscales separately (Huberty & Morris, 1989). For example, Wampold
et al. (1995) hypothesized that social-coping skills would vary as a function
of Holland type, whereas problem-focused social skills would not; conse-
quently they analyzed the social-coping skills subscales of the Social Skills
Inventory (Riggio, 1989) separately from the problem-focused social skills
subscales, rather than using a total social skills score. Finally, subscale scores
should never be used if there is not persuasive evidence that they are measur-
ing distinct constructs, a point on which we will elaborate subsequently.
*ZungD = Zung Self-Rating Depression Scale; BDI = Beck Depression Inventory; DACL = Depression Adjective
Checklist; ZungA = Zung State Anxiety Measure; SAI = State Anxiety Inventory; TAI = Trait Anxiety Inventory;
MAS = Manifest Anxiety Scale; EHE = Endler-Hunt Examination Anxiety; EHS = Endler-Hunt Speech Anxiety.
Source: Tanaka-Matsumi, J., & Kameoka, V. A., (1986). Reliabilities and concurrent validities of popular self-report
measures of depression, anxiety, and social desirability Journal of Consulting and Clinical Psychology, 54, 328–333.
.55
.70 EHE
.56
.69 EHS
chapter 13 designing or choosing the dependent variable 327
are 0.83, 0.83, and 0.68, respectively. The latent variable “Depression” is a
statistical entity representing the construct operationalized by the three mea-
sures of depression. This latent variable represents the construct depression
measured without error, because in a sense it is the variance that the three
measures have in common, excluding specific or error variance. This proce-
dure statistically accomplished what is pictured in Figure 13.1 (that is, it uses
the common variance among measures to produce a variable that represents
the construct). Similarly, the construct of anxiety (the ellipse “Anxiety”) is the
statistically developed measurement of the construct of anxiety from the six
observed measures, with factor loadings ranging from 0.55 to 0.90.
The correlation of the constructs of depression and anxiety is then es-
timated from the latent variables of anxiety and depression. The curved dou-
ble arrow between “Depression” and “Anxiety” represents this correlation,
which was calculated to be 0.98. This says that the estimate of the correlation
of the constructs of depression and anxiety, as measured by the three mea-
sures of depression and the six measures of anxiety, is 0.98. This correlation is
not attenuated by unreliability. The conclusion here is that, at least as measured
by these commonly used instruments, the constructs of depression and anxiety
are not distinct. If these nine measures were used in an outcome study, it
would only make sense to talk about the effect of treatment on the aggregate
of the measures because it has been shown that the constructs operationalized
are not distinct. It would not make sense to talk about the relative effectiveness
of one treatment on one of the constructs but not the other (for example, to
say that Treatment A is more appropriate for the treatment of depression)
regardless of the pattern of results on the individual measures, a pattern that
is surely random (that is, is due to error). It would be even more problematic
to perform individual tests on the nine measures and make conclusions about
individual measures, as this example has shown that they are measuring the
same construct. Moreover, conducting nine statistical tests dramatically increases
the probability of obtaining a statistically significant result by chance alone (that
is, it inflates the alpha level). With nine variables, the probability of obtaining
at least one significant result by chance is approximately 0.40, dramatically
higher than is desired to make valid conclusions (see Hays, 1988).
Although it has been shown that latent variables can be used to calculate
correlations that are not attenuated by unreliability, these correlations may be
inflated by the fact that measures use the same method of data collection. In
the next section we show how this method variance can be removed.
affect responses to the instruments. Students attending the testing session just
after receiving grades on their mid-term may experience transient feelings
induced by the results of the exam. Because only one method was used, these
possibilities are likely to affect responses to all instruments similarly, increas-
ing the correlations among them. Variance common to all measures using the
same method is called method variance. Method variance inflates relation-
ships among variables; that is, the relationship between two measures is due
not only to a conceptual relationship in the constructs of interest, but also to
a relationship in how the constructs were measured. Whereas unreliability
attenuates correlations, method variance inflates correlations, as the follow-
ing example demonstrates.
Table 13.2 displays a multitrait-multimethod correlation matrix in which
two traits, A and B, are measured with three different methods, forming six
measured variables, A1, A2, and A3 (Trait A measured using the three methods)
and B1, B2, and B3 (Trait B measured using the three methods). In this fabri-
cated example, the correlations correspond to the convergent and discriminant
validity presented in Table 13.1. The correlations of the same trait with differ-
ent methods are relatively high (0.57 to 0.64 for Trait A and 0.54 to 0.56 for
Trait B), and the correlations between different traits using different methods
are relatively low (0.35 to 0.54). However, as described earlier, correlations of
different traits using the same method are inflated by method variance and are
high (0.72 to 0.78). Furthermore, all the correlations are attenuated by unrelia-
bility. From this matrix, we want to estimate the correlation between traits A
and B to determine whether they are independent, related but distinct, or essen-
tially the same. To this end, we again use structural equation modeling.
The first structural equation model, shown in Figure 13.3, examines the
correlation of the latent traits in the same manner as we did for depression
and anxiety. It appears that both Trait A and Trait B are measured well
because the loadings on the observed variables are high, suggesting conver-
gent validity. (Structural equation modeling provides model fit indexes that
chapter 13 designing or choosing the dependent variable 329
.47 A1
.73
.27 A2 .86
Trait A 1.00
.46 A3 .73
.95
.45 B1 .74
Trait B 1.00
.37 B2 .79
.68
.54 B3
assess the adequacy of measurement. Although not discussed here, these fit
indexes should be examined to determine whether the constructs are being
measured well.) Moreover, the two traits are highly correlated (namely, 0.95),
indicating that Trait A and Trait B likely are not distinct.
It should be kept in mind, however, that the three correlations that esti-
mate the relationship between different traits measured with the same method
were inflated by method variance. Structural equation modeling can reflect
this method variance by calculating correlations of the same method across
traits, shown by the two-headed arrows in Figure 13.4. Essentially, paths have
.37 A1
.78
.34 A2 .79
.25 Trait A 1.00
.47 A3 .76
.32 .76
.38 B1 .81
.37 Trait B 1.00
.48 B2 .69
.71
.50 B3
been added to the model to take method variance into account. The correlations
0.25, 0.32, and 0.37 in the figure reflect the method variance for methods
1, 2, and 3, respectively. As expected, the correlation between the traits
dropped from 0.95 to 0.76 when the method variance was modeled, indicat-
ing that the traits are distinct, although not independent.
Method variance appears often in counseling research when various
aspects of counseling are rated from the same perspective. For example, if the
supervisor rates both cultural competence of the counselor and therapeutic
progress of the client, then the correlation between the cultural competence and
outcome is influenced in part by the rating perspective. If the supervisor has a
generally favorable attitude toward the counselor, then that supervisor will
tend to rate all aspects of the counselor and the client as positive.
Reactivity
The dependent variable should be sensitive to some characteristic of the par-
ticipant, but the assessment process itself should not affect the characteristic
directly; that is, the dependent measure should indicate how the participant
functions normally. Sometimes, something about obtaining scores on the
dependent measure alters the situation so that “false” readings are obtained.
Variables that affect the characteristics of the participants they are intended to
measure are said to be reactive. For example, a test-anxious participant may
report increased anxiety on a self-report instrument because completing the
instrument is like taking a test; an aggressive child may display less aggressive
behavior when being observed by an adult than at other times; a person may
smoke less when asked to record the number of cigarettes smoked; a counseling
trainee may demonstrate more culturally sensitive behaviors because he or she
is aware of being observed. Clearly, the reactive nature of dependent variables
must be considered in designing research; again, knowledge of the substantive
area is vital. Later in this chapter we discuss unobtrusive measures, which are
designed to be nonreactive.
Procedural Considerations
A number of procedural issues must be considered when selecting or designing
the dependent variable. Often, the time involved with the assessment is critical to
the success of the study. Participants will be reluctant to volunteer for studies that
demand a long time to complete forms and instruments, or if they do volunteer
they may respond carelessly to items (increasing error variance), especially toward
the end of a long assessment period. As mentioned previously, the readability of
instruments is critical to their psychometric performance. Any instrument admin-
istered should be checked to ensure that participants can adequately read the
materials. Often the manuals of published tests contain references to the reading
level required. Alternatively, the researcher can use one of the relatively easy-to-
use methods for determining readability (see Klare, 1974–1975).
The order of the administration of instruments can have an effect on
responses; one instrument may sensitize or otherwise influence responses on
another instrument. An instrument that draws attention to a participant’s own
pathology (for example, the Minnesota Multiphasic Personality Inventory)
may well affect how the participant rates other measures (for example, the
counselor on the CRF). Order is also important when the same instrument is
administered repeatedly. Performance at a given time may be due to previous
responding (a testing effect) rather than to the amount of the characteristic.
For instance, on an intelligence test, participants may acquire knowledge
about specific questions or tasks (such as picture completion) that improves
performance, even though intelligence remains unchanged.
When repeated measures are used in a study, use of alternative forms is
desirable if testing effects are anticipated. Alternative forms enable the researcher
to give a pretest and a posttest without having to use the identical instrument.
332 part 3 methodological issues
Students often ask questions about what to include when describing the
psychometric properties of an inventory for something like a Methods section
of their thesis, dissertation, or journal article. Although all of the above con-
ceptual issues are essential considerations in the design of research, when de-
scribing the psychometric properties of a particular instrument that will be
used, authors typically provide a description of the inventory, roughly in the
following order:
1. Description of measure itself
a. Instrument name
b. Acronym
c. Author(s)
d. Key references
e. A brief description of the construct the instrument measures
f. Self-report, behavioral observation, interview, or Internet
g. Number of items and examples of items
h. Type of items (e.g., Likert items)
i. Factors or subscales, and their definitions
j. Indication of the direction of scoring, and what a high score means
2. Validity estimates
a. Convergent and discriminant validity
b. Samples on which measure was validated
3. Reliability estimates
a. Cronbach’s alpha coefficients
b. Test-retest (if applicable)
c. Reliability is a property of the scores resulting from a particular admin-
istration of a test, so researchers should report reliability estimates for
the current data set.
Readers are encouraged to consult Heppner and Heppner (2004) for spe-
cific written examples of instruments.
other methods of classification are discussed. Note also that these seven data
collection methods may sometimes overlap; the interview method also is a
form of self-report, and ratings of other people are sometimes behavioral
measures. The main point is that there is a broad range of data collection methods,
each of which has its respective advantages and disadvantages. Finally, each
method of data collection should be evaluated in terms of its congruence with
the research question, psychometric properties, relation to other methods used
in the study, reactivity, and so forth.
Self-Reports
In self-report measures, the participant assesses the degree to which some
characteristic is present or to which some behavior has occurred. The self-
report may be accomplished by responding to items in an inventory, complet-
ing a log, or keeping a journal. In any case, the sine qua non of self-reports is
that participants themselves make the observation or report. Generally, the
assumption is made that the report accurately reflects the true state of af-
fairs—that participants respond honestly and accurately. In this section we
discuss advantages, disadvantages, types of inventories, and scoring formats
for self-report inventories.
and smoothness of counseling sessions (see, for example, Stiles, Shapiro, &
Firth Cozens, 1988).
Ratings of other persons and events share many of the advantages of self-
reports, particularly their ease of administration and flexibility. When raters
are experts, their judgments are particularly valuable because they are made
with a rich background and deep understanding. Experienced counselors’
judgments take into account years of experience with many types of clients.
Another advantage is that many rating scales (for example, the SEQ) have
proven psychometric properties under various conditions.
The primary problem with ratings of other persons and events is that the
ratings may be systematically biased. This is especially a problem when the raters
are aware of the hypotheses and cognizant of the conditions to which partic-
ipants belong. If counselors are raters who also are involved in the experimental
treatment, they may rate the progress of clients higher because they have an
interest in the outcome of the study. If at all possible, raters should be blind to
as many factors of the experiment as possible.
When raters are used to make judgments about events, the ratings can
reflect characteristics of the rater as well as those of the event. When the par-
ticipants (counselors and clients) judge the depth and smoothness of interviews
on the SEQ, they are actually reporting their perceptions of the interview, and
in that respect their ratings are self-reports. Thus, when interpreting ratings of
events (or of other persons, for that matter), researchers must be careful to
separate the variance due to differences in the event from the variance due to
the raters themselves.
One strategy to examine the variance due to raters is to use neutral or
multiple observers, and then test for differences across raters. For example,
Hill, Carter, and O’Farrell (1983) compared observers’ ratings as well as the
counselor’s and the client’s ratings of counseling sessions.
Another problem with ratings is that because they often are relatively gen-
eral, it is not possible to determine what led to them. In the SEQ, raters respond
to the stem “The session was” on seven-point scales anchored by adjectives such
as “bad/good,” “dangerous/safe,” and “difficult/easy.” However, it is unknown
which aspects of a session lead to a rater’s responding with “difficult” as opposed
to “easy.”
An imaginative way to use ratings of events is to have participants
respond to a stimulus and rate their responses in some way. Tracey, Hays,
Malone, and Herman (1988) used the Therapist Response Questionnaire to
obtain counselors’ reactions to various client statements. The counselors indi-
cated how they would normally respond, and then these responses were rated
on eight dimensions: dominance versus submission, approach versus avoid-
ance, focus on cognition versus affect, immediacy, breadth versus specificity,
the extent to which the counselor met the client’s demand, verbosity, and con-
frontation. In this way, Tracey et al. were able to obtain a set of counselor
responses to various client statements and then obtain additional dimensional
ratings on those counselor responses, which allowed for greater precision and
interpretability of the ratings.
chapter 13 designing or choosing the dependent variable 337
Behavioral Observations
Behavioral measures are derived from observations of overt behavior, most
typically by a trained observer. Behavioral psychology has emphasized the
importance of overt behavior and deemphasized intrapsychic phenomena.
Accordingly, observing and recording behavior is the key component of ap-
plied behavior analyses. (See the Journal of Applied Behavior Analysis for ex-
amples of this type of research.) Essentially, behavioral observations are the
same as ratings of other persons or events, except that behavioral measures
focus on overt, observable behavior and presumably do not rely on inferences
by raters.
As is the case with other modalities of assessment, behavioral assessment
encompasses a wide variety of methods (Barlow, 1981; Mash & Terdal,
1988). Generally, behavioral assessment requires an operational definition of
the behaviors of interest, direct observation of participants’ behavior, recording
of occurrences of the targeted behavior, and some presentation or summa-
rization of the data.
The general advantages of behavioral observations are that they are direct
and objective measures. Although there can be systematic biases in the ob-
servation and recording of overt behavior, behavioral measurements are not
typically as subject to the personal biases inherent in self-reports. Another
advantage of behavioral measures is that participants can be assessed in various
environments. Studies have repeatedly shown that behavior is situation-specific;
behavioral measures can be used to assess functioning in several situations.
Finally, for many dysfunctions the behavior itself is problematic (for example,
stuttering, social skills deficits, sexual dysfunction, physical avoidance, substance
abuse) and thus warrants specific attention.
Among the disadvantages of behavioral observations is the fact that problems
and concerns of clients frequently do not center around readily observable
behavior. Marital satisfaction is a construct that is difficult to operationalize
behaviorally (although there are many behavioral correlates of marital satisfaction).
The central question, as with any operationalization, is whether the behavior
chosen reflects the construct of interest. Another disadvantage of behavioral
observations is related to representativeness.
A presumption of behavioral assessment is that the behavior sampled is
representative of behavior at other times. However, for a number of reasons,
this may not be the case. For instance, nonrepresentativeness can occur when
behavior is recorded at fixed but unusual times (for example, classroom
behavior on Friday afternoons). In addition, the reactivity that results when
participants are aware that they are being observed leads to observations that
may not be representative.
Issues related to reliability are problematic for behavioral assessment. An
observer’s decision that a particular behavior occurred may be idiosyncratic
to that observer. In the context of behavioral assessment, these reliability
issues are judged by calculating indexes of agreement; that is, how well do
observers agree about the occurrence of targeted behavior? As was the case
338 part 3 methodological issues
Physiological Indexes
Biological responses of participants can often be used to infer psychological
states. Many psychological phenomena have physiological correlates that can
be used as dependent variables. In fact, physiological responses often can be
thought of as direct measures of a construct. For example, whereas self-
reports of anxiety can be biased by a number of factors, measures of physio-
logical arousal can be made directly and can be presumed to be free of bias.
However, although physiological arousal is a focus in the theoretical concep-
tualization of anxiety, the relation between physiological states and psycho-
logical phenomena is not as straightforward as was anticipated in the early
years of this research. Moreover, physiological measures are expensive, re-
quire special expertise, may be reactive, and may be subject to error due to a
number of mechanical and electronic factors (such as electrical interference).
As a result, physiological measures are infrequently used in counseling research.
chapter 13 designing or choosing the dependent variable 339
Interviews
Interviews are a straightforward means of obtaining information from partici-
pants. In Chapter 11 we discussed using interviews in qualitative research.
Essentially, the process of using interviews to obtain data on a dependent vari-
able is similar, except that the goal is to quantify some construct. In everyday
life, interviewing is a pervasive activity; we simply ask people to supply infor-
mation. Interviews typically involve an interpersonal interaction between the
interviewer and the interviewee or participant. Kerlinger (1986) advocated
using personal interviews because of the greater control and depth of infor-
mation that can be obtained. The depth of information most often results
from carefully planning and developing the interview schedule.
Personal interviews allow flexibility in questionnaire design; the interviewer
can provide explanations (and thus reduce participant confusion), make deci-
sions during the interview about the adequacy of a particular response (and
probe if necessary), and evaluate the motivation of the participant. The flexibil-
ity of the personal interview can be a real advantage if the topic is complex and
if participants are unaware of their psychological processes; interviewer probing
can then be extremely beneficial and add considerable depth to the information
obtained. Babbie (2001) also observed that personal interviews that are prop-
erly executed typically achieve a completion rate of at least 80–85% of the
participants targeted. Even though interviews rely on the self-report of the par-
ticipant, the human interaction with the interviewer provides another facet to
the self-report. In short, the interviewer can also make observations about the
participant, which is an additional data source (Babbie, 1979).
Interviews, however, are costly in terms of money and time. If the topics are
sensitive (for instance, sexual behavior), then participants may be more reluctant
to divulge information than if they were allowed to respond to an anonymous
questionnaire. Interviewers must be recruited and trained. It is also important in
quantitative research to standardize procedures across interviews to avoid intro-
ducing confounding variables due to different interviewer behavior or biases.
Often considerable training is needed to standardize procedures (general
greeting, introduction of the interview schedule, methods of recording exact
responses, manner of asking questions, responses to participants’ questions,
handling of unusual participant behavior, and termination of the interview).
Thus, interviewer training is another task for the experimenter (see Babbie,
1979, for more details regarding interviewer behavior and training).
340 part 3 methodological issues
Projective Techniques
The rationale behind projective techniques is that participants’ responses to
ambiguous stimuli will reveal some facet of their personality. The Thematic
Apperception Test (which uses ambiguous drawings) and the Rorschach
(which uses inkblots) are probably the two most well-known projective tests.
However, a wide variety of possibilities exist, including drawing pictures, writing
essays, completing sentences, playing with dolls, associating words, and so
forth. The assumption is that because the method is indirect, participants will
not censor themselves. In turn, participants’ responses are indirect measures
and need to be interpreted in some way. Scoring of projective tests is typically
subjective, although there are some very objective systems for scoring them,
such as the Exner system for scoring Rorschach responses (Exner, 1974).
Historically, projective techniques have been associated with psychody-
namic approaches to understanding human behavior. As the popularity of
psychodynamic approaches has decreased, however, so has the use of projec-
tive techniques. One of the most troublesome aspects of these techniques is
that their scoring is subject to systematic biases that tend to confirm precon-
ceived (but incorrect) conceptions about people (see, for example, Chapman
& Chapman, 1969). Furthermore, the connection between underlying per-
sonality characteristics and overt behavior is tenuous.
In spite of their disadvantages, some forms of projective techniques have
made useful contributions to several areas of counseling research. For instance,
one of the conspicuous themes in counseling research involves the matching of
environmental structure with conceptual level (Holloway & Wampold, 1986).
The conceptual level theory states that high-conceptual thinkers will perform
best in low-structured environments, whereas low-conceptual thinkers will per-
form best in high-structured environments. Studies in this area typically have
used the Paragraph Completion Method (PCM; Hunt, Butler, Noy, & Rosser,
1978) to measure conceptual level. The PCM asks participants to respond to six
sentence stems; scores are based on the cognitive complexity of the responses.
Unobtrusive Measures
To eliminate reactivity, it is often possible to collect data on participants without
their awareness of this process. Measures used in such a way that participants
chapter 13 designing or choosing the dependent variable 341
Although there are several types of validity, the projective techniques, and unobtrusive measures.
one most germane to research design is construct Ratings of other persons and events are useful
validity, the degree to which scores reflect the because experts or participants can be used to
desired construct rather than some other con- judge important aspects of counseling, such as
struct. Establishing construct validity is compli- the counseling interview itself. Behavioral mea-
cated and indirect, but nevertheless vital to the sures reflect overt behavior and thus are not sub-
integrity of a study. Construct validity can be ject to the distortions that can plague self-reports
investigated in a number of ways, including recent and ratings of other persons and events; further-
applications of structural equation modeling. more, they are compatible with behavioral ap-
Commonly, a single dependent variable is unable proaches to counseling, even though they may
to adequately operationalize a construct; multiple be incompatible with other approaches (such
dependent variables are often recommended. as psychodynamic approaches). Physiological
The hope is that each variable reflects some aspect responses can be used to infer psychological
of the construct of interest, and that together states because many psychological phenomena
they measure the essence of the construct. In any (for example, anxiety) have physiological corre-
study, the researcher must be cognizant of both lates; however, due to lack of reliability, signifi-
the attenuation of true relationships due to unre- cant expense, and other problems, physiological
liability and the inflation of true relationships indexes are infrequently used in counseling re-
due to method variance. However, the depen- search. Interviews are advantageous because much
dent variables are designed or chosen such that information can be obtained quickly and because
they do not react with the treatment. the interviewer can pose follow-up questions,
There are many methods of collecting data but they are relatively expensive, depend on the
related to dependent variables, each of which has skill of the interviewer, and can be biased. Pro-
its advantages and disadvantages. The most widely jective techniques, which use ambiguous stimuli to
used measure in counseling research is the self- reveal some facet of personality, can be useful to
report. The sine qua non of the self-report is that uncover unconscious aspects of the personality.
each participant makes his or her own observa- Unobtrusive measures are designed to eliminate
tions or reports. The advantages of self-reports reactivity because the participant is unaware that
are that they are relatively easy to administer, can any measurement is being conducted.
access areas that otherwise would be impossible Given the multitude of data collection methods,
or difficult to measure (such as sexual behavior), the task of the informed researcher is to collect data
and are compatible with phenomenological views with a method that provides the type of informa-
of counseling. The most conspicuous problem with tion that is most relevant to the research question.
self-reports is that they are vulnerable to distor- Obviously, the selection of the dependent
tions by the participant. However, participants variable and the method of data collection
may not be consciously aware of the construct require considerable forethought and examina-
being measured, and self-reports are incompati- tion of the research literature. Moreover, these
ble with several theoretical approaches to coun- tasks often require creative thinking to tailor
seling (for example, behavioral approaches). measurements to the constructs of interest. Un-
Self-report instruments may either be published, fortunately, sometimes researchers spend very
by professional publishers or in the literature, or little time in selecting dependent variables, and
tailor made for a specific study, and they can be weak and disappointing findings often result. We
written in a number of formats. firmly believe that careful deliberation and con-
Less frequently used dependent measures in- sultation with colleagues can greatly facilitate
clude ratings of other persons and events, behav- the selection of dependent variables and enhance
ioral measures, physiological indexes, interviews, the overall quality of research in counseling.
chapter 13 designing or choosing the dependent variable 343
STIMULUS QUESTIONS
The Dependent Variable
1. Randomly select four recent counseling research 3. As discussed in this chapter, the nature of
articles. Classify the dependent measures the sample on which a measure was normed
according to the method of data collection is important. Consider measures commonly
(self-report, ratings of others or events, be- used in outcome research (e.g., the Beck
havioral observation, physiological indices, inter- Depression Inventory, the Brief Symptom
views, projectives, or unobtrusive measures). Inventory, the Outcome Questionnaire) and
Discuss the adequacy of the method used to characterize the samples on which the mea-
capture the nature of the construct in question. sure was normed and validated.
2. Select a commonly used measure in your area 4. Suppose a researcher is interested in the con-
of interest. Research the development and struct of cultural competence and wishes to
validation of the measure and discuss the avoid self-report measures. Indicate how the
adequacy of the measure for the purpose it researcher could use raters for this purpose.
was intended. What issues would be considered?
14 CHAPTER Population Issues
344
chapter 14 population issues 345
(4) establishing the validity of research in the absence of random selection, and
(5) determining the number of participants. Finally, we examine the relation-
ship of external validity and population issues by considering factorial designs
involving factors related to person or status variables.
SAMPLING THEORY
Selecting participants for a study typically involves selecting samples from a
population of interest. For example, it would be too cumbersome for an inves-
tigator interested in homophobia to interview all Americans about homophobia,
so instead the investigator selects a sample of participants that presumably
reflects the American population as a whole. For example, Barrett and McWhirter
(2002) conducted an analogue study of how client sexual orientation and
counselor-trainee homophobia and gender influence perceptions of clients. The
researchers solicited responses from a sample of counseling graduate students
so that they might be able to make conclusions about counseling students in
general. However, as will be described below, a number of practical concerns lead
one to question whether this was accomplished in this and other related studies.
Sampling theory provides the foundation for understanding the process
and the implications of selecting participants for a particular study. We briefly
discuss sampling theory and elucidate some of the real life restrictions and
subsequent problems that investigators encounter. The essence of sampling
theory involves selecting samples that reflect larger or total populations. We
typically think of a population as a well-defined set of people, such as college
students seeking help at a counseling center, depressed adolescents, or
counselors-in-training, but technically a population is a set of observations.
Put another way, it is the observations (or scores) of the people, rather than
the people themselves, that constitute the population. The important aspect of
populations, whether viewed as people or observations, is that conclusions
reached from the research sample should apply to the population. By neces-
sity, counseling research is conducted with a limited number of participants;
the results for these particular participants alone are rarely of primary inter-
est. The object of most research is to generalize from the observations of these
study participants to some larger population; that is, some inference is made
about the population based on a small number of observations.
The concept of population, however, is elusive. Some populations are
quite real. For example, consider a population that includes the cumulative
grade point averages of all college students enrolled as of January 3, 2005,
and who have completed at least one term of college. The grade point aver-
ages (that is, the observations) exist and can be obtained from student records.
The size of the population in this instance is fixed and finite, although quite
large. Other populations are more ambiguous. For example, examination of
depression in college students might involve a population that includes
scores on the Beck Depression Inventory (BDI; Beck, Ward, Mendelson,
Mock, & Erbaugh, 1961) for all college students enrolled as of January 3, 2005.
346 part 3 methodological issues
Clearly, not every college student has taken the BDI, so in some sense this is
a hypothetical population. Nevertheless, it is not difficult to imagine having
each of these students take the BDI; the population would consist of all these
scores. However, it probably would be unwise to limit the population to students
enrolled as of January 3, 2005, because to be useful the results of the study
should be applicable to students enrolled at different times. A truly hypothetical
population might involve college students enrolled both currently (at the time
of the research) and in the future. This hypothetical population is infinite.
Clearly, some problems arise when generalizing to infinite hypothetical
populations, some of whose scores exist in the future; however, it is just as clear
that limiting conclusions to populations in existence only at a given time re-
stricts the generalizability (and practical utility for that matter) of the results.
Inferences about populations are made on the basis of samples selected
from populations. Technically, a sample is a subset of the population; that is,
the observations in the sample are taken from the set of observations that con-
stitute the population. This process is called sampling. Again, inferences about
the population of observations are made from the observations of the sample;
the validity of the inferences about the population depend on how well the
sample in fact represents the population. Representativeness is a complex con-
cept that requires further explanation.
Certainly, selecting 20 males at an Ivy League college and recording their
scores on the BDI would poorly represent the population of BDI scores for all
college students nationally. Samples that systematically differ from the popu-
lation in some way are said to be biased. More technically, a biased sample is
a sample selected in such a way that all observations in the population do not
have an equal chance of being selected. In the example of male Ivy Leaguers,
the sample is biased because female students do not have the same chance of
being selected as males (that is, the probability of selecting a female is zero),
and students in non-Ivy League colleges do not have the same chance of being
selected as students in the Ivy League. However, as we shall describe, sample
bias often operates in much more subtle ways than the above.
Samples that are not biased are random samples—that is, samples in which
each observation in the population has an equal chance of being selected.
Logistically, random samples can be selected by assigning each observation a
consecutive number (1, 2, 3 . . .) and then choosing the observations by select-
ing numbers from a random numbers table or by using a computer-assisted
random numbers generator. To randomly select a sample of size 20 from all
college students, each student could be assigned an eight-digit number, a com-
puter could be used to generate 20 eight-digit random numbers, and the BDI
scores for the students whose numbers were generated would compose the
sample. Clearly, this would be a laborious process (and could never realisti-
cally be accomplished), but it illustrates the ideal that researchers seek to
approach when utilizing random selection.
Although random selection eliminates systematic bias, there is no guar-
antee that a random sample will be representative of the population. For
example, even though it is highly unlikely, the random selection process just
chapter 14 population issues 347
Population
Random selection
Sample
Random assignment
Null hypothesis μT = μC
Selecting Participants
The next step is to determine those participants in the participant pool who will
participate in the study. Ideally, participants are randomly selected from the par-
ticipant pool. For example, if the participant pool comprised students seeking
help at a counseling center at a particular university, then the researcher could
assign each such student a number and, with the aid of a random numbers table
or a computer-assisted random numbers generator, randomly select the partici-
pants for the experiment. However, even this process can be pragmatically trou-
blesome. Often the researcher needs all participants to be at the same stage of
counseling, but if there are not enough qualified clients, the researcher may
solicit participants as they become available. For example, Tracey, Glidden, and
Kokotovic (1988) had university counseling center clients evaluate their coun-
selor immediately after the intake session. To obtain a sufficient sample, the
researchers asked all clients who presented themselves at the center to partici-
pate in the study; in all, 192 of 430 clients agreed to participate and completed
the study. Thus, in this study, random selection from a participant pool did not
occur because all available participants were used.
in one context, that the treatment works; future researchers should examine
generalizability.
Now consider a hypothetical status study whose purpose is to compare
the anxiety, stress, and academic performance in two samples—European-
American and African-American adolescents. Suppose that samples of
European-American and African-American students were obtained from a
school to which the European-American students were transported from sub-
urban areas, and that significant differences were found between the groups
in levels of anxiety, stress, and achievement. Because there was no random
assignment, the two populations might well have differed on many character-
istics other than ethnicity, such as socioeconomic status (SES), family status,
parental involvement and supervision, community crime rates, and so forth.
Consequently, it would be difficult to attribute the differences in dependent
variables to ethnicity, and therefore the sampling method causes a problem for
both internal and external validity. It might well be that the differences
between the two groups of students were due to the characteristics mentioned,
and had the researchers selected samples that held these factors constant
(which would be extremely difficult), the differences would not have been
present. The point here is that when status variables are used, internal as well
as external validity problems arise, whereas if a study involves a true inde-
pendent variable, only external validity is problematic.
Because the generalizability of results relies on the characteristics of the
participants, researchers must carefully document the important characteris-
tics of the sample. Given that race and ethnicity are always important consid-
erations in generalizability for studies in the United States, Ponterotto and
Casas (1991) made the following recommendation:
Knowing simply the ethnic make-up and mean ages of one’s sample is insufficient
in assessing result generalizability. Describe the sample fully: mean and median
age; educational level (and in immigrant groups, where the education was re-
ceived); socioeconomic status; gender; preferred language and level of accultura-
tion in immigrant samples; the level of racial identity development; geographic
region of the study, and any other sample characteristics you believe your reader
would consider when interpreting the results. As a rule of thumb, the more accu-
rately you can describe your sample, the more accurate you can be in determining
the generalizability of your results. (p. 107)
for exclusion except when the study would duplicate data from other sources. . . .
Under this statute, when a . . . clinical trial is proposed, evidence must be reviewed
to show whether or not clinically important gender or race/ethnicity differences in
the intervention effects are to be expected. (NIH, 1994, p. 14509)
The directionality of the test also affects power. If a two-tailed (that is,
nondirectional) test is used, the researcher reserves the option of rejecting the
null hypothesis in either direction. This is helpful when a researcher is inter-
ested in results in both directions and/or is unclear about the direction. For
instance, when comparing two treatments, knowing whether Treatment A or
Treatment B is superior is important; however, keeping options open in both
directions costs the researcher power because it is more difficult to detect
effects in this case than when one direction or the other is specified. One-tailed
(directional) tests are more powerful when the effect is in the expected direction.
For example, when testing the efficacy of a treatment vis-à-vis a control group,
it makes sense to test only whether the treatment is more effective than no treatment
(one is rarely interested in knowing whether the treatment is less effective than
no treatment). By specifying the direction (that the treatment is superior to no
treatment), the researcher increases the power of the statistical test.
The most difficult factor to specify in any determination of power is the
size of the true effect. When a treatment is extraordinarily effective, the effect
of the treatment is relatively easy to detect, and thus power is high. For example,
if a treatment of depression reduces self-deprecating statements from an average
of 20 per hour to zero, achieving a statistically significant finding will be easy.
However, if the reduction is from an average of 20 self-deprecating statements
to 18, then detecting this small change will be difficult. Specifying the size of
the effect before the study is conducted is problematic—if one knew the effect
size for any experiment beforehand, there would be no need to conduct the
study. Nevertheless, the effect size must be stipulated before the number of
participants can be determined. The effect size can be stipulated in a number
of ways. First, prior research in relevant areas often provides clues about the
size of effects. For instance, if the effect of cognitive-behavioral treatments of
test anxiety is known to be a certain size, it is reasonable to expect that the
effect of a cognitive-behavioral treatment of performance anxiety would be
approximately the same size. Haase, Waechter, and Solomon (1982) surveyed
the effect sizes obtained in the counseling psychology research in general,
although it is unclear how applicable these results are to specific areas within
the field. A second way to stipulate effect size is to specify the effect size con-
sidered to have practical or clinical significance. For example, in a treatment
study involving a treatment group and a control group, the researcher might
want to stipulate the percentage of those treated that exceeds the mean of
those untreated. Using normal distributions, it can be shown that an effect size
of 1.00 indicates that at the end of treatment, 84% of the treatment group
functioned better than the mean of the control group (assuming normality);
an effect size of 1.5 indicates that 93% functioned better than the mean of the
control group; an effect size of 2.0 indicates that 98% functioned better than
the mean of the control group. Translation of effect size into indexes of clini-
cal improvement allows the researcher to gauge how large the effect must be
to have clinical significance. Finally, based on a number of considerations,
Cohen (1988) classified effects into three categories: small, medium, and
large. This scheme makes it possible for a researcher to determine the number
chapter 14 population issues 357
of participants needed to detect each of these three effect sizes. Of course, the
researcher must still stipulate which of the three sizes of effects the study
should detect. Furthermore, Cohen’s determination of effect size is arbitrary
and cannot apply equally well to all areas of social and behavioral research.
Nevertheless, in the absence of other guiding lights, stipulation of a “medium”-
sized effect has guided many a researcher.
The last determination needed before deciding how many participants to
use in an experiment is the level of power desired. Power of 0.80 has become
the accepted standard (although, again, this level is arbitrary). A level of
power of 0.80 refers to a probability level; that is, 80% of the time the stipu-
lated effect size will be detected (i.e., the test will be statistically significant).
It also means that there is a 20% chance that no statistically significant results
will be found when the effect in fact exists!
Once the researcher has selected the statistical test to be used, chosen
whether to use a one-tailed or two-tailed test, set alpha, stipulated a desirable
level of power, and determined the effect size to be detected, he or she can ascer-
tain the number of participants needed to obtain the stipulated level of power.
Typically this is accomplished by using tables, such as those provided by
Cohen (1988) or Kraemer and Thiemann (1987). Cohen provided extensive
examples, but his format uses different tables for different tests. By using ap-
proximations, Kraemer and Thiemann were able to reduce the complexity of the
process of determining the number of participants needed. Perhaps the simplest
way to make this important determination is to use computer programs de-
signed for this purpose (see, for example, Borenstein & Cohen, 1988).
Some caveats are needed about determining sample size. First, all of the
procedures presume that the assumptions of the chosen statistical test are met.
When assumptions are violated, power is typically decreased, so beware. Second,
even though one often hears rules of thumb about sample sizes—10 participants
for each variable in a multiple regression, 15 participants to a cell in a facto-
rial design, and so forth—be warned that such rules are almost always mis-
leading, as Table 14.1, an abbreviated power table for multiple regression,
shows. In some instances fewer than 10 participants per variable are needed,
and in other instances many more than 10 are needed. Third, the general rule
that the more participants for an experiment, the better, is also misleading.
Certainly, the researcher wants to have a sufficient number of participants to
have a reasonable opportunity (say 80%) to detect an effect of a specified size.
However, using too many participants raises the possibility that a very small
effect size can be detected (see Meehl, 1978, for an excellent discussion of this
issue). Although small effects can be interesting, they often mislead the researcher
into believing that something important has occurred when in fact only a triv-
ial finding has been obtained. Given a large enough sample, a researcher
might find a significant (although meaningless) correlation between, say,
hours of television watched and shoe size. An interesting exercise might be to
obtain the World Values Survey (n > 200,000) available from www.worldval-
uessurvey.org and attempt to find two variables that are not significantly
related. For example, in a regression problem, a statistically significant finding
358 part 3 methodological issues
Table 14.1
Number of independent variables
Number of
Participants K=3 K=6
Needed to
Achieve Various R2 Power = .30 .50 .70 .90 .30 .50 .70 .90
Levels of Power
(α = .05) .10 34 56 83 132 47 74 107 164
.30 11 17 25 37 17 24 33 45
.50 7 10 13 18 11 14 18 24
with a large number of participants that accounts for only 2% of the variance
in the dependent variable will likely add little to our understanding of psy-
chological processes. Because statistical significance can be obtained for triv-
ial effects, it is often recommended that researchers report effect size and
power in addition to significance levels (Fagley, 1985; Folger, 1989; Shadish,
Cook, & Campbell, 2002).
Independent variable
I II III
I
Persons
II
To make this strategy more clear, consider the three levels of the inde-
pendent variable to be three treatments, and the two levels of the status vari-
able related to persons to be gender. Interpretation of the main effects and the
interaction effects of this factorial design will illustrate how it establishes the
generality of the results across persons.
Treatments
Treatment A Treatment B Treatment C
Males
Gender
Females
Suppose that it was found that there was no treatment effect; that is, there
was insufficient evidence to establish that one treatment was more effective
than any other. External validity involves answering the question of whether
this result applies equally to males and females. It may well be that there was
no main effect for gender as well (e.g., no effect of gender across treatments).
However, the presence of an interaction (e.g., does the effect of a particular
treatment vary depending on the gender of a client?) effect most clearly
addresses issues related to external validity. For example, Treatment A may
have been most effective with males, whereas Treatment C may have been
most effective with females, indicating that the results are not generalizable
across gender. Clearly, considerations of person variables can be vital to the
proper understanding of research results. Parenthetically, both gender and
racial/ethnic variables are receiving increased attention in the counseling liter-
ature (interested readers could examine Arredondo et al., 1996; Betz &
Fitzgerald, 1993; Cook, 1990; Cournoyer & Mahalik, 1995; Fouad, 2002;
Fouad & Mohler, 2004; Good, Gilbert, & Scher, 1990; Good et al., 1995;
Heppner, 1995; Kim & Atkinson, 2002; Melby, Hoyt, & Bryant, 2003; Mintz
& O’Neil, 1990; Ponterotto, 1988b; Ponterotto, Rieger, Barrett, & Sparks,
1994; Richardson & Johnson, 1984; Scher & Good, 1990; Williams &
Chambless, 1994).
360 part 3 methodological issues
anxiety for the two groups different?). One of the most persistent but contro-
versial findings in psychology involves just such a comparison—namely, the
group differences found on traditional tests of intelligence.
The research design point that we want to make is that simply identifying
group differences on some variable is of limited usefulness; ultimately, the goal
of research is understanding, and research could be profitably used to explore
observed differences among groups.
When differences between groups are found, a vital question arises: Are
the processes that lead to these differences the same for both groups? Suppose
there are two groups, A and B, that reliably show differences in some con-
struct, say antisocial behavior (Group A > Group B). A critical piece of the
puzzle is to determine whether the factors that lead to antisocial behavior in
the two groups are the same. Given that we know inadequate parental super-
vision to be a key causal construct in antisocial behavior, the question could
be expressed as follows: “Is the rate of antisocial behavior in Group A higher
because there is less adequate supervision, or because there is some other vari-
able that uniquely influences behavior in Group A but not Group B?” In a
comprehensive reanalysis of several large data sets, Rowe, Vazsonyi, and
Flannery (1994) found racial similarity in developmental processes related to
achievement and behavior; that is, they found that the variables that predicted
achievement and behavior were common across groups, and that no variables
were uniquely important to specific racial groups.
Clearly, then, if a researcher is going to examine group differences, an
understanding of the processes that lead to those differences is paramount to
making sense of the results. Basically, if an examination across populations
reveals that the causal factors are similar, then the efficacy of interventions
would be similar; if causal factors are not similar, then group-specific interven-
tions would be needed. As is becoming clear, such research is needed to establish
the efficacy of culture-specific educational and psychological interventions.
As an example of systematic examination of group differences, Fouad
(2002) has investigated the cross-cultural validity of the Strong Interest In-
ventory and the RIASEC model of career interests, demonstrating basic sup-
port for similarity of interests and structure of interests. Although these findings
have made important contributions to our understanding of career interests
across cultures, they do not comment on the processes through which interests
may have developed cross-culturally.
As an example of unique processes in different groups, Eugster and
Wampold (1996) found that the factors that predicted global evaluation of a
counseling session were different for the counselor and the client. Although
clients and therapists had some factors in common, the researchers found that
clients’ global evaluation of a session was positively related to their perceptions
of therapist interpersonal style and of the real relationship, whereas therapists’
evaluations were positively related to their perceived expertness and negatively
related to perceptions of the real relationship. These differences were verified
by statistically testing the differences in regression equations. The goal of this
research was not to determine whether there were mean differences between
chapter 14 population issues 363
client and therapist evaluation of sessions, but to determine whether the factors
that form the bases of these evaluations were different.
The statistical and methodological issues involved in testing common
versus unique processes are complex. Eugster and Wampold (1996) examined
the equivalence of regressions to test for unique processes. Rowe, Vazsonyi,
and Flannery (1994) present a particularly cogent explanation of research issues
related to testing models involving the hypothesis that psychological processes
differ among groups.
Another, although subtle, issue related to group differences involves the
proportions of participants assigned when status variables are included in a
factorial design. It is desirable in a factorial design to have equal numbers of
participants in each cell; in this way, main and interaction effects are inde-
pendent (provided the assumptions of the analysis of variance are met; see
Wampold & Drew, 1990). Accordingly, equal numbers of participants of each
type should be selected (recruited). For instance, equal numbers of males and
females should be selected and then randomly assigned to the three treatments.
For research on gender, this presents no problems; however, when the base
rates of various person variables are different, conceptual (as well as statistical)
problems occur. For example, selecting equal numbers of Native Americans
and European Americans for a study results in a sample that is not represen-
tative of the general population (that is, it is biased toward inclusion of Native
Americans). This is problematic because the results are peculiar to the manner
in which the sample was obtained. For example, correlations are affected by
the proportions of types of participants included (see Cohen & Cohen, 1983).
This is a complex issue and is beyond the scope of this text, but a simple
example will illustrate the problem. If one were conducting a study to exam-
ine how various types of political television commercials affected voting pref-
erences of European Americans and Native Americans, equal numbers of
these persons would be selected. The design of the experiment might involve
exposing participants to various types of commercials and then measuring
their attitudes. The results of this study might be theoretically interesting and
applicable to real political campaigns. However, if one were polling to deter-
mine the expected outcome of an election, one would not want to bias the sample
by having equal numbers of Native Americans and European Americans in the
sample. Generally, experimental studies that examine theoretical phenomena
or treatment efficacy should have equal numbers of participants. However,
for studies that examine relations between variables in society, participants
should be selected so that the proportions of the various person variables reflect
the proportions in the population.
The factorial design approach to external validity also has philosophical
implications for various classes of persons. Typically, studies are either con-
ducted on predominantly majority samples (for example, European Americans)
or include person variables that contrast a majority sample with a minority
sample (for example, European Americans versus African Americans). The
latter design includes the assumption that European Americans are somehow the
norm, and that all other groups are to be contrasted with them (see Delgado-Romero,
364 part 3 methodological issues
Galván, & Maschino, 2005; Heppner, Casas, Carter, & Stone, 2000; Ponterotto,
1988b). It is also assumed that each of these groups is homogeneous. African
Americans comprise a diverse population, and typically such diversity (in, for
example, level of acculturation, racial identity, and generational status; again,
see Delgado-Romero, Galván, & Maschino, 2005; Heppner et al., 2000;
Ponterotto, 1988b) must be considered in research designs. Furthermore, there
are phenomena that may be culture-specific and for which it is not optimal or
even sensible to use a design that contrasts various groups. For instance, the
reasons for the underutilization of mental health clinics by ethnic minorities,
which is a major concern to providers of service, may best be understood by
examining intra-ethnic processes rather than ethnic differences.
External validity is often established using multiple studies. For example,
single-participant designs preclude the inclusion of person variables because only
one or a few participants are used. Therefore, researchers use a strategy called
systematic replication (Barlow & Hersen, 1984), which involves replicating an
experiment while varying a single element. By systematically varying the one
element over time, the researcher can identify its effects. For example, a single-
participant design that establishes the efficacy of social reinforcement in the
attenuation of off-task behavior with students in a primarily European-American
urban school might be replicated in a rural school with a European-American
student body, in an urban school with a Hispanic student body, and so forth.
The idea of systematic replication can apply to group studies as well. For
example, interaction effects might be identified by two studies that differed only
insofar as one had female participants and the other had male participants.
However, there are disadvantages of such a strategy. True replication is difficult
to accomplish, and thus the differences between two studies may be due to factors
other than different types of participants. Furthermore, examination of external
validity in one study allows direct estimation of the size of the interaction effect,
a procedure that is precluded in the systematic replication strategy.
With regard to estimating statistical power (and settings, or times. In this way, external validity
thus the number of participants needed), we can be investigated by examining the interaction
discussed (1) the particular statistical test used, of the independent variable and a variable related
(2) alpha level, (3) directionality of the statistical to persons, settings, or times.
test, and (4) effect size. Thus, investigators must Because counseling is an applied profession
make a number of decisions in selecting partici- designed to help a broad array of individuals in
pants for a study. Typically, researchers are inter- various settings across different times, external
ested in generalizing the results of a particular validity is very important. Far too often, con-
study to a larger population of individuals, venience samples consisting of predominantly
which is the essence of external validity. External European-American undergraduate participants
validity also relates to the generalizability of find- are used in counseling research. We strongly en-
ings across persons, settings, or times. We sug- courage efforts to broaden the external validity
gested that the most direct means to increase the of research in counseling in order to develop the
external validity of findings is to build into the more extensive databases that are needed in the
design factors that represent relevant persons, counseling profession.
STIMULUS QUESTIONS
Exercises on Population Issues
1. A researcher is considering the relative 3. Select a recent issue of a counseling journal.
efficacy of two treatments for depression. Describe the samples used in the research in
What population issues are important the issue and rate the adequacy of the sam-
when considering the interpretation of the ples vis-à-vis the purpose of the research.
results? 4. Suppose that you are designing a study to test
2. Discuss the differences between random theoretical propositions related to career de-
selection and random assignment. How are velopment. Discuss issues related to the defi-
these concepts related to the validity of a nition of the population and recruitment of
study? participants.
15 CHAPTER Conceptual and
Methodological Issues
Related to Multicultural
Research
Kevin Cokley and Germine H. Awad
366
chapter 15 conceptual and methodological issues 367
Race
Race has been defined as a “. . . presumed classification of all human groups on
the basis of visible physical traits or phenotype and behavioral differences”
(Carter, 1995, p. 15), “. . . a sociopolitical designation in which individuals are
assigned to a particular racial group based on presumed biological or visible
characteristics such as skin color, physical features, and in some cases, language”
(Carter, 1995, p. 15), and “. . . an inbreeding, geographically isolated population
that differs in distinguishable physical traits from other members of the species”
(Zuckerman, 1990, p. 1297). Most definitions of race include a biological com-
ponent, where the assumption is that people can be divided into groups defined
by similar physical features and behavioral tendencies. These groups are gener-
ally classified as Mongoloid, Caucasoid, and Negroid, where Mongoloid is de-
fined as anyone whose ancestors were born in East Asia, Caucasoid is defined as
anyone whose ancestors were born in Europe, and Negroid is defined as anyone
whose ancestors were born in sub-Saharan Africa (Rushton, 2000).
However, it has been documented that the creation of racial taxonomies has
occurred in a distinctly unscientific manner (Gould, 1994), and there does not
exist any agreed upon racial classification system. There are no physical traits
that are inherently and exclusively found in combination with other physical
traits (e.g., “black” skin is not always linked with full lips; “white skin” is not
368 part 3 methodological issues
always linked with straight hair, etc.). Most social scientists agree that race is
not a biological reality, but rather a social construct used to divide people and
perpetuate power relations and social inequities. Several researchers have
discussed at length the problematic nature of race as a biological reality (At-
kinson, 2004; Betancourt & Lopez, 1993; Helms & Cook, 1999; Zuckerman,
1990). Race is often described as a social construct because there are real social
consequences of being perceived to be a part of an identifiable racial group.
These consequences include being the targets of prejudice and discrimination,
which have long-term quality of life and mental health implications.
Ethnicity
Similar to race, there has not always been agreement or consistency on the
definition of ethnicity or ethnic group. Carter (1995) defined ethnic group as
“one’s national origin, religious affiliation, or other type of socially or geo-
graphically defined group” (p. 13). Betancourt and Lopez (1993) defined eth-
nicity as “. . . groups that are characterized in terms of a common nationality,
culture, or language” (p. 7), while Helms and Cook (1999) characterized
ethnicity as a euphemism for race, and defined ethnicity as “. . . the national,
regional, or tribal origins of one’s oldest remembered ancestors and the cus-
toms, traditions, and rituals (i.e., subjective culture) handed down by these
ancestors . . .” (p. 19). Phinney (1996) used the term ethnicity to encompass
race, and indicates that ethnicity refers to broad groups of people on the basis
of culture of origin and race. There are also broad and narrow interpretations
of ethnicity. When ethnicity is characterized as sharing cultural characteristics
and physical features, the broad interpretation of ethnicity is being used. This
interpretation of ethnicity is similar to the way in which race is discussed
(Atkinson, 2004). When ethnicity is restricted to cultural characteristics and
differences, the narrow interpretation of ethnicity is being used (Atkinson).
From our perspective, a broad interpretation of ethnicity is problematic because
it primarily relies on physically distinguishable features as markers of ethnic
group membership, and minimizes the psychological and cultural processes
believed to play a critical role in ethnic group membership. Thus, in this chap-
ter we will use a narrow interpretation of ethnicity.
Culture
Culture is perhaps the most difficult of the three terms to define because there
are well over 100 different definitions of culture found in various psychological,
anthropological, and sociological literatures. Although the definitions overlap in
many instances, it is important to note that there are some important differences.
These differences often depend on whether the scholar is defining culture psy-
chologically, anthropologically, or sociologically. The scientific study of
culture is the domain of anthropologists, who focus on objective culture (e.g.,
the human-made part of the environment such as buildings, roads, homes,
tools, etc.) (Herskovits, 1955) and subjective culture (e.g., values, beliefs, attitudes,
chapter 15 conceptual and methodological issues 369
Phenotype
Race Ethnicity
Identity
Worldview Value
Culture
© 2004 Kevin Cokley, Ph.D., kcokley@siu.edu. Created using the online Create-A-
Venn system at www.venndiagram.com.
role definitions, etc.) (Triandis, 1972). Subjective culture falls within the interests
of counselors and psychologists. Helms and Cook (1999), using a psychological
definition, defined culture as “. . . the values, beliefs, language, rituals, tradi-
tions, and other behaviors that are passed from one generation to another
within any social group . . .” (p. 24). Matsumoto (2000), also using a psy-
chological definition, defined culture as
a dynamic system of rules, explicit and implicit, established by groups in order to
ensure their survival, involving attitudes, values, beliefs, norms, and behaviors,
shared by a group but harbored differently by each specific unit within the group,
communicated across generations, relatively stable but with the potential to
change across time. (p. 24)
THEORETICAL CONSIDERATIONS
Research involving race, ethnicity, or culture goes by many names (e.g., cross-
cultural research, ethnic minority research, multicultural research). Cross-cultural
research is interested in determining which aspects of behavior are culture-
specific (i.e., emic) and which aspects of behavior are universal across cultures
(i.e., etic) (Matsumoto, 1994). Multicultural research is interested in examin-
ing emics and etics of ethnic minority and majority groups living within one
country (e.g., African Americans and European Americans living in the United
States). Typically, cross-cultural researchers are interested in examining be-
haviors between two or more countries. For example, Stipek (1998) found
that Chinese people feel more ashamed for behavior related to family mem-
bers than do Americans, and also feel more pride for an affiliated other’s
achievement than do Americans. Studies like this are published in the Journal
of Cross-Cultural Psychology, which is devoted to publishing cross-cultural
research studies; however, in some instances, multicultural research is also
published. For example, Ayalon and Young (2003) found group differences
in depressive symptomatology among African Americans and European
Americans, with African Americans reporting less pessimism, dissatisfaction,
self-blame, and suicidal ideation relative to European Americans. In another
example, Okazaki and Kallivayalil (2002) found that cultural norms about
depression were not related to depressive symptoms in European Americans,
but were related in Asian Americans.
Kurt Lewin is best known for his field theory, which proposes that human
behavior is the result of both the person’s characteristics and the environment
or social situation that a person finds him- or herself in, as represented by the
symbolic terms B = f(P, E), where B = behavior, P = personality, and E = environment.
chapter 15 conceptual and methodological issues 371
Lewin’s often-quoted statement about theory is the driving force behind theory-
driven research. Although multicultural and cross-cultural research continues
to become increasingly popular, a lingering criticism is that much of the
research is atheoretical and primarily descriptive (Betancourt & Lopez, 1993;
Fisher & Moradi, 2001). Betancourt and Lopez make the following observa-
tions about multicultural/ethnic minority research:
Ethnic minority research shares conceptual problems similar to those of cross-
cultural psychology. Direct measures of cultural elements are frequently not
included, yet cultural factors are assumed to underlie ethnic group differences.
Furthermore, ethnic minority research often lacks sufficient attention to psycho-
logical theory. It appears that investigators of ethnicity are more inclined toward
description than testing theoretically derived hypotheses. (p. 634)
Although these criticisms are in part valid, researchers who are critical of
multicultural/ethnic minority research must consider two issues. The first
issue is the sociohistorical context within which ethnic minority research
developed. Early research pathologized ethnic minorities, operating out of a
genetically deficient model whereby racial and ethnic minorities were consid-
ered to be inferior (Sue & Sue, 2003). Eventually this model was challenged
by “well-meaning social scientists” (Sue & Sue, p. 55) and replaced with the
“culturally deficient model,” which assumed that the problems of racial and
ethnic minorities could be attributed to being culturally deprived of middle-class
culture. This model was limited in that it perpetuated the “myth of minority
inferiority” (Sue & Sue, p. 56). The current zeitgeist emphasizes a culturally
diverse model, in which it is not necessary for racial and ethnic groups to be
compared with European Americans as the standard, but rather they are
understood on their own cultural terms in and of themselves.
Multicultural research was born essentially out of the necessity to conduct
research on ethnic minority populations that was not pathologically oriented or
deficit driven. In many ways, its initial purpose was solely to refute the nega-
tive nature of much of the early research conducted on racial and ethnic minority
populations. To that end, much of the research has been characterized by providing
descriptive information about racial and ethnic minority populations, and describ-
ing in nonpathological terms how they differ from European Americans.
The second issue to consider is the level of existing knowledge on a par-
ticular research topic. Some topics have been heavily researched, such as the
counseling process (see Chapter 19). Other topics, especially with racial/ethnic
minority populations, have not been researched extensively, and thus there is
presently a considerably smaller knowledge base. As suggested in Chapter 4,
sometimes atheoretical, descriptive research can be very informative for topics
where there is a small knowledge base.
Descriptive research is concerned with simply describing phenomena.
Tools of descriptive research include the use of descriptive and inferential sta-
tistics. Descriptive statistics refer to data that are summarized (e.g., means,
medians, standard deviations, percentages). For instance, calculating a mean
and standard deviation on scores from the Beck Depression Inventory for
372 part 3 methodological issues
let’s say that Mexican Americans had significantly higher scores on the Beck
Depression Inventory than European Americans, and we are interested in why
that difference exists. As previously mentioned, ethnic minority research has
been criticized for being primarily descriptive in nature. Research that is
descriptive in nature usually relies on distal factors to explain differences in
behavior. The word distal means farthest from the point of origin. Distal fac-
tors are factors that, in and of themselves, do not directly explain observed
phenomena. They are often not psychological in nature, but rather, they tend
to be demographic. For example, when researchers collect data, they usually
incorporate demographic information. The demographic information will
usually include information about race and/or ethnicity, sex, socioeconomic
status, age, and so on. When researchers analyze their data, descriptive statis-
tics are usually the first set of analyses conducted. It is common to compute
the mean scores on the variables or constructs of interest, and then, using
inferential statistics, compare demographic groups to see whether there are
statistically significant differences. When differences are found, researchers often
attribute the differences to the demographic variables themselves. Continuing
with our earlier example, the difference in depression scores between Mexican
Americans and European Americans is attributed to cultural differences.
The researchers relied on the self-reported ethnic categorization of the partic-
ipants as a necessary and sufficient explanation for the observed differences.
However, as Phinney (1996) reminds us, ethnic categorization is problematic
because of its imprecise and arbitrary nature. Ethnic categories can vary over
time and context, such that an individual can identify with one ethnic group
in a certain context and identify with another ethnic group in a different con-
text. Furthermore, even if the researchers were 100% certain that the indi-
viduals in the study categorized themselves in a way that was consistent with
societal norms, attributing the differences to ethnic status is insufficient.
Phinney stated that ethnic categories, or ethnicity, has psychological dimen-
sions. She says that ethnicity can be thought of as culture, identity, or minor-
ity status, with each dimension having a specific psychological impact.
As ethnic minority research starts to mature, it begins to move away from
purely descriptive research and reliance on distal explanations to more theo-
retically driven research with proximal explanations. The word proximal
means situated nearest the point of origin. Proximal factors are ones that
more directly and parsimoniously explain observed phenomena. Unlike distal
factors, proximal factors are usually more psychological in nature. In our
example, ethnic identity, as a psychological dimension of ethnicity, is a prox-
imal factor that could be hypothesized to account for the differences in
depression between Mexican Americans and European Americans. Using the
scientific method, we would test this hypothesis against other plausible or
rival hypotheses. We may find that ethnic identity does not account for the
differences in depression, and instead, the answer may lie with the importance
of familial roles in the lives of both groups. We may discover that a dispro-
portionate number of Mexican-American students are first generation college
students, and that leaving home and going to college far away from their
374 part 3 methodological issues
Table 15.1
Proximal Explanations Distal Explanations
Proximal
and Distal Racial identity Race
Explanations
for Group Ethnic identity Ethnicity
Differences Worldview Culture
Acculturation Sex
Values Age
Communication patterns Socioeconomic status*
Spirituality/religiosity Education
Individualism/collectivism Generation status
Independence/interdependence Year in school
Familial roles Major
*If used simply as a demographic, categorical variable (e.g., low, middle, high), socioeconomic
status is a distal explanation. However, socioeconomic status may also be used as a proximal
explanation if race and ethnicity differences are observed, and these differences can best be
explained because of differences in socioeconomic status.
families is causing not only higher rates of depression, but also guilt because
of the desire to want to work and financially help the family. Thus, it can be
seen that although the distal factor of ethnicity is not sufficient to explain why
differences in depression exist between the two groups, a proximal factor
related to familial roles provides the specific reason for the difference. A list
of variables that serve as proximal or distal explanations for observed phe-
nomena is provided in Table 15.1.
Later, Cokley (2003b) reported that the relationship between grade point
average (i.e., predictor) and self-esteem (i.e., outcome) differed by ethnic
group. He found that for European-American students, grade point average
significantly predicted self-esteem; however, for African-American students,
grade point average did not predict self-esteem. In this instance, Cokley was
using ethnicity as a moderator variable, because ethnicity moderated the rela-
tionship between grade point average and self-esteem.
Recall that mediators are variables that account for or explain the rela-
tionship between a predictor (independent variable) and criterion (dependent
variable). Mediators are usually involved in establishing “how” or “why” a
predictor variable causes or predicts a criterion variable (Baron & Kenny,
1986; Frazier et al., 2004). Mediators are usually psychological in nature
(Baron & Kenny). In Table 15.1, many of the variables that are proximal in
nature can also be considered as mediators. For example, Lee (2003) reported
that discrimination was negatively correlated with psychological well-being
and positively correlated with distress. He hypothesized that ethnic identity
mediated the effects of discrimination. However, contrary to his hypothesis,
ethnic identity did not mediate the effects of discrimination.
Regarding our earlier point about the importance of conducting theoret-
ically driven research, we want to emphasize the point that when multicul-
tural researchers are able to consistently identify when and under what
conditions a variable predicts or causes another variable, the level of under-
standing about a particular multicultural topic will be evident. Similarly, when
multicultural researchers are able to consistently explain how and why vari-
ables like ethnic identity and acculturation explain relationships between pre-
dictors and criterions, multicultural research on a particular topic will be seen
as reaching a high level of understanding.
METHODOLOGICAL CONSIDERATIONS
The comparison of groups—whether they are different ethnic groups; males
and females; low, middle, or high socioeconomic status; high achievers or low
achievers; or clinical or nonclinical samples—is an inherently interesting
research endeavor. As mentioned previously, any differences found in compar-
isons of ethnic groups should be explained by proximal factors. However,
before conducting comparisons using ethnic groups, a clear theoretically
driven rationale should guide the decision to compare ethnic minority and
majority groups.
a researcher must ensure that the research design is amenable to finding behav-
ioral determinants that may be due to culture. Some validity issues become
more crucial when culture is considered in research. Some scholars contend
that an additional category of validity, namely cultural validity, should be
added to the list of discussed validities in research (Quintana, Troyano, &
Taylor, 2001). In an effort to increase awareness of multicultural issues in re-
search, the addition of such a term to denote a separate validity only further
marginalizes concerns of culture in research. The issues that are subsumed
under the characterization of cultural validity (e.g., construct operationaliza-
tion, selection issues) are already part of the extant discussion on validity.
Instead of creating a new term to denote the importance of culture on valid-
ity, a paradigm shift toward the acceptance of culture as a part of everyday
life and as a result a part of everyday human behavior needs to occur.
Chapter 5 presented the types of and threats to the four categories of
validity outlined by Shadish et al. (2002). Although these four types of validity
are for the most part exhaustive of general validity concerns in research
design, we will highlight some important aspects of these four types of validity
that should be considered when incorporating culture into research.
an incident occurred where on a Friday night, black college students were seem-
ingly targeted by university police and told that their house parties must end.
Imagine that one black male house tenant argued with the officer and refused
to send individuals home, citing that there was a white fraternity on their
street that was also having a party but was not told to send its guests home.
As a response to the apparent anger displayed by the black male student, the
officer became alarmed and decided to use mace to subdue the student.
Although the incident was not publicized in the school newspaper, the news
spread like wildfire to all the black students and faculty at the university. It
appeared that many of the white and Latino students were unaware that this
incident occurred. When the diversity training intervention was completed
and the posttest was given, scores significantly decreased for the black stu-
dents, indicating a decrease in awareness, knowledge, and acceptance of other
groups, whereas scores for whites and Latinos increased, indicating the oppo-
site finding. The researcher, unaware of any of the events that occurred, may
conclude that the intervention did not work for blacks but was successful for
Latinos and whites. In this case, the threat of local history or selection × history
interaction occurred and led to invalid conclusions by the researcher. Strategies
for dealing with such issues will be discussed later in the chapter.
Another threat that may have occurred in the previous example is attri-
tion. Suppose that instead of staying in the study, the black students were so
upset with what had happened that they refused to participate in the research,
and as a result dropped out of the study. This threat would also lead to invalid
conclusions about the intervention due to the altered composition of partici-
pants at the posttest.
Some interventions that are performed with immigrant populations or
individuals for whom English is not their first language are especially suscep-
tible to internal validity threats. If individuals are tested more than once
during the implementation of an intervention, there is a possibility of testing
threats. This holds true for individuals who may not understand all the items
on a test and because of increased exposure to the measure actually begin to
understand the language used on the pre- or posttests. Therefore, the change in
score at the intervention may be due to increased levels of acculturation as
opposed to the effectiveness of the intervention.
Similarly, level of acculturation may also introduce maturation threats to
internal validity. Let us suppose, for example, that an intervention was imple-
mented with immigrant women that was designed to prevent domestic
violence. Part of the intervention included assertiveness training and education
about domestic violence laws in the United States. The intervention was part
of a three-year longitudinal study assessing several factors related to personal
relationships; every six months participants were contacted and asked to com-
plete a series of measures. At almost every data collection time, scores on
assertiveness and knowledge of domestic violence laws increased. Researchers
concluded that the intervention was successful in increasing knowledge and
level of assertiveness among participants. What the researchers have not con-
sidered is that as the length of the participant’s residence in the United States
chapter 15 conceptual and methodological issues 379
increased, their level of acculturation also increased. Some women may have
come from more patriarchal societies where domestic violence is more common
and laws to protect women from abuse are nonexistent or not enforced. As
they continued to live in and learn about the culture in the United States, they
learned that what may have been more acceptable in their former countries of
residence is not customary in the United States.
One of the most pernicious threats to internal validity is selection. Selection
is particularly threatening to multicultural research in that studies using a
comparative framework analyze data based on group membership. Sometimes
individuals are chosen on the basis of their membership in a cultural or minority
group and other times analysis is conducted post-hoc (after the fact with no
theoretically based hypotheses) after the researcher realizes that he or she has
enough participants from each group to conduct analyses. One reason that
selection threats are dangerous to internal validity is that one cannot randomly
assign race or ethnicity; therefore, selection biases are inherent in these types of
studies. Another reason is that race or group membership is used as a proxy for
numerous cultural variables. As discussed earlier, differences found among
cultural groups are erroneously attributed to group membership as opposed to
more proximal variables that may actually be responsible for differences (e.g.,
socioeconomic status, acculturation). As stated in Chapter 5, selection biases may
also interact with other threats to internal validity to impact results of a study.
external validity violations (e.g., not including individuals from diverse back-
grounds in the study sample) (Sue), or the issue is relegated to future research
suggestions.
What we believe is more important with regards to external validity is to
explicitly define the researcher’s population, time, and setting of interest. These
aspects of external validity should be chosen with the research question in mind.
If a researcher wants to know the extent that a certain counseling technique is
applicable to individuals who experience depression, then the sample must be het-
erogeneous to allow for valid conclusions about different populations. Therefore,
the researcher should include individuals whose differences (e.g., ethnicity,
socioeconomic status, gender) reflect their population of interest. For multicul-
tural research, given that some psychological phenomena operate differently
based on cultural group membership, it may be inappropriate to combine par-
ticipants together into one “representative” population. Assuming that different
cultural groups were equally represented numerically in a study, critical differ-
ences between these populations may be obscured and as a result increase the
error variance in the study. Error variance may be random, but some of the error
that occurs is systematic, meaning that a variable has not been accounted for.
When cultural group differences are not accounted for in a study, a selection by
treatment threat may have occurred.
(Reynolds, C. R., 1982; Tabachnick & Fidell, 1996; Valencia & Suzuki, 2001).
This procedure identifies the degree that clusters of test items or clusters of
subsets of psychological test items correlate with one another, and in turn can
provide some indication that individuals are conceptualizing the meaning of
item subsets in similar or different ways (Tabachnick & Fidell). Furthermore,
if the factor structure differs between groups then the test does not demon-
strate construct validity, and thus is biased.
Although factor analysis is a useful tool for tapping into construct validity,
it should not be used as the only method for determining construct validity.
Furthermore, the replicability of factor structure across groups is not sufficient
evidence of measurement invariance and instrument equivalence (Byrne &
Watkins, 2003; also see Chapter 20). Byrne and Watkins contended “even though
a factor structure may yield a similar pattern when tested within each of two
or more groups, such findings represent no guarantee that the instrument will
operate equivalently across these groups” (p. 156), that is, with the same
meaning across groups. Furthermore, for any given measure there are sources
of error that sometimes can be identified, but most times are not. For example,
a measure may not cover all the important aspects of a construct across different
cultural groups, and thus result in construct under-representativeness (Shadish
et al., 2002) for some groups. Therefore, due to possible imperfections in a
given measure, one cannot fully assess construct validity; one can only hope
to strengthen it.
STIMULUS QUESTIONS
An Integration Exercise
The purpose of this exercise is to promote inte- 1. Why is it problematic to use the terms race,
gration and utilization of the material presented ethnicity, and culture interchangeably?
in Chapter 15. Students should respond to each 2. What is the difference between distal and
question, and then discuss their responses with proximal explanations for behavior in multi-
a peer. cultural research?
384 part 3 methodological issues
3. A researcher hypothesizes that there are 5. What are the advantages and disadvantages
racial differences in intelligence test scores. of adding cultural validity as an additional
What issues would the researcher need to category of validity in research?
address, and what evidence would need to be 6. What are some ways individuals can conduct
presented, in order to provide support for the more culturally competent research studies?
hypothesis? 7. Think of a situation where the validities dis-
4. What are the advantages and disadvantages cussed in this chapter would be at odds for a
of conducting multicultural research that is person trying to conduct culturally competent
primarily descriptive versus theoretically research (i.e., increasing one type of validity at
driven? the expense of another type of validity).
Investigator,
Experimenter,
CHAPTER
16
and Participant Bias
385
386 part 3 methodological issues
this chapter examines participant bias, particularly with regard to (1) demand
characteristics, (2) participant characteristics, and (3) introspective abilities.
Throughout both sections we use examples from previous research efforts to
clarify different types of bias. Moreover, in discussing the various sources of bias,
we also discuss some strategies for controlling or minimizing these variables.
that Hans could reliably answer questions (that is, nine times out of ten) only
when the interrogator knew the correct answers, but his performance dropped
to one out of ten if the interrogator was ignorant of the answer. As it turned
out, Hans had not learned math, or music, or the German language, but
rather had learned to read subtle cues in the interrogator’s posture, breathing,
and facial expressions (Pfungst, 1911). Research in the 1980s revealed that
laboratory animals can learn to read a wide variety of subtle behavioral cues
in trainers that give away the intended answer.
In short, investigators and experimenters can be the source of extraneous
variables and error variance that may very well bias the results of a study. We
next discuss three major types of bias: experimenter attributes, investigator
and experimenter expectancies, and experimental procedures. In addition, we
offer some strategies for reducing such biases.
Experimenter Attributes
Experimenter attributes are primarily biological and interpersonal characteristics
of the experimenter that may cause differential responses in participants.
Examples include the experimenter’s age, gender, race/ethnicity, physical ap-
pearance, and interpersonal style. For example, some participants might re-
spond more honestly to a female researcher investigating sexual harassment
than to a male. Likewise, an experimenter of age 50 might inhibit younger
participants but facilitate disclosure for older participants. In another case,
the experimenter’s interpersonal style (say, unfriendly and dominant) might
interact with the independent variable (for example, expertness cues) such
that some participants feel uncomfortable or even threatened during the
experiment. Race and ethnicity can also influence bias in a study’s results if a
participant has preconceptions or stereotypes about the experimenter’s racial
or ethnic group. Another potential biasing characteristic pertains to previous
contact between the experimenter and participants. Some participants may
feel that having some prior knowledge about an experimenter would make it
easier for them to respond, whereas others may feel much less likely to disclose
personal information.
In short, a wide range of experimenter characteristics might influence
some or all participants to respond differentially in a particular experiment,
thereby confounding its results. In fact, several writers (see, for example,
Christensen, 1980; Kazdin, 1980) have reported empirical investigations that
document that experimenter characteristics can affect responses given by
participants on various tasks, such as self-report inventories, projective tests,
laboratory tasks, and measures of intelligence (see, for example, Barber, 1976;
Johnson, 1976; Masling, 1966; Rumenik, Capasso, & Hendrick, 1977). For
example, one study found some African-American clients tend to rate white
counselors as less proficient and credible (Watkins & Terrell, 1988).
In short, experimenter attributes create threats to validity. Consider the
following example, which raises questions about construct validity. Imagine
an investigator (Professor Kay Tharsis) who has reason to believe that a
388 part 3 methodological issues
studies, reaction time studies, inkblot test studies, structured laboratory studies,
and person perception studies (see Rosenthal, 1966). Although some writ-
ers have argued that the effect of experimenter bias has been overstated
(Barber, 1976; Barber & Silver, 1968), it is generally concluded that investi-
gators and experimenters can and do influence participant responses (Barber;
Christensen, 1980).
Expectancies can be positive or negative and occur in many different
ways. Investigator and experimenter bias can affect participants or clients at
any stage of an investigation, such as during participant recruitment, during
data collection, or after treatment interventions. For example, an exuberant
experimenter (serving as a therapist) may subtly or not-so-subtly promote the
effectiveness of one intervention over others. This form of bias is operative
when an investigator’s enthusiasm for a new treatment intervention (that per-
haps he or she has developed) gets communicated to the experimenter, who
then might consciously or unconsciously be too enthusiastic in conducting the
therapeutic role within a study.
Sometimes the desire to be effective even leads experimenters serving as
therapists to break protocol and engage in activities outside of normal thera-
peutic or experimental procedures. For example, in one study that examined
the counseling process, a relatively inexperienced therapist was found engag-
ing in a friendly conversation with the client immediately after the counseling
session, during which time the client was completing forms evaluating the
counseling session and the counselor! This type of interaction could certainly
influence participants to give more favorable responses.
Investigator and experimenter bias also can influence clients in a negative
way. For example, if an experimenter is not very motivated or interested in
conducting a study, the lack of enthusiasm can very well affect client recruitment
and willingness to participate. Halfhearted efforts by experimenters can result
in halfhearted responses by clients. Biases resulting from lack of experimenter
motivation can be a particular problem with doctoral dissertations, notably
when the author asks a friend to assume total responsibility for certain aspects
of a study (such as recruitment, data collection, or monitoring data flow);
even most “good friends” will not be as motivated as the researcher, especially
as problems arise and frustrations mount. Sometimes inexperienced investiga-
tors feel shy or guilty about asking clients to participate in a study, but telling
clients three or four times in an apologetic tone that they do not have to par-
ticipate is not especially persuasive. Another probable source of bias can arise
during the administration of inventories to participants or clients; a busy
counseling center receptionist, for example, might be tired at the end of each
working day, given the added demands of the research, and verbally or non-
verbally convey impatience to the final clients.
In summary, there are many opportunities for investigator and experimenter
bias to contaminate the results of a study. Experimenter and investigator
expectancies can bias participants whenever anyone involved in conducting
the study interacts with participants, can bias the results to favor one treatment
over another, or can affect the generalizability of findings. Thus, experimenter
chapter 16 investigator, experimenter, and participant bias 391
and investigator expectancies can affect both construct and external validity
in similar ways to those discussed with respect to experimenter attributes.
Some strategies to lessen the effects of investigator and experimenter ex-
pectancies include the following:
• Perhaps the most common strategy to offset experimenter biases is to keep
experimenters “blind” as to the purpose of the study. Thus, for example, an
investigator who is comparing the effectiveness of two treatment ap-
proaches would not communicate to the experimenters serving as thera-
pists the specific purposes and hypotheses of the study, thereby reducing
the probability that the therapists will unintentionally influence partici-
pants in the hypothesized direction. In fact, the investigator may want to keep
anyone involved in the study who has contact with participants (for example,
receptionists, therapists, or assistants who collect data from participants)
blind as to the specific purpose of the study.
• Because keeping various personnel blind is more difficult in some studies
than others, the investigator may need to resort to a partial-blind strategy.
For example, it would be very difficult to keep therapists blind as they
administer a cognitive-behavioral group treatment for bulimics that will be
compared with a placebo nondirective discussion group. In such cases, the
investigator might try to keep the therapists as blind as possible, and espe-
cially with regard to the specific hypotheses and variables involved in the
study. Another strategy is to restrict the amount of contact the partial-
blind therapists have with participants; therapists might administer the
treatments only and not be involved in participant selection, data collection,
and debriefing. In short, even when one can achieve only a partial-blind
situation, the goal is to keep the experimental personnel as blind as possible
to the purpose of the study.
• Because experimenter expectancies can affect participant responses, another
strategy is for investigators to assess the accuracy of experimenter
expectancies. For instance, an investigator may assess whether experi-
menters correctly surmise the purpose of the study, or even the general
hypotheses. If experimenters have accurately pieced together the purpose
of the investigation, the potential bias from experimenter expectancies is
generally much higher than if the experimenters have not adequately dis-
cerned the nature of the study. Although experimenters might still bias par-
ticipants in subtle ways even when kept blind about the purpose of a study,
the probability that participants will be biased toward the hypotheses is
reduced. In short, assessing experimenter expectancies allows for both an
evaluation of the degree to which experimenters have been kept blind and
an assessment of the accuracy of their expectancies.
• Use strategies for decreasing negative experimenter bias due to halfhearted
efforts. Investigators should be wary of recruiting “friends” to assume
total responsibility for various aspects of a study, especially if the investi-
gator is physically removed from the location of the study. Given our expe-
rience, which suggests that no one will do the same quality of work as the
392 part 3 methodological issues
Experimental Procedures
Once investigators have carefully developed a study, operationalized constructs,
identified variables, and controlled as many extraneous variables as possible, they
usually attempt to conduct the experimental procedure in a constant and consistent
manner. A typical study involves a wide range of experimental procedures—
recruiting participants, greeting participants, obtaining informed consent,
administering instructions, providing a rationale, administering interventions,
recording observations, reminding participants about returning question-
naires, interviewing participants, administering questionnaires, and debrief-
ing; imprecision or inconsistencies in the manner in which the experimental
procedures are conducted can therefore be a major source of bias and con-
tamination. Procedural imprecision occurs when the activities, tasks, and
instructions of an experiment are not specifically defined. As a result, experi-
menters might treat participants differently because they are unclear exactly
how to conduct the experiment in specific situations. Thus, experimenters
introduce bias (or error variance, if it occurs randomly) into an experiment
due to variability in experimental procedures.
If, for example, an investigator wants to study the effects of training res-
idence hall staff to be career resource specialists, then the training given to the
staff should be clearly specified, and the specific ways the staff are to interact
with students on their hall floors should be clearly delineated as well. Are the
staff supposed to actually dispense career information? If so, what kind of
information? Should they make referrals to the career center? If so, to anyone
in particular? Should they administer career assessment inventories? If so,
which ones? Should they engage in actual counseling sessions with students?
If so, are there any guidelines for topics and number of sessions? Should they
promote information-gathering activities? If so, what kind? This example
indicates only some of the ways residence hall staff acting as career resource
specialists might behave. If the investigator does not specifically delineate
what the staff should and should not do, it is very likely that a wide variability in
activities will occur across the different residence hall staff members. Moreover,
the lack of delineation of staff responsibilities increases the probability of
systematic staff biases, such as preferential treatment for brighter students, for
same-gender or opposite-gender students, or for students who come from
families of higher socioeconomic status.
Likewise, investigator and experimenter biases pertaining to biological
sex, race, ethnicity, sexual orientation, age, and physical disability might inad-
vertently affect participants’ responses, either negatively or positively. For
example, a male investigator who is insensitive to gender issues and makes
sexist comments might systematically bias the recruitment of participants.
chapter 16 investigator, experimenter, and participant bias 393
In another case, racial bias can influence the interaction and communication
between the experimenter and participants (see Dovidio, Gaertner, Kawa-
kami, & Hodson, 2002). In short, any of a number of culturally insensitive
behaviors, comments, instructions, or procedures by investigators and exper-
imenters might bias participants’ responses.
Even if an investigator has carefully specified the procedures for a particular
study, variability across experimenters can occur for a number of reasons, of
which we discuss three: fatigue, experimenter drift, and noncompliance. An
experimenter who becomes fatigued over time, resulting in different perfor-
mances across participants, can be a particular problem if experimenters
engage in an intensive activity, such as interviewing several participants or
clients in a short period of time. An experimenter may also gradually and
unconsciously alter his or her performance over time (experimenter drift); this
is of special concern if an experimenter is involved in repetitive tasks over
time. Or an experimenter may fail to comply with the exact experimental pro-
cedures over time, perhaps because of a lack of awareness of the importance
of some procedures. For example, toward the end of a data collection process
in one study, an experimenter began to estimate a timed task to “about the
nearest minute,” obviously an imprecise procedure! In another study designed
to compare the effects of two therapists, a researcher was chagrined to find
one counselor complying with the normal 50-minute sessions while the other
counselor, trying very hard to do well, conducted therapy sessions for as long as
70 minutes each! In short, even when procedures are carefully specified, exper-
imenter variability can still occur.
At least three problems are associated with procedural imprecision. First,
as suggested in the preceding example, experimenters most likely vary among
themselves in addition to introducing systematic biases over time. Thus, not
only is there a good chance that each residence hall staff member would
engage in quite different activities in the role of career resource specialist, it
also is likely that they would act differently from one student to another,
thereby adding a confounding variable to the study.
Second, if the procedures are unclear, the investigator does not know
what actually occurred. It may be unclear whether the independent variable
was administered consistently, or whether other variables might have inter-
vened (recall the discussions of construct validity in Chapter 5 and independent
variables in Chapter 12). If significant results are found, it is not clear whether
they are to be attributed to the independent variable or to other variables. Or
if a participant responds negatively in a survey, it is unknown as to whether
this is the participant’s true response or if he or she responded negatively due
to frustration, such as completing the survey in a loud environment. In short,
if investigators do not know what was done in an experiment, their conclu-
sions are confounded, and it is difficult, if not impossible, to discuss the
study’s results with much precision.
A third problem of procedural imprecision pertains to the statistical issue
of introducing error variance or “noise” into the data due to experimenter
variability. Statistically speaking, the variability due to the experimenter increases
394 part 3 methodological issues
PARTICIPANT BIAS
Returning again to our ideal world, the perfect participant is an honest, naive
person who comes to an experiment without any preconceived notions, will-
ingly accepts instructions, and is motivated to respond in as truthful and help-
ful a way as possible (Christensen, 1980). Such participants would not be
afraid to be seen in a negative light and would be willing to disclose personal
information concerning their innermost secrets. Likewise, within counseling,
ideal clients would openly discuss their experiences in counseling, the problems
they have been unable to solve on their own, the reasons they chose to enter
counseling, their perceptions of their counselor, and the ways in which they
chapter 16 investigator, experimenter, and participant bias 397
Demand Characteristics
One major source of participant bias pertains to what is commonly referred
to as demand characteristics, which are cues within an experiment that may
influence participants to respond in a particular way apart from the inde-
pendent variable. Demand characteristics may or may not be consistent with
an experimenter’s expectancies; they often include events other than the exper-
imenter’s expectancies. Examples of demand characteristics include instructions
on a personal problems questionnaire that give the impression that most col-
lege students do not have personal problems, a receptionist’s nonverbal behav-
iors that seem intended to make potential participants feel guilty if they do not
complete a questionnaire, or an experimenter’s apologetic nonverbal cues
toward participants in the control group.
Demand characteristics are typically subtle influences or pressures, al-
though sometimes they are not so subtle. For example, in one study of client
preferences, clients coming to a counseling center were asked very leading
questions, such as “What kind of counselor do you expect to see, expert or
inexpert?” Not surprisingly, the results of this study revealed most clients pre-
ferred expert counselors, although it is unlikely that counselor expertness
would have been consistently cited by clients if a more general question were
used. Demand characteristics can occur at any point during an experiment,
such as in recruiting participants, during interactions with any personnel
involved with the experiment, during the completion of inventories or exper-
imental tasks, and in debriefing.
Demand characteristics operating within a particular study are often dif-
ficult to identify. Even though investigators’ intentions are to be objective and con-
duct a rigorous study, they may be unaware that the specific instructions on an
inventory might influence some participants to withhold personal information.
398 part 3 methodological issues
Likewise, very minor comments in recruiting participants may mask the effects
of the independent variable. For example, consider the story of a group of
researchers interested in the effects of breaches of confidentiality on partici-
pants’ perceptions of counselor trustworthiness. In the pilot study, the re-
searchers were shocked to find that blatant breaches of confidentiality within
a counseling analogue did not seem to affect how participants rated the coun-
selor on trustworthiness. For example, even if the counselor began an interview
by commenting on the previous participant, saying something like, “Did you
see that person who just left? Boy, he has some serious problems!” the second
participant still rated the counselor as quite trustworthy. Close examination
of all of the experimental procedures revealed that when participants were
initially contacted, they were asked if they would be willing to participate in a
counseling study in which they would talk with a “highly competent profes-
sional counselor, a person who is well liked and has a very positive reputation
with students on campus.” Additional piloting of participants revealed that
omitting this emphasis on prestige in the introduction resulted in more accurate
perceptions of the counselor’s behaviors. In sum, demand characteristics
operating on participants are often subtle and unintentional.
Participant Characteristics
Not all participants respond in the same way to different experimental tasks
and procedures. Some participants may respond more than others to subtle
cues or pressures. A broad range of participant characteristics may affect how
participants respond not only to demand characteristics, but also on a broader
level to the experimental situation. We now briefly discuss five participant
characteristics that may bias participants’ responses: self-presentation style,
motivation level, intellectual skills, psychological defenses, and worldview.
of the counselor, particularly if they believe that their comments will be disclosed
to or cause trouble for the counselor.
Participants with a strong desire to present themselves in a positive
manner may be more susceptible to being influenced by demand characteristics.
Christensen (1980) has argued that such participants use available demand
characteristics to identify the types of responses that make them appear most
positively. In short, participants who want to present themselves well may try,
either consciously or unconsciously, to be the “good participant” and respond
in a way they believe is consistent with the experimenter’s desires or wishes.
Interested readers might examine Friedlander and Schwartz (1985) for a
theory of self-presentation in counseling.
At other times some participants may feel defensive or paranoid about reveal-
ing their feelings or thoughts about sensitive topics (for example, sexual orienta-
tion, race relations, or feelings of inadequacy or embarrassment). Likewise,
participants such as prison inmates may be suspicious of any experimenter
and withhold or temper their responses because of perceived danger. In short,
bias may be introduced into a study because participants perceive some real
or imaginary threat, which consciously or unconsciously tempers their
responses.
unsuccessful in finding a solution, Maier, who had been wandering around the
room, would walk by the cords and swing one of them into motion. Some of
these participants then would subsequently pick up some object, tie it to the
end of the cord, swing the cord like a pendulum, and shortly thereafter solve
the problem. But when Maier asked these participants how they had arrived
at the solution, he got answers such as “It just dawned on me” or “I just realized
the cord would swing if I fastened a weight to it.” In short, most of Maier’s
participants could not accurately report the causal events involved in their
cognitive and affective processing. More recent research in experimental infor-
mation processing also suggests that participants have difficulty explaining
the causal chain of events in their cognitive processes, particularly with the
passage of time (Ericsson & Simon, 1984).
Even though Nisbett and Wilson’s (1977) work has been the subject of
some debate, their observations should be carefully considered when counsel-
ing researchers attempt to examine clients’ cognitive processes, especially
within the often-ambiguous situation we call counseling. For example, why
clients changed or what influenced them to change in the course of counseling
may be a very difficult question for them to answer accurately and reliably.
Likewise, as one researcher learned, asking clients how they decided to seek help
at a counseling center resulted in a very broad range of responses—some of
them incomprehensible! In short, counseling researchers in particular must
carefully consider the type of information that can be accurately and reliably
obtained about clients’ mental processes, particularly retrospectively. Researchers
may well ask clients to tell them more than the clients can know, and thus
clients may provide misleading self-reports.
no right or wrong answers” and that “this inventory is not a test.” The
actual title of an inventory might also be altered if it is found to arouse
anxiety or introduce demand characteristics. In addition, some researchers
attempt to reduce threat by explicitly normalizing typical participant fears.
For example, in a study of the coping process, participants might be told
that it is normal to have unresolved personal problems, and that in fact
“everyone has them.”
• Several procedures can be used to increase participants’ honesty and reduce
participants’ fears about confidentiality. First and foremost, researchers
often make honest and direct statements about their desire or need to obtain
honest responses to increase the profession’s understanding of a phenomenon.
In addition, researchers typically communicate to participants that their
responses will be strictly confidential, and then they explain the mecha-
nisms for safeguarding confidentiality (for example, coding). Moreover, in
research using groups of participants, it is often helpful for participants to
know that the researcher is interested in how whole groups of participants
respond to the items, rather than specific individuals. Sometimes participants
are asked to omit their name from their questionnaires, thereby maintaining
anonymity. To ensure confidentiality when a researcher needs to collect
data on several occasions and then compare a given participant’s responses
over time, participants can be asked to supply a code name in lieu of their
actual name, or to generate an alias or develop a code based on some com-
bination of numbers related to their birth dates, age, social security num-
ber, or the like (see Kandel, 1973).
• Often researchers will make appeals to participants to increase their moti-
vation level. For example, participants can be briefly told the importance
of the research and its possible outcomes (for example, honest responses
to the questionnaires will clarify changes or growth within supervision, or
learning about one’s experience as an Asian American will provide infor-
mation about potential ethnic-related stressors). In exchange, researchers
sometimes promise to mail participants the results of the study, particu-
larly if the study is of a survey nature.
• Sometimes researchers concerned about demand characteristics and
participant bias conduct a postexperimental inquiry. As per our discussion
in Chapter 14, after the experiment the experimenter assesses potential
participant bias by asking participants questions about their understanding
of the purpose of the experiment, about their beliefs concerning how the
experimenter wanted them to respond, and about any problems they encoun-
tered. Such a postexperimental inquiry might consist of a brief question-
naire (with perhaps follow-up inquiries of those participants who
identified the true purpose of a study or those who felt pressured in some
way) or be conducted via direct interviewing. Because direct interviewing
can introduce demand characteristics in and of itself, it may best be done
by someone who has not previously interacted with the participants. Even
though there is considerable value in postexperimental inquiries, they have
their limits: If participants have been biased by some demand characteristic
chapter 16 investigator, experimenter, and participant bias 403
This chapter discussed investigator and exper- important issue for counseling researchers because
imenter biases that may well affect the results of much of our research is conducted in applied
a study. We discussed three major types of exper- settings, where fewer experimental controls are
imenter bias: experimenter attributes, investiga- available and thus more sources of bias are
tor and experimenter expectancies, and experi- potentially operative.
mental procedures. We also discussed three major Because minimizing and eliminating bias is a
sources of participant bias—demand characteris- matter of degree, and because the researcher is
tics, participant characteristics, and participants’ often uncertain whether various sources of bias
abilities to report their experiences—and described are operative, it is useful to qualify the results of
several strategies for controlling or minimizing one’s research (typically done in the discussion
such biases. section) concerning the possible sources of bias
Participant, investigator, and experimenter that might have been operative. An increasingly
biases represent serious problems for the coun- commonplace research strategy is for inves-
seling researcher. These sources of bias can quickly tigators to statistically test for the existence of
reduce the results of many long hours spent various biases and confounds in a series of pre-
developing and conducting a study into a pile of liminary analyses. Hogg and Deffenbacher
meaningless information. The veteran researcher (1988) provide a good example of using such
remains vigilant of various sources of bias and preliminary analyses. Such statistical procedures
constantly monitors the study to detect any biases are recommended, as are frank discussions of
that might have crept in. Bias is a particularly other possible sources of bias that were untested.
STIMULUS QUESTIONS
Reflections on Bias
It is often very difficult to clearly identify the 4. List four ways you could most significantly
myriad of biases that threaten the validity of a reduce biases related to the experimental pro-
study. The purpose of this exercise is to reflect cedures in the type of research topics you are
on a wide range of biases with the goal of be- considering.
coming more aware and sensitive to biases that 5. Identify potential demand characteristics in
threaten the validity of a study. the research topics you are considering.
6. Discuss how the participants’ worldviews
1. Briefly discuss how your values and beliefs
might create some confounding biases in the
could result in cultural biases.
type of research topics you are considering.
2. Are there particular topics in which your
7. In what ways could participants’ need for
values and beliefs could result in cultural
self-presentation, or saving face, affect the
biases?
results of your research?
3. In considering your research interests, which
8. Discuss the most important things you have
of your personal attributes could cause dif-
learned about investigator, experimenter, and
ferential responses in participants?
participant bias in scholarly research.
Analogue Research CHAPTER
17
The first section of this chapter provides a brief historical overview of the
use of analogue methodology in counseling; the second section provides
examples of recent analogue studies. The third and fourth sections discuss the
advantages and disadvantages of this particular methodology. The fifth sec-
tion proposes that the external validity of analogue research be evaluated by
examining variables that depict real-life counseling, most notably those related
to the counselor, the client, and the counseling process. The sixth section main-
tains that the ultimate utility of analogue methodology must be evaluated
405
406 part 3 methodological issues
within the context of current knowledge bases and existing research method-
ologies used in a particular topic area. The use of analogue methodology in
social influence research in counseling is analyzed to demonstrate this point.
HISTORICAL OVERVIEW
Real-life counseling is a tremendously complex process. Clients differ, therapists
differ, and counseling is such a highly interactive, emotionally charged, and
complex communication process that it is difficult to describe, much less inves-
tigate. Over 30 years ago such complexity led Heller (1971) to conclude that
the counseling interview, “while an excellent source of research hypotheses, is a
poor context for isolating factors responsible for behavior change. The varied
complexity of the therapeutic interaction and the inability to specify and con-
trol therapeutic operations make it difficult to obtain reliable information
concerning exact agents of change” (p. 127). Heller, a strong believer in the
scientific method and its use of experimental control, contended that part of
the solution to investigating and understanding the complexity within coun-
seling was to exercise the experimental control offered in what he called lab-
oratory research: “The purpose of clinical laboratory research is to determine
what factors produce change, under what conditions they operate best, and
how they should be combined to produce an effective therapeutic package”
(Heller, p. 127).
Basically, a counseling analogue is an experimental simulation of some
aspect of the counseling process involving manipulation of some aspects of the
counselor, the client, and/or the counseling process. In the past, analogues
have been referred to as “miniature therapy” (Goldstein, Heller, & Sechrest,
1966) or as a “simplification strategy” (Bordin, 1965).
Keet (1948) has been credited with one of the first uses of the analogue
methodology in psychotherapy research (see, for example, Bordin, 1965; Heller,
1971; Kushner, 1978). Keet used volunteer participants and examined the effi-
cacy of expressive (reflective) versus interpretative therapeutic statements in
overcoming previously identified memory blocks on word association tasks.
Interpretative statements were found to be more effective. It is interesting to
note that even though subsequent research did not replicate Keet’s findings
(Grummon & Butler, 1953; Merrill, 1952), Bordin observed that “so great is the
attractiveness of control and simplification as a research strategy that this failure
[to replicate] only spurred further efforts in this direction” (p. 494). In general,
however, the use of analogue research has been a relatively recent development
within counseling research, a phenomenon Heller attributed to the fact that
analogues are rather foreign to psychotherapy because knowledge in the
clinical fields has normally been accumulated in the more naturalistic tradition
(as, for example, in the case studies of Freud). Munley (1974) attributed the
slow emergence of the analogue to the reluctance of counselors to accept a
contrived and highly controlled methodology that might well be too artificial
and unrealistic.
chapter 17 analogue research 407
Hayes and Erkis (2000) gathered the data from 425 psychologists who
were licensed, practicing therapists and members of the APA. Participants
received a packet in which they were randomly assigned to one of the eight
vignettes, and scales measuring homophobia and the dependent variables.
They found that therapist homophobia and the source of HIV infection pre-
dicted therapist attributions of client responsibility for problem cause. In addi-
tion, they found that when therapists were more homophobic, when the client
was gay, and when the client’s source of HIV infection was other than drugs,
the therapists responded with less empathy, attributed less responsibility to
the client for solving his problems, assessed the client’s functioning to be
worse, and were less willing to work with the client.
Mohr et al. (2001) investigated the influence of counselors’ attitudes
toward bisexuality on counselors’ clinical responses to a fictitious intake report.
Specifically, the independent variable was the counselor’s attitude toward bi-
sexuality and the dependent variables were reactions to the client, anticipated
reactions to the bisexual client, and psychosocial functioning.
Unlike Hayes and Erkis’s (2000) study, in which two independent variables
were manipulated using eight vignettes, Mohr and his colleagues used one
fictitious intake report about a bisexual woman, which was used as a stimulus
for each counselor’s assessment and response to the client. They summarized
the vignette as follows:
. . . a two-paragraph fictional scenario summarizing a clinical intake session with
a client named Alice, who was a 20-year-old White bisexual woman. . . . In the
scenario, which was set in a university counseling center, the presenting problems
included (a) difficulty making a career choice, (b) grief over the end of a 2-year
romantic relationship with a woman, (c) a boyfriend who was having difficulty
accepting the client’s bisexuality. Thus, the predominant clinical issues involved
issues with career indecision, negotiating emotional boundaries with parents, and
romantic relationships. (Mohr et al., 2001, p. 213)
The data were gathered from 97 counselor trainees in master and doctoral
programs. The authors found that attitudes regarding bisexuality were related
to counselors’ clinical judgments and reactions. Counselors who had a negative
attitude toward bisexuality assessed the function of clients to be low, antici-
pated biased and judgmental responses to the client, and were more likely to
have negative reactions to the client.
Kim and Atkinson (2002) investigated the effect of client adherence to
Asian cultural values, counselor expression of cultural values, and counselor
ethnicity on the career counseling process among Asian-American college
students. They used a quasi-counseling interview, and the sessions were con-
ducted by trained counselors with students who were recruited for this study
but had real career issues. A 2 × 2 × 2 factorial design was used with two levels
of client adherence to Asian cultural values (low and high), two levels of
counselor expressions of cultural values (Asian cultural values and U.S. cul-
tural values), and two levels of counselor ethnicity (Asian American and European
American). Client-perceived counselor effectiveness, counselor empathetic
410 part 3 methodological issues
example, counselor interventions and client reactions and disclosures), and to the
particular situation (for example, room decor and arrangement, cost of therapy,
and reasons for seeking help). In analogue designs, variables extraneous to the
particular research problem can be eliminated or controlled. For example, par-
ticipants easily can be randomly assigned to treatment conditions, thereby reduc-
ing confounds from participant variability. Participants also can be selected based
on a particular variable (such as locus of control or level of depression), which
can be either held constant across treatment conditions or varied to create levels
of an independent variable. Therapists’ theoretical orientation and interview
behaviors can be controlled or even standardized across conditions. In short,
analogue research allows for a great deal of situational control by enabling the
researcher to manipulate one or more independent variables, to eliminate or hold
extraneous variables constant, and to use random assignment.
Along with providing situational control, analogue methodology has the
advantage of enabling the experimenter to achieve a high degree of specificity
in the operational definitions of a variable. For example, level of counselor
self-disclosure could be manipulated to enable examination of, say, three dis-
tinct levels of self-disclosure (no self-disclosure, 5 self-disclosures per session,
10 self-disclosures per session). In this sense, analogue methodology often
offers greater precision—not only in terms of the variables under examina-
tion, but also in terms of experimental procedures.
Such increased specificity is a major advantage in isolating specific events
or processes in the complex activity that is counseling. Thus, within a labora-
tory analogue it is possible to isolate and examine the effects of rather small
events, including self-involving counselor disclosures, specific counselor intro-
ductions, the counselor’s professional titles, and the client’s perceptions of the
counselor (see Andersen & Anderson, 1985; Strong & Dixon, 1971).
Another advantage of analogue methodology is often the reduction of prac-
tical and ethical obstacles in experimentally examining some aspect of the coun-
seling process. Many times, real-world constraints such as financial limits and
unavailability of participants can be substantially reduced by using client surro-
gates in an analogue design (see, for example, Dixon & Claiborn, 1981). Given
that real-life counseling involves clients with real-life problems, stresses, and anx-
ieties, experimental procedures such as randomly assigning clients to placebo
groups or to waiting-list control groups can cause problems. Or certain experi-
mental manipulations, such as varying the type of counselor feedback or type of
counselor self-disclosure, may pose serious ethical dilemmas when clients with
real problems are involved. Creating a situation analogous to counseling or using
transcripts, audiotapes, or videotapes of counseling interactions sidesteps ethical
problems with manipulations, especially with clients under some kind of duress.
Client
Expectations Client expects treatment Person expects experi- Participant expects course
of change and change mental treatment credit or to learn about
psychology
Motivation and Client is distressed enough Person is distressed Participant is not distressed
distress level to seek help at a treatment enough to seek relevant and does not seek help;
center academic experiences participant has ulterior
and psychological motivation (such as course
experiments credit) other than seeking
psychological help and
change
Selection Client often chooses ther- Person selects relevant Participant is assigned to
of treatment apists or type of treatment psychological experiments treatments and therapists/
providing treatment interviewers
Presenting Real-life problem typically Hypothetical problems None, or some experi-
problem seen in counseling mental task(s)
Knowledge Relevant and current con- Relevant but not pressing Irrelevant or new issue;
of problem cern; high level of infor- concern; moderate level low level of information
mation processing and of information processing processing and knowledge
knowledge and knowledge
Counselor
Counselor Client change Moderate expectation of Successful role play or
expectations client change interview
Role credibility High status; appearance is Moderate level of status Absence of status cues;
role congruent role incongruent
Knowledge bases Broad range of knowledge Moderate levels of Low level of knowledge
about assessments, per- knowledge about assess- about assessments, per-
sonality and counseling ments, personality and sonality and counseling
theories, and the counsel- counseling theories, and theories, and the counsel-
ing process the counseling process ing process
Counseling skill High levels of procedural Moderate levels of pro- Low levels of procedural
skills within the counseling cedural skills within the skills within the counseling
process counseling process process
Motivation level Highly motivated to pro- Moderately motivated Not motivated to provide
vide therapeutic relation- to provide therapy; pos- therapy; primary goal is
ship and facilitate change sibly some motivation to conduct an interview
for experimental change
Experience level 10 years + 3rd-year doctoral student 1st-year M.A. student
(Continued)
414 part 3 methodological issues
Counseling
process and
setting
Assessment Client is carefully diagnosed Person may be assessed Participant is not assessed;
and goals established to determine congruence goals for specific individual
with treatment goals lacking
Interventions Specifically targeted Relevant to person’s Not relevant to participant’s
to client’s presenting problem concerns or problems
problems
Duration Several normal-length A few normal-length A single brief (10 minutes
therapy sessions over time sessions or so) session
Interpersonal Counselor and client Counselor and client/ Participant views counsel-
interchange interact and exchange participant interact on ing scenario but does not
information restricted topic or in interact with a counselor
some defined manner
Client reactions Client processes the coun- Person reacts to restricted Participant views counsel-
seling experience and topic or semirelevant ing scenario and responds
reacts in some way to the topic hypothetically
relevant information
Client change Client changes or is Person may change in Participant does not
or outcome different in some way some way, providing the change in any way
because of the counseling treatment is successful because the counseling
interchange scenario was not person-
ally relevant
Environment Professional treatment Facility that may not Laboratory setting or
center offer regular treatment classroom
services
Client Variables
A number of variables pertaining to clients or participants directly relate to
the generalizability of research findings to actual counseling practice. We now
chapter 17 analogue research 415
discuss several client variables that illustrate important aspects of clients seek-
ing counseling and then relate these variables to evaluating the generalizabil-
ity of a particular study.
In most actual counseling situations, clients experience personal problems
that they have been unable to resolve (Fretz, 1982). These personal problems
typically cause anxiety and distress of some sort, as people find themselves
“failing” where they want to “succeed” in some way. As people cope with
their “current concerns” (Klinger, 1971), they typically engage in a wide range
of cognitive, affective, and behavioral trial-and-error processes (Heppner &
Krauskopf, 1987). Because these clients typically have thought about their
problem and tried a number of possible solutions, they have compiled some
kind of knowledge base (whether accurate or inaccurate) pertaining to this
problem. After unsuccessful problem solving and the accompanying distress,
a person might seek a wide variety of resources for assistance (see Wills, 1987,
for a review of client help-seeking) and may even end up at a counseling center
or some kind of treatment facility. Most important, people seeking psycho-
logical help have expectations about being treated. They often choose a certain
therapist based on a recommendation or reputation, and they are motivated
to change in some way. In short, typically clients seeking psychological help
enter therapy (1) with expectations about change; (2) with expectations about
the therapist and treatment; (3) under distress, and thus in a motivated state;
(4) with the intention of discussing specific problematic situations; and (5) with
a range of information or knowledge about their particular problems. Although
there may well be other variables that depict other aspects of clients seeking
help, we recommend that researchers evaluating the relevance of analogue
methodology begin by considering client variables.
Table 17.1 lists these five client variables and what might constitute relatively
high, moderate, and low degrees of resemblance of each to real-life counseling.
A relatively high degree of resemblance for client expectations might, for example,
entail a client expecting treatment and change, as opposed to a participant
simply expecting course credit (low degree of resemblance). Also related to
client expectations is the way in which treatment is selected. Clients often
choose a type of treatment or therapist based on their presenting problem or
a counselor’s reputation (high resemblance), rather than being assigned to par-
ticular treatments and therapists/interviewers (low resemblance). Distress and
motivation levels may also be polarized; a client is distressed enough to seek
help at a treatment center (high resemblance), whereas a participant is part of
a convenient or captive participant pool and merely seeks course credit (low
resemblance) rather than seeking psychological help and change. Perhaps
most important, actual clients have both “current concerns” or real problems
and a high level of information processing and knowledge about that problem
(high resemblance); conversely, participants assigned to a potentially irrelevant
task have relatively low knowledge levels about the task and thus represent
low resemblance to real-life counseling.
The main point is that several client variables might be considered in
evaluating the generalizability of particular analogue methodologies within
416 part 3 methodological issues
Counselor Variables
A number of variables pertaining to counselors or interviewers also directly
relate to the generalizability of analogue research findings to actual counseling
practice. In the ideal therapeutic counseling relationship, the counselor is
experienced and has a broad range of knowledge about assessments, person-
ality and counseling theories, and the counseling process in general. In addi-
tion, the therapist has high levels of procedural skill—the interpersonal and
counseling skills required to in fact be therapeutic with a client. The therapist
also is highly motivated to provide a therapeutic relationship, as reflected in
establishing Rogerian conditions such as empathy and unconditional positive
regard, or perhaps through other ways of establishing a strong working al-
liance. Thus, the therapist approaches counseling with the expectation that
the therapy will be successful, and that the client will change in some desired
way(s). Finally, the therapist appears to be a credible professional, an expert
and trustworthy person who can provide therapeutic assistance.
Table 17.1 suggests relatively high, moderate, and low degrees of resemblance
of six counselor variables to actual counselors. For example, high degrees of
resemblance characterize therapists possessing a broad range of relevant
knowledge about counseling and high levels of procedural skill. Such coun-
selors have a considerable amount of counseling experience. By contrast, rel-
atively low resemblance to actual therapists characterizes interviewers or
inexperienced counselors who lack both essential knowledge about counseling
and the skills to actually do counseling. The other variables can also be polarized,
so that people resemble actual therapists when they (1) are highly motivated
to provide a therapeutic relationship and facilitate change, (2) expect coun-
seling to be successful and the client to change, and (3) appear credible and
congruent within a therapeutic role. Conversely, a person having a relatively
low resemblance to actual therapists may be characterized as not intending to
provide a therapeutic and caring relationship, but rather being motivated
solely to conduct an interview. Moreover, often the interviewer reflects an
absence of status and credibility cues.
In some previous research in counseling, the therapist variables under
examination did not closely resemble the role or behaviors of a typical thera-
pist; several examples are apparent within what is referred to as the social or
interpersonal influence area in counseling (see Corrigan, Dell, Lewis, &
Schmidt, 1980; Heppner & Claiborn, 1989; Heppner & Dixon, 1981). In the
past, researchers have manipulated a broad range of cues associated with per-
ceived counselor expertness, attractiveness, and trustworthiness. One goal of
chapter 17 analogue research 417
much of this research has been to identify behaviors and cues that enhance the
counselor’s credibility and subsequent ability to affect the client. A common
research strategy has been to examine extreme levels of an independent vari-
able to ascertain whether that particular variable has an effect on client per-
ceptions of the counselor, but all too often the therapist variables have not
resembled the role of a typical therapist closely enough.
For example, in attempting to lower the perceived expertness of an
interviewer, participants have been told, “We had originally scheduled
Dr. __________ to talk with you, but unfortunately he notified us that he
wouldn’t be able to make it today. In his place we have Mr. __________, a student
who unfortunately has had no interviewing experience and has been given
only a brief explanation of the purpose of this study. We think he should work
out all right, though. Now, if you would step this way . . .” (Strong &
Schmidt, 1970, p. 82).
Likewise, in some cases the procedural skills of interviewers have been
manipulated to produce interviewer behaviors that do not closely resemble
those of actual therapists. For example, a counselor portraying an unattrac-
tive role “ignored the interviewee when he entered the office, did not smile at
him, did not look beyond a few cold glances, leaned away from him, and por-
trayed disinterest, coldness, and boredom” (Schmidt & Strong, 1971, p. 349).
Gelso (1979) referred to such procedures as “experimental deck stacking”
and raised questions about the utility of research on such atypical therapist
behaviors. In short, although a considerable amount of information was
obtained about events contributing to clients’ perceptions of counselor credi-
bility, the generalizability of some of this knowledge to actual counseling prac-
tice is questionable because of the relatively low resemblance of the events to
actual therapist behaviors.
Note that the focus here is on the extent to which a person in a therapeutic
role resembles an experienced therapist in terms of knowledge, skill, and
expectations. Trainees—perhaps beginning-level practicum students counsel-
ing their first actual client—may at best only poorly resemble an actual ther-
apist, and thus, it is important not to confuse a trainee with a therapist just
because they both counsel an actual client seeking psychological help at a
treatment center. In sum, to evaluate the generalizability of analogue research
to actual counseling practice, it is important to consider several variables per-
taining to therapists, notably their knowledge bases, skills, expectations, and
role credibility.
Most important, the counselor and client freely interact and exchange a
wealth of information. The client is not a tabula rasa, but instead assimilates
the new information into his or her existing conceptual framework and reacts
in some way (see Hill, Helms, Spiegel, & Tichenor, 1988, for a taxonomy of
client reactions). In a positive counseling situation, the client changes in some
desirable manner, such as learning new behaviors; altering beliefs, attitudes,
or feelings; and adapting to environmental demands more effectively. The
environmental context for the therapeutic situation is typically a professional
treatment center of some sort, a university counseling center, or a community
mental health center.
Table 17.1 provides examples of relatively high, moderate, and low
degrees of resemblance of seven counseling process variables to actual coun-
seling practice. In terms of assessment and interventions, high resemblance
characterizes those situations in which the client is carefully diagnosed and
interventions are specifically targeted to the client’s problems. Low resem-
blance involves a lack of assessment, as well as interventions that are not rel-
evant to a participant’s concerns or problems. Analogues that resemble the
actual therapy process involve multiple 50-minute sessions extended over
several weeks (as opposed to one-shot, 10-minute counseling scenarios). In
addition, analogues that resemble actual counseling include rather extended
interactions between the counselor and client during which a broad range
of information is exchanged, as distinct from analogues that do not include live
interactions between counselor and client. The analogue also can be evaluated in
terms of how much and what kind of information the client processes; high
resemblance entails the client’s processing the counseling experience repeatedly
over time, whereas low resemblance entails the participant’s responding to
counseling scenarios in a hypothetical and often irrelevant manner. The ana-
logue might also be evaluated in terms of therapeutic outcomes: Did the client
change in some desired way? High resemblance involves change of personally
relevant behaviors, thoughts, or feelings, whereas low resemblance entails a lack
of change on the part of the participant, most likely because the counseling
scenario was not personally relevant. Finally, the analogue can be evaluated in
terms of the environment or context of the counseling situation. Analogues
involving a high resemblance to actual practice take place in a professional
environment, such as a treatment or counseling center, whereas an experimental
laboratory setting or classroom offers relatively low resemblance.
The major question surrounding analogue method- But does this mean that all analogues should
ology in counseling research pertains to the closely resemble the conditions of actual coun-
external validity of the results; sometimes ana- seling practice? We believe not. In our opinion,
logue methodology examines circumstances so a considerable amount of information can be
far removed from actual counseling practice that obtained about counseling from tightly con-
the research becomes oversimplified and artifi- trolled analogue studies that do not closely resem-
cial. We propose that the external validity of ble actual counseling. This may well be the case
analogue research can be evaluated in part by early in a line of research, when relatively little
examining variables that depict real-life coun- is known about certain variables. For example,
seling in three categories: (1) the client, (2) the researchers have collected a substantial amount
counselor, and (3) the counseling process and of knowledge from tightly controlled analogue
setting. studies about events that affect clients’ percep-
We suggest that analogues fall on a contin- tions of counselor expertness, attractiveness, and
uum from low to high resemblance to the coun- trustworthiness (see reviews by Corrigan et al.,
seling situation. Given the sparsity of empirical 1980; Heppner & Claiborn, 1989; Heppner &
research, the relationship between analogues Dixon, 1981).
with various degrees of resemblance to actual The extent to which an investigator empha-
counseling is unclear. Investigations that have sizes external validity, and perhaps sacrifices
examined the comparability of analogue studies internal validity when examining events in coun-
and more applied research have produced mixed seling, depends on the knowledge base that cur-
results (see, for example, Elliott, 1979; Helms, rently exists in that particular line of research.
1976, 1978; Kushner, 1978). Clearly, more research One argument states that if relatively little is
is needed. empirically known, the researcher should avoid
Nonetheless, as Kerlinger (1986) has indicated, sacrificing internal validity (see Kerlinger, 1986).
the temptation to incorrectly interpret the results This reasoning emphasizes the role of inter-
of analogue (laboratory) research as they apply nal validity in making scientific advancement.
to real-life phenomena is great. When an investi- Another argument holds that the powerful ana-
gator obtains highly statistically significant re- logue methodology can be used to refine knowl-
sults in the laboratory, it is tempting to assume edge obtained from less internally valid field
that these results would also be applicable to situations (see Gelso, 1979). In this way, the
actual counseling practice. As a general rule, it is strength of the analogue (precision and experi-
questionable to generalize beyond the conditions mental control) can be taken full advantage of,
or population used in a given study. Thus, if an and the results may be more readily interpreted
investigator is primarily interested in generalizing within the existing base of knowledge collected
about clients, counselors, and/or the counseling in the field. Both lines of reasoning have merit
process, then the analogue methodology must and pertain to a central theme of this book—
be evaluated with those particular conditions or namely, that the strengths or weaknesses of any
populations in mind. Depending on the degree of particular methodology for a specific research
resemblance to actual counseling practice, the area are related to the existing knowledge base
investigator may be able to conclude that the and prior research methods used in that area. In
analogue results apply to actual counseling. line with Bordin’s (1965) recommendation, we
Again, as a general rule, relationships found un- suggest that analogue research be combined with
der laboratory conditions must be tested again in empirical investigations conducted in a field set-
the context to which we wish to generalize— ting to create knowledge bases that emphasize
typically, actual counseling. both internal and external validity.
chapter 17 analogue research 423
STIMULUS QUESTIONS
Analyzing Analogue Studies
This exercise is designed to give you practice in 3. What were the main advantages of using an
analyzing an analogue study, weighing the advan- analogue methodology for this study?
tages and limitations of this research method. We 4. In order to make the analogue study similar
suggest that you pair yourself with a colleague in to actual counseling, what did researchers do
your class, respond to the following questions in this study?
individually, and then compare your responses 5. What would be some of the possible practical
with your colleague. Pick one of the three recent and ethical obstacles that the researchers might
examples of analogue studies in the second sec- well have encountered if they would have tried
tion of this chapter; copy and read that article, to examine the same or similar research ques-
and then answer the following questions. tions in an actual counseling situation?
6. Strong (1971) suggested five “boundary con-
1. One of the advantages of an analogue study
ditions” that would increase the external validity
is control of the experimental situation. In the
or generalizability of analogue studies. Discuss
study you chose, what independent variable(s)
the study in terms of these five criteria.
was (were) manipulated by researchers? What
7. The major disadvantage of analogue studies
extraneous variables were eliminated? Was
is the threats to external validity related to
random assignment used in any way?
the artificial circumstances of the study. In
2. Another advantage of an analogue study is a
your view, what factors were the most serious
high degree of specificity in the operational
threats to the external validity of the study?
definition of the dependent variables. What
8. Discuss the main conclusions from the study
was the operational definition of the dependent
relative to the limitations of the study. What do
variables in this study?
you believe can be concluded from this study?
18 CHAPTER Outcome Research:
Strategies and
Methodological Issues
424
chapter 18 outcome research: strategies and methodological issues 425
researchers only test pre- to postchange, have argued for the need to examine
important events from session to session and within sessions. (See Chapter 19
for more details on counseling process research.)
The assumption is that any differences between the groups at the end of the
study were due to treatment.
There are several issues inherent in this design. One is that it is assumed
that a difference in the groups at the end of treatment, should a difference be
found, was due to the efficacy of the treatment. However, as discussed often
in this book, such a difference may be due to chance. This is, of course, a
problem encountered in any experimental design and one that is addressed by
the statistical test. That is, one sets an alpha level that is tolerable; for example,
if alpha was set at 0.05, then the probability of falsely claiming that the treatment
works (i.e., rejecting the null hypothesis) would be less than 5 out of 100.
A second issue relates to the assumption that mental health status at the
end of treatment of the no-treatment control group is truly representative of
how clients would fare without treatment. If the treatment is for a mental dis-
order (e.g., depression), then a no-treatment group is intended to represent the
natural history of the disorder (i.e., the course of the disorder without any
intervention). An issue here is that the experimental situation attenuates the
validity of this assumption. Suppose that the study is examining a treatment
for depression, and those who are enrolled in the study are eager to avail
themselves of the latest treatment. Some clients, when notified that they have
been assigned to the no-treatment group, will become demoralized: “Story of
my life—I never have any good luck. Even when I enroll in a study I get the short
end of the stick.” Other clients may seek treatment elsewhere. In these ways,
the status of the control group clients might not represent how they would
have fared had they never heard of and enrolled in the experimental protocol.
A third issue is an ethical one. If viable treatments are available for a
particular disorder, then withholding a treatment from control group clients
is problematic. One solution is to promise that, should the treatment being
studied be found to be effective, it will be made available to the clients at the
end of the study. Typically, novel treatments being studied are not generally
available, and clients in the control group are informed that they are being
placed on a waiting list for the treatment, in which case the control condition
is labeled “wait-list control group.” However, the mental health status of
those on the wait lists should be monitored so if serious sequelae of the dis-
order should arise, such as suicidal behavior of depressed clients, appropriate
action can be taken. The benefits to society that accrue from comparing a treatment
to a no-treatment control group should always be weighed against the risks of
withholding treatment. However, it could be argued that until a treatment for
a disorder has been established as effective, not providing a treatment does
not disadvantage a client.
This design attempts to take apart the treatments to identify the critical components,
and thus aptly has been labeled as a “dismantling” study. In a study using the
dismantling strategy, the researcher compares the treatment package to the
treatment package with one or more of the critical components removed.
Perhaps the best example of a dismantling study is Jacobson et al.’s (1996)
study of cognitive therapy (CT) (Beck, Rush, Shaw, & Emery, 1979). Al-
though Beck et al. are explicit in their claim that the alteration of core cogni-
tive schemas drives the effectiveness of CT, the treatment contains numerous
components that may be responsible for treatment efficacy. If the hypothesis
is that alteration of core schemas is crucial to the effectiveness of CT, then the
full package of CT should be significantly more effective than a treatment
without the purported critical components. In order to test the mechanisms
involved in the efficacy of CT, Jacobson et al. used a three-group pretest-
posttest experimental design. The first treatment contained the full package of
CT (i.e., containing all its components). The second treatment contained both
the behavioral activation (BA) and automatic thought components of CT, but
no therapeutic actions related to altering core schemas. The third treatment con-
tained only the BA component (i.e., no therapeutic actions related to reducing
automatic thoughts or changing core schemas). All therapists were well
trained and adhered to treatment protocols (see the section on methodological
issues in this chapter for a discussion of adherence in outcome research).
Despite an admitted researcher bias toward CT, results indicated no differences
in the efficacy of the three treatments at termination or at 6-month follow-up,
suggesting that the cognitive components are not critical for the success of
cognitive therapy for depression (see Wampold, 2001).
active ingredients of a medication are responsible for the benefits and not
hope, expectation, or other psychological factors. In medicine, the effects of
the psychological factors are often called placebo effects.
The “gold standard” for establishing the efficacy of drugs is to compare
a pill with active ingredients to a placebo (sugar) pill in a randomized double-
blinded design. Double blind refers to the conditions that render both the
recipient of the pill and the administrator of the pill “blind” to whether it is
the pill with the active medication or a pill with inert ingredients (i.e., the
placebo). (Actually, the best design is a triple blind in which evaluators are
also blind to treatment administered.) To maintain the blind, the active med-
ication and the placebo must be indistinguishable so that no cues are given to
the patient or the administrator that the pill is active or a placebo. If the
design is successful, any superiority of the drug over and above the placebo is
due to the effects of the active ingredients and not due to hope, expectation,
or receipt of an explanation. In this case, the “specificity” of the drug is estab-
lished because the benefits of the drug are due, in part, to the specific ingredients.
Medical researchers are not interested in whether the placebo pill is superior
to no treatment (i.e., the existence of a placebo effect) (see Wampold, Minami,
Tierney, Baskin, & Bhati, 2005).
In 1956, Rosenthal and Frank suggested that psychotherapy borrow the
randomized placebo control group design for the study of psychotherapy to
control for the factors that are common to all treatments, such as the rela-
tionship. The logic of the placebo control group in psychotherapy is that it
contains all these common factors but none of the specific factors of the treat-
ment. For example, if the researcher was testing a cognitive-behavioral treatment
for a particular disorder, the placebo control would have no ingredients re-
lated to cognitive-behavioral treatments (such as behavioral activation, reality
testing, changing core schemas, and so forth) but would likely involve a com-
passionate and caring therapist who responded empathically to the client
(Wampold, 2001; Wampold et al., 2005). Often these control groups are referred
to as “alternative treatments,” “supportive counseling,” or “common-factor
controls,” the latter to indicate that they control for the common factors; the
term placebo control is no longer typically used in psychotherapy research.
There are a number of problems with common factor control groups (see
Wampold, 2001; Wampold et al., 2005), as will be illustrated by the examples
presented below. Foremost is that they cannot be employed in a double-blind
design. The therapists providing the treatment are always aware of whether
they are providing the “real” treatment or the “sham” treatment (the term
sham is often used in medical studies to denote the placebo). Moreover, some
of the common factors are not present in such a control group. For example,
the provision of a cogent rationale is common to all treatments but is typically
absent in common factor controls (see Wampold, 2001). Nevertheless, the
common factor control group design is often used in psychotherapy research.
In one such study, Markowitz, Kocsis, Bleiberg, Christos, and Sacks
(2005) compared the effect of interpersonal psychotherapy (IPT), Sertraline, and
brief supportive psychotherapy (BSP) in the treatment of individuals diagnosed
432 part 3 methodological issues
the validity of the study could be criticized (i.e., a lack of efficacy could be due
to the treatment itself or to a failure to implement the treatment competently).
For example, Snyder and Wills (1989) compared insight-oriented marital
therapy (IOMT) with behavioral marital therapy (BMT). The authors found
that both treatments resulted in positive relationship and intrapersonal vari-
ables and that the two treatments were equally effective. However, at three-
year follow-up it was found that there were significantly fewer divorces in the
IOMT condition (Snyder, Wills, & Grady-Fletcher, 1991). Jacobson (1991)
criticized the study because he contended that BMT was inadequately deliv-
ered: “The data suggest that BMT was practiced with insufficient attention to
nonspecifics” (p. 143). If this is the case, the results are difficult to interpret.
There are several steps to developing and deploying an adequately valid
treatment, including (1) specifying the treatment, (2) training the therapists to
adequately deliver the treatment, and (3) checking whether the treatment was
delivered as intended. The degree to which these steps are completed success-
fully is often referred to as treatment integrity or treatment fidelity.
One of the advances related to specifying the treatment has been the
development of treatment manuals. A treatment manual, according to Luborsky
and Barber (1993), contains three components: (1) a description of the prin-
ciples and techniques that characterize the particular treatment, (2) detailed
examples of how and when to apply these principles and techniques, and (3)
a scale to determine how closely a specific session or treatment conforms
(adherence measure) to the principles and techniques described in the manual.
Luborsky and Barber traced the advent of treatment manuals to Kelerman
and Neu’s (1976) unpublished manual describing an interpersonal approach
for treating depression; the best known manual is the Cognitive Behavioral
Treatment of Depression (Beck, Rush, Shaw, & Emery, 1979). The treatment
manuals developed to date involve far more precise specification of the exper-
imental treatment. Still, treatment manuals in and of themselves do not guar-
antee that the treatment in any particular study is delivered as the researcher
intended. Adherence measures attempt to assess the degree of match between
the intended treatment (as described in the manual) and the treatment actually
delivered in a study. Although at least rudimentary treatment manuals have
existed for some 30 years, in the late 1980s researchers began to develop
adherence measures that are associated with specific treatment manuals
(see Waltz et al., 1993). One adherence measure that has been used in several
counseling studies is Butler, Henry, and Strupp’s (1992) Vanderbilt Therapeutic
Strategies Scale. Developed to measure adherence to Strupp and Binder’s
(1984) treatment manual, Psychotherapy in a New Key, the Vanderbilt
Therapeutic Strategies Scale consists of two scales, labeled Psychodynamic
Interviewing Style (with 12 items) and Time-Limited Dynamic Psychotherapy
(TLDP) Specific Strategies (with 9 items). Trained observers typically watch a
videotape or listen to an audiotape of a counseling session and then use the
Vanderbilt Therapeutic Strategies Scale to indicate the degree to which each
of the 21 items was descriptive of what the therapist had observed.
436 part 3 methodological issues
Measuring Change
As noted by Francis et al. (1991), measuring and analyzing change plays a
central role in many areas of study, but especially when trying to identify
counseling outcomes. There is a voluminous literature dealing with the mea-
surement of change (see, for example, Keller, 2004; Willett, 1997; Willett &
Sayer, 1994). Although we cannot review this entire literature here, three
areas have particular relevance to counseling researchers: (1) clinical versus
statistical significance, (2) hypothesis testing, and (3) growth curve analysis.
In the following subsections we discuss how each of these issues relates to the
conduct of counseling outcome research.
A score greater than 1 on this index indicates that there was more change
from pretest to posttest than the measurement error in the instrument. Using
these criteria, an individual participant in a treatment study is considered to
have improved if her or his posttest score is in the functional distribution and
her or his reliable change index is greater than 1. In a study calculating clinical
significance for a treatment group versus control group design, the researcher
would compare the percentage of participants in the treatment group who had
improved versus the percentage of participants in the control group who had
improved.
Paivio and Greenberg (1995) used the two criteria proposed by Jacobson
and Truax (1991) when examining the effects of “the empty chair” on resolv-
ing “unfinished business.” Thirty-four clients completed either 12 weeks of
counseling using the empty-chair technique or an attention-placebo minimal
treatment condition. Although several measures in the outcome battery were
used in this research, only the SCL-90-R (Derogatis, 1983) and the Inventory
of Interpersonal Problems (Horowitz, Rosenberg, Baer, Ureno, & Villasenor,
1988) were used in calculating clinically significant change. Paivio and Greenberg
averaged the SCL-90-R and the Inventory of Interpersonal Problems scores in
their calculation of clinically significant change. Using normative information,
Paivio and Greenberg determined that before treatment, 44% of their clients
in both experimental conditions were in the functional distribution of SCL-
90-R and Inventory of Interpersonal Problems scores. After treatment, however,
89% of the clients receiving 12 weeks of counseling using the empty-chair
technique were in the functional distribution, whereas only 59% of the clients
in the attention-placebo minimal treatment condition were in this distribu-
tion. The improvement rate for the counseling using the empty-chair tech-
nique was 45% (89% at posttest – 44% at pretest), whereas the improvement
rate for the attention-placebo minimal treatment condition was only 15%
(59% at posttest – 44% at pretest). In terms of the reliable change index, 73%
of the clients in the counseling using the empty-chair technique had a reliable
change index greater than 1, but only 10% of the clients in the attention-
placebo minimal treatment condition had reliable change indexes greater than
1. These results led Paivio and Greenberg to conclude that the counseling
using the empty-chair technique resulted in clinically significant changes in
client unfinished business.
Tingey, Lambert, Burlingame, and Hansen (1996) presented an extension
of Jacobson and colleagues’ method for calculating clinically significant change.
440 part 3 methodological issues
For example, for step 1, Tingey et al. used the SCL-90-R (Derogatis, 1983)
as the specifying factor. According to these authors, one important factor
related to the SCL-90-R is the type of psychological treatment a person
receives, because more symptomatic clients generally receive more intensive
treatment (e.g., inpatient versus outpatient services). Consequently, for step 2
they identified four samples of people who differed in the intensity of psy-
chological treatment they were receiving: (1) asymptomatic (a specially col-
lected healthy sample), (2) mildly symptomatic (unscreened community
adults), (3) moderately symptomatic (people receiving outpatient counseling),
and (4) severely symptomatic (people receiving inpatient counseling). Next,
for step 3, Tingey et al. utilized t and d tests to ascertain whether the four
identified samples were in fact distinct. In this example, their tests showed the
four samples to be distinct. RCIs were calculated for the 10 possible pairs of
samples (step 4). Finally, the authors established cut points and confidence
intervals for each pair of adjacent samples (step 5); these cutoff points repre-
sent the point at which a score is more likely to be in one distribution as
opposed to the adjacent distribution.
One of the most exciting features of Tingey et al.’s (1996) approach is that
any researcher who uses the SCL-90-R as a pretest-posttest measure can use
the information provided in the article to calculate the amount of clinically
significant change for each individual client in her or his study. In addition,
clinicians can use the information to ascertain whether their clients are
making clinically significant change. To illustrate the usefulness of Tingey et al.’s
calculations, Figure 18.1 both illustrates the cutoff points identified in the
article and plots the pretest-posttest changes for two clients involved in the
Missouri Psychoanalytic Psychotherapy Research Project (Patton, Kivlighan,
& Multon, 1997). In this project, community clients seen for 20 sessions of
psychoanalytically oriented counseling were given an extensive battery of tests
prior to and upon completion of counseling; one of these measures was the
chapter 18 outcome research: strategies and methodological issues 441
2.0
Severely symptomatic
1.5
Posttreatment score
Client A
1.0
Moderately symptomatic
.
.5
Mildly symptomatic
Client B
.1 Asymptomatic
SCL-90-R. As can be seen in the figure, Client A made a substantial and clin-
ically significant improvement, as indicated by movement from the severely
symptomatic distribution at pretest to the asymptomatic distribution at posttest.
By contrast, Client B, who both began and ended treatment in the moderately
symptomatic range, exhibited no clinically significant change.
Lambert and his colleagues (Condon & Lambert, 1994; Grundy &
Lambert, 1994b; Seggar & Lambert, 1994) have used this method of defining
clinical significance with several other well-validated psychometric instru-
ments (State-Trait Anxiety Inventory, Auchenbach Child Behavior Checklist,
and Beck Depression Inventory). However, an attempt to define clinically sig-
nificant change groups for the Hamilton Rating Scale for Depression was
unsuccessful because the authors could not find appropriate samples (Grundy
& Lambert, 1994a). Even though the method described by Tingey et al.
(1996) is an exciting extension of Jacobson et al.’s (1984) methodology, for
the most part the instruments used are not commonly used in counseling
outcome research. It would be interesting to examine this method with in-
struments such as the Career Beliefs Inventory (Krumboltz, 1991) or the
Rosenberg Self Esteem Scale (Rosenberg, 1965).
442 part 3 methodological issues
The pretest score can be used in several ways for the analysis of data
obtained in outcome studies. First, the pretest score can be used as a covari-
ate in an analysis of covariance. Second, change scores can be calculated and
an analysis of variance conducted on the change score. Third, the pretest score can
be used to calculate a residualized change score. A description of the statistical
theory that forms the bases of these three methods and the relative advantages
and disadvantages of the methods is beyond the scope of this text (see Keller,
2004; Willett, 1997; Willett & Sayer, 1994).
The most stringent method with which to measure client progress in any
intervention is to collect data over time. Measuring client functioning solely at
pretest and posttest restricts knowledge of client functioning to two points—
at the beginning of therapy and at the end of therapy. More sophisticated methods,
often called growth curve modeling, can be used to investigate therapeutic
effects if multiple measurements across the course of therapy are collected. For
example, Doss, Thum, Sevier, Atkins, and Christensen (2005) were interested
in the mechanisms of change that lead to response in two forms of couple’s
therapy (traditional vs. integrative couple’s therapy). In order to do so they
assessed both measures of relationship satisfaction, communication, behavioral
change, and emotional acceptance four times throughout the course of therapy.
The authors found that change in relationship satisfaction during the first half
of both therapies was related to both emotional acceptance and behavior
change, but that during the second half of treatment acceptance remained
important while changes in behavior became less critical.
Statistical methods that use multiple measurements are able to estimate
the rate and nature of change rather than simply estimating the amount of
change, increase the power of statistical tests, handle missing data points, and
model other important aspects of therapy (such as therapist effects—see the
following section). However, these methods are complicated and can obscure
relatively obvious conclusions. Interested readers should consult Raudenbush
and Bryk (2002) and Snijders and Bosker (1999).
Therapist Effects Recently, Wampold and Bhati (2004) examined the history
of psychotherapy and clinical trials and noted that many important aspects of
psychotherapy have been omitted in current theory and research. One of the
important omissions is the therapist. Because the randomized control group
design was originally used in the fields of education, agriculture, and medicine,
the provider of service was thought to be unimportant. Education was interested
in which programs were effective; relatively unimportant were the teachers,
primarily women, who were thought to be interchangeable. Agriculture was
focused on fertilizers, seed varieties, and irrigation and not on the farmer.
Finally, medicine, attempting to distance itself from charismatic healers such
as Mesmer, emphasized the potency of medicines and purposefully ignored
physician effects. Psychotherapy research, borrowing the randomized design,
similarly ignored the provider of service—that is, the therapists. Indeed,
apparently there are no clinical trials of psychotherapy that have considered
therapist effects in the primary analyses of the data (Wampold & Bhati, 2004).
chapter 18 outcome research: strategies and methodological issues 445
design has the advantage that the general skill of the therapist is balanced.
However, problems arise if the therapists have more allegiance to or skill in
delivering one of the treatments. In the nested design, therapists deliver one of
the treatments only (i.e., the therapists are nested within treatment). Ideally,
the therapists delivering a particular treatment would have allegiance to and
skill in delivering the respective treatment and allegiance and skill would be
balanced. A commendable effort to balance allegiance and skill was shown
by Watson et al. (2003), who compared cognitive behavioral treatment and
process experiential treatment for depression. The therapists for each treatment
were adherents to their respective treatment and then were trained by inter-
nationally recognized experts in this treatment.
Appropriately including therapists in the design and analysis of outcome
research is more complex than simply ignoring therapists. However, the extra
effort is needed to correctly estimate treatment effects, and it has the potential
to detect therapist effects should they exist. Generally it is recommended that
therapists be considered a random factor in the model; that is, therapists are
assumed to be randomly selected from the population of therapists. Of course
this is rarely done, but the issues are comparable to the ones created by the
fact that we do not randomly select study participants from a population of
participants (see Chapter 8 and Serlin, Wampold, & Levin, 2003). In the
nested design, the analysis would be a mixed model in which the treatments
are considered to be fixed (i.e., not drawn from a population of treatments)
and the therapists are a random factor (Wampold & Serlin, 2000). In a bal-
anced design, the analysis would be a relatively simple mixed model analysis
of variance; when the number of clients per therapist and the number of ther-
apists per treatment varies, it is recommended that a multilevel model (some-
times called a hierarchical linear model) analysis be conducted (Raudenbush
& Bryk, 2002; Snijders & Bosker, 1999).
STIMULUS QUESTIONS
Three Exercises to Enhance Students’
Understanding of Outcome Research
1. Suppose that you have developed a new 2. Find three outcome studies. Describe the
treatment for a particular disorder or problem. design of each and classify them into one of
Discuss how you would design a series of the types of outcome research discussed in
outcome studies to establish that this treatment this chapter. Discuss the degree to which each
is the treatment of choice. What steps would need study handled the issues presented in this
to be taken before the first study was undertaken? chapter appropriately.
What precautions would need to be instituted 3. Find several outcome studies and discuss how
to protect the research from threats to validity? allegiance was handled in each study.
19 CHAPTER Design Issues Related
to Counseling Process
Research
448
chapter 19 design issues related to counseling process research 449
some early and current counseling process research, summarizing some of the
major research findings in the counseling process literature. The second section
of this chapter focuses on major design issues of particular relevance to coun-
seling process research, such as the types of measurements to use in examining
the counseling process or ways of analyzing process data. The third section
discusses representative examples of instruments used to assess different aspects
of counseling process research.
tend to involve more abstract behaviors that occur over longer periods of
time:
1. Ancillary behaviors such as speech dysfluency, or nonverbal behaviors
such as the body lean of the counselor
2. Verbal behaviors, such as therapist response modes, including therapist
self-disclosure, interpretation, and confrontation (see Hill, 1992, for
reviews of this literature)
3. Covert behaviors, such as therapist intentions “to support” or “to challenge”
(see Hill, 1992, for a partial review)
4. Content, which examines the topics of discussion and typically focuses on
client behaviors
5. Strategies, which focus on therapist techniques such as identifying maladaptive
cognitions
6. Interpersonal manner, such as therapist involvement, empathy, and congruence
7. The therapeutic relationship, such as the working alliance and control of
the topic of discussion
For example, Wampold and Kim related patterns of client interaction to eval-
uations of sessions made by counselor, client, and observers.
Two conceptual systems established in early process research remain
subjects of current research. F. P. Robinson’s book Principles and Procedures
in Student Counseling (1950) had a seminal influence on early process research.
In addition to his emphasis on session content, Robinson also devised a
system of classifying counseling behavior based on the concept of counselor
roles and subroles. This concept is still the basis for category systems used to
classify counselor actions. The second conceptual system that influenced early
process research was T. Leary’s (1957) system of interpersonal diagnosis.
Unlike Robinson’s ideas, which led to the classification of counselor behavior
in terms of actions or response modes, Leary’s system classified counselor
behavior in terms of its interpersonal style. W. J. Mueller (1969) used the
Leary system to study the manifestation of transference and countertransfer-
ence in counseling interviews. Researchers continue to apply Leary’s system
in examining client and counselor interpersonal style (see, for example, Strong
et al., 1988).
Even though several studies conducted in the 1950s could be classified as
counseling process research, Carl Rogers’s (1957) formulation of the neces-
sary and sufficient conditions for positive therapeutic change probably served
as the major stimulus for much of the process research in the 1960s and 1970s
(Toukmanian & Rennie, 1992). Rogers identified what he believed to be the
core therapeutic conditions, which in essence were largely process variables,
such as therapist empathy and unconditional positive regard. The large volume
of process and outcome research that followed empirically tested Rogers’s
postulations. Although the research initially supported Rogers’s ideas, subse-
quent research revealed that the effective elements of therapeutic change are
more complex than he first postulated (see reviews by Orlinsky & Howard,
1978; Orlinsky et al., 1994, 2004; Parloff et al., 1978).
have suggested for some time that race and cultural factors affect the dynamics
of the counseling process (e.g., Helms, 1990; Sue & Sue, 1990; Vontress,
1970), relatively few empirical studies have been conducted on racial and
cultural factors in the therapy process (Helms, 1994). Some early research
suggested that black clients report lower levels of rapport with white coun-
selors than with black counselors (see Atkinson, 1983), although subsequent
research suggested that other variables, such as attention to cultural issues,
similar values, attitudes, and personalities, also are important variables affect-
ing client reactions (e.g., Atkinson & Lowe, 1995; Thompson, Worthington,
& Atkinson, 1994; Zane et al., 2004). Ridley, Mendoza, Kanitz, Angermeier,
and Zenk (1994) provided an excellent conceptualization integrating race and
cultural factors (particularly cultural sensitivity) into the counseling process
by focusing on the cognitive schema of counselors. Some useful resources for
beginning process researchers interested in racial and cultural factors in coun-
seling are Chapter 15 as well as Atkinson and Lowe; Grieger and Ponterotto
(1995); Leong, Wagner, and Tata (1995); Ponterotto and Casas (1991); Pon-
terotto, Fuertes, and Chen (2000); Ponterotto and Grieger, in press; Ridley et al.
(1994); Sue, Zane, and Young (1994); Thompson et al. (1994); and Zane et al.
Clearly, race and cultural factors are important topics in process research and
merit additional attention.
Although process research in psychotherapy has a long history and much
has been learned about process variables that contribute to effective outcome,
the specialty of career counseling has not had a focus on the examination of
process variables. As Swanson (1995) commented: “Although we know a
great deal about the effectiveness of career interventions in general, we know
considerably less about career counseling specifically, and further, we know
almost nothing about the career counseling process” (p. 217). M. Heppner and
Heppner (2003) published a research agenda for career counseling process
research in which they identify 10 avenues for future investigations in this
area. In the article they review some of the areas that have led to the richest
information in psychotherapy process research and discuss possible applica-
tions to career counseling.
When process research has been conducted in the area of career counseling,
researchers have discovered both similarities and differences with psychother-
apy processes. For example Kirschner, Hoffman, and Hill (1994) found that,
similar to psychotherapy research, when counselors used challenge and
in-sight it was most helpful. Other research indicates that psychotherapy and
ca-reer counseling verbal responses were similar and tended to be active, directive,
and focused on problem solving (Nagel, Hoffman, & Hill, 1995). Conversely,
in their career counseling study, Multon, Ellis-Kalton, M. Heppner, and
Gysbers (2003) found that self-disclosure, which has been found to be helpful
to building the relationship in psychotherapy, was actually negatively related to
the working alliance. On closer examination it appears it might be the type of
self-disclosure used that accounted for these contrasting findings.
Similarities and contrasts have also been found between career counseling
and psychotherapy process research regarding the importance of the working
454 part 3 methodological issues
topics after beginning a process study. Beginning researchers may have the
misperception that because process research can be relevant and exciting, it
will also be easy. The many details and the complexity can translate into more
opportunities for problems to arise. Hill (1992) recommended that beginning
process researchers become apprentices to an experienced process researcher
to facilitate learning the methodological skills needed to conduct process
research. Being a part of a research team is another way of making the demands
more manageable while learning about the complexity of the methodological
issues.
and a naturalistic setting is not a categorical either/or decision, but rather one
that involves a continuum. Some analogue designs emphasize internal validity
more than others (see Chapter 17), and some naturalistic designs incorporate
analogue methods to create quasi-naturalistic settings (see Chapter 8). In the
final analysis, the decision regarding the type of context or setting is thus a
matter of the degree to which the research will be conducted in a context that
maximizes either internal or external validity.
Another key design issue is whether to use a quantitative or qualitative
approach (see Chapters 10 and 11). It is critical to carefully contemplate both
the type of research questions of interest and the type of data that will best
answer those questions at this time. For example, despite an abundant litera-
ture on reducing client resistance, there has been very little research on either
willingness to work through interpersonal difficulties in family therapy or a
family’s level of engagement/disengagement. Because of the lack of empirical
data, Friedlander et al. (1994) used a qualitative, discovery-oriented approach
to examine events related to successful, sustained engagement in family ther-
apy. Moreover, they were able to develop a conceptual model of successful,
sustained engagement based on their qualitative analysis. Researchers are
encouraged not to dichotomize their research approach as either quantitative
or qualitative, but to integrate these approaches into their research to provide
the most informative data possible (see, for example, Cummings, Martin,
Halberg, & Slemon, 1992).
What to Measure?
Once a general research topic has been identified, a subsequent issue concerns
the aspect of the counseling process to examine. At the most basic level,
researchers must decide whether they want to examine aspects of the individ-
ual participants’ behaviors or aspects of the developing relationship or system.
Process research focuses on either a participant (counselor or client), the rela-
tionship, or some combination thereof. Indeed, some of the most common
activities examined in process research are therapist techniques, client behav-
iors, and the therapeutic interaction. In group and family counseling, the
clients are obviously multiple individuals, and the counselor dimension often
involves a measurement of the activities of co-counselors. In individual coun-
seling, the relationship involves how the client and counselor work together.
In group and family process research, this relationship dimension is usually
referred to as the group (cohesion, norms) or family (closeness, involvement)
process. We will discuss issues pertaining to what to measure specifically with
regard to individual participants and the client–counselor relationship.
Suppose that a researcher, Dr. B. Famous, decides to conduct a process
study that examines the counselor’s behavior. But there are many counselor
behaviors: what the counselor said, how the counselor said it (for example,
emphatically or angrily), how often the counselor said it, when the counselor
said it, and so on. So the insightful Dr. Famous knows it is necessary to con-
sider specific types of counselor behaviors that might be examined. Elliott’s
chapter 19 design issues related to counseling process research 457
Whose Perspective?
Another major question that process researchers must address is: From whose
perspective should the counseling process be evaluated? A considerable
amount of evidence suggests that client, counselor, and observer perspectives
on the counseling process may offer quite diverse views of what happens in
counseling (see Eugster & Wampold, 1996; Lambert, DeJulio, & Stein, 1978;
Wampold & Poulin, 1992). For example, Dill-Standifond, Stiles, and Rorer
(1988) found that clients and counselors had very different views of session
impact (in terms of depth, smoothness, positivity, and arousal).
In part, differences and similarities across perspectives may be related to
the type of rating used. For example, Lambert et al. (1978) found only minimal
relationships among ratings of facilitative conditions across the three per-
spectives. Likewise, Gurman (1977) found that client ratings of empathy were
positively related to therapy outcome, but ratings from outside observers were
not related to outcome. Other research reveals little difference between, for
example, client and counselor perspectives on the rating of therapist behavior
(Carkhuff & Burstein, 1970). Eugster and Wampold (1996) found that regression
458 part 3 methodological issues
intentions), should you collect data from the whole counseling session, or per-
haps from only the first (or last) 15 minutes of the session? Although it seems
desirable to collect a lot of data from multiple sessions, which would increase
the external validity or generalizability of the findings, such a strategy quickly
results in an overwhelming amount of data. Hill et al. (1983) for example,
analyzed over 11,000 responses from one intensive single-subject study;
another study that examined data from only three group counseling sessions
had 5,833 adjacent turns in the group communication to categorize (Friedlander,
Thibodeau, Nichols, Tucker, & Snyder, 1985). Process researchers must decide
how much of a session or how many sessions to use in their analyses. Even
though little empirical evidence pertains to this issue, we briefly discuss sam-
pling and its role in counseling process research.
Friedlander and colleagues (1988) empirically examined the issue of how
much to measure. Specifically, they asked: (1) What fraction of an interview,
if any, best represents the entire interview? and (2) At what point in the interview
should this “best” sample be drawn? Friedlander et al. decided that the answer
to these questions depended on the researcher’s purpose. If the researcher was
interested in group data, such as the process across several different counseling–
client dyads, then fairly small segments of sessions (as little as 10%) were rea-
sonable representations of an entire session. For a group design, the point in
the interview at which the researcher drew the sample did not seem to matter.
Friedlander et al. did recommend, however, that a “mini-generalizability” study
be routinely conducted when using a group design; that is, a subsample of the
data should be analyzed to ascertain that the use of small segments is appro-
priate in that particular case. On the other hand, Friedlander et al. recom-
mended that entire sessions be used when an intensive single-subject strategy
is used; their data suggested that sampling of single-subject data leads to enormous
differences in the conclusions.
Additional data suggest that clients from different diagnosis groups show
different patterns of experiencing the counseling process; for example, one
study found that neurotic clients showed an upward trend in experiencing
during a counseling session, whereas schizophrenics showed a saw-toothed
pattern (Kiesler, Klein, & Mathieu, 1965). Other data suggest that patterns of
client responding change over the course of many sessions (see, for example,
Kiesler, 1971). Likewise, some data suggest that counselors change their
behavior across counseling sessions (see, for example, Hill & O’Grady, 1985;
Mintz & Luborsky, 1971). These studies underscore potential differences
across clients and counselors over time.
Thus, the question of whether to use sample session segments seems to
depend on the type of research question and the nature of the design.
Certainly, investigators must be alert to individual differences across clients,
particularly across very different diagnostic groups. Moreover, some differences
in counselor behavior seem likely over time. Accordingly, researchers might
choose to include both individual differences and time intervals in their research
questions. If these variables are not desired in the research questions, then the
researcher could decide to control these variables to reduce potential confounds.
460 part 3 methodological issues
Apart from the individual differences and time intervals, segments of as little
as 10% of the session will likely yield acceptable results with group designs,
whereas with single-subject designs it is best to use an entire session without
sampling. Clearly, the issue of how much to measure is relatively unexplored
and needs additional empirical research. Sampling and its effects on external
validity merit serious attention in counseling process research.
Researchers typically use more than one rater to assure some reliability
and validity of the ratings. The recommended number of raters ranges from 2
(the most common number) to 10 (Mahrer, Paterson, Theriault, Roessler, &
Quenneville, 1986). Lambert and Hill (1994) recommended that the number
of judges be linked to the difficulty of the rating task: The higher the reliability
expected, the fewer judges are needed, whereas if lower reliability is expected,
pooling ratings from several judges increases the probability of getting closer
to the “true” score. Because the skills of the raters can have a major effect on
the results of a study (Lambert & Hill), it behooves the researcher to carefully
select skillful raters. But what constitutes a “skillful” rater? Although empirical
evidence is lacking, previous researchers have selected raters based on levels
of clinical experience, intelligence, theoretical sophistication, interpersonal
skills, motivation for the specific project, attentiveness to detail, dependability,
trustworthiness, and a sense of ethics (see, for example, Hill, 1982; Lambert
& Hill; Mercer & Loesch, 1979; Moras & Hill, 1991).
Raters are “two-legged meters” (Lambert & Hill, 1994, p. 99) who are
asked to interpret and make judgments on observed events. As such, judges
bring biases to the rating task and are subject to the broad range of inference
errors that humans often make (Gambrill, 1990). Thus, to reduce “error” cre-
ated by the judges, researchers typically provide raters training concerning
description of the rating system, discussion of sample items, rating of sample
items, and monitoring of rater drift over the course of the rating task. To
ensure that all judges receive exactly the same training, all judges should be
trained together in one group.
In the first step, the rater must be introduced to the rating system. This
step often includes a discussion of the theory base from which the rating
system is derived, as well as a description of the specific system. Introduction
to the rating system is enhanced by the use of manuals describing examples of
expert ratings. Next, the rater uses the rating system on a set of sample items.
This aspect of the training is often enhanced if each rater does the ratings
independently and if raters are given the whole range of events to be rated so
they can become familiar with all points of the ratings scales. Once these rat-
ings are made, the raters discuss the ratings they made and the reasons for
them. This step is enhanced if “expert” ratings of the sample items are avail-
able so trainees can compare their ratings to those of the experts. Discussion
can help judges refine their knowledge of the rating system and make finer
distinctions in their ratings. The previous two steps (rating sample items and
discussing the ratings) continue until a specified level of agreement exists
among the raters. Suppose that after the judges rate the first set of sample
items (but before the general discussion of the items) the experimenter calcu-
lates the inter-rater reliability between two raters and finds that the level of
agreement is .50. (That is, the judges agreed in their ratings only half the
time.) Typically, this rate of agreement is considered to be too low. Rather
than allowing the judges to proceed to rate all of the responses (and hope their
inter-rater reliability goes up!), the experimenter would typically discuss the
items and then provide another sample test. Although the “acceptable” level
462 part 3 methodological issues
of inter-rater agreement depends on the difficulty of the rating task, .80 is gen-
erally considered acceptable.
After an acceptable level of inter-rater reliability is obtained, the judges
begin the actual rating task. It is essential that judges do their ratings totally
independent of each other. If each judge does not rate all of the data, then judges
should be randomly assigned to various rating tasks to reduce potential error
due to any particular judge. Even though judges typically begin the rating task
conscientiously, the accuracy of their judgments might decline over time due
to fatigue or repetition, or their understanding of the anchor points on rating
scales might change. This phenomenon is called rater drift. Marmar (1990)
recommended recalibration sessions during the rating task to maintain fidelity
to the original training standards. Similarly, Elliott (1988) has recommended
“the care and feeding of raters”—that is, listening to raters, reinforcing them,
and if possible, removing administrative or procedural problems that make the
rating task more cumbersome. The “care” of raters is a very important process that
requires close monitoring.
At the end of the rating, the experimenter calculates the inter-rater relia-
bility of all ratings. In some ways this is an assessment of the rater training
program. Hill (1982) stated that researchers should calculate both inter- and
intra-rater reliability. Intra-rater reliability assesses whether an individual rater
makes consistent ratings over time, whereas inter-rater reliability assesses the
amount of agreement among raters. (See Tinsley and Weiss, 1975, for meth-
ods of calculating inter- and intra-rater reliability.) Once rating has concluded,
the experimenter can debrief judges concerning the specific purpose and
hypotheses of the study. To ascertain whether the judges were indeed “blind”
to the nature of the study (see Chapter 16), the experimenter can first ask
judges to relate their ideas about the purpose of the study.
It is appropriate here to introduce the issue of training participants to
make certain ratings at the time of data collection. For example, Martin,
Martin, and Slemon (1989) found that counselors used different categories of
the Hill and O’Grady (1985) Intentions List to record nominally similar
descriptions of their intentions. The authors’ solution to this problem was to
ask therapists to give verbal descriptions of their intentions and then have
raters classify these descriptions. This solution is very expensive in terms of
time and effort. An alternative approach might be to train the counselors/raters
in the use of the intention system. Although providing people training in intro-
spection has a long history in psychology (Wundt, 1904), it is rarely used in
modern process research. In this case, counselors could be trained to use the
Intentions List by first learning to classify a series of intention statements to a
specified level of accuracy. Only after they reached this level on the practice
statements would they start to rate their own intentions. Rater drift could be
assessed by administering additional series of intention statements to be rated
throughout the study.
link between counseling process and outcome. This trend has raised impor-
tant questions about statistically analyzing process data. Traditionally, process
has been linked to outcome using a correlational strategy. If, for example, a
researcher found a significant positive relationship between counselors’ use
of interpretation and session depth, then he or she could infer that interpre-
tation was an important process element. Unfortunately, correlational designs
cannot enable the researcher to identify causal relationships (whether coun-
selor interpretations caused session depth).
A second example of a correlational design (an ex post facto design)
involves comparisons across successfully and unsuccessfully treated cases. In
this instance the researcher would examine whether the frequency of a re-
sponse (for example, open-ended questions) differed across the two kinds of
cases. Again, more frequent occurrence of open-ended questions in the suc-
cessful cases may or may not mean that counselors who use more open-ended
questions will be more successful.
Thus, correlational designs in counseling process research, although help-
ful, have limited utility. In fact, Gottman and Markman (1978) are quite crit-
ical of these types of correlational designs in process research. They argued
instead for analyses that examine the direct effect of counselor behavior on
subsequent client behavior. Stiles (1988) also observed that counselors often
use a behavior more frequently not because it is working well, but because it
is not working well. For instance, a counselor might offer a lot of interpreta-
tions because the client is being “resistant” and is rejecting the interpretations.
This situation would cause a higher number of counselor interpretations and
likely a low rating for session depth. Thus, the correlation between these two
variables alone presents rather misleading information.
An alternative way of examining the relationship between counselor
behavior and client response is sequential analysis, a set of statistical tech-
niques that examine the mutual influence of counselor and client behaviors.
For example, sequential analysis can be used to examine the likelihood of a
client response (for example, self-disclosure) given a prior counselor response
(for example, interpretation). At a more sophisticated level, sequential analy-
ses can examine issues such as control and power (that is, is the counselor’s
response made more predictable by knowing the client’s preceding response,
or vice versa?). Excellent descriptions of these types of analyses can be found
in Bakeman and Gottman (1986), Claiborn and Lichtenberg (1989), Gottman
and Roy (1990), and Wampold (1995).
A study by Wampold and Kim (1989) illustrates the power of sequential
analysis in examining counseling process. Wampold and Kim reanalyzed
data from Hill, Carter, and O’Farrell’s (1983) study using an intensive single-
subject design across 12 counseling sessions. The process variables used in this
study were Hill’s counselor and client verbal response category systems. Cat-
egories for counselor responses included minimal encouragement, silence,
approval-reassurance, information, direct guidance, closed question, open
question, restatement, reflection, interpretation, confrontation, nonverbal ref-
erent, self-disclosure, and other. Client categories included simple response,
464 part 3 methodological issues
data points into predetermined phases (for example, early, middle, and late
periods) and used repeated measures analysis of variance or repeated mea-
sures multivariate analysis of variance. As discussed by Willett, Ayoub, and
Robinson (1991) and Francis, Fletcher, Stuebing, Davidson, and Thompson
(1991), these methods have proven problematic from both a logical and a sta-
tistical perspective. Growth modeling, also known as growth curve analysis,
offers a powerful alternative method for analyzing repeated measures types of
counseling process data. In the first step of growth curve analysis, a theoreti-
cal or heuristic growth model (for example, linear or quadratic curve) is fit
both to the data from an individual counseling dyad or a counseling group
and to the sample of dyads or groups as a whole. The parameters from these
initial models are then used in the second step of the modeling process as out-
comes (that is, dependent variables) onto which other variables are regressed.
For example, if a linear growth model is used in group counseling research,
each group will have (1) an intercept term that represents that group’s process
score (for example, level of cohesion) at a particular time, and (2) a slope term
that represents the linear change in the group’s process score (for example,
cohesion) over time.
Growth modeling has typically been applied to outcome data. Kivlighan
and Shaughnessy (1995) and Kivlighan and Lilly (1997) are two examples of
studies that have used growth modeling with counseling process data. In the
Kivlighan and Lilly study, hierarchical linear modeling (HLM) was used to
estimate growth curves from group climate data. Based on the theory of hier-
archical linear models developed by Bryk and Raudenbush (1992), growth
modeling was used in a two-level analysis to estimate growth curves from the
within-group, group climate data and to relate the growth parameters from
this within-group analysis to the between-group variable of “group success.”
Conceptually, HLM involves a two-stage analysis. In the first or uncon-
ditional model, the growth trajectory of each individual group is modeled or
characterized by a unique set of parameters. This set of parameters, which is
assumed to vary randomly, is then used in the second or conditional model as
dependent variables in a series of regressions. Arnold (1992) has summarized
this analytic technique by describing HLM as “regressions of regressions” (p. 61).
Hierarchical linear modeling differs from trend analysis in analysis of
variance (ANOVA) or multivariate analysis of variance (MANOVA) in that
individuals (as opposed to group means) are being modeled; that is, in trend
analysis, individual variance is subsumed under the error term. Variance in the
individual growth parameters (that is, across groups) can then be plotted
against correlates of change. This type of analysis allows for “a different set
of research questions” (Francis et al., 1991, p. 31) than is found in more tra-
ditional research (that is, based on group means only). For a detailed dis-
cussion of the statistical aspects of hierarchical linear models, see Arnold
(1992), Bryk and Raudenbush (1992), Francis et al., and Raudenbush and
Chan (1993).
Kivlighan and Lilly (1997) investigated how the shape or function of
group climate growth patterns were related to the amount of participant-rated
chapter 19 design issues related to counseling process research 467
benefit from the group experience. HLM analyses were run three times, once
for each of the three aspects of group climate (engaged, conflict, and avoiding).
Examination of the relationship between group climate ratings and group
benefit proceeded through a series of model-building steps.
Several specific findings related group climate to outcome. For example,
a quadratic, high-low-high pattern of engaged (cohesion) development was
related to increased member benefit. In addition, the pattern of engaged devel-
opment was more predictive of member benefit than was the absolute level of
engaged feeling. In fact, the pattern of group climate development consistently
accounted for more of the variance in member benefit than did the absolute
level of the group climate dimension. This suggests that pattern over time is
an important dimension of counseling process. We believe that growth mod-
eling provides a powerful methodology for counseling researchers interested
in examining repeated measures types of data. Also see Kivlighan, Coleman,
and Anderson (2000) for examples of growth modeling in group counseling.
Table 19.1
Level of Measurement
Instruments
Representative Counseling Process Micro Global
of the
Counseling Ancillary behaviors Client Vocal Quality Arousal Scale (Burgoon,
Process by (Rice & Kerr, 1986) Kelly, Newton, & Keeley-
Level of Dyreson, 1989)
Measurement
Verbal behaviors Hill Counselor Verbal Supervision Questionaire
Response Modes Category (Worthington & Roehlke,
System (Hill, 1985) 1979)
Covert behaviors Thought-Listing Form Brief Structured Recall
(Heppner et al., 1992) (Elliott & Shapiro,
1988); Session Evaluation
Questionnaire (Stiles
& Snow, 1984)
Comprehensive Scale
of Psychotherapy Session
Constructs (Eugster &
Wampold, 1996)
Content Hill Interaction Matrix Discussion Units
(Hill, 1965) (Dipboye, 1954)
Strategies Counselor Intentions Therapist Session
List (Hill & O’Grady, Intentions (Stiles et al.,
1985) 1996)
Client Reactions System
(Hill et al., 1988)
Interpersonal manner Interpersonal Check List of
Communication Rating Interpersonal
Scale (Strong et al., 1988) Transactions (Kiesler,
1984)
Therapeutic relationship Categories of Semantic Working Alliance
Cohesion Analysis Inventory (Horvath &
(Friedlander et al., 1985) Greenberg, 1989)
Quality Client Experiencing Scale Penn Adherence and
(Klein, Mathieu-Coughlan, Competence Scale for
& Kiesler, 1986) Supportive-Expressive
Therapy (Barber & Crits-
Christoph, 1996)
their conscious control. This lack of conscious control suggests that nonverbal
behaviors may provide a more sensitive indicator of client or counselor emotional
state than self-report measures. In line with this assumption, Mohr, Shoham-
Salomon, Engle, and Beutler (1991) found that nonverbal behaviors provided
a more accurate assessment of client arousal than did verbal reports.
Two classes of nonverbal behaviors have interested counseling process
researchers. Vocalics involve the voice qualities of the interactants; examples
are voice tone or rate of speech. Kinesics are body movements such as facial
expressions and gestures. Counseling researchers have measured both kinesics
and vocalics in their process studies.
Rice and Kerr’s (1986) Client Vocal Quality (CVQ) measure is probably
the most widely used vocalic index. The CVQ rater uses auditory cues to cat-
egorize client or counselor statements into one of four categories: (1) focused,
(2) external, (3) limited, and (4) emotional. The focused category describes an
inner-directed exploratory voice quality. An outer-directed, lecturing type of
voice quality is considered external. The emotional category is used when the
voice “breaks its normal platform and expresses emotions.” Finally, a low-
energy, wary type of voice is classified as limited. The four CVQ categories are
mutually exclusive, so any individual speech act can have only one vocal quality
rating. Rice and Kerr have shown that the CVQ has good reliability. In addition,
CVQ ratings have been shown to predict productive engagement in both client-
centered and Gestalt counseling; even though the CVQ system has mainly
been used to rate client speech, it could equally apply to counselor speech.
Greenberg and Foerster (1996) used the CVQ in their task analysis of the
process of resolving unfinished business. Task analysis is a specific approach
to process research that incorporates eight steps, as outlined by Greenberg and
Newman (1996): (1) explicating the implicit cognitive map of expert clinicians;
(2) describing the therapeutic task and the context in which this task exists;
(3) verifying that the task is therapeutically significant; (4) constructing ra-
tional models of successful task performance; (5) empirically describing cases
of actual task performance; (6) revising the rational model by comparing the
rationally derived and empirically derived task performances; (7) validating the
task model by comparing successful and unsuccessful performances; and (8)
relating process to outcome.
In the Greenberg and Foerster (1996) study, the CVQ was a critical com-
ponent of the verification phase of the task analysis paradigm. The focus of
this study was the use of the Gestalt empty-chair dialogue to resolve unfin-
ished business, defined as lingering bad feelings toward a significant other.
Fourteen counselors provided 46 audiotaped counseling sessions containing
empty-chair dialogues. Eleven “resolved” and 11 “unresolved” sessions were
identified through client, counselor, and judge ratings on a five-point resolu-
tion scale. Once these 22 sessions had been identified, clinical judges selected
2-minute sections from each tape for CVQ rating by a second set of judges.
Greenberg and Foerster found that the client dialogue was more focused and
emotional in the “resolved” sessions than in the “unresolved” sessions.
According to the authors, the CVQ data confirmed that both (1) the intense
470 part 3 methodological issues
Burgoon et al. (1993) also examined the relationship between clients’ end
of session ratings of resolution (the extent to which the problem worked on
was resolved) and the nonverbal arousal measures for clients in the FEP
groups. Random movement and vocal relaxation were both associated with
client-judged resolution. Specifically, greater amounts of random movement
and moderate amounts of vocal relaxation during middle sessions were asso-
ciated with greater subjective resolution. The results of the Burgoon et al.
(1993) study suggest that level of nonverbal arousal is differentially associated
with various treatment modalities. In addition, the results suggest that some
aspects of nonverbal arousal may be linked to client outcome. The results are
encouraging for the continued assessment of macro-level ancillary behaviors
in counseling process research.
see, for example, Holloway, Freund, Gardner, Nelson, & Walker, 1989) have
relied on supervisor or supervisee ratings of supervision behavior. In this type
of research, supervisors and/or supervisees are presented with a list of items
describing supervisor behaviors and asked to indicate the general frequency
with which the behaviors occurred in the supervision session or during the
course of supervision. An example of an instrument in this area is the Su-
pervision Questionnaire (Worthington & Roehlke, 1979).
An example of this type of research is provided by Krause and Allen
(1988), who were interested in examining Stoltenberg’s (1981) developmental
model, which predicted that supervisors would use different behaviors with
trainees at different developmental levels. These authors gave a list of 37 items
describing supervisor behavior to 87 supervisors and 77 supervisees. The re-
sponses of the supervisees and supervisors were factor-analyzed separately.
Eight factors accounted for the variance in supervisors’ ratings: teacher, coun-
selor, respectful sharing, satisfied colleague, dynamic counselor, perceived
impact, laissez-faire, and preparation. A factor analysis of the same items
from the supervisees’ perspective yielded five factors: supervisor as mentor,
supervisor as counselor, directive supervision, supervisor as dynamic coun-
selor, and process supervision.
Krause and Allen (1988) used these factors to examine the hypothesis that
supervisors would vary their behaviors with supervisees. Using the eight
supervisor-derived factors to examine this hypothesis, supervisors rated three
of these factors differently depending on the supervisees’ training levels.
Specifically, structuring and directing behaviors decreased, and collegial and
consulting relationships increased as supervisees advanced in development.
An identical analysis using the five supervisee-derived clusters yielded no
differences in supervisees’ ratings of supervisor behaviors for supervisees at
different developmental levels. In sum, the supervisors saw themselves as
varying their behaviors with supervisees of different developmental levels, but
the supervisees did not.
Did the supervisors actually vary their behaviors with supervisees at dif-
ferent levels? Unfortunately, we cannot provide a complete answer to this
question because of the measures used to examine supervisor verbal behavior.
Without some outside perspective from which to examine the process at the
micro level, it is impossible to determine the reliability of either the supervi-
sors’ or supervisees’ accounts of the supervisors’ behavior.
more detail later in this chapter when we address process measures of the rela-
tionship.) The Positivity and Arousal scales measure postsession mood. Pos-
itivity, as the name implies, is a measure of how positive or negative the client
feels upon completing the session. The Arousal scale is a measure of how much
emotional arousal the client feels after completing the session. These SEQ scales
have been used to measure consequences (that is, the relationship between ther-
apist response modes and client postsession mood) and antecedents (that is, the
relationship between postsession mood and counseling outcome).
Counselor reactions have been measured at a global level with instru-
ments parallel to those used to measure global client reactions. For instance,
the SEQ has been used to record the counselor’s session-by-session reactions.
A final instrument in this category is the Comprehensive Scale of Psycho-
therapy Session Constructs (CSPSC; Eugster & Wampold, 1996). Basically, the
CSPSC measures nine aspects of the session (patient involvement, patient comfort,
patient learning, patient real relationship, therapist involvement, therapist
comfort, therapist expertness, therapist interpersonal style, and therapist real
relationship) from the perspective of both patient and therapist, as well as a global
evaluation of the session from both perspectives. Eugster and Wampold found
that therapist session evaluation was best predicted by therapist expertness, and
patient session evaluation was best predicted by the therapist real relationship.
The therapist real relationship negatively predicted therapist session evaluation
when all process variables were considered simultaneously. Patient learning
and patient involvement significantly and positively predicted both therapist and
patient evaluations of the session.
area are counselor intentions and client reactions, which are the focus of our
discussion in this category.
Counselor and client intentions are a relatively new and unexplored area
of examination in process research. At present, all of the research has exam-
ined intentions at the speech act level, and most of the work has focused on
counselor (as opposed to client) intentions. Hill and O’Grady (1985) define a
counselor intention as the covert rationale for (or the “why” of) counselor
behavior. These authors have developed a list of 19 pantheoretical, nominal,
nonmutually exclusive intentions: set limits, get information, give information,
support, focus, clarify, hope, cathart, cognitions, behaviors, self-control, feel-
ings, insight, change, reinforce change, resistance, challenge, relationship, and
therapist needs. Because intentions are covert, they are available only through
counselor introspective reports. To obtain these introspective reports, researchers
have counselors review a videotape or audiotape of a recently completed session.
(Hill and O’Grady recommend that this review take place within 24 hours.)
For each counselor turn, counselors list up to five intentions that described
their goals for that intervention.
This intention measure has been used in numerous studies examining dif-
ferent aspects of counseling process (see, for example, Fuller & Hill, 1985;
Kivlighan, 1989; Kivlighan & Angelone, 1991). For instance, Hill and O’Grady
(1985) found that counselor theoretical orientation was related to differential
intention use, and that intention use changed both within and across sessions.
More important was Hill, Helms, Tichenor, Spiegel, O’Grady, and Perry’s
(1988) finding that counselor intentions were more adequate descriptors of
counselor behavior than were response modes. Specifically, intentions either
alone or in conjunction with response modes (counselor actions) accounted
for significantly more of the variance in immediate outcome ratings than did
response mode measures alone. This finding supports the continued use of
intention measures for examining counselor behavior.
Some controversy surrounds the measurement of counseling intentions.
Hill and colleagues have simply asked counselors to record their intentions
while reviewing taped sessions, sometimes with little or no training in the use
of the intentions measure. This method of ascertaining intentions makes it
virtually impossible to obtain estimates of reliability. Martin, Martin, and
Slemon (1989) have contended that counselors often use different intention
categories to describe nominally identical reasons for behaving; consequently
they have modified Hill and O’Grady’s (1985) procedure for collecting inten-
tions. Martin et al. had counselors review each videotape and for each turn
describe their intentions or reasons for the intervention. These descriptions
were then transcribed and submitted to judges, who placed the descriptions
into the categories in the Hill and O’Grady intentions list. One advantage of
this procedure is that the reliability of category placement can be assessed by
examining agreement across judges.
Given the concerns of Martin et al. (1989) about counselors using the
intentions list in an idiosyncratic manner, we recommend that researchers use
one of two procedures for examining counselor intentions. First, researchers
480 part 3 methodological issues
in terms of validity, the extent to which client reactions are influenced by, for
instance, self- or other-deception is presently unknown.
Although counselors certainly have reactions to what their clients do, few
systematic investigations have explored this important area. Most researchers
seem to view counseling as a one-way interaction, with the counselor influ-
encing the client and the client passively accepting this influence. Heppner and
Claiborn (1989) maintained that the client should be considered an active par-
ticipant in the counseling process who also exerts influence on the counselor.
From this perspective, counselors would be expected to have reactions to their
clients. Unfortunately, the development of statement-by-statement or micro-
level measures of counselor reactions has not kept pace with the theoretical
formulations.
1987) use the control and affiliation dimensions to describe client or coun-
selor interpersonal manner. The Impact Message Inventory (IMI; Kiesler,
1987) assesses client interpersonal manner by recording the engagements or
pulls that a client has on another interactant or observer (that is, what the
person feels, thinks, or wants to do as he or she interacts or observes this
client). The Check List of Psychotherapy Transactions (CLOPT) and the
Check List of Interpersonal Transactions (CLOIT) define the client’s style by
describing overt interpersonal actions. The IMI, CLOPT, and CLOIT can
characterize clients in terms of 16 interpersonal scores: dominant, competi-
tive, mistrusting, cold, hostile, detached, inhibited, unassured, submissive,
deferent, trusting, warm, friendly, sociable, exhibitionistic, and assured. These
interpersonal scores can be combined to form octant, quadrant, or axes
descriptions of client interpersonal style. We believe research in this area is
promising. For example, Orlinsky and Howard (1986) noted a positive asso-
ciation between client openness and therapeutic outcome.
ties produced in a group, and that groups with more cohesive ties had better
client outcomes. These findings suggest that this speech act measure of cohesion
may be a useful means of studying this elusive group phenomenon.
such a plan for each client using assessment instruments and transcripts from
an assessment interview. The second step involved having a second group of
judges rate the extent to which each counselor interpretation fit the plan that
had been developed. These judges used a seven-point Likert scale ranging
from –3 (strongly anti-plan) to +3 (strongly pro-plan) to measure the extent
of agreement between the plan and the individual interpretation. Silberschatz
et al. found that the compatibility of an interpretation with a plan was a better
predictor of immediate outcome (change in client experience) than was type
of interpretation (transference versus nontransference).
Silberschatz et al. (1986) suggested that their methods or procedures for
assessing quality of counselor interventions were transferable to a variety of
conceptual frameworks. We agree with this suggestion. To reiterate, this
method of assessing quality requires two steps. First, an independent and reli-
able conceptualization of the client, including a specification of his or her par-
ticular learning needs, is needed. Second, an independent and reliable rating of
the extent to which particular interventions fit the conceptualization is needed.
As was true for assessing the quality of therapist interventions, there is
relatively little consensus concerning measures of client response quality. One
obstacle to the development of standardized quality measures is that various
theories define the client role differently, and hence the appropriateness or
quality of various client responses is viewed in divergent ways. For example,
a behavior therapist might construe a client response of questioning or dis-
agreeing with the therapist as a sign of resistance to implementing a treatment
protocol, and thus assign it a low quality rating. A dynamic therapist, on the
other hand, might construe the same response as a sign of autonomy, and thus
assign it a high quality rating. Despite these theoretical differences, there is
surprising consistency in usage of two particular speech act ratings of client
response quality, which we will now discuss.
One of the most widely used ratings of client response quality is the Client
Experiencing Scale (Klein, Mathieu-Coughlin, & Kiesler, 1986), which is a
seven-point scale used by trained raters to describe a client’s level of involve-
ment. Low levels are characterized by client disclosures that are impersonal or
superficial. At high levels of experiencing, feelings and exploration are a ba-
sis for problem resolution and/or self-understanding. Klein et al. reported
high inter-rater reliability and validity, as evidenced by the relationships be-
tween client experiencing and self-exploration, insight, working through, the
absence of resistances, and high-quality free association.
The second measure that has been used to operationalize the quality of
client responding at the speech act level is the Client Vocal Quality Scale
(CVQ; Rice, Koke, Greenberg, & Wagstaff, 1979). Like the Client Experiencing
Scale, the CVQ is designed to measure the quality of the client’s involvement
in the counseling process. Vocal quality is a measure of how the energy of a
speech act is expressed. According to this system, voice quality can be characterized
as limited, externalizing, emotional, or focused (from low to high voice quality).
In limited vocal quality, there is a lack of energy; the voice has a thinness that
seems to suggest a withdrawal of energy. Externalizing vocal quality involves
488 part 3 methodological issues
therapist competence. To do this, the researcher must first carefully and specif-
ically define the parameters of the treatment. Once the treatment has been
specified, then the therapist’s behaviors within a session can be examined on
the extent to which they match or represent competent expression of the spec-
ified behaviors. The critical question for counseling researchers concerns the
qualifications of the competence raters/judges. For example, Barber (1998,
personal communication) emphasizes that use of the PACSE scale requires
training in both the use of the scale and supportive-expressive therapy. It is
likely that only judges who are experts in a particular modality can make
accurate competency judgments. The assessment of counselor quality is an
exciting and virtually untapped area for process researchers. The methods and
procedures developed by Silberschatz et al. (1986) and Barber and Crits-
Christoph (1996) to assess counselor quality can serve as models for researchers
seeking to measure the quality of counselor behavior from other theoretical
perspectives.
The examination of client response quality at the global level has received
even more limited attention. One of the most promising dimensions for char-
acterizing client response quality at the global level is formulated in terms of
openness versus defensiveness. Orlinsky and Howard (1986) reported that
88% of the studies examining client openness found a positive relationship
between this variable and outcome measures. There is, however, little agreement
on how this construct should be measured at the global level. In the 16 studies
that examined client openness, the same process measure was seldom used to
assess this construct. Nor is it known how the different measures that assess
client openness relate to one another. Given the positive association between
client openness and therapeutic outcome noted by Orlinsky and Howard,
development of more standardized ways of assessing this construct at a global
level seems important.
Two Postscripts
We have some final comments concerning two topics: client process measures
and assessment strategies. With regard to the former, Hill (1982) observed
that counselor process variables have attracted more theoretical and empiri-
cal attention than client process variables, perhaps because we as counselors/
researchers are most interested in examining what we do, as evidenced by the
differential attention focused on counselor and client response modes.
We believe this relative lack of attention to client process variables is a
critical omission that hampers our understanding of the counseling process.
Moreover, we strongly recommend that the client be conceptualized not as a
passive agent to whom interventions are administered, but rather as an active
processor of information in the change process (see Heppner & Claiborn,
1989; Heppner & Krauskopf, 1987; Martin, 1984, 1987; McGuire, 1985;
Petty & Cacioppo, 1986). The Martin and Stelmaczonek (1988) investigation
is an excellent example of conceptualizing the client as an active processor of
information. Because human reasoning is a key activity in how people cope
chapter 19 design issues related to counseling process research 491
with their problems, attention to the active process of human reasoning de-
serves a central place in counseling process research.
Such a perspective might best be examined in both the content and quality
aspects of process research. With regard to content, it may be useful to examine
how clients represent their presenting problems to themselves (for example,
via schemas), and whether the client’s views of the problem change over time
in counseling. Are there changes in the client’s knowledge of the problem over
time, or in the way the client’s knowledge is organized? (See Martin, 1985, for
an excellent case illustration.) It also may be important to investigate how
clients deal with their affective reactions and how they appraise the signifi-
cance of their problems. In short, we believe it may be very fruitful to examine
the internal processes that clients engage in (cognitive, affective, and physiological)
as they struggle with their problems before, during, and after counseling.
Whereas client satisfaction or perceptions of counselor expertness provide
some information about the client, these variables tell us little about how
clients process information about the pressing concerns that brought them
into counseling (Heppner & Claiborn, 1989; Heppner & Frazier, 1992).
Finally, some comments on assessment strategies are warranted given the
multifaceted nature of counseling process and outcome research. Whereas
most instruments can be categorized within one of the cells in Table 19.1,
some instruments contain items that provide assessments across more than
one aspect of the counseling process. Given the need to obtain a broad assess-
ment of the counseling process, and the time-consuming nature of multiple
assessments, the advantages of such inventories are clear. For example, con-
sider the Comprehensive Scale of Psychotherapy Session Constructs (CSPSC;
Eugster & Wampold, 1996). The CSPSC measures session-level occurrences
from either the client’s or the therapist’s perspective of nine counseling process
components (patient involvement, client involvement, comfort, learning, ther-
apist involvement, expertness, interpersonal style, real relationship) and ses-
sion evaluation. In short, an 80-item, six-point Likert scale was developed by
adapting items from the California Psychotherapy Alliance Scales (Barkham,
Andrew, & Culverwell, 1993), the Empathy Scale (Burns, 1981), the Experiencing
Scales (Klein, Mathieu, Kiesler, & Gendlin, 1969), the Patient and Therapist
Behavior Ratings (Bennum, Hahlweg, Schindler, & Langlotz, 1986), the
Patient and Therapist Therapy Session Reports (Orlinsky & Howard, 1978),
the Penn Helping Alliance Scales (Luborsky et al., 1983), the Relationship
Inventory (Barrett-Lennard, 1962), the Session Evaluation Questionnaire
(Stiles & Snow, 1984), the Session Impacts Scale (Elliott & Wexler, 1994), the
Therapist and Patient Action Scales (Hoyt, Marmar, Horowitz, & Alvarez,
1981), the Vanderbilt Psychotherapy Process Scale (Suh, O’Malley, Strupp, &
Johnson, 1989), and the Working Alliance Inventory (Horvath & Greenberg,
1986). Eight items were devised for each of the nine constructs and for global
session evaluation. Alpha coefficients revealed an average of .73 and .76 for
subscales on the therapist and patient forms, respectively. Assessment strate-
gies such as this, and instruments such as the CSPSC in particular, hold a great
deal of promise for future research.
492 part 3 methodological issues
STIMULUS QUESTIONS
An Exercise to Analyze Two Counseling
Process Studies
The following exercise is designed to help readers 3. How adequately do you think the constructs/
not only to become more aware of the intricacies variables were measured or assessed?
of counseling process research, but also to become 4. From whose perspective was the counseling
more aware of different types of counseling process process examined?
methods. 5. What were the pros and cons of examining
First, identify two published articles that in- the counseling process from these perspectives?
vestigated the counseling process. One article 6. What were the pros and cons of the quan-
should employ quantitative methods and the titative and the qualitative methodologies
second qualitative methods. Study these two used across the two articles?
articles, and then respond to the following ques- 7. What conclusions were drawn from each
tions. In addition, if you want, ask a peer to join article about the counseling process?
you in this exercise; independently respond to 8. What limitations do you see about the con-
the questions for the same two articles, and then clusions from each study?
discuss and compare your responses. 9. What would be some exciting “next best
studies” to follow each of the two process
1. What constructs or variables were investi-
articles?
gated in each article?
10. What types of methodologies would you use
2. How were the constructs or variables for each
to conduct those studies, and why?
of these two articles measured or assessed?
20 CHAPTER Scale Construction
Dong-gwi Lee and Hyun-Woo Lim
494
chapter 20 scale construction 495
On the contrary, in Asian cultures, the majority of items for parental criticism
disappeared. Specifically, Cheng et al. (1999) suggested that Chinese parents in
Hong Kong do not frequently use criticism as a medium of parenting, which
would inhibit the emergence of the parental criticism factor in Chinese adoles-
cents’ responses. They further suggested that Chinese adolescents would interpret
the occasional existence of parental criticism “in a positive way such as parental
care” (p. 1059). This hypothesis may be applicable to Korean culture as well,
although it is unclear whether Korean parents do not perceive criticism as an
effective ways of parenting. A plausible hypothesis is that Korean students do
experience critical feedback from parents, but may not perceive it as criticism.
This can be understood through the lens of unique parent-child relationships in
Korea, which is best explained by the phrase, “om bu ja mo” that means discipli-
nary father and lenient mother (Kim & Choi, 1994, p. 244). It would be rare to
regard Korean parents as critical or harsh. (p. 8)
In short, even though a scale has been carefully translated and back-trans-
lated, the items may mean different things in different cultures.
This item contains two elements (i.e., avoidance of family shame and only
told a few people about the stressful event), and thus the respondents may be
confused in determining which content they should respond to. If this item
were literally translated into different languages where the order between the
subordinate phrase and the main clause became different from that of the
original item, then it would add more complexity. Specifically, after the factor
analysis, if this item loaded on the avoiding family shame factor, it is unclear
whether this item truly belongs to this factor or merely is an artifact of a different
word order across the two cultures. Therefore, we strongly advise the researcher
to avoid any item that contains two or more elements at any time, particularly
when translating it into another language.
chapter 20 scale construction 499
utility of the scale and the appropriateness of the definition. In order for the
definition to be operationalized, the construct should be written in a statement
that is measurable (Vogt, 1993). For example, Lee, Choe, Kim, and Ngo
(2000) developed a scale that was designed to measure typical intergenera-
tional family conflicts in Asian Americans. The construct of family conflicts
was defined as “acculturation differences between parents and late adolescent
and young adult children [in Asian American populations across various
areas such as cultural values and behaviors]” (p. 212). Note that the re-
searchers focused on two things: (1) the prevalent but underexplored phe-
nomenon of family conflicts in Asian immigrants, and (2) particularly, the
Asian-American adolescent population that is at risk of intergenerational
gaps that lead to a broad array of mental health problems. Conceptualizing and
operationalizing a construct necessitates a researcher’s extensive knowledge of
the target population.
For example, in Heppner et al.’s (2006) CCS project, identifying the rele-
vant literature or theoretical background for the construct they purport to
measure was a crucial task because it identified relevant theories related to the
construct. For example, the CCS was guided by three theoretical/conceptual
502 part 3 methodological issues
grounds or previous research findings: (1) Kim, Atkinson, and Yang’s (1999)
work on Asian values; (2) Weisz, Rothbaum, and Blackburn’s (1984) concepts
regarding primary and secondary control; and (3) Zeidner and Saklofske’s (1996)
adaptational model of coping. For example, Heppner et al. described how the
construct, Collectivist Coping Styles, and items were derived from Weisz et
al.’s work on both primary control (individuals achieve control by directly
influencing their realities) and secondary control (individuals obtain control
by reframing or accommodating their realities). This was an important com-
ponent in generating items for the CCS project because the researchers’ goal
was to develop a broad range of coping items specifically for Asians, which
would include items related to secondary control as well as primary control
(which is prominent in Western coping instruments).
program, we identified the reading level of the scale as slightly above 6th grade
comprehension. To assess further the readability and clarity of the items, one
primary and one secondary school teacher, a high school student, and a newspaper
editor evaluated the scale. On the basis of feedback from the content experts and
community persons, four items were reworded for clarification. (p. 61)
In essence, content analysis and pilot testing helped the authors to fine
tune and revise their items. Readers should keep in mind that the revision
processes are neither linear nor clear-cut; sometimes, the researchers repeat-
edly engage in numerous activities to revise and alter the items to achieve
more clarity in content.
The sampling issue is a critical one. For example, when Heppner et al.
(2006) were developing and validating the CCS inventory, they specifically
sought an East Asian sample that would assess Asian-specific ways of coping
with traumatic life events. A total of 24 universities in Taiwan were involved in
the data collection procedures. The college student samples were also relevant for
the study because several categories among the 17 common stressful or trau-
matic events assessed academic situations. It should be noted that the research
team sought data from the four different geographic regions in Taiwan (north,
south, central, and east) in order to obtain a geographically representative
sample of Taiwanese college students. This was extremely important because
the geographic region is one of the most distinct demographic characteristics in
Taiwan. The data collection procedures required a great deal of work for a
senior Taiwanese faculty member and one of the research team members,
which was far from collecting the data from a “convenience” sample. As a
result, college students from around the island participated in the study. For
more information about sampling and screening, see Kline (2005).
Data collection is also a critical step in scale construction. A typical
method of data collection involves distributing questionnaires to a group of
participants; however, it is not uncommon to collect data through the mail,
email, or the Internet. Internet-based surveys are increasingly being used be-
cause of their distinctive advantages, such as being less geographically con-
strained, quick and economical, and easy to convert into statistical software.
However, the Internet’s disadvantages should also be considered; these include
limited accessibility to the Internet for some people, a tendency to obtain low
response rates, and issues with privacy and confidentiality. For more infor-
mation on survey methods, see Chapter 10 as well as Dillman (2000) and
Heppner and Heppner (2004).
Furnham & Procter, 1988), two established measures of social justice attitudes.
They expected moderate to high correlations with these two scales. In addition,
they examined the discriminant validity of the CoBRAS by calculating its cor-
relations with the Marlowe-Crowne Social Desirability Scale (MCSDS; W. M.
Reynolds, 1982), expecting a small correlation. Following is an excerpt from
Neville et al. Two points are noteworthy. First, in order to establish the dis-
criminant validity of the CoBRAS, the researchers used a short version of the
MCSDS (13 true-false items), which in the past was one of the most widely used
measures of social desirability. More recently, another measure of social desir-
ability, the 20-item Impression Management (i.e., deliberate self-presentation
to an audience) of the Balanced Inventory of Desirable Responding (BIDR;
Paulhus, 1984, 1991) is being used more frequently. (Those interested in the
psychometric properties of the BIDR should refer to Paulhus, 1991.) Second,
note that Neville et al. found one significant correlation between one of the
CoBRAS factors and the MCSDS. This is not an ideal situation because it indi-
cates some items of the CoBRAS may reflect that respondents’ responses are
associated with social desirability. However, this does not nullify the utility of
the CoBRAS because the amount of variance accounted for in the correlation
was very small (.20 × .20 = 4%). This is called a “coefficient of determination,”
which refers to “an indication of how much variance is shared by the X and
the Y variable” (Cohen, Swerdlik, & Phillips, 1996, p. 133). We recommend that
researchers report the amount of variance when the correlation between their
new measure and social desirability is statistically significant.
Concurrent Validity
The correlations among the CoBRAS factors and the two Belief in a Just World
scales were examined to investigate the concurrent validity of the CoBRAS.
Results indicate significant correlation among the GBJW, MBJWS-sociopolitical
subscales (SS), the three CoBRAS factors, and the CoBRAS total score. Correlations
ranged from .39 (between Institutional Discrimination and GBJW) to .61 (among
MBJWS and Racial Privilege as well as the CoBRAS total).
Discriminant Validity
The correlations among the CoBRAS factors and the MCSDS were examined to
provide estimates of discriminant validity. Results suggest that generally there is
no strong association among the MCSDS and the CoBRAS factors. There was
one statistically significant relation: MCSDS was associated with Blatant Racial
Issues (r = .20); however, the maximum amount of variance accounted for was
4%. (p. 65)
properties developed in a Western culture is uni- (1987) stated, “scales are ubiquitous features of
versally valid, (6) a literal translation ensures counseling psychology research” (p. 481), and in
linguistic and cultural equivalence, and (7) essence promote the development of psychomet-
structural elements of a scale, such as a Likert rically sound ways of assessing and promoting
rating scale, or even ancillary forms like a con- understanding of a new construct relevant to the
sent form, are universal across cultures. In addi- counseling profession. Heppner (2006), in his
tion, our discussion of the seven steps in scale presidential address to the Society of Counseling
construction along with examples from several Psychology of the American Psychological Asso-
inventories developed in the field of counseling ciation, suggested many challenges to becoming a
(e.g., CCS and CoBRAS) guides researchers and globally competent counseling psychologist.
students who may develop their own scales, Some of the challenges, such as (1) ethnocen-
particularly in a cross-cultural or multicultural tricity, (2) difficulty in accepting others’ world-
context. view, (3) accepting cultural differences across
Creating a new scale is very important in edu- cultures as simply differences, and (4) universal
cation and psychology in general because a psy- assumptions, are also pertinent to the challenges
chometrically sound inventory can provide an that many face in developing a culturally sensi-
effective tool to (1) assess individuals’ differences tive scale. It is hoped that researchers and students
across various domains such as personality, atti- in the field not only increase their awareness of
tudes, cognitions, emotions, and behaviors; and the complex nature of scale construction, but
(2) examine the interface of such domains and also incorporate multicultural and cross-cultural
psychological adjustment. Furthermore, as Dawis issues into the construction of new inventories.
18. Enter the data into a statistical analysis 21. Determine reliability estimates (internal
package (e.g., SPSS, SAS) and clean the data consistency, e.g., alpha coefficients) as well
(see Heppner & Heppner, 2004; Meyers, as various estimates of validity (e.g., conver-
Gamst, & Guarino, 2006)? gent and discriminant), calculating correla-
19. Calculate descriptive statistics (e.g., means, tion coefficients?
standard deviations)? 22. Collect additional data and examine the sta-
20. Identify the factor structure of the construct bility of the factor structure through confir-
and finalize the items through a series of matory factor analysis and administer other
exploratory factor analyses including paral- inventories to promote additional estimates
lel analysis? of validity?
Program Evaluation
Matrese Benkofske and Clyde C. Heppner
CHAPTER
21
The focus of this book throughout has been on research design, its application
in the counseling profession, and the conditions needed to conduct quality
research. Turning to a slightly different topic, this chapter describes program
evaluation—what it is, how it is similar to and differs from counseling research,
and how it is used within counseling settings. The chapter describes in con-
siderable detail the phases of program evaluation to introduce professionals
to the field. The first step is to define program evaluation.
511
512 part 3 methodological issues
2. What will this stress-reduction program involve? How long will the pro-
gram last? How many sessions will a participant typically attend? What
type of stress-reduction techniques will be used? This information helps
the evaluator to shape the evaluation, the time line, and the types of sta-
tistics (nonparametric or parametric) used in the analyses.
3. What types of effects do program planners anticipate? When should these
effects be measurable? Does the program manager anticipate long-term
benefits of the program? What type of evidence would the program man-
ager need to claim that the program’s mission has been met? Information
about these questions enables the evaluator to tailor the evaluation methods
to the informational needs of stakeholders and program planners alike.
4. Has anyone written a proposal for funding? Are any program require-
ments tied to program funding (for example, including a segment of a spe-
cial population as program recipients, paying program providers, or using
some specific treatment in the program)? If, for example, the women’s
center on campus provided a portion of the program’s funding, they may
require the program to address issues identified by women students such
as finding child care for a few hours while attending class rather than
paying for an entire day. These kinds of stipulations can have dramatic
implications on the evaluation design, and it is better to know this before
the evaluation has been set in motion.
5. What is the funding level for the program, and how much has been set
aside for the evaluation process? A guideline used at times is that the fund-
ing for the evaluation should constitute approximately 10% of the cost of
the program. This helps to prevent a very small project from having a far
too expansive evaluation, and vice versa. Designers of programs often
severely underestimate the cost of conducting a program evaluation.
6. What are the program manager’s expectations for the evaluation? Does he
or she have a preferred data collection strategy? What role will the evalu-
ator play? Some agencies have staff members capable of helping with data
collection, which reduces the billing cost of the evaluation. In some cases,
the program manager expects the evaluator to be highly visible throughout
the program’s direction. Knowing what the program manager feels com-
fortable with in terms of evaluator exposure helps in planning the evaluation.
7. Why is an evaluation of the program being undertaken? Is it required by
the funders? Have the outcome measures and the sampling procedures
been predetermined? Knowing the reasoning behind the evaluation enables
the evaluator to tailor the evaluation to the needs of the program manager
and stakeholders. If the funders of the evaluation have clearly stated that
objective, measurable, and scientifically rigorous data must be collected in
order to be considered for continued funding, the evaluation will be quite
different from one designed to help program implementers better serve and
understand their clients.
8. What resources are available to the evaluator? Will counseling staff help
with data collection? Are interns or clerical help available? What records
are available to the evaluator? Given that the stress-reduction program is
chapter 21 program evaluation 517
the evaluation because it is the stakeholders who ultimately make the final
decisions about the utility of the program.
Including stakeholders in the initial planning meetings allows the evalua-
tor to include data collection strategies that meet these stakeholders’ need for
information when determining a program’s worth. Inclusion helps to decrease
the temptation of some stakeholders to claim that the evaluation was flawed,
especially if the program does not fair well in the final report.
It is important, however, not to be disheartened if stakeholders show a
lack of interest in becoming involved in the program evaluation planning
process. Some people have a propensity to be silent or even uninvolved unless
a program’s continuance is threatened. In these cases it is helpful to actively
solicit the opinions and suggestions of stakeholders. The key word here is
“actively”; it may take more than one formal invitation of marginal stake-
holders, plus a well-placed phone call from the evaluator, during which the
importance of the stakeholder’s participation is emphasized. Another strategy
for garnering stakeholder input is for the same individuals to be involved in
both the initial planning of the program and the planning of the evaluation,
assuming that this group also includes people who will be using the program.
At the least, the evaluation plan, once prepared in first draft, should be
presented publicly through existing meetings typically attended by program
recipients and/or the general public. When evaluating the counseling program
for stress reduction, for example, the evaluation plan should be presented
to governing bodies involved in funding the counseling center. For other
programs, this public body may be the PTA of a school program, the city
council (for a city-sponsored program), or a steering committee for campus
programs. It is very important that the input of such meetings be incorporated
into the evaluation plan; this is not the time to force a plan without an ade-
quate discussion or to solicit input only to dismiss it. If evaluators seek input
from stakeholders, it is their responsibility to consider suggestions and input
seriously.
During discussions in which the focus of the evaluation is being gener-
ated, a few questions should be kept in the forefront: What aspects of the pro-
gram should be evaluated? Will the whole program be evaluated or only
newly added components? At what point in the program will the evaluation
begin, and when will it end? Will it focus on the conceptualization stage, the
design stage, or the implementation stage of the program? Is the program suf-
ficiently mature for the evaluator to expect any measurable efficacy? By
reflecting on these questions while formulating the evaluation plan, the eval-
uator is more likely to keep the evaluation on track. Moreover, by keeping the
program’s time frame and maturity in mind while designing the evaluation, it
is more likely that the evaluation will be appropriate for the program.
This is also the time that the evaluator heeds the specific questions to be
answered during the evaluation. Conducting an evaluation with only the very
general questions, “Is this program worthwhile?” or “Has this program
been successful?” is not sufficient and violates the first standard of program
evaluation—the need for utility (Joint Committee on Standards for Educational
chapter 21 program evaluation 521
Evaluation, 1994). Put another way, the evaluation must answer the questions
of interest to the stakeholders.
These initial meetings are a good time for the evaluator to come to
understand the needs and interests of the primary stakeholders, and their rela-
tionships with one another. During the meetings the evaluator should pay atten-
tion to how people are interacting with one another. Are there factions within
various groups? How deep are these divisions, and on what are they based?
How large an effect will these factions have on the evaluation process, and on
the reception of the evaluation results? How are differing points of view dealt
with? Are viewpoints in opposition to the key stakeholders’ allowed to surface
and be fully considered, or are they quickly dismissed? If the latter occurs, the
evaluator should steer the conversation back to the overlooked viewpoint, per-
haps by saying, “I’d like to come back to a point just made. Could you elabo-
rate on . . . ?” It is the evaluator’s responsibility to ensure that the concerns
raised at meetings represent not just the opinions of the vocal, but also those of
potentially disenfranchised, underrepresented, or overlooked stakeholders.
Throughout these initial meetings, the evaluator is typically considered by
the program managers and other stakeholders to be an “expert” whose words
are taken as fact. Thus, it is important that evaluators allow others to speak
and facilitate the process. If the evaluator becomes too vocal, the evaluation
can quickly become one devised by the evaluator, not the one envisioned by
stakeholders.
a mutually exclusive list for each of the categories (stressor, behavior, and con-
sequences) and ask participants to check off one item in each category. A more
complex log might involve collecting information about the conditions sur-
rounding the stressful event, the duration of the stressful feelings, and the time
of day the stressor occurred. Logs and journals can provide specific data about
behaviors as they occur rather than the broader statements about behaviors and
feelings collected in more general surveys.
Content testing, yet another method for collecting data on a program’s
effect, evaluates program participants’ knowledge about a topic, typically
using some paper-and-pencil activity. Course exams are a form of content test-
ing. This type of data is useful when distinct facts or information is being con-
veyed through the program. In the stress-reduction program, respondents
might be asked to name the stress-reduction techniques introduced in the
workshop, the physiological symptoms associated with stress, the psycholog-
ical manifestations of stress, and the five most common stressors faced by
people in their age group. An even better data set would result from adminis-
tering the same test before and after the workshop to measure the amount of
information respondents acquired from it. Of course, using content testing
requires the test to be highly correlated with the material presented in the
workshop, and the individual items must be specific and difficult enough that
only workshop attendees can answer the questions correctly.
Another use of content testing is to determine respondents’ familiarity
with a specific topic. One evaluator used a content test to evaluate a city’s biking
and walking trails. Respondents were asked how familiar they were with the
biking and walking trail system (using a 1 to 5 scale to assess the respondent’s
self-report of familiarity), as well as 20 questions about the trails. Only
respondents who had used the trails were able to correctly answer a majority
of the questions. These data then were used to subsequently classify the respon-
dents into “high,” “medium,” and “low” trail users, a categorization that was
subsequently used for other statistical analyses.
Planning the Data Analyses The final column in the matrix of evaluation
questions and data collection methods in Table 21.1 is “data analyses.” If, for
example, a survey is being used, will subscores be calculated, or will frequen-
cies for each item be calculated? Will a respondent’s survey data be correlated
with workshop attendance, and if so, is there a way to match these two pieces
of data, through either a name or a code? What statistical analysis will be per-
formed on the data, and are the data in a form that allows the analysis to
occur without extensive manipulation? If the data are qualitative, how will
the data be analyzed? Will there be case studies or a summary of findings?
Time spent planning for the data analyses, whether the data are qualitative,
quantitative, or archival, can save considerable time in conducting the data
analyses. Extensive planning can also prevent fatal data collection errors from
occurring, such as collecting pre- and posttest data only to find that there is
no way to link the data because no identifying names or codes were included
on the survey instrument.
Estimating the Financial and Time Costs of the Evaluation “The cost of an
evaluation is difficult to predict accurately” (Herman, Morris, & Fitz-Gibbon,
1987) is an assessment with which almost anyone who has conducted an eval-
uation will likely agree without hesitation. There are no hard-and-fast rules
about how much time a survey, for example, will take to design, print, dis-
tribute, analyze, and interpret. Herman et al. suggest first determining the
fixed costs—those over which you have no control, such as postage, consul-
tant time, overhead costs (heat, electricity, phone, and so on), printing, test
and instrument purchase, and supplies or equipment needed to complete the
project. They also suggest that you then calculate the “per unit” cost for each
of these items. Thus, printing costs should be “per page,” and overhead costs
might be calculated “per month.” Similarly, “person-hours” should also be
calculated on a monthly or daily basis. Per person costs should include all the
expenses associated with individuals, such as benefits, income tax, and social
security tax that routinely are charged for a person’s employment.
After the per unit costs have been calculated, the evaluator can begin to
create the actual budget. The exact level of detail and layout of the budget
depend on funders’ needs, any forms required by your own organization, and
your experience. This process can be quite complex, especially for a large eval-
uation, but it can be made less daunting by creating a detailed list of specific
tasks needed to complete each component of the program evaluation. Thus,
for a survey, the evaluator would estimate the costs for each step needed to
move the survey from creation to data interpretation. Experience makes this
528 part 3 methodological issues
process easier; knowing some of the areas that can cause problems allows the
evaluator to allocate sufficient resources within the budget. It is rare when an
evaluator says, “I overestimated the amount of time this project would take”
or “I have more resources than I know what to do with!” The more detailed
this list of tasks is, the less likely it is that you will underestimate the time
associated with each task. Clearly, the more information you can add to this
list, the better estimate you will make. Some details result from specific de-
cisions, such as how many surveys will be given out and how they will be
distributed and returned; other details come from talking with others. For
example, if you learn that the data entry specialist charges by the keystroke
rather than by the hour, you might want to know what the survey will look
like before moving forward on that line in the budget. Colleagues can be good
resources in helping to create realistic time lines and budgets. Each component
should be broken into its individual tasks, and then estimates of time to com-
plete each task and the subsequent cost for that task should be made. Even
though this procedure creates a budget with a built-in time line, the evaluator
may wish to create a time line in calendar form, with target dates for each
component’s completion highlighted. Several commercially available com-
puter programs can create a master calendar of projects, in addition to
monthly calendars. This software can be very helpful for keeping track of
target dates in projects, especially if the evaluator is working on several proj-
ects simultaneously.
and embarrassment when, while visiting the program site, she was asked
when the data collector was going to observe the program; she had just
looked over the data supposedly collected from that site the previous week!
The combination of that and similar experiences has led the evaluator to hang
above her desk the following sign, which summarizes her philosophy about
data collection:
If you haven’t seen it, heard it, touched it, smelled it, rolled in it, felt it, and tasted
it, you don’t know it.
Periodic checks of the program site, both announced and unannounced, are
essential for monitoring the program and helping to keep the data clean.
The task of checking the data does not end once they arrive on the eval-
uator’s desk. Data entry must be checked for accuracy, especially if performed
by someone new to the evaluator. Are missing data being coded properly?
How many errors are there per 100 data entry keystrokes? A skilled data
entry person probably has no more than one or two errors. Are the data being
entered using standardized procedures to make data analyses easy and to min-
imize data transformation or manipulation? The evaluator should never as-
sume that the person entering the data knows how to do so in a format that
is usable by a given statistical computer package.
Once all of the data have been collected and readied for analyses (that is,
quantitative data have been entered into a spreadsheet or statistical analysis
package, field notes have been typed into case notes, tapes of focus groups have
been transcribed, and either paper copies or computer files are available), then
the evaluator is now ready to conduct the data analysis. We divide data analy-
sis into two distinct steps: primary analyses and secondary analyses.
Primary Data Analyses The first analyses, called primary analyses, are iden-
tical to those presented in the results sections of empirical articles. The data
from each component of the evaluation are analyzed in isolation from other
data. Thus, for example, the results of the survey that measured respondent
happiness are analyzed and reported separate from the descriptions of the
stress-reduction workshop activities prepared by the participant-observer;
each component has its own methods, results, and summary sections, as if
each were the only data collected for the evaluation.
There are several advantages to analyzing and interpreting each data
component in isolation from the other components. First, these primary data
reports are excellent vehicles for periodically presenting data to stakeholders.
They give stakeholders some preliminary indication of some of the findings;
moreover, they begin to prepare stakeholders for the final report. Second, they
prevent the data from becoming backlogged. By analyzing and writing the
report for each data set as soon as it is ready, program evaluators are not faced
with piles and piles of data as the completion deadline approaches. Third, pri-
mary data reports help the evaluator see holes in the data or areas that need
additional examination. Programs are fluid entities; despite the best planning,
changes in the program, failed data collection procedures, or ambiguous results
chapter 21 program evaluation 531
Secondary Data Analyses A secondary data analysis ties together the individual,
primary analyses to describe a component of the program. Because multiple
sources of data are used, the program evaluator must reexamine each of the pri-
mary analyses to determine where findings support one another and where there
are discrepancies. When answering the question, “Did the stress-reduction pro-
gram provide program participants with strategies they could implement?” the
evaluator would look at all the data collected. The evaluator would look at the
survey items (which had been summarized into frequencies and percentages in
the primary analyses) that asked respondents how often they used a particular
technique, as well as examine journals kept by the respondents and the field
notes taken during the participant-observations. If program participants said
they used meditation frequently to reduce stress, was this technique emphasized
in the workshops? If “effective decision making” was a technique emphasized
during training but did not appear in the participants’ journals, was this because
they did not understand the importance of decision making in stress reduction
or because it was not presented clearly enough to be useful?
Although there are several differences between how a researcher and an
evaluator conduct the study of a social program, one of the elemental differ-
ences comes in the collection and interpretation of several data sources.
Multiple data sources that provide insight into the worth of a social program
are a hallmark of program evaluation, and the multiplicity of data sources
requires some method for weaving together these sources. This method in-
volves secondary data analyses.
The evaluator begins with the program’s goals and objectives and exam-
ines the primary data reports to see how the results of each data collection
component support or refute the supposition that the program met its goals.
What evidence is there that college students who have taken part in the stress-
reduction program have learned several new stress-reduction techniques? Did
items on the questionnaire given in the workshop address this question? What
did respondents say in focus groups when asked if they learned new tech-
niques? According to the field notes of the participant-observer, were work-
shop attendees introduced to “several” techniques? Perhaps another goal of
the stress-reduction workshop was to have respondents feel better able to
handle stressful situations. What evidence either supports or refutes this claim
of the program? If workshop attendees kept journals, do they contain evi-
dence that attendees improved their stress-coping skills over time?
The secondary analysis moves back and forth from stated program goals
and objectives to data sources. The importance of reducing the data via pri-
mary data analyses should now be evident: Wading through raw data at this
point would be inefficient and cumbersome, if not impossible; relying on the
532 part 3 methodological issues
summary statements made in the primary report helps to expedite the writing
of the secondary analyses.
The secondary analysis—this moving back and forth from the program’s
objectives to the evidence, the weaving together of the individual pieces of
data into a holistic picture—is program evaluation. Combined with recom-
mendations and comparisons to similar programs, the secondary analysis cre-
ates the evaluation of the program. A program evaluator should never leave
the interpretation of the evaluation to the stakeholders. It is the program eval-
uator’s role to pull the pieces together and to clearly state what he or she
means in terms of the program’s implementation, anticipated outcomes, and
implications for the successful conduct of this or future programs.
have had an opportunity to digest the contents of the final report—and stake-
holders will do this at different levels of intensity—the evaluator schedules a
meeting at which the evaluator facilitates the group in the generation of rec-
ommendations based on the final report. As recommendations are written by
the stakeholders, the evaluator notes the differences from his or her own pre-
written recommendations. The evaluator also may need to draw the group’s
attention to points not yet brought into the discussion. By using this method of
including stakeholders in writing the recommendations, two important goals
are achieved. First, stakeholders are reminded that program evaluation is not
something that happens to them and their program, but instead is a collabora-
tive discussion in which several perspectives shape the final recommendations.
Second, the evaluation report and the subsequent recommendations are not
viewed as being the sole property of the evaluator but rather as reflecting the
thoughts of several professionals closely related to the program.
Utilizing the experience of stakeholders to help formulate the recommen-
dations can be very beneficial, but a successful meeting does not occur
without proper planning. First, because it is likely that many stakeholders
assembled have not read the report prior to the meeting, the evaluator should
provide a 15- to 20-minute presentation that outlines the evaluation findings.
Moreover, the evaluator should have a list of recommendations he or she feels
are absolutely crucial for inclusion and should ensure their inclusion in the
discussion. A block of uninterrupted time must be set aside for this meeting.
No one has ever complained if a meeting does not fill the whole block of time,
but trying to hurry through the discussion can leave stakeholders feeling that
ideas have been thrust upon them. The evaluator should be the keeper of the
agenda and move the meeting along at a reasonable pace, allowing for dis-
cussion but preventing it from straying from the stated purpose. Finally, it is
extremely important to invite the full range of stakeholders to this meeting,
including those with dissenting views, and everyone invited should be notified
as to who else will be in attendance. The evaluator should act as a facilitator,
encouraging people to express their opinions within the constraints of the pro-
gram. This meeting is not the vehicle for settling longstanding disagreements.
Recommendations include both the positive and negative aspects of
the program as it was implemented and the subsequent outcomes. The
strengths of the program are always listed first, separate from the weaknesses
(sometimes called the limitations) of the program. Careful wording of the
recommendation section of the report is crucial; the goal is to write a recom-
mendation that adequately describes the strengths and weaknesses of the pro-
gram without being overly critical or mired in detail. One good perspective
addresses the future—either in the next year of the program or the next time
the program is replicated: What aspects of the program would be absolutely
critical for success next time? What are the key, beneficial features of the
program that need to be incorporated into future programs? What parts of
the program were not implemented as anticipated, and what does this mean
for future replications of the program? What is missing from the current
program that, if included in future years, would make it stronger? What can
534 part 3 methodological issues
Writing the Executive Summary The other aspect of the evaluation report
that may be unfamiliar to researchers is the executive summary. Attention to
the style of this section and the recommendations is time well spent, because
although few people read a report from cover to cover, the executive summary
and the recommendations are read by most people who come in contact with
the report. To illustrate this point, Fitzpatrick, Sanders, and Worthen (2004)
described an experience in evaluating a statewide “controversial program” for
which three separate reports were prepared for review by stakeholders: (1) the
full report containing all of the technical detail of how the evaluation was con-
ducted; (2) a medium-size summary of major interpretations drawn from the
data; and (3) a brief executive summary:
Availability of these three reports was broadly announced in the newspapers and
on television. . . . Nearly 400 individuals [requested and] read the executive sum-
mary, 40 read the mid-size interpretive report, and only one person ever even
requested the complete report (and he was an expert methodologist hired by
opponents of the evaluation to see if he could find fault with it). As these results
show, shorter reports will often be most widely disseminated. (Worthen et al.,
1997, p. 416)
The executive summary should be no more than three pages, give the reader
an overview of the goals of the program, and indicate the services provided, the
outcomes anticipated, and the extent to which these objectives were met. An exec-
utive summary should stand on its own; it should contain enough detail so that
a person can grasp the program, its purpose, and its impact by reading it alone.
Disseminating the Report Who owns the data and to whom the written
findings are disseminated are points negotiated prior to signing a letter of
agreement. The evaluator and program administrators should come to an
understanding about how many copies of the final report are needed, the
number of presentations the evaluator is expected to make, and how the eval-
uation results should be disseminated. Some agencies are bound by the Public
Information Act, which makes the report automatically available to anyone
who requests a copy. But who is the spokesperson for the evaluation and its
results? What are the limitations for using the evaluation data and the subse-
quent report? This answer lies in the Program Evaluation Standards (Joint
Committee on Standards for Educational Evaluation, 1994). Suppose the
local newspaper chooses to print only that portion of an evaluation that out-
lines a controversial program’s weaknesses, or suppose the program adminis-
trator requests from the evaluator an updated summary but then forwards to
a federal funding agency only those portions of the evaluator’s report that
favor the program. In both cases, the evaluator and the program administrator
could have taken steps to prevent or minimize the effect of these actions.
chapter 21 program evaluation 535
In the Program Evaluation Standards, the “formal parties [in this case, the
evaluator and program administrator] should ensure that the full set of eval-
uation findings along with pertinent limitations are made accessible to the
persons affected by the evaluation, and any others with expressed legal rights
to receive the results” (Propriety Standard 6). Both the evaluator and the pro-
gram administrator are responsible for monitoring the release and use of the
evaluation. In the second situation, it was the evaluator’s responsibility to
require the program administrator to provide a copy of the summary prior to
releasing it to the funding agency. An evaluator should never give up editing
responsibility to others; editing, summarizing, or releasing portions of an eval-
uation should remain exclusively in the evaluator’s direct control.
also the program’s implementation (process). An this phase the program evaluator also plans how
examination of how the program is implemented the data will be collected and prepared for analy-
often reveals the reasons a program met or failed ses, and trains the program staff to ensure reli-
to meet specified outcomes. Documenting the able collection.
factors that both limit and enhance the program The third phase entails the actual collection and
as it was administered can help to strengthen the analyses of the data. The evaluator pilot-tests all
program or similar programs in the future. A newly created or modified instruments and obser-
defining feature of program evaluation is the use vation forms, monitors data collection by program
of and interpretation of triangulated data—data staff or evaluation staff, and ensures that agree-
collected from more than one source, at more ments of anonymity and/or confidentiality are
than one setting, using more than one data col- upheld. Primary data analyses are conducted and
lection methodology. By using data triangula- reported in a manner similar to those conducted in
tion, program evaluators increase the validity of empirical research. Secondary analyses examine
the evaluation’s findings. the various primary data analyses to focus the
This chapter describes four phases of pro- findings on specific evaluation questions.
gram evaluation. An evaluator begins the The final phase of evaluating a program
process by setting the boundaries of the evalua- requires the evaluator to report the findings of the
tion. A crucial step in this phase involves allow- evaluation. Methods of dissemination, report
ing all stakeholders to have a voice in the scope deadlines, and types of reports to be created
of the evaluation to ensure that the evaluation’s should be agreed upon by the evaluator and client
findings are fair and unbiased. Utility is also as part of the evaluation contract. All evaluation
enhanced by linking the program’s goals and reports should contain sections that describe (1)
objectives to specific evaluation questions. the program, (2) the evalua-tion process, (3) the
After setting the boundaries of the evaluation, data collection procedures, and (4) the results and
the second phase involves selecting appropriate findings of the evaluation. In addition, all reports
evaluation methods. The selection of the should contain an executive summary and a list of
methodology depends on the information needs recommendations that include the positive aspects
of the evaluation audience; the time, personnel, of the program as well as the areas that may need
and financial resources available to the evalua- improvement. These two sections should not
tor; and the constraints of the program. During exceed five or six pages.
STIMULUS QUESTIONS
An Exercise in Program Evaluation
You have been asked to evaluate a week-long, 3. How would the evaluation questions change
overnight camping experience for girls ages 7 to 14. if the evaluation were conducted during the
Use this example when answering the following conceptualization phase of the program?
questions: 4. Who are the possible stakeholders for your
program? Are there any silent stakeholders?
1. What sources of information will you use to
Are there any disenfranchised stakeholders?
describe the program and understand its
5. The second phase in the evaluation of a pro-
mission and goals?
gram involves outlining data collection steps
2. Outline a couple of evaluation questions you
and time lines. What is the relevance of this
might pose during the implementation phase
procedure?
of the program.
chapter 21 program evaluation 537
6. What data collection strategies will you use? 10. What is secondary data analysis, and how
7. What might be some of the advantages and does it differ from primary data analysis?
disadvantages of using participant observa- 11. What are some of the advantages of involv-
tion in your evaluation? ing the stakeholders when writing the evalu-
8. What does it mean to “triangulate the data ation recommendations?
sources”? 12. What is an “evaluation trail,” and what
9. When does primary data analysis occur, and does it include?
why is it important in the evaluation of a 13. How do research and program evaluation
program? differ?
This page intentionally left blank
Professional Issues PART
4
539
22 CHAPTER Professional Writing
540
chapter 22 professional writing 541
including review articles (Bem, 1995), methodological articles (Maxwell & Cole,
1995), and meta-analytic reviews (Rosenthal, 1995).
Writing style is personal; the opinions of authors, critics, and educators in
this matter differ. Furthermore, style varies according to the constraints of the
publication in which the writing will appear. For example, the American Psy-
chological Association (APA) carefully details the required elements of manu-
scripts submitted to APA journals in its Publication Manual of the American
Psychological Association (APA, 2001). This reference book is indispensable for
most authors who publish in counseling and related professional journals.
Graduate schools typically have their own requirements for dissertations.
One chapter cannot cover the myriad of details involved in writing theses,
dissertations, or manuscripts for publication. Our goal in this chapter is to
introduce writers to the basic components of a research report. First, we present
the major sections of a report and discuss their relation to the principles of
research design previously discussed in this book. Although we could discuss
various organizational formats, we focus on the title, abstract, introduction,
method, results, and discussion. We also present some general principles for
writing research reports. For a more detailed discussion of writing research
reports, we suggest interested readers consult Heppner and Heppner (2004),
which provides a more in depth guide to writing and publishing thesis and dis-
sertation research, and offers many examples to illustrate writing each section
of a research report.
It is also important to acknowledge at the beginning that everyone has
difficulty in writing at one time or another. It may be due to writer’s block,
fatigue, a particularly complex writing task, or a myriad of fears and appre-
hensions about the product (for example, “I will make a mistake, which will
be there for everyone to see”; “What I have to say is really not very important”;
“If I can’t write it perfectly, I won’t write anything”). For inexperienced au-
thors, many of these fears are particularly acute but developmentally normal.
However, even the veteran author experiences fears and apprehensions. Writing
is just one more skill the researcher needs to acquire, and like any other skill,
it takes practice, feedback, analysis of successful models of writing such as
published articles, and rewriting.
• Participants
• Measures (or Variables or Instruments)
• Materials
• Design (or Design and Analysis)
• Procedure
• Results
• Discussion (or Conclusions)
• References
• Tables
• Figures
The nature of each of these components depends on the publication and its
readership. Nevertheless, some aspects of each of these components are crucial
to well-written research reports. In this discussion we will focus on those
aspects of the components that relate to research design. (See APA, 2001;
Dorn, 1985; Heppner & Heppner, 2004; and Sternberg, 1988, for other dis-
cussions related to these components.) We will not discuss reference lists,
tables, and figures, because these are clearly described in the APA Publication
Manual. As we discuss these components sequentially, keep in mind that they are
held together by a common focus—the research questions and/or hypotheses—
which help the reader comprehend the study as a whole. For this reason, pres-
entation of the results, for example, should mirror presentation of the design
used to examine the research questions and/or hypotheses, as discussed in the
method section.
Title
The title of a research report does much more than just give it a name; it is
used in a variety of ways to make the report accessible to readers. Many services
depend on titles to index and abstract publications; users of these services often
see only the title. Furthermore, readers perusing journals or other sources (such
as Current Contents or PsychLit) often decide to read a report solely on the
basis of its title. Therefore, choose the title carefully.
A title should accurately summarize the research, describing the topic, the
variables, the design, and the outcome succinctly (in 12 to 15 words, according
to most sources). Given the limitations on the length of titles, redundancies
and irrelevancies must be avoided. Omit such phrases as “A Study of” or “The
Relation Between.” However, even within these constraints, interesting and
stylistic titles can be developed.
Alloy and Abramson (1979) provide an example of a seminal article with
a title that is both descriptive and attention-getting: “Judgment of Con-
tingency in Depressed and Nondepressed Students: Sadder but Wiser?” The title
describes the independent variable (depressed versus nondepressed), the depend-
ent variable (judgment of contingency), the participants (students), and the results
(sadder but wiser). The phrase “Sadder but Wiser?” not only adds interest,
but also has become a mnemonic; it is often used to refer to this article.
chapter 22 professional writing 543
Abstract
The abstract is a summary of the research report. Like the title, the abstract is
used for indexing and retrieving articles. After the title, the abstract is the
most-often read portion of a research report, and thus it should be succinct,
accurate, and comprehensive. It should summarize the content of each of the
major sections of the report, including the hypotheses, method (participants,
measures, materials, design, and procedure), results, and conclusions. Re-
member that nonsignificant results are informative; typically, it can be useful
to report and discuss all results, not just those that were expected. The length
of abstracts depends on the publication; APA journals require abstracts of
empirical studies to be between 100 and 120 words. Writing accurate and
informative abstracts is a difficult task.
The abstract should be self-contained; the reader should not need to
depend on any other section of the report to make sense of the abstract.
Therefore, avoid abbreviations and acronyms to the extent possible, or at the
very least make sure that they are defined in the abstract. Explain unique and
technical terms. Because readers of abstracts often do not have access to the
entire research report, do not refer to other articles in an abstract. Exceptions
to this rule include abstracts of reaction articles (for instance, an article written
in response to a previous article or a reanalysis of data from a published article).
Introduction
The introduction to a research report sets the stage for the study. It should
orient the reader to the problem, develop the rationale behind the study, and
indicate as specifically as possible the hypotheses being tested. Several ques-
tions should be answered in the introduction: Why is this an important topic
to study? What previous work (empirical and theoretical) bears on the topic?
How does this previous work logically lead to the author’s research question
and hypotheses? How will this question be researched? What predictions can
be made? To answer these questions, the introduction of a research report typ-
ically contains three elements: (1) an introduction to the problem, (2) a devel-
opment of the framework for the study, and (3) a statement of the research
questions and/or hypotheses.
The introduction to the problem should begin by orienting the reader to
the topic, including why this is an important area to investigate. This discussion
should be achieved with broad strokes. A study of child abuse might begin with
some general statistics about the prevalence of child abuse and its implications
for society; a study of supervision might begin by discussing difficulties in-
volved in understanding the supervisory process (see, for example, Friedlander,
Siegel, & Brenock, 1989); a study of social support vis-à-vis the effectiveness
of group therapy might begin with a statement about the widespread finding
that social support mediates stressful life changes (see Mallinckrodt, 1989).
The introduction to the problem should begin to narrow the topic by identifying
important developments relevant to the topic area. With regard to child abuse,
544 part 4 professional issues
the identification of stress as a precursor of abuse could be cited (if that is the
direction the researcher will pursue). The introduction to the problem can be
concluded with a statement of the problem in general terms.
The second element in an introduction is the development of a rationale
for the study. The framework for the rationale is built through the logical inter-
connectedness of empirical results and theory that leads to a critical, unan-
swered research question. We quite deliberately avoid the use of the more
traditional term literature review, because it implies (at least to some) a syn-
opsis of one study after another, along with considerable integration and syn-
thesis of the findings; in contrast, a framework consists of elements connected
in some logical way. In some ways, the development of a framework is similar
to that of a legal brief. When the case is solid, there is one inescapable con-
clusion; similarly, by the time the readers have completed reading this section,
they should be saying, “I know exactly what research question should be
asked” and then find that the author asked that research question.
The literature cited in the development of the framework should be pertinent
to this particular research study. If reviews of the literature exist, cite the
review articles rather than the original studies if this is sufficient to build the
case. In any discussion of a previous study, discuss only the pertinent aspects
of that study which build the rationale of your study—these are typically
those aspects related to the findings of that study, but perhaps methodological
issues (such as type of participants), design, and statistical tests as well. For
example, reference to a study would not mention the number of participants
used in that study unless those data were pertinent to the rationale of your
study, which would be the case if the point being made was related to inade-
quate power to detect an effect. Discussion of previous findings and theories
needs to be intertwined logically. When you discuss a particular study or
theory, the purpose of the discussion vis-à-vis the framework should be clear
to the reader. Do not hesitate to inform the reader directly: “Left and Write’s
(1980) application of the Direct Counseling Approach (DCA) is important
because it extends previous studies using DCA to children.” Furthermore, you
should integrate the material reviewed. Often studies result in contradictory
findings; you need to speculate about the reasons for the discrepancy and to
indicate how these discrepancies (and the reasons behind them) relate to the
present study.
A word about completeness is needed. Certainly, the scope of the framework
and the literature discussed in it is determined by the publication for which it
is intended; more completeness is needed for a dissertation than for a journal
article. Nevertheless, the goal is the same: to develop a logical argument that
will lead to a research question and/or hypothesis, not to review all the studies
ever written on a topic. Generally, this element of the introduction should be
complete enough that it touches on issues raised elsewhere in the research
report; it should not be necessary to review literature when hypotheses are
stated or when dependent variables are operationalized. One exception to this
rule is that it is common to report psychometric information under the measures
subsection of the method section.
chapter 22 professional writing 545
If the framework has been developed properly, the purpose of the present
research should be readily apparent. One purpose might be to reconcile dis-
crepancies in previous research; in that case the study will explain why con-
tradictory findings have been obtained. Another purpose might be to apply
results from some noncounseling area to problems in counseling (see, for example,
the social influence model, Strong, 1968). Still another purpose of the study
might be to extend previous results to a different population.
The final element of the introduction is the statement of the research questions
and/or hypotheses. The research questions and/or hypotheses should follow
logically from the framework built previously. Research questions are typically
used in descriptive studies, especially of a qualitative nature. In those studies
the author may not have specific hypotheses because little is known about the
topic, and the purpose of the study is more exploratory than predictive of rela-
tionships. Conversely, authors often use hypotheses as a critical test of some
important theoretical or practical question (Shadish, Cook, & Campbell,
2002; Wampold, Davis, and Good, 1990). A good theory will allow several
implications to be drawn from it; a hypothesis is critical to the extent that it
tests an implication from a theory and that the implication is unique to that
theory (see Wampold, Davis, and Good, 1990). In that way, knowledge pro-
gresses because unsubstantiated theories are winnowed out.
Research questions and/or hypotheses are written in terms of theoretical
constructs; operationalization of the constructs comes later. For example, the
research hypothesis might state that a cognitive-behavioral treatment will
lower state anxiety but not trait anxiety. Unless there is some particular issue
with operationalization that this study addresses directly, operationalization
of state and trait anxiety is typically discussed under the measures subsection
of the method section. Thus, discussion of particular measures used to opera-
tionalize a construct should not be included in the introduction, unless a par-
ticular operationalization is related to the research hypothesis (for example,
the study differs from previous studies in the manner in which a construct is
operationalized).
Research hypotheses should be stated unambiguously. Of course, the
degree to which a hypothesis can be specific depends on the specificity of the
theory. However, statements such as “the purpose of the present study is to
explore the relation between” or “the purpose is to determine the relation
between” are insufficient because some relation will always be discovered
(even if the relation is null—that is, no relation). Such statements do not make
clear how the discovered relation relates to theoretical predictions, and the
researcher must create post-hoc explanations.
An example of an unambiguous research hypothesis is provided by Ponce
and Atkinson (1989). The purpose of this study was to examine the effects of
counselor ethnicity, participant acculturation, and counseling style on Mexican-
American participants’ perceptions of counselor credibility and influence. It
was hypothesized that participant acculturation would interact with both
counselor ethnicity and counseling style in such a way that unacculturated
participants would prefer a Mexican-American counselor who used a directive
546 part 4 professional issues
style, whereas acculturated clients would have no preference for either coun-
selor ethnicity or counseling style.
An example of a research question that guides the direction of a study is:
What are the coping processes Mexican immigrants use in response to the
stressors of acculturation in a rural Midwest community? For more on this
topic, see Suarez-Renaud, 2002. Heppner and Heppner (2004) devote an entire
chapter to discussing writing research questions and hypotheses, and provide
examples of how the literature review can be used to establish research questions
and hypotheses.
Method
The method section describes how the research questions and/or hypotheses
were examined, including all aspects of how the study was conducted. Enough
detail is needed so that a reader can (1) determine the validity of the study and
(2) replicate the study. Of all the sections of a manuscript, the method section
often has the highest correlation with editorial decisions concerning rejection
or acceptance for publication (Munley, Sharkin, & Gelso, 1988); therefore, it
is imperative that the method section be well written.
The method section is typically divided into subsections. Research in
counseling usually contains subsections on participants, measures (or variables
or instruments), materials, design, and procedures, each of which we briefly
describe here. Organization of the method section depends, to a great extent,
on the nature of the study. The order of the subsections may be altered (for
instance, design often appears as either the first or last subsection), but they
should fit together like a puzzle, providing at the end a very clear picture of
what was done and how it was done. Furthermore, they should explain how
the researcher analyzed the data so that the reader is primed for the presentation
of the results.
Design (or Design and Analysis) This subsection describes the design of the
study. After this section, readers should have a clear understanding of, for
example, the independent, status, and dependent variables and the manner in
which participants were assigned to levels of the independent variables. For
example, if a factorial design is used, the factors should be labeled, the
number of conditions within a factor should be indicated, and the nature of
the factors (such as between versus within, or random versus fixed) should be
stated. If the analyses are mentioned (for example, a two-way analysis of vari-
ance applied to a factorial design), then the name of the subsection should
include “Analysis.” A power analysis, if conducted, can also be included in
this subsection.
The connection between the design and the research questions and/or
hypotheses should be clear. For example, after reading this subsection, the
reader should understand how the hypotheses were tested (see Wampold,
Davis, and Good, 1990). Often (especially in a dissertation), the research hy-
potheses are restated in operationalized form, such as “To test the research
hypothesis that [statement of hypothesis], a two-way analysis of variance was
conducted. [Explain factors and indicate dependent variable.] It was predicted
that a statistically significant F test for the interaction would be obtained.”
Procedure The procedure subsection describes how the research was con-
ducted with the participants. Discuss everything that the researchers did
with the participants, from beginning to end, including instructions to the
participants, the formation of groups, experimental manipulations, interview
questions, development of questionnaire items, and so on. The procedure sub-
section is typically organized chronologically. The goal of the procedure subsec-
tion is to provide enough detail so that the study can be replicated by another
researcher. If a separate design subsection is not included, details related to
randomization and other features of the design can be included in this subsec-
tion. The procedure section can vary greatly depending on whether the study
is a survey study, a correlational study, a scale construction study, or a coun-
seling process study (see examples in Heppner & Heppner, 2004).
chapter 22 professional writing 549
Results
The purpose of the results section is to summarize the data and the results of
the analyses utilized in the study. It is important to note that the writing style
is quite different for a results section depending on whether the methodology
is quantitative or qualitative (see Heppner & Heppner, 2004). Generally, for
a quantitative study, two types of results are reported: (1) data screening, sum-
mary statistics, and results of preliminary analyses; and (2) results related to
the research hypotheses. The summary statistics typically include the means
and standard deviations of the dependent variables for each level of the inde-
pendent variable (or combination of levels, as in a factorial design) and the
correlation matrix for all variables, when appropriate (for example, regression
analyses, factor analyses, or structural equation modeling). Results of prelim-
inary analyses may be related to manipulation checks, dropouts, verification
of stimulus materials, and so forth (see Heppner & Heppner, 2004).
Results related to research hypotheses should be organized so that the
reader can tie the results of specific statistical tests to research hypotheses stated
earlier in the report. There should be a one-to-one correspondence between the
hypotheses and the statistical tests (see Wampold, Davis, and Good, 1990). This
organization is facilitated by appropriately titled subsections, such as “Preliminary
Analyses,” “Summary Statistics,” or “Test of Hypothesis 1.”
The results section should report the findings, but discussion of the results
is saved for the discussion section. Nevertheless, introduce each result so that
its purpose is clear. For example, indicate why a preliminary analysis was con-
ducted and what various outcomes mean: “A preliminary analysis on the
observers’ ratings of the therapy sessions was conducted to determine whether
or not the treatments were perceived as different. The [name of statistical test]
was statistically significant [give information about test statistic], indicating
that the observers did indeed rate the treatments differently.”
Some confusion often exists about what information to include concern-
ing statistical tests. The APA Publication Manual (2001) gives specifics about
reporting statistical tests in the text and in tables. However, these specifica-
tions are inadequate because for the most part they indicate how to report
rather than what to report. The trend has been to report less; for example, one
rarely sees analysis of variance source tables anymore. More disturbing is the
tendency not to report important information (such as size of test statistic and
probability levels) when results are nonsignificant. This minimalist point of
view puts the emphasis on statistical significance and ignores concepts such as
effect size, estimation, and power. Failure to report information other than
significance levels makes it difficult or impossible for readers to verify results,
calculate other indexes (such as effect size or power), and conduct meta-
analyses. A persuasive argument can be made for reporting size of test statis-
tics, exact probability levels (rather than, say, p < .05) for significant and
nonsignificant findings, some measure of effect size, power, and other indexes
as appropriate (Rosnow & Rosenthal, 1988). An article by Ellis, Robbins,
Shult, Ladany, and Banker (1990) provides a good example of reporting a
550 part 4 professional issues
variety of indexes other than level of significance. For more details about writ-
ing the results section for a wide range of research designs and corresponding
statistical analyses, see Heppner and Heppner (2004).
For qualitative studies, the results section will typically differ based on the
chosen strategy of inquiry (e.g., grounded theory, consensual qualitative research,
or phenomenology/hermeneutics). Thus, the results section might contain
subsections such as a dimensional analysis model, descriptions of categories or
domains that emerge from the data, excerpts from the interviews to explain the
meaning of the categories, cross-analysis of the results, and detailed descriptions
of participants’ lived experiences. For concrete examples of writing qualitative
results sections, see Heppner and Heppner (2004).
For both quantitative and qualitative studies, results are often presented
in tables or figures, which are useful because results can be understood while
taking up a minimal amount of space. Locating important results in the text
is much more difficult than finding them in a well-organized table. Figures are
particularly useful for illustrating patterns of results, such as an interaction in
an analysis of variance. When tables or figures are used, total recapitulation
of the results in the text should be avoided, although sometimes it is useful to
summarize key results presented in the tables. The writer must, however, tell
the reader what to look for in the tables. For example, one might say, “The
main effects of the 3 (treatment) by 2 (gender) analysis of variance were sta-
tistically significant, as shown in Table 1”; it is then unnecessary to note, for
example, the degrees of freedom, the F statistics, and so forth, because that
information is in the table.
Discussion
The discussion typically includes: (1) an explanation of the results of the data
analyses; (2) if appropriate, whether or not the data support the research
hypotheses; (3) a statement of the conclusions; (4) an indication of the study’s
limitations; and (5) a discussion of the implications. This section allows the
author to expand on the findings and to place them in the context of previous
research and theory on this topic.
The discussion typically begins with a description of the strongest findings
from the data analyses that were presented in the results section. For example,
sometimes authors structure the discussion around the hypotheses, addressing
each one sequentially (see Heppner & Heppner, 2004, for examples). In such
cases, there are usually clear statements about whether the research hypothe-
ses were supported. Has new knowledge been generated by support of the
hypotheses? Do the results support or refute particular theories? If the data
lead to reformulated hypotheses, state them, but indicate that they have been
formulated post hoc (that is, they have not been tested directly). If future
research is needed, indicate what types of hypotheses should be tested and how
future studies should be designed (see Heppner & Heppner).
Every discussion of results should include a statement of the limitations, which
typically are related to methodological issues such as low power, the analogue
chapter 22 professional writing 551
Principle 1: Be Informative
The goal of the research report is to inform the reader about the study. Provide
enough detail for the reader to understand what you are explaining, but not so
much that the reader becomes bogged down. Of course, the publication format
determines the level of detail. Dissertations require the most detail. Some journals
are more technical than others and thus require more detail. Are the readers of
a particular journal primarily practitioners, other researchers, or statisticians?
The best way to include the right amount of detail is to emulate the publication
of interest. Most journals periodically present editorials or other information
about the materials to be contained in their articles (see the inside covers of
most journals for descriptions of appropriate articles).
In any report, discuss the central points and minimize digressions. Obviously,
you should not report everything that you did, but only information that readers
need to understand the study. For example, preparation of stimulus materials
often involves many successive stages of development. It is not necessary to
explain each iteration; rather, describe the final product and summarize the
development process in a sentence or two. However, do not omit important
details, either. Authors frequently refer to a decision that was made but do not
discuss the decision rule; if, for example, “three outliers were omitted from the
analysis,” on what basis were the three data considered outliers? Was this decision
rule formulated before the data were analyzed? (For more details on writing the
discussion section, see Heppner and Heppner, 2004.)
Principle 2: Be Forthright
Every study has flaws (Gelso, 1979). It is rare that a researcher would not
alter a study, given the chance to do so. Authors should be forthright and discuss
552 part 4 professional issues
justified, Wampold and Kim provided several possible explanations for the
differences and made the following conclusion: “Whether Hill et al.’s obser-
vations or the patterns revealed by the sequential analysis [conducted by
Wampold & Kim] are reflective of the essential nature of the counseling process
in this study remains an unanswered epistemological question” (p. 362). Readers
are left to draw their own conclusions.
(participants, measures, materials, design, and and experienced authors alike. One of the most
procedure), a presentation of the results, and a effective strategies to improve one’s professional
discussion or conclusion. The content of these writing is to work closely with a successful author,
sections is determined, to a great extent, by the writing drafts, receiving feedback, rewriting, and
design of the research. polishing. It is not atypical for graduate students
The style of the written report varies, for each to spend two to three years co-writing with a
of us has our own style, which we in turn alter faculty member to enhance both their profes-
depending on the publication outlet for which sional writing skills and their research skills in
our writing is intended. Nevertheless, as a rule general. We strongly urge students to actively solicit
authors should (1) be informative, (2) be forth- feedback on their writing and seek co-writing ex-
right, (3) not overstate or exaggerate, (4) be log- periences with established authors.
ical and organized, (5) write in an interesting In closing, it is our sincere hope that students
way, and (6) revise their reports to obtain a pol- are able to utilize scholarly inquiry to promote
ished product. an inquisitive and pioneering attitude. As Enos
Professional writing is a complex skill that Mills, an explorer of Rocky Mountain National
takes years of practice and feedback. Moreover, Park, observed, pioneers lives suffer from no dull
it is a skill that involves one’s personal style of com- existence, but rather “are rich with hope and
municating. Procrastination and other avoidance their future has all the promise of spring” (Mills,
patterns are common reactions of inexperienced 1924, p. 9). We wish you happy trails!
STIMULUS EXERCISE
Writing an Introduction to an Article
Learning to write a compelling introduction is dif- area of interest and receives feedback on their
ficult. We have found that the following exercise is critique from the instructor.
helpful in learning this complex skill. This exercise 3. The next assignment involves a published
is best done as a class, or with a small group of four article that contains an introduction citing
to six students. It is best if you can have a faculty seven other articles (but no books or book
advisor or advanced student with expertise in sci- chapters). The students are given the pub-
entific writing leading the exercise or helping the lished article without the introduction and
group work through it together. We will describe the seven articles cited.
the exercise using a class format, but it can be 4. The next assignment is to read the seven cited
adapted for a small group of students as long as articles and the published article (without the
one of the students is more advanced in their writ- introduction). At the next class meeting, the
ing skills and can give appropriate feedback to class discusses various structures that could be
group members. In addition, the exercise can be used to construct the introduction to the pub-
done with all seven steps or just a few of the steps. lished article (that is, the logic of the intro-
duction, how the paragraphs would follow
1. First, the class critiques an introduction to an each other, how the case would be built for
article the instructor selected from a recent the hypotheses, and so on). In this discussion,
journal. it is emphasized that there is no one correct
2. Then, as an individual assignment, each student way to write this introduction, but the class
critiques the introduction to an article in their agrees on a common structure.
556 part 4 professional issues
5. The next assignment is for each student to level (that is, grammar, sentence construction,
write, in APA style (that is, in accordance with the paragraph construction, logic, proper citation,
Publication Manual of the American Psychological and so forth) within one week.
Association, 2001), an in.troduction that leads 6. The final assignment is to revise the intro-
logically to the hypothesis, using only the seven duction given the instructor’s feedback. This
articles as supporting evidence (this assignment revision is again evaluated by the instructor.
is completed in one week). The instructor 7. At the end, the original introduction is cri-
grades the papers, providing feedback at every tiqued in a class discussion.
ACA Code of Ethics APPENDIX
PREAMBLE
The American Counseling Association is an educational, scientific, and profes-
sional organization whose members work in a variety of settings and serve in
multiple capacities. ACA members are dedicated to the enhancement of human
development throughout the life span. Association members recognize diver-
sity and embrace a cross-cultural approach in support of the worth, dignity,
potential, and uniqueness of people within their social and cultural contexts.
Professional values are an important way of living out an ethical com-
mitment. Values inform principles. Inherently held values that guide our be-
haviors or exceed prescribed behaviors are deeply ingrained in the counselor
and developed out of personal dedication, rather than the mandatory requirement
of an external organization.
557
558 appendix a
3. The Code establishes principles that define ethical behavior and best practices
of association members.
4. The Code serves as an ethical guide designed to assist members in construct-
ing a professional course of action that best serves those utilizing counseling
services and best promotes the values of the counseling profession.
5. The Code serves as the basis for processing of ethical complaints and in-
quiries initiated against members of the association.
The ACA Code of Ethics contains eight main sections that address the following
areas:
Section A: The Counseling Relationship
Section B: Confidentiality, Privileged Communication, and Privacy
Section C: Professional Responsibility
Section D: Relationships With Other Professionals
Section E: Evaluation, Assessment, and Interpretation
Section F: Supervision, Training, and Teaching
Section G: Research and Publication
Section H: Resolving Ethical Issues
Each section of the ACA Code of Ethics begins with an Introduction. The
introductions to each section discuss what counselors should aspire to with
regard to ethical behavior and responsibility. The Introduction helps set the tone
for that particular section and provides a starting point that invites reflection
on the ethical mandates contained in each part of the ACA Code of Ethics.
When counselors are faced with ethical dilemmas that are difficult to
resolve, they are expected to engage in a carefully considered ethical decision-
making process. Reasonable differences of opinion can and do exist among
counselors with respect to the ways in which values, ethical principles, and
ethical standards would be applied when they conflict. While there is no specific
ethical decision-making model that is most effective, counselors are expected
to be familiar with a credible model of decision making that can bear public
scrutiny and its application.
Through a chosen ethical decision-making process and evaluation of the
context of the situation, counselors are empowered to make decisions that
help expand the capacity of people to grow and develop.
A brief glossary is given to provide readers with a concise description of
some of the terms used in the ACA Code of Ethics.
A.1.c. Counseling Plans Counselors and their clients work jointly in devising
integrated counseling plans that offer reasonable promise of success and are
consistent with abilities and circumstances of clients. Counselors and clients
regularly review counseling plans to assess their continued viability and effec-
tiveness, respecting the freedom of choice of clients. (See A.2.a., A.2.d., A.12.g.)
A.2.a. Informed Consent Clients have the freedom to choose whether to enter
into or remain in a counseling relationship and need adequate information
about the counseling process and the counselor. Counselors have an obligation
560 appendix a
to review in writing and verbally with clients the rights and responsibilities of
both the counselor and the client. Informed consent is an ongoing part of the
counseling process, and counselors appropriately document discussions of in-
formed consent throughout the counseling relationship.
A.4.b. Personal Values Counselors are aware of their own values, attitudes,
beliefs, and behaviors and avoid imposing values that are inconsistent with
counseling goals. Counselors respect the diversity of clients, trainees, and research
participants.
A.10.d. Bartering Counselors may barter only if the relationship is not ex-
ploitive or harmful and does not place the counselor in an unfair advantage,
564 appendix a
if the client requests it, and if such arrangements are an accepted practice
among professionals in the community. Counselors consider the cultural im-
plications of bartering and discuss relevant concerns with clients and docu-
ment such agreements in a clear written contract.
Wide Web, the Internet, online assessment instruments, and other communi-
cation devices.
A.12.e. Laws and Statutes Counselors ensure that the use of technology does
not violate the laws of any local, state, national, or international entity and
observe all relevant statutes.
10. Discuss time zone differences, local customs, and cultural or language dif-
ferences that might impact service delivery.
11. Inform clients when technology-assisted distance counseling services are
not covered by insurance. (See A.2.)
A.12.h. Sites on the World Wide Web Counselors maintaining sites on the
World Wide Web (the Internet) do the following:
1. Regularly check that electronic links are working and professionally ap-
propriate.
2. Establish ways clients can contact the counselor in case of technology failure.
3. Provide electronic links to relevant state licensure and professional certification
boards to protect consumer rights and facilitate addressing ethical concerns.
4. Establish a method for verifying client identity.
5. Obtain the written consent of the legal guardian or other authorized legal
representative prior to rendering services in the event the client is a minor
child, an adult who is legally incompetent, or an adult incapable of giving
informed consent.
6. Strive to provide a site that is accessible to persons with disabilities.
7. Strive to provide translation capabilities for clients who have a different primary
language while also addressing the imperfect nature of such translations.
8. Assist clients in determining the validity and reliability of information
found on the World Wide Web and other technology applications.
B.2. Exceptions
B.2.a. Danger and Legal Requirements The general requirement that counselors
keep information confidential does not apply when disclosure is required to
protect clients or identified others from serious and foreseeable harm or when
legal requirements demand that confidential information must be revealed.
Counselors consult with other professionals when in doubt as to the validity
of an exception. Additional considerations apply when addressing end-of-life
issues. (See A.9.c.)
B.2.d. Minimal Disclosure To the extent possible, clients are informed be-
fore confidential information is disclosed and are involved in the disclosure
decision-making process. When circumstances require the disclosure of confi-
dential information, only essential information is revealed.
existence and composition, information being shared, and the purposes of sharing
such information.
B.6. Records
B.6.a. Confidentiality of Records Counselors ensure that records are kept
in a secure location and that only authorized persons have access to records.
B.6.e. Assistance with Records When clients request access to their records,
counselors provide assistance and consultation in interpreting counseling records.
B.8. Consultation
B.8.a. Agreements When acting as consultants, counselors seek agreements
among all parties involved concerning each individual’s rights to confidential-
ity, the obligation of each individual to preserve confidential information, and
the limits of confidentiality of information shared by others.
C.2.g. Impairment Counselors are alert to the signs of impairment from their
own physical, mental, or emotional problems and refrain from offering or pro-
viding professional services when such impairment is likely to harm a client or
others. They seek assistance for problems that reach the level of professional
impairment, and, if necessary, they limit, suspend, or terminate their professional
responsibilities until such time it is determined that they may safely resume their
work. Counselors assist colleagues or supervisors in recognizing their own pro-
fessional impairment and provide consultation and assistance when warranted
with colleagues or supervisors showing signs of impairment and intervene as
appropriate to prevent imminent harm to clients. (See A.11.b., F.8.b.)
C.3.b. Testimonials Counselors who use testimonials do not solicit them from
current clients nor former clients nor any other persons who may be vulnerable
to undue influence.
aca code of ethics 573
C.5. Nondiscrimination
Counselors do not condone or engage in discrimination based on age, culture,
disability, ethnicity, race, religion/spirituality, gender, gender identity, sexual
orientation, marital status/ partnership, language preference, socioeconomic
status, or any basis proscribed by law. Counselors do not discriminate against
clients, students, employees, supervisees, or research participants in a manner
that has a negative impact on these persons.
C.6.b. Reports to Third Parties Counselors are accurate, honest, and objec-
tive in reporting their professional activities and judgments to appropriate
third parties, including courts, health insurance companies, those who are the
recipients of evaluation reports, and others. (See B.3., E.4.)
D.1.i. Protection from Punitive Action Counselors take care not to harass
or dismiss an employee who has acted in a responsible and ethical manner to
expose inappropriate employer policies or practices.
D.2. Consultation
D.2.a. Consultant Competency Counselors take reasonable steps to ensure
that they have the appropriate resources and competencies when providing
consultation services. Counselors provide appropriate referral resources when
requested or needed. (See C.2.a.)
E.1. General
E.1.a. Assessment The primary purpose of educational, psychological, and
career assessment is to provide measurements that are valid and reliable in
either comparative or absolute terms. These include, but are not limited to,
measurements of ability, personality, interest, intelligence, achievement, and
performance. Counselors recognize the need to interpret the statements in this
section as applying to both quantitative and qualitative assessments.
E.1.b. Client Welfare Counselors do not misuse assessment results and in-
terpretations, and they take reasonable steps to prevent others from misusing
the information these techniques provide. They respect the client’s right to
know the results, the interpretations made, and the bases for counselors’ con-
clusions and recommendations.
E.2.b. Appropriate Use Counselors are responsible for the appropriate appli-
cation, scoring, interpretation, and use of assessment instruments relevant to the
needs of the client, whether they score and interpret such assessments themselves
or use technology or other services.
E.5.d. Refraining from Diagnosis Counselors may refrain from making and/or
reporting a diagnosis if they believe it would cause harm to the client or others.
Counselors have theoretical and pedagogical foundations for their work and
aim to be fair, accurate, and honest in their assessments of counselors-in-training.
G.2.h. Explanations After Data Collection After data are collected, coun-
selors provide participants with full clarification of the nature of the study to
aca code of ethics 591
G.4.d. Identity of Participants Counselors who supply data, aid in the research
of another person, report research results, or make original data available
take due care to disguise the identity of respective participants in the absence
of specific authorization from the participants to do otherwise. In situations
where participants self-identify their involvement in research studies, researchers
take active steps to ensure that data are adapted/changed to protect the identity
and welfare of all parties and that discussion of results does not cause harm
to participants.
G.5. Publication
G.5.a. Recognizing Contributions When conducting and reporting research,
counselors are familiar with and give recognition to previous work on the topic,
observe copyright laws, and give full credit to those to whom credit is due.
G.5.b. Plagiarism Counselors do not plagiarize; that is, they do not present another
person’s work as their own work.
G.5.f. Student Research For articles that are substantially based on students’
course papers, projects, dissertations or theses, and on which students have
been the primary contributors, they are listed as principal authors.
H.1.a. Knowledge Counselors understand the ACA Code of Ethics and other
applicable ethics codes from other professional organizations or from certifi-
cation and licensure bodies of which they are members. Lack of knowledge or
594 appendix a
INTRODUCTION
The American Psychological Association’s (APA’s) Ethical Principles of Psy-
chologists and Code of Conduct (hereinafter referred to as the Ethics Code)
consists of an Introduction, a Preamble, six General Principles (A–F), and spe-
cific Ethical Standards. The Introduction discusses the intent, organization,
procedural considerations, and scope of application of the Ethics Code. The
Preamble and General Principles are aspirational goals to guide psychologists
toward the highest ideals of psychology. Although the Preamble and General
Principles are not themselves enforceable rules, they should be considered by
psychologists in arriving at an ethical course of action and may be considered
by ethics bodies in interpreting the Ethical Standards. The Ethical Standards
set forth enforceable rules for conduct as psychologists. Most of the Ethical
Standards are written broadly, in order to apply to psychologists in varied
roles, although the application of an Ethical Standard may vary depending on
the context. The Ethical Standards are not exhaustive. The fact that a given
conduct is not specifically addressed by the Ethics Code does not mean that it
is necessarily either ethical or unethical.
This version of the American Psychological Association Ethics Code was adopted by the
American Psychological Association’s Council of Representatives during its meeting on August
13 and 16, 1992, and was published in American Psychologist, 1992, Vol. 47, 1597–1611.
Copyright 1992. Reprinted by permission.
596
ethical principles of psychologists and code of conduct 597
Membership in the APA commits members to adhere to the APA Ethics Code
and to the rules and procedures used to implement it. Psychologists and students,
whether or not they are APA members. should be aware that the Ethics Code may
be applied to them by state psychology boards, courts, or other public bodies.
This Ethics Code applies only to psychologists’ work-related activities,
that is, activities that are part of the psychologists’ scientific and professional
functions or that are psychological in nature. It includes the clinical or coun-
seling practice of psychology, research, teaching, supervision of trainees, de-
velopment of assessment instruments, conducting assessments, educational
counseling, organizational consulting, social intervention, administration, and
other activities as well. These work-related activities can be distinguished
from the purely private conduct of a psychologist, which ordinarily is not
within the purview of the Ethics Code.
The Ethics Code is intended to provide standards of professional conduct
that can be applied by the APA and by other bodies that choose to adopt
them. Whether or not a psychologist has violated the Ethics Code does not by
itself determine whether he or she is legally liable in a court action, whether a
contract is enforceable, or whether other legal consequences occur. These
results are based on legal rather than ethical rules. However, compliance with
or violation of the Ethics Code may be admissible as evidence in some legal
proceedings, depending on the circumstances.
In the process of making decisions regarding their professional behavior,
psychologists must consider this Ethics Code, in addition to applicable laws
and psychology board regulations. If the Ethics Code establishes a higher
standard of conduct than is required by law, psychologists must meet the
higher ethical standard. If the Ethics Code standard appears to conflict with
the requirements of law, then psychologists make known their commitment to
the Ethics Code and take steps to resolve the conflict in a responsible manner.
If neither law nor the Ethics Code resolves an issue, psychologists should con-
sider other professional materials* and the dictates of their own conscience, as
well as seek consultation with others within the field when this is practical.
*Professional materials that are most helpful in this regard are guidelines and standards that have
been adopted or endorsed by professional psychological organizations. Such guidelines and stan-
dards, whether adopted by the American Psychological Association (APA) or its Divisions, are not
enforceable as such by this Ethics Code, but are of educative value to psychologists, courts, and
professional bodies. Such materials include, but are not limited to, the APA’s General Guidelines
for Providers of Psychological Services (1987), Specialty Guidelines for the Delivery of Services
by Clinical Psychologists, Counseling Psychologists, Industrial/Organizational Psychologists, and
School Psychologists (1981), Guidelines for Computer Based Tests and Interpretations (1987),
Standards for Educational and Psychological Testing (1985), Ethical Principles in the Conduct of
Research With Human Participants (1982), Guidelines for Ethical Conduct in the Care and Use
of Animals (1986), Guidelines for Providers of Psychological Services to Ethnic, Linguistic, and
Culturally Diverse Populations (1990), and the Publication Manual of the American Psychological
Association (3rd ed., 1983). Materials not adopted by APA as a whole include the APA Division
41 (Forensic Psychology)/American Psychology–Law Society’s Specialty Guidelines for Forensic
Psychologists (1991).
598 appendix b
PREAMBLE
Psychologists work to develop a valid and reliable body of scientific knowl-
edge based on research. They may apply that knowledge to human behavior
in a variety of contexts. In doing so, they perform many roles, such as re-
searcher, educator, diagnostician, therapist, supervisor, consultant, adminis-
trator, social interventionist, and expert witness. Their goal is to broaden
knowledge of behavior and, where appropriate, to apply it pragmatically to
improve the condition of both the individual and society. Psychologists respect
the central importance of freedom of inquiry and expression in research,
teaching, and publication. They also strive to help the public in developing
informed judgments and choices concerning human behavior. This Ethics
Code provides a common set of values upon which psychologists build their
professional and scientific work.
This Code is intended to provide both the general principles and the deci-
sion rules to cover most situations encountered by psychologists. It has as its
primary goal the welfare and protection of the individuals and groups with
whom psychologists work. It is the individual responsibility of each psychol-
ogist to aspire to the highest possible standards of conduct. Psychologists
respect and protect human and civil rights, and do not knowingly participate
in or condone unfair discriminatory practices.
The development of a dynamic set of ethical standards for a psychologist’s
work-related conduct requires a personal commitment to a lifelong effort to
act ethically; to encourage ethical behavior by students, supervisees, employees,
and colleagues, as appropriate; and to consult with others, as needed, con-
cerning ethical problems. Each psychologist supplements, but does not violate,
the Ethics Code’s values and rules on the basis of guidance drawn from personal
values, culture, and experience.
ethical principles of psychologists and code of conduct 599
GENERAL PRINCIPLES
Principle A: Competence
Psychologists strive to maintain high standards of competence in their work.
They recognize the boundaries of their particular competencies and the limi-
tations of their expertise. They provide only those services and use only those
techniques for which they are qualified by education, training, or experience.
Psychologists are cognizant of the fact that the competencies required in serving,
teaching, and/or studying groups of people vary with the distinctive charac-
teristics of those groups. In those areas in which recognized professional stan-
dards do not yet exist, psychologists exercise careful judgment and take
appropriate precautions to protect the welfare of those with whom they work.
They maintain knowledge of relevant scientific and professional information
related to the services they render, and they recognize the need for ongoing
education. Psychologists make appropriate use of scientific, professional,
technical, and administrative resources.
Principle B: Integrity
Psychologists seek to promote integrity in the science, teaching, and prac-
tice of psychology. In these activities psychologists are honest, fair, and
respectful of others. In describing or reporting their qualifications, services,
products, fees, research, or teaching, they do not make statements that are
false, misleading, or deceptive. Psychologists strive to be aware of their
own belief systems, values, needs, and limitations and the effect of these on
their work. To the extent feasible, they attempt to clarify for relevant par-
ties the roles they are performing and to function appropriately in accordance
with those roles. Psychologists avoid improper and potentially harmful
dual relationships.
ETHICAL STANDARDS
1. General Standards
These General Standards are potentially applicable to the professional and sci-
entific activities of all psychologists.
state the nature of the research or service provided, the fees or charges, and
where applicable, the identity of the provider, the findings, and the diagnosis.
(See also Standard 5.05, Disclosures.)
1.27 Referrals and Fees When a psychologist pays, receives payment from,
or divides fees with another professional other than in an employer-employee
relationship, the payment to each is based on the services (clinical, consulta-
tive, administrative, or other) provided and is not based on the referral itself.
4. Therapy
4.01 Structuring the Relationship
(a) Psychologists discuss with clients or patients as early as is feasible in the
therapeutic relationship appropriate issues, such as the nature and anticipated
course of therapy, fees, and confidentiality. (See also Standards 1.25, Fees and
Financial Arrangements, and 5.01, Discussing the Limits of Confidentiality.)
(b) When the psychologist’s work with clients or patients will be supervised,
the above discussion includes that fact, and the name of the supervisor, when
the supervisor has legal responsibility for the case.
(c) When the therapist is a student intern, the client or patient is informed
of that fact.
(d) Psychologists make reasonable efforts to answer patients’ questions
and to avoid apparent misunderstandings about therapy. Whenever possible,
psychologists provide oral and/or written information, using language that is
reasonably understandable to the patient or client.
(b) When persons are legally incapable of giving informed consent, psy-
chologists obtain informed permission from a legally authorized person, if such
substitute consent is permitted by law.
(c) In addition, psychologists (1) inform those persons who are legally inca-
pable of giving informed consent about the proposed interventions in a manner
commensurate with the persons’ psychological capacities, (2) seek their assent to
those interventions, and (3) consider such persons’ preferences and best interests.
4.06 Therapy with Former Sexual Partners Psychologists do not accept as therapy
patients or clients persons with whom they have engaged in sexual intimacies.
5.05 Disclosures
(a) Psychologists disclose confidential information without the consent of
the individual only as mandated by law, or where permitted by law for a valid
purpose, such as (1) to provide needed professional services to the patient or
the individual or organizational client, (2) to obtain appropriate professional
consultations, (3) to protect the patient or client or others from harm, or (4) to
obtain payment for services, in which instance disclosure is limited to the
minimum that is necessary to achieve the purpose.
(b) Psychologists also may disclose confidential information with the
appropriate consent of the patient or the individual or organizational client
(or of another legally authorized person on behalf of the patient or client),
unless prohibited by law.
with whom they have a confidential relationship unless they have obtained the
prior consent of the person or organization or the disclosure cannot be avoided,
and (2) they share information only to the extent necessary to achieve the pur-
poses of the consultation. (See also Standard 5.02, Maintaining Confidentiality.)
6.07 Responsibility
(a) Psychologists conduct research competently and with due concern for
the dignity and welfare of the participants.
(b) Psychologists are responsible for the ethical conduct of research con-
ducted by them or by others under their supervision or control.
(c) Researchers and assistants are permitted to perform only those tasks
for which they are appropriately trained and prepared.
(d) As part of the process of development and implementation of re-
search projects, psychologists consult those with expertise concerning any spe-
cial population under investigation or most likely to be affected.
6.08 Compliance with Law and Standards Psychologists plan and conduct
research in a manner consistent with federal and state law and regulations,
as well as professional standards governing the conduct of research, and
particularly those standards governing research with human participants
and animal subjects.
6.25 Sharing Data After research results are published, psychologists do not
withhold the data on which their conclusions are based from other competent
professionals who seek to verify the substantive claims through reanalysis and
who intend to use such data only for that purpose, provided that the confi-
dentiality of the participants can be protected and unless legal rights concerning
proprietary data preclude their release.
7. Forensic Activities
7.01 Professionalism Psychologists who perform forensic functions, such as
assessments, interviews, consultations, reports, or expert testimony must comply
with all other provisions of this Ethics Code to the extent that they apply to
such activities. In addition, psychologists base their forensic work on appropri-
ate knowledge of and competence in the areas underlying such work, includ-
ing specialized knowledge concerning special populations. (See also Standards
1.06, Basis for Scientific and Professional Judgments; 1.08, Human Differences;
1.15, Misuse of Psychologists’ Influence; and 1.23, Documentation of
Professional and Scientific Work.)
ethical principles of psychologists and code of conduct 619
7.06 Compliance with Law and Rules In performing forensic roles, psychol-
ogists are reasonably familiar with the rules governing their roles. Psy-
chologists are aware of the occasionally competing demands placed upon
them by these principles and the requirements of the court system, and
attempt to resolve these conflicts by making known their commitment to this
Ethics Code and taking steps to resolve the conflict in a responsible manner.
(See also Standard 1.02, Relationship of Ethics and Law.)
620 appendix b
Abramson, L. Y. (Ed.). (1988). Social Alexander, L. B., & Luborsky, L. (1986). American Psychological Association.
cognition and clinical psychology: The Penn Helping Alliance Scales. (1994). Publication manual of the
A synthesis. New York: Guilford In L. S. Greenberg and W. M. American Psychological Association
Press. Pinsof (Eds.), The Psychotherapeutic (4th ed.). Washington, DC: Author.
Adler, P. A., & Adler, P. (1991). Process: A Research Handbook American Psychological Association.
Backboards and blackboards. (pp. 325–366). New York: Guilford. (2001). Publication manual of the
New York: Columbia University Alloy, L. B., & Abramson, L. Y. American Psychological Association
Press. (1979). Judgment of contingency (5th ed.). Washington, DC: Author.
Adler, P. A., & Adler, P. (1994). in depressed and nondepressed American Psychological Association.
Observational techniques. In N. K. students: Sadder but wiser? Journal (2002). Ethical principles of psy-
Denzin & Y. S. Lincoln (Eds.), of Experimental Psychology, 108, chologists and code of conduct.
Handbook of qualitative research 441–485. Washington, DC: Author.
(pp. 377–392). Thousand Oaks, Alvidrez, J., Azocar, F., & Miranda, J. American Psychological Association.
CA: Sage. (1996). Demystifying the concept of (2003). Guidelines on multicul-
Albee, G. W. (1970). The uncertain ethnicity for psychotherapy researchers. tural education, training, research,
future of clinical psychology. American Journal of Consulting and Clinical practice, and organizational change
Psychologist, 25, 1071–1080. Psychology, 64, 903– 908. for psychologists. American Psy-
Aldenderfer, M. S., & Blashfield, R. K. American Cancer Society. (2003). Cancer chologist, 58, 377–402.
(1984). Cluster analysis. Beverly facts and figures. Atlanta: Author. American Psychological Association
Hills, CA: Sage. American Counseling Association. Ethics Committee. (1983, Febru-
Alexander, C. H., & Suzuki, L. A. (1995). Code of ethics. Washington, ary 19). Authorship guidelines for
(2001). Measurement of multicul- DC: Author. dissertation supervision. Washington,
tural constructs: Integration and American Counseling Association. DC: Author.
research directions. In J. G. (2005). Code of ethics. Alexandria, Andersen, B., & Anderson, W. (1985).
Ponterotto, J. M. Casas, L. A. VA: Author. Client perceptions of counselors
Suzuki, & C. M. Alexander (Eds.), American Psychological Association. using positive and negative self-
Handbook of multicultural coun- (1983). Publication manual of the involving statements. Journal of
seling (2nd ed., pp. 499–505). American Psychological Association Counseling Psychology, 32,
Thousand Oaks, CA: Sage. (3rd ed.). Washington, DC: Author. 462–465.
621
622 references
Anderson, J. R. (1983). The architecture Atkinson, D. R., Morten, G., & Sue, nary report. Psychotherapy Research,
of cognition. Cambridge, MA: Har- D. W. (Eds.). (1989). Counseling 6, 79–92.
vard University Press. American minorities: A cross- Barber, J. P., Crits-Christoph, P., &
Anderson, W. P., & Heppner, P. P. cultural perspective (3rd ed.). Luborsky, L. (1996). Effects of
(1986). Counselor applications of Dubuque, IA: William C. Brown. therapist adherence and compe-
research findings to practice: Atkinson, D. R., & Wampold, B. E. tence on patient outcome in brief
Learning to stay current. Journal (1982). A comparison of the dynamic therapy. Journal of Con-
of Counseling and Development, Counselor Rating Form and the sulting and Clinical Psychology, 64,
65, 152–155. Counselor Effectiveness Rating 619–622.
Anton, W. D., & Reed, J. R. (1991). Scale. Counselor Education and Barber, J. P., Gallop, R., Crits-
College adjustment scales. Odessa, Supervision, 22, 25–36. Christoph, P., Frank, A., Thase,
FL: Psychological Assessment Re- Ayalon, L., & Young, M. (2003). A M. E., Weiss, R. D., et al., (2006).
sources. comparison of depressive symp- The role of therapist adherence,
Areán, P. A., & Gallagher-Thompson, toms in African Americans and therapist competence, and alliance
D. (1996). Issues and recommen- Caucasian Americans. Journal of in predicting outcome of individ-
dations for the recruitment and Cross-Cultural Psychology, 34, ual drug counseling: Results from
retention of older ethnicity minor- 111–124. the National Institute Drug Abuse
ity adults into clinical research. Azibo, D. A. (1988). Understanding Collaborative Cocaine Treatment
Journal of Consulting and Clinical the proper and improper usage of Study. Psychotherapy Research,
Psychology, 64, 875–880. the comparative research frame- 16, 229–240.
Argyris, C. (1968). Some unintended work. Journal of Black Psychology, Barber, T. X. (1976). Pitfalls in human
consequences of rigorous research. 15, 81–91. research: Ten pivotal points. New
Psychological Bulletin, 70, 185–197. Babbie, E. (2001). The practice of York: Pergamon Press.
Arnold, C. L. (1992). An introduction social research (9th ed.). Belmont, Barber, T. X., & Silver, M. J. (1968).
to hierarchical linear models. CA: Wadsworth/Thomson Learning. Fact, fiction, and the experimenter
Measurement and Evaluation in Babbie, E. R. (1979). The practice of bias effect. Psychological Bulletin
Counseling and Development, 25, social research (2nd ed.). Belmont, Monograph, 70, 1–29.
58–90. CA: Wadsworth. Barkham, M., Andrew, R. M., &
Arredondo, P., Toporek, R., Brown, Bakeman, R., & Gottman, J. M. Culverwell, A. (1993). The California
S. P., Sanchez, J., Locke, D. C., (1986). Observing interaction: An psychotherapy alliance scales: A
Sanchez, J., & Stadler, H. (1996). introduction to sequential analysis. pilot study of dimensions and
Operationalization of the multi- Cambridge: Cambridge University elements. British Journal of Medical
cultural counseling competencies. Press. Psychology, 66, 157–165.
Journal of Multicultural Counseling Baldwin, S. A., Murray, D. M., & Barlow, D. H. (Ed.). (1981).
& Development, 24, 42–78. Shadish, W. R. (2005). Empirically Behavioral assessment of adult
Asner-Self, K. K., & Marotta, S. A. supported treatments or type I disorders. New York: Guilford
(2005). Developmental indices errors? Problems with the analysis Press.
among Central American immi- of data from group-administered Barlow, D. H., & Hersen, M. (1984).
grants exposed to war related treatments. Journal of Consulting Single case experimental designs:
trauma: Clinical implications for and Clinical Psychology, 73, Strategies for studying behavior
counselors. Journal of Counseling 924–935. change (2nd ed.). New York:
and Development, 83, 162–171. Bamberger, M. (2000). Integrating Pergamon Press.
Atkinson, D. (Ed.). (2004). Counseling quantitative and qualitative research Baron, R. M., & Kenny, D. A. (1986).
American minorities. Boston, MA: in development projects. Washington, The moderator-mediator variable
McGraw-Hill. DC: World Bank. distinction in social psychological
Atkinson, D., & Hackett, G. (2004). Bangert, A. W. & Baumberger, J. P. research: Conceptual, strategic,
Counseling diverse populations. (2005). Research designs and and statistical considerations. Jou-
Boston, MA: McGraw-Hill. statistical techniques used in the rnal of Personality and Social Psy-
Atkinson, D. R. (1983). Ethnic simi- Journal of Counseling & Development, chology, 51, 1173–1182.
larity in counseling psychology: A 1990–2001. Journal of Counseling Barrett, K. A., & McWhirter, B. T.
review of the research. The Counsel- & Development, 83, 480–487. (2002). Counselor trainee’s per-
ing Psychologist, 11, 79–92. Barak, A., & LaCrosse, M. B. ceptions of clients based on cli-
Atkinson, D. R., & Lowe, S. M. (1975). Multidimensional percep- ent sexual orientation. Counselor
(1995). The role of ethnicity, cul- tion of counselor behavior. Journal Education and Supervision, 41,
tural knowledge, and conventional of Counseling Psychology, 22, 219–232.
techniques in counseling and psy- 471–476. Barrett-Lennard, G. T. (1962). Di-
chotherapy. In J. G. Ponterotto, Barber, J. P., & Crits-Christoph, P. mensions of therapist response as
J. M. Casas, L. A. Suzuki, & C. M. (1996). Development of a thera- causal factors in therapeutic
Alexander (Eds.), Handbook of mul- pist adherence competence rating change. Psychological Monographs:
ticultural counseling (pp. 387–414). scale for supportive-expressive dy- General and Applied, 76, Whole
Thousand Oaks, CA: Sage. namic psychotherapy: A prelimi- No. 562.
references 623
Baskin, T. W., Tierney, S. C., Minami, Bem, D. J. (1995). Writing a review the scientist-practitioner model:
T., & Wampold, B. E. (2003). article for Psychological Bulle- Implementation and implications.
Establishing specificity in psy- tin. Psychological Bulletin, 118, The Counseling Psychologist, 21,
chotherapy: A meta-analysis of 172–177. 136–151.
structural equivalence of placebo Benjamin, L. S. (1974). Structural Betancourt, H., & Lopez, S. R.
controls. Journal of Consulting analysis of social behavior. Psy- (1993). The study of culture, eth-
and Clinical Psychology, 71, chological Review, 81, 392–425. nicity, and race in American psy-
973–979. Benn, S. I. (1967). Justice. In P. Ed- chology. American Psychologist,
Baumrind, D. (1976). Nature and wards (Ed.), The encyclopedia of 48, 629–637.
definition of informed consent philosophy (Vol. 4, pp. 298–302). Betz, N., & Fitzgerald, L. (1987). The
in research involving deception. New York: Macmillan. career psychology of women. New
Background paper prepared for Bennum, I., Hahlweg, K., Schindler, York: Academic Press.
the National Commission for the L., & Langlotz, M. (1986). Ther- Betz, N. E. (1986). Research training
Protection of Human Subjects of apist’s and client’s perceptions in in counseling psychology: Have
Biomedical and Behavioral Re- behavior therapy: The develop- we addressed the real issues?
search. Washington, DC: Depart- ment and cross-cultural analysis of The Counseling Psychologist, 14,
ment of Health, Education, and an assessment instrument. British 107–113.
Welfare. Journal of Clinical Psychology, 25, Betz, N. E., & Fitzgerald, L. F. (1993).
Beauchamp, T. L., & Childress, J. F. 275–283. Individuality and diversity: Theory
(2001). Principles of biomedical Berdie, R. F. (1972). The 1980 coun- and research in counseling psy-
ethics (5th ed.). New York: selor: Applied behavioral scientist. chology. Annual Review of Psy-
Oxford University Press. Personnel and Guidance Journal, chology, 44, 343–381.
Beck, A. T., Rush, A. J., Shaw, B. F., & 50, 451–456. Betz, N. E., & Hackett, G. (1981).
Emery, G. (1979). Cognitive ther- Berg, I. A. (1954). Ideomotor res- The relationship of career-related
apy of depression. New York: ponse set: Symbolic sexual gesture self-efficacy expectations to per-
Guilford Press. in the counseling interview. ceived career options in college
Beck, A. T., Ward, C. H., Mendelson, Journal of Counseling Psychology, women and men. Journal of Coun-
M., Mock, J., & Erbaugh, J. 1, 180–183. seling Psychology, 28, 399–410.
(1961). An inventory for measur- Bergin, A. E., & Garfield, S. L. (Eds.). Betz, N. E., & Hackett, G. (1987).
ing depression. Archives of Gen- (1971). Handbook of psychother- The concept of agency in educa-
eral Psychiatry, 4, 561–571. apy and behavior change. New tional and career development.
Beck, K. A. (2005). Ethnographic deci- York: Wiley. Journal of Counseling Psychology,
sion tree modeling: A research Bergin, A. E., & Garfield, S. L. (Eds.). 34, 299–308.
method for counseling psychology. (1994). Handbook of psychother- Betz, N. E., & Taylor, K. M. (1982).
Journal of Counseling Psychology, apy and behavior change. New Concurrent validity of the Strong-
52, 243–249. York: Wiley. Campbell Interest Inventory for
Bednar, R. L., & Kaul, T. J. (1978). Bergin, A. E., & Lambert, M. J. graduate students in counseling.
Experiential group research: (1978). The evaluation of thera- Journal of Counseling Psychology,
Current perspectives. In S. L. peutic outcomes. In S. L. Garfield 29, 626–635.
Garfield & A. E. Bergin (Eds.), & A. E. Bergin (Eds.), Handbook Beutler, L. E., Brown, M. T., Crothers,
Handbook of psychotherapy and of psychotherapy and behavior L., Booker, K., & Seabrook, M. K.
behavior change (2nd ed., pp. change (2nd ed., pp. 139–190). (1996). The dilemma of factitious
769–816). New York: Wiley. New York: Wiley. demographic distinctions in psy-
Behrens, J. T. (1997). Does the white Bergin, A. E., & Strupp, H. H. (1970). chological research. Journal of
racial identity attitude scale New directions in psychotherapy Consulting and Clinical Psychology,
measure racial identity? Journal research. Journal of Abnormal 64, 892–902.
of Counseling Psychology, 44, Psychology, 76, 13–26. Beutler, L. E., Moleiro, C. M., &
3–12. Berkowitz, A. D. (2003). Applications Talebi, H. (2002). Resistance. In
Behrens, J. T., Leach, M. M., Franz, of Social Norms Theory to other J. C. Norcross (Ed.), Psychotherapy
S., & LaFleur, N. K. (1999, health and social justice issues in relationships that work: Therapist
August). Revising the Oklahoma H. W. Perkins (Ed.). The Social contributions and responsiveness
racial attitudes scale: Work in Norms approach to preventing to patients (pp. 129–144). New
progress. Paper presented at the school and college substance York: Oxford University.
annual meeting of the American abuse. San Francisco: Jossey- Bhaskar, R. (1975). A realist theory
Psychological Association, Boston. Bass. of science. Leeds, England: Leeds
Belar, C. D., & Perry, N. W. (1992). Bernard, J. M., & Goodyear, R. K. Books.
National conference on scientist- (1998). Fundamentals of clinical Bishop, J. B. (1992). The changing
practitioner education and train- supervision (2nd ed.). Boston: student culture: Implications for
ing for the professional practice of Allyn & Bacon. counselors and administrators.
psychology. American Psychologist, Bernstein, B. L., & Kerr, B. (1993). Journal of College Student Psy-
47, 71–75. Counseling psychology and chotherapy, 6, 37–57.
624 references
Blashfield, R. K. (1984). The classifi- Bradley, J. V. (1968). Distribution-free Brown, S. D., & Krane, N. E. R.
cation of psychopathology: Neo- statistical tests. Englewood Cliffs, (2000). Four (or five) sessions
Kraepelinian and quantitative NJ: Prentice-Hall. and a cloud of dust: Old assump-
approaches. New York: Plenum. Brennan, K. A., Clark, C. L., & tions and new observations about
Blos, P. (1946). Psychological counsel- Shaver, P. R. (1998). Self-report career counseling. In S. D. Brown
ing of college students. American measurement of adult attachment: & R. W. Lent (Eds.), Handbook
Journal of Orthopsychiatry, 16, An integrative overview. In J. A. of counseling psychology (3rd
571–580. Simpson & W. S. Rholes (Eds.), ed., pp. 740–766). New York:
Bohus, M., Haaf, B., Simms, T., Attachment theory and close rela- Wiley.
Limberger, M. F., Schmahl, C., tionships (pp. 46–76). New York: Brown, W., & Holtzman, W. (1967).
Unckel, C., Lieb, K., & Linehan, Guilford Press. Manual, survey of study habits
M. M. (2004). Effectiveness of Breuer, J., & Freud, S. (1955). Studies and attitudes. New York: The
inpatient dialectical behavioral on hysteria. In J. Strachey (Ed. and Psychological Corporation.
therapy for borderline personality Trans.), The standard edition of Bryk, A. S., & Raudenbush, S. W.
disorder: A controlled trial. Be- the complete psychological works (1992). Hierarchical linear mod-
haviour Research and Therapy, of Sigmund Freud (Vol. 2). London: els: Applications and data analy-
42, 487–499. Hogarth Press. (Original work sis methods. Newbury Park, CA:
Bollen, K. A. (1989). Structural equa- published 1893–1895.) Sage.
tions with latent variables. New Bridgewater, C. A., Bornstein, P. H., Buboltz, W. C., Jr., Miller, M., &
York: John Wiley. & Walkenbach, J. (1981). Ethical Williams, D. J. (1999). Content
Bordin, E. S. (1965). Simplification as issues and the assignment of publi- analysis of research in the Jour-
a research strategy in psychother- cation credit. American Psychologist, nal of Counseling Psychology
apy. Journal of Consulting Psychology, 36, 524–525. (1973–1998). Journal of Counseling
29, 493–503. Brislin, R. W., Lonner, W. J., & Psychology, 46, 496–503.
Bordin, E. S. (1979). The generaliz- Thorndike, R. M. (1973). Cross- Buchler, J. (Ed.). (1955). Philosophical
ability of the psychoanalytic concept cultural research methods. New writings of Peirce. New York:
of working alliance. Psychotherapy: York: Wiley. Dover.
Theory, Research and Practice, Broad, W., & Wade, N. (1982). Burgoon, J. K., Beutler, L. E., Le
16, 252–260. Betrayers of the truth. New York: Poire, B. A., Engle, D., Bergan, J.,
Borenstein, M., & Cohen, J. (1988). Simon & Schuster. Salvio, M., & Mohr, D. C. (1993).
Statistical power analysis: A com- Brock, T. C. (1967). Communication Nonverbal indices of arousal in
puter program. Hillsdale, NJ: discrepancy and intent to persuade group therapy. Psychotherapy, 30,
Erlbaum. as determinants of counter- 635–645.
Borgen, F. H. (1984a). Counseling argument production. Journal of Burgoon, J. K., Kelly, D. L., Newton,
psychology. Annual Review of Experimental Social Psychology, D. A., & Keeley-Dyreson, M. P.
Psychology, 35, 579–604. 3, 269–309. (1989). The nature of arousal
Borgen, F. H. (1984b). Are there nec- Brophy, J. E., & Good, T. L. (1974). and nonverbal indices. Human
essary linkages between research Teacher-student relationships: Causes Communication Research, 16,
practices and the philosophy of and consequences. New York: 217–255.
science? Journal of Counseling Holt, Rinehart & Winston. Burkard, A. W., & Knox, S. (2004).
Psychology, 31, 457–460. Brotemarkle, R. A. (1927). College Effect of therapist color-blindness
Borgen, F. H. (1992). Expanding student personnel problems. Journal on empathy and attributions in
scientific paradigms in counsel- of Applied Psychology, 11, 415–436. cross-cultural counseling. Journal
ing psychology. In S. D. Brown Brown, F. (1982). The ethics of psy- of Counseling Psychology, 51,
& R. W. Lent (Eds.), Handbook chodiagnostic assessment. In M. 387–397.
of counseling psychology (2nd Rosenbaum (Ed.), Ethics and values Burns, B. D. (1981). Feeling good:
ed., pp. 111–139). New York: in psychotherapy. New York: The new mood therapy. New
Wiley. Free Press. York: The New American Library.
Borgen, F. H., & Barnett, D. C. Brown, S. D., & Lent, R. W. (Eds.). Butler, S. F., Henry, W. P., & Strupp,
(1987). Applying cluster analysis (2000). Handbook of counseling H. H. (1992). Measuring adher-
in counseling psychology research. psychology (3rd ed). New York: ence and skill in time-limited
Journal of Counseling Psychology, Wiley. dynamic psychotherapy. Unpublished
34, 456–468. Brown, S. D., Lent, R. W., Ryan, manuscript, Vanderbilt University.
Borgen, F. H., & Weiss, D. J. (1971). N. E., & McPartland, E. B. (1996). Byrne, B. M., & Watkins, D. (2003).
Cluster analysis and counseling Self-efficacy as an intervening The issue of measurement invari-
research. Journal of Counseling mechanism between research ance revisited. Journal of Cross-
Psychology, 18, 583–591. training environments and schol- Cultural Psychology, 34, 155–175.
Bracht, G. H., & Glass, V. V. (1968). arly productivity: A theoretical Campbell, A., Converse, P. E., Miller,
The external validity of experiments. and methodological extension. W. E., & Stokes, D. E. (1990).
American Educational Research The Counseling Psychologist, 24, The American voter. New York:
Journal, 5, 437–474. 535–544. Wiley.
references 625
Campbell, D. T., & Stanley, J. C. methods. In N. K. Denzin & Y. S. Journal of Nursing Education,
(1963). Experimental and quasi- Lincoln (Eds.), Handbook of 26(4), 138–143.
experimental designs for research. qualitative research (2nd ed., Cohen, J. (1968). Multiple regression
Chicago: Rand McNally. pp. 509–536). Thousand Oaks, as a general data-analytic strategy.
Campbell, L. M., III. (1973). A varia- CA: Sage. Psychological Bulletin, 70, 426–443.
tion of thought stopping in a Chartrand, J., Martin, W., Robbins, Cohen, J. (1988). Statistical power
twelve-year-old boy: A case report. S., McAuliffe, G., Pickering, J., & analysis for the behavioral sciences
Journal of Behavior Therapy and Calliotte, J. (1994). Testing a level (2nd ed.). Hillsdale, NJ: Erlbaum.
Experimental Psychiatry, 4, 69–70. versus an interactional view of Cohen, J., & Cohen, P. (1983).
Campbell, M. K., James, A., Hudson, career indecision. Journal of Ca- Applied multiple regression/corre-
M. A., Carr, C., Jackson, E., reer Assessment, 2, 55–69. lation analysis for the behavioral
Oates, V., Demissie, S., Farrell, D., Chen, C., Lee, S-Y., & Stevenson, sciences (2nd ed.). Hillsdale, NJ:
& Tessaro, I. (2004). Improving H. W. (1995). Response style and Erlbaum.
multiple behaviors for colorectal cross-cultural comparisons of Cohen, M. R., & Nagel, E. (1934).
cancer prevention among African rating scales among East Asian An introduction to logic and scien-
American church members. Health and North American students. tific method. New York: Harcourt,
Psychology, 23, 492–502. Psychological Science, 6, 170–175. Brace & Company.
Caple, R. B. (1985). Counseling and Cheng, S. K., Chong, G. H., & Wong, Cohen, R. J., Swerdlik, M. E., &
the self-organization paradigm. C. W. (1999). Chinese Frost Phillips, S. M. (1996). Psychological
Journal of Counseling and Devel- Multidimensional Perfectionism testing and assessment: An intro-
opment, 64, 173–178. Scale: A validation and prediction duction to tests and measurement.
Carkhuff, R. R., & Burstein, J. W. of self-esteem and psychological Mountain View, CA: Mayfield.
(1970). Objective therapist and distress. Journal of Clinical Psy- Cokley, K. (2003a). What do we
client ratings of therapist-offered chology, 55, 1051–1061. know about the motivation of Af-
facilitative conditions of moder- Chia, R. C., Allred, L. J., & Jerzak, rican American students? Chal-
ate to low functioning therapist. P. A. (1997). Attitudes toward lenging the “anti-intellectual”
Journal of Clinical Psychology, 26, women in Taiwan and China: myth. Harvard Educational Review,
394–395. Current status, problems, and sug- 73, 524–558.
Carney, C. G., & Barak, A. (1976). A gestions for future research. Psy- Cokley, K. (2003b). What do we
survey of student needs and stu- chology of Women Quarterly, 21, really know about the academic
dent personnel services. Journal of 137–150. motivation of African American
College Student Personnel, 17, Chickering, A. W. (1969). Education college students? Challenging the
280–284. and identity. San Francisco: “anti-intellectual” myth. Presented
Carter, R. (1995). The influence of Jossey-Bass. at the 16th Annual National
race and racial identity in psy- Christensen, L. B. (1980). Experimental Conference on Race and Ethnicity,
chotherapy: Toward a racially methodology (2nd ed.). Boston: San Francisco, CA.
inclusive model. New York: Wiley. Allyn & Bacon. Cole, D. A. (1987). The utility of con-
Casas, J. M. (2005). Race and racism: Chwalisz, K. (2003). Evidence-based firmatory factor analysis in test
The efforts of counseling psychol- practice: A framework for the validation research. Journal of
ogy to understand and address the twenty-first century scientist prac- Consulting and Clinical Psychology,
issues associated with these terms. titioner training. The Counseling 55, 584–594.
The Counseling Psychologist, 33, Psychologist, 31, 497–528. Cole, D. A., Lazarick, D. L., &
501–512. Claiborn, C. D. (1982). Interpretation Howard, G. S. (1987). Construct
Casas, J. M., Ponterotto, J. G., & and change in counseling. Journal validity and the relation between
Gutierrez, J. M. (1986). An ethical of Counseling Psychology, 29, depression and social skill. Journal
indictment of counseling research 439–453. of Counseling Psychology, 34,
and training: The cross-cultural Claiborn, C. D. (1985). Harold B. 315–321.
perspective. Journal of Counseling Pepinsky: A life of science and Coleman, H. L. K., Wampold, B. E.,
and Development, 64, 347–349. practice. Journal of Counseling & Casali, S. L. (1995). Ethnic
Chang, P. (1994). Effects of inter- and Development, 64, 5–13. minorities’ ratings of ethnically
viewer questions and response Claiborn, C. D. (1987). Science and similar and European American
type on compliance: An analogue practice: Reconsidering the Pep- counselors: A meta-analysis.
study. Journal of Counseling Psy- inskys. Journal of Counseling and Journal of Counseling Psychology,
chology, 41, 74–82. Development, 65, 286–288. 42, 55–64.
Chapman, L. J., & Chapman, J. P. Claiborn, C. D., & Lichtenberg, J. W. Condon, K. M., & Lambert, M. J.
(1969). Illusory correlations as an (1989). Interactional counseling. (1994, June). Assessing clinical sig-
obstacle to the use of valid psy- The Counseling Psychologist, 17, nificance: Application to the State-
chodiagnostic tests. Journal of 355–453. Trait Anxiety Inventory. Paper
Abnormal Psychology, 74, 271–280. Cobb, A. K., & Hagemaster, J. N. presented at the annual meeting
Charmaz, K. (2000). Grounded the- (1987). Ten criteria for evaluating of the Society for Psy-chotherapy
ory: Objectivist and constructivist qualitative research proposals. Research, York, En-gland.
626 references
Constantine, M. G., Quintana, S. M., Creswell, J. W. (2003). Research design: Cummings, A. L., Martin, J., Halberg,
Leung, S. A., & Phelps, R. E. Quantitative, qualitative, and E., & Slemon, A. (1992). Memory
(1995). Survey of the professional mixed methods approaches (2nd for therapeutic events, session
needs of division 17’s ethnic and Ed.). Thousand Oaks, CA: Sage. effectiveness, and working alliance
racial minority psychologists. Crews, J., Smith, M. R., Smaby, M. H., in short-term counseling. Journal
The Counseling Psychologist, 23, Maddux, C. D., Torres-Rivera, E., of Counseling Psychology, 39,
546–561. Casey, J. A., & Urbani, S. (2005). 306–312.
Cook, E. P. (1990). Gender and psy- Self-monitoring and counseling Daniels, L. K. (1976). An extension of
chological distress. Journal of skills-based versus Interpersonal thought stopping in the treatment
Counseling and Development, 68, Process Recall Training. Journal of of obsessional thinking. Behavior
371–375. Counseling and Development, 83, Therapy, 7, 131.
Cook, T. D., & Campbell, D. T. 78–85. Danskin, D. G., & Robinson, F. P.
(1979). Quasi-experimentation: Crites, J. O. (1978). Career maturity (1954). Differences in “degree of
Design and analysis issues for field inventory. Monterey, CA: McGraw- lead” among experienced coun-
settings. Boston: Houghton Mifflin. Hill. selors. Journal of Counseling Psy-
Cooke, R. A. (1982). The ethics and Crits-Christoph, P., Baranackie, K., chology, 1, 78–83.
regulation of research involving Kurcias, J. S., Beck, A. T., Carroll, Dar, R. (1987). Another look at
children. In B. B. Wolman (Ed.), K., Perry, K., Luborsky, L., Meehl, Lakatos, and the scientific
Handbook of developmental psy- McLellan, A. T., Woody, G. E., practices of psychologists. American
chology. Englewood Cliffs, NJ: Thompson, L., Gallagher, D., & Psychologist, 42, 145–151.
Prentice-Hall. Zitrin, C. (1991). Meta-analysis of Daus, J. A., III. (1995). Changes in
Coopersmith, S. (1981). SEI; self- therapist effects in psychotherapy counseling efficacy across a semes-
esteem inventories. Palo Alto, CA: outcome studies. Psychotherapy ter of group supervision: A time
Consulting Psychologists Press. Research, 1, 81–91. series analysis. Unpublished doc-
Corazzini, J. (1980). The theory and Crits-Christoph, P., Siqueland, L., toral dissertation, University of
practice of loss therapy. In B. Mark Chittams, J., Barber, J. P., Beck, Missouri, Columbia.
Schoenberg (Ed.), Bereavement A. T., Frank, A., Liese, B., Luborsky, Dawis, R. V. (1987). Scale construc-
counseling: A multi-disciplinary L., Mark, D. M. D., Onken, L. S., tion. Journal of Counseling Psy-
handbook (pp. 71–85). Westport, Najavits, L. M., Thase, M. E., & chology, 34, 481–489.
CT: Greenwood Press. Woody, G. (1988). Training in Dawis, R. V. (2000). Scale construc-
Corning, A. F. (2002). Self-esteem as a cognitive, supportive-expressive, tion and psychometric considera-
moderator between perceived dis- and drug counseling therapies for tions. In H. E. A. Tinsley & S. D.
crimination and psychological cocaine dependence. Journal of Brown (Eds.), Handbook of
distress among women. Journal Consulting and Clinical Psychology, applied multivariate statistics and
of Counseling Psychology, 49, 66, 484–492. mathematical modeling (pp.
117–126. Crosbie, J. (1993). Interrupted time 65–94). San Diego, CA: Academic
Corrigan, J. D., Dell, D. M., Lewis, series analysis with brief single Press.
K. N., & Schmidt, L. D. (1980). subject data. Journal of Consulting Day, S., & Schneider, P. L. (2002).
Counseling as a social influence and Clinical Psychology, 61, 966–974. Psychotherapy using distance tech-
process: A review [monograph]. Cross, W. E. (1971). The Negro to nology: A comparison of face to
Journal of Counseling Psychology, black conversion experience: Toward face, video, and audio treatment.
27, 395–441. a psychology of black liberation. Journal of Counseling Psychology,
Corrigan, J. D., & Schmidt, L. D. Black World, 20(9), 13–27. 49, 499–503.
(1983). Development and valida- Cross, W. E. (1978). The Cross and Deffenbacher, J. L., & Stark, R. S.
tion of revisions in the counselor Thomas models of psychological (1992). Relaxation and cognitive-
rating form. Journal of Counseling nigresence. Journal of Black Psy- relaxation treatment of general
Psychology, 30, 64–75. chology, 5(1), 13–19. anger. Journal of Counseling Psy-
Cournoyer, R. J., & Mahalik, J. R. Cross, W. E., & Vandiver, B. J. (2001). chology, 39, 158–167.
(1995). Cross-sectional study of Nigrescence theory and measure- Deffenbacher, J. L., Thwaites, G. A.,
gender role conflict examining ment: Introducing the cross racial Wallace, T. L., & Oetting, E. R.
college-aged and middle-aged men. identity scale (CRIS). In J. G. (1994). Social skills and cognitive-
Journal of Counseling Psychology, Ponterotto, J. M. Casas, L. A. relaxation approaches to general
42, 11–19. Suzuki, & C. A. Alexander (Eds.), anger reduction. Journal of Coun-
Creswell, J. W. (1994). Research Handbook of multicultural coun- seling Psychology, 41, 386–396.
design: Qualitative and quantita- seling (2nd ed., pp. 425–456). Delgado-Romero, E. A., Galván, N.,
tive approaches. Thousand Oaks, Thousand Oaks, CA: Sage. & Maschino, P. (2005). Race and
CA: Sage. Cummings, A. L. (1989). Relationship ethnicity in empirical counseling
Creswell, J. W. (1998). Qualitative of client problem type to novice and counseling psychology re-
inquiry and research design: counselor response modes. Journal search: A 10-year review. The
Choosing among five traditions. of Counseling Psychology, 36, Counseling Psychologist, 33,
Thousand Oaks, CA: Sage. 331–335. 419–448.
references 627
Dennin, M. K., & Ellis, M. V. (2003). Diener, E., & Crandall, R. (1978). Mechanisms of change in couple
Effects of a method of self-supervision Ethics in social and behavioral therapy. Journal of Consulting and
for counselor trainees. Journal of research. Chicago: University of Clinical Psychology, 73, 624–633.
Counseling Psychology, 50, 69–83. Chicago Press. Douglas, J. D. (1985). Creative inter-
Denzin, N. K. (1978). The research Dillman, D. A. (2000). Mail and viewing. Thousand Oaks, CA: Sage.
act: A theoretical introduction to Internet surveys: The tailored Dovidio, J. F., Gaertner, S. L.,
sociological methods. New York: design method (2nd ed.). New Kawakami, K., & Hodson, G.
McGraw-Hill. York: John Wiley & Sons. (2002). Why can’t we just get
Denzin, N. K. (1997). Coffee with Dillon, F. R., & Worthington, R. L. along? Interpersonal biases and
Anselm. Qualitative family research, (2003). The Lesbian, Gay, and interracial distrust. Cultural Di-
11, 16–18. Bisexual Affirmative Counseling versity and Ethnic Minority Psy-
Denzin, N. K., & Lincoln, Y. S. (Eds.). Self-Efficacy Inventory (LGB-CSI): chology, 8, 88–102.
(1998). The landscape of qualita- Development, validation, and train- Dowd, E. T., & Boroto, D. R. (1982).
tive research: Theories and issues. ing implication. Journal of Counseling Differential effects of counselor
Thousand Oaks, CA: Sage. Psychology, 50, 235–251. self-disclosure, self-involving state-
Denzin, N. K., & Lincoln, Y. S. (Eds.). Dill-Standifond, T. J., Stiles, W. B., ments, and interpretation. Journal
(2000). Handbook of qualitative & Rorer, L. G. (1988). Counselor- of Counseling Psychology, 29,
research (2nd ed.). Thousand client agreement on session impact. 8–13.
Oaks, CA: Sage. Journal of Counseling Psychology, Drane, J. F. (1982). Ethics and psy-
Department of Health and Human 35, 47–55. chotherapy: A philosophical per-
Services. (1989). Responsibilities Dipboye, W. J. (1954). Analysis of spective. In M. Rosenbaum (Ed.),
of awardee and applicant insti- counselor style by discussion units. Ethics and values in psychother-
tutions for dealing with and re- Journal of Counseling Psychology, apy: A guidebook (pp. 15–50).
porting possible misconduct in 1, 21–26. New York: Free Press.
science. Federal Register, 54(151), Dixon, D. N., & Claiborn, C. D. Drew, C. F. (1980). Introduction to
32446–32451. (1981). Effects of need and com- designing and conducting research
DeProspero, A., & Cohen, S. (1979). mitment on career exploration (2nd ed.). St. Louis: C. V. Mosby.
Inconsistent visual analysis of intra- behaviors. Journal of Counseling Edgington, E. (1982). Nonparametric
subject data. Journal of Applied Psychology, 28, 411–415. tests for single-subject multiple
Behavior Analysis, 12, 573–579. Dixon, D. N., & Glover, J. A. (1984). schedule experiments. Behavioral
Derogatis, L. R. (1983). SCL-90-R Counseling: A problem-solving Assessment, 4, 83–91.
administration, scoring and pro- approach. New York: Wiley. Edgington, E. S. (1980). Randomization
cedures manual. Towson, MD: Dixon, D. N., Heppner, P. P., Petersen, tests. New York: Marcel Dekker.
Clinical Psychiatric Research. C. H., & Ronning, R. R. (1979). Edgington, E. S. (1987). Randomized
Derogatis, L. R. (1992). The Brief Problem-solving workshop training. single-subject experiments and sta-
Symptom Inventory (BSI): Admin- Journal of Counseling Psychology, tistical tests. Journal of Counseling
istration, scoring, and procedures 26, 133–139. Psychology, 34, 437–442.
manual II. Baltimore, MD: Clinical Dixon, W. A. (1989). Self-appraised Elder, N. C., & Miller, W. L. (1995).
Psychometric Research. problem solving ability, stress, Reading and evaluating qualitative
DeRubeis, R. J., Hollon, S. E., Evans, and suicide ideation in a college research studies. The Journal of
M. D., & Bemis, K. M. (1982). population. Unpublished masters Family Practice, 41(3), 279–285.
Can psychotherapies for depres- thesis, University of Missouri, Elkin, I., Falconnier, L. Martinovich,
sion be discriminated? A system- Columbia. Z., & Mahoney, C. (2006).
atic investigation of cognitive Dolliver, R. H. (1969). Strong voca- Therapist effects in the National
therapy and interpersonal therapy. tional blank versus expressed Institute of Mental Health Treat-
Journal of Consulting and Clinical vocational interests: A review. ment of Depression Collaborative
Psychology, 50, 744–756. Psychological Bulletin, 72, 95–107. Research Program. Psychotherapy
DeSena, P. A. (1966). Problems of Dorn, F. J. (1985). Publishing for pro- Research, 16, 144–160.
consistent over-, under-, and fessional development. Muncie, Elkin, I., Shea, M. T., & Watkins, J. T.
normal-achieving college students IN: Accelerated Development. (1989). National Institute of
as identified by the Mooney Dorn, F. J. (Ed.). (1986). Social influ- Mental Health Treatment of De-
Problem Checklist. Journal of Ed- ence processes in counseling and pression Collaborative Research
ucational Research, 59, 351–355. psychotherapy. Springfield, IL: Program: General effectiveness of
DeVellis, R. F. (2003). Applied social Charles C Thomas. treatments. Archives of General
research methods (Vol. 26). Thousand Dorn, F. J. (1988). Utilizing social Psychiatry, 46, 971–982.
Oaks, CA: Sage. influence in career counseling: A Elliot, R., & Wexler, M. M. (1994).
Dhami, M. K., Hertwig, R., & case study. Career Development Measuring the impact of sessions
Hoffrage, U. (2004). The role of Quarterly, 36, 269–280. in process-experiential therapy of
representative design in an ecolog- Doss, B. D., Thum, Y. M., Sevier, M., depression: The session impacts
ical approach to cognition. Psy- Atkins, D. C., & Christensen, A. scale. Journal of Counseling Psy-
chological Bulletin, 130, 959–988. (2005). Improving relationships: chology, 41, 166–174.
628 references
Elliott, E. (1989). Comprehensive process cognitive-behavioral and psychody- Eysenck, H. J. (1965). The effects of
analysis: Understanding the change namic-interpersonal psychotherapies. psychotherapy. International Journal
process in significant therapy Journal of Counseling Psychology, of Psychology, 1, 97–178.
events. In M. J. Packer & R. B. 41, 449–463. Eysenck, H. J. (1969). The effects of
Addison (Eds.), Entering the circle: Ellis, A. (1962). Reason and emotion psychotherapy. New York: Science
Hermeneutic investigation in psy- in psychotherapy. New York: Lyle House.
chology (pp. 165–184). Albany, Stuart. Fabrigar, L. R., Wegener, D. T.,
New York: State University of Ellis, J. V., Robbins, E. S., Shult, D., MacCallum. R. C., & Strahan,
New York Press. Ladany, N., & Banker, J. (1990). E. J. (1999). Evaluating the use of
Elliott, E., Fischer, C. T., & Rennie, Anchoring errors in clinical judg- exploratory factor analysis in psy-
D. L. (1999). Evolving guidelines for ments: Type I error, adjustment, or chological research. Psychological
publication of qualitative research mitigation. Journal of Counseling Methods, 4, 272–299.
studies in psychology and related Psychology, 37, 343–351. Fagley, N. S. (1985). Applied statisti-
fields. British Journal of Clinical Ellis, M. V., Ladany, N., Krengel, M., cal power analysis and the inter-
Psychology, 38, 215–229. & Schult, D. (1996). Clinical pretation of non-significant results
Elliott, R. (1979). How clients per- supervision research from 1981 to by research consumers. Journal
ceive helper behaviors. Journal of 1993: A methodological critique. of Counseling Psychology, 32, 391–
Counseling Psychology, 26, 285–294. Journal of Counseling Psychology, 396.
Elliott, R. (1985). Helpful and non- 43, 35–50. Farmer, H. (1985). Model of career
helpful events in brief counseling Enns, C. Z., McNeilly, C. L., Corkery, and achievement motivation for
interviews: An empirical taxon- J. M., & Gilbert, M. S. (1995). women and men. Journal of Coun-
omy. Journal of Counseling Psy- The debate about delayed memo- seling Psychology, 32, 363–390.
chology, 32, 307–322. ries of child sexual abuse: A femi- Farmer, H. S., Wardrop, J. L.,
Elliott, R. (1988, June). Issues in the nist perspective. The Counseling Anderson, M. Z., & Risinger, R.
selection, training, and manage- Psychologist, 23, 181–279. (1995). Women’s career choices:
ment of raters. In R. Moras & C. E. Erdur, O., Rude, S. S., & Baron, A. Focus on science, math, and tech-
Hill (Co-chairs), Selecting raters for (2003). Symptom improvement nology careers. Journal of Coun-
psychotherapy process research. and length of treatment in ethi- seling Psychology, 42, 155–170.
Workshop conducted at the So- cally similar and dissimilar client- Farrell, A. D. (1999). Evaluation of
ciety for Psychotherapy Research, therapist pairings. Journal of the Computerized Assessment
Santa Fe, NM. Counseling Psychology, 50, 52–58. System for Psychotherapy Eva-
Elliott, R. (1991). Five dimensions of Erickson, C. D., & Al-Timimi, N. R. luation and Research (CASPER) as
therapy process. Psychotherapy (2001). Providing mental health a measure of treatment effective-
Research, 1, 92–103. services to Arab Americans: Recom- ness in an outpatient training
Elliott, R., Hill, C. E., Stiles, W. B., mendations and considerations. clinic. Psychological Assessment,
Friedlander, M. L., Mahrer, A. R., Cultural Diversity and Ethnic 11, 345–358.
& Margison, F. R. (1987). Primary Minority Psychology, 7, 308–327. Fassinger, R. (1990). Causal models
therapist response modes: Comparison Ericsson, K. A., & Simon, H. A. (1984). of career choice in two samples
of six rating systems. Journal of Protocol analysis: Verbal reports of college women. Journal of
Consulting and Clinical Psychology, as data. Cambridge, MA: MIT Press. Vocational Behavior, 36, 225–248.
55, 218–223. Eugster, S. L., & Wampold, B. E. Fassinger, R. E. (1987). Use of struc-
Elliott, R., & James, E. (1989). (1996). Systematic effects of par- tural equation modeling in cousel-
Varieties of client experience in ticipant role on evaluation of the ing psychology research. Jour- nal
psychotherapy: An analysis of the psychotherapy session. Journal of of Counseling Psychology, 34,
literature. Clinical Psychology Review, Consulting and Clinical Psychology, 425–436.
9, 443–467. 64, 1020–1028. Fassinger, R. E. (2005). Paradigms,
Elliott, R., James, E., Reimschuessel, Exner, J. E., Jr. (1974). The Rorschach: praxis, problems, and promise:
C., Cislo, D., & Sack, N. (1985). A comprehensive system (Vol. 1). Grounded theory in counseling
Significant events and the analysis New York: Wiley. psychology research. Journal of
of immediate therapeutic impacts. Eysenck, H. J. (1952). The effects of Counseling Psychology, 52(2),
Psychotherapy, 22, 620–630. psychotherapy: An evaluation. 156–166.
Elliott, R., & Shapiro, D. A. (1988). Brief Journal of Consulting Psychology, Fassinger, R. E., & Richie, B. S.
structured recall: A more efficient 16, 319–324. (1997). Sex matters: Gender and
method for studying significant Eysenck, H. J. (1960). Behavior ther- sexual orientation in training for
therapy events. British Journal of apy and the neuroses. Oxford: multicultural counseling compe-
Medical Psychology, 61, 141–153. Pergamon Press. tency. In D. B. Pope-Davis &
Elliott, R., Shapiro, D. A., Firth- Eysenck, H. J. (1961). The effects of H. L. K. Coleman (Eds.), Multi-
Cozens, J., Stiles, W. B., Hardy, psychotherapy. In H. J. Eysenck cultural counseling competencies:
G. E., Llewelyn, S. P., & Margison, (Ed.), Handbook of abnormal psy- Assessment, education and train-
F. R. (1994). Comprehensive pro- chology (pp. 697–725). New York: ing, and supervision (pp. 83–110).
cess analysis of insight events in Basic Books. Thousand Oaks, CA: Sage.
references 629
Feldman, D. A., Strong, S. R., & Fong, M. L. (1992). When a survey individual growth. Journal of
Danser, D. B. (1982). A compari- isn’t research. Counselor Education Consulting and Clinical Psychology,
son of paradoxical and nonpara- and Supervision, 31, 194–195. 59, 27–37.
doxical interpretations and directives. Fong, M. L., & Malone, C. M. (1994). Frank, G. (1984). The Boulder model:
Journal of Counseling Psychology, Defeating ourselves: Common errors History, rationale, and critique.
29, 572–579. in counseling research. Counselor Professional Psychology: Research
Fine, M. (1992). Disruptive voices: Education and Supervision, 33, and Practice, 15, 417–435.
The possibilities of feminist re- 356–362. Frank, J. D. (1961). Persuasion and
search. Ann Arbor, MI: University Fontana, A., & Frey, J. H. (1994). healing: A comparative study of
of Michigan Press. Interviewing: The art of science. In psychotherapy. Baltimore: Johns
Fisher, A. R., & Moradi, B. (2001). N. K. Denzin & Y. S. Lincoln Hopkins University Press.
Racial and ethnic identity: Recent (Eds.), Handbook of qualitative Frank, J. D. (1974). Therapeutic com-
development and needed direc- research (pp. 361–376). Thousand ponents of psychotherapy. Journal
tions. In J. Ponterotto, J. Casas, Oaks, CA: Sage. of Nervous and Mental Disease,
L. Suzuki, & C. Alexander (Eds.), Fontana, A., & Frey, J. H. (2000). The 159, 325–342.
Handbook of multicultural coun- interviewing: From structured Frankena, W. K. (1963). Ethics.
seling (pp. 341–370). Thousand questions to negotiated text. In Englewood Cliffs, NJ: Prentice-
Oaks, CA: Sage. N. K. Denzin & Y. S. Lincoln (Eds.), Hall.
Fiske, S. T., & Taylor, S. E. (1984). Handbook of qualitative research Frankenberg, R. (1993). White women,
Social cognition. Reading, MA: (2nd ed., pp. 645–672). Thousand race matters: The social construc-
Addison-Wesley. Oaks, CA: Sage. tion of whiteness. Minneapolis:
Fitzpatrick, J., Sanders, J., & Ford, D. H. (1984). Reexamining University of Minnesota Press.
Worthen, B. (2004). Program eval- guiding assumptions: Theoretical Frazier, P. A., Tix, A. P., & Barron,
uation: Alternative approaches and methodological implications. K. E. (2004). Testing moderator
and practical guidelines. Boston: Journal of Counseling Psychology, and mediator effects in counsel-
Pearson/Allyn & Bacon. 31, 461–466. ing psychology research. Journal
Flaherty, J. A., Gaviria, F. M., Pathak, Forgy, E. W., & Black, J. D. (1954). A of Counseling Psychology, 51,
D., Mitchell, T., Wintrob, R., & follow-up after three years of 115–134.
Richman, J. A. (1988). Developing clients counseled by two methods. Fremont, S., & Anderson, W. P.
instruments for cross-cultural Journal of Counseling Psychology, (1986). What client behaviors make
psychiatric research. Journal of 1, 1–8. counselors angry? An exploratory
Nervous and Mental Disorders, Forum Qualitative Sozialforschung study. Journal of Counseling and
176, 257–263. (n.d.). Retrieved September 20, Development, 65, 67–70.
Flaskerud, J. H., & Nyamathi, A. M. 2006, from http://www.qualitative- Fretz, B. R. (1982). Perspective and
(2000). Collaborative inquiry with research.net/fqs/fqs-eng.htm definitions. The Counseling Psy-
low-income Latina women. Journal Fouad, N. A. (2002). Cross-cultural chologist, 10, 15–19.
of Health Care for the Poor and differences in vocational interests: Friedlander, M. L., Ellis, M. V.,
Underserved, 11, 326–342. Between-group differences on the Siegel, S. M., Raymond, L., Haase,
Foa, E. B., Ehlers, A., Clark, D. M., Strong Interest Inventory. Journal R. F., & Highlen, P. S. (1988).
Tolin, D. F., & Orsillo, S. M. of Counseling Psychology, 49, Generalizing from segments to ses-
(1999). The Posttraumatic Cognitions 283–289. sions: Should it be done? Journal
Inventory (PTCI): Development Fouad, N. A., Cudeck, R., & Hansen, of Counseling Psychology, 35,
and validation. Psychological As- J. (1984). Convergent validity of 243–250.
sessment, 11, 303–314. Spanish and English forms of the Friedlander, M. L., Heatherington, L.,
Foa, E. B., & Rauch, S. A. (2004). Strong-Campbell Interest Inventory Johnson, B., & Showron, E. A.
Cognitive changes during pro- for bilingual Hispanic high school (1994). Sustaining engagement: A
longed exposure versus prolonged students. Journal of Counseling change event in family therapy.
exposure plus cognitive restructur- Psychology, 31, 339–348. Journal of Counseling Psychology,
ing in female assault survivors Fouad, N. A., & Mohler, C. J. (2004). 41, 438–448.
with posttraumatic stress disorder. Cultural validity of Holland’s Friedlander, M. L., & Schwartz, G. S.
Journal of Consulting and Clinical theory and the Strong Interest (1985). Toward a theory of self-
Psychology, 72, 879–884. Inventory for five racial/ethnic presentation in counseling and
Folger, R. (1989). Significance tests groups. Journal of Career Assessment, psychotherapy. Journal of Counseling
and the duplicity of binary deci- 12, 423–439. Psychology, 32, 483–501.
sions. Psychological Bulletin, 106, Fowler, F. (2002). Survey research Friedlander, M. L., Siegel, S. M., &
155–160. methods (3rd ed.). Thousand Brenock, K. (1989). Parallel pro-
Folkman, S., & Lazarus, R. S. (1980). Oaks, CA: Sage. cesses in counseling and supervi-
An analysis of coping in a middle- Francis, D. J., Fletcher, J. M., sion: A case study. Journal of
aged community sample. Journal Stuebing, K. K., Davidson, K. C., Counseling Psychology, 36, 149–157.
of Health and Social Behavior, 21, & Thompson, N. M. (1991). Friedlander, M. L., Thibodeau, J. R.,
219–239. Analysis of change: Modeling Nichols, M. P., Tucker, C., &
630 references
Snyder, J. (1985). Introducing high school population. Jour-nal Annual Review of Psychology, 41,
semantic cohesion analysis: A of Counseling Psychology, 35, 355–386.
study of group talk. Small Group 183–186. Gelso, C. J., Mallinckrodt, B., &
Behavior, 16, 285–302. Gambrill, E. (1990). Critical thinking Judge, A. B. (1996). Research
Friedlander, M. L., Thibodeau, in clinical practice. San Francisco: training environment, attitudes
J. R., & Ward, L. G. (1985). Jossey-Bass. toward research, and research
Discriminating the “good” from Gambrill, E. (2005). Critical thinking self-efficacy: The revised research
the “bad” therapy hour: A study in clinical practice; Improving the training environment scale. The
of dyadic interaction. Psychotherapy, quality of judgments and decisions Counseling Psychologist, 24,
22, 631–642. (2nd Ed.). Hoboken, NJ: Wiley. 304–322.
Friedman, N. (1967). The social Garfield, S. L. (1993). Major issues in Gelso, C. J., Raphael, R., Black,
nature of psychological research. psychotherapy research. In D. K. S. M., Rardin, D., & Skalkos, O.
New York: Basic Books. Freedheim (Ed.), History of (1983). Research training in coun-
Friesen, W. V., & Ekman, P. (1984). psychotherapy (pp. 335–360). seling psychology: Some prelimi-
EMFACS-7: Emotional facial action Washington, DC: American Psy- nary data. Journal of Counseling
coding system. Unpublished manual. chological Association. Psychology, 30, 611–614.
Frontman, K. C., & Kunkel, M. A. Garfield, S. L., & Bergin, A. E. (Eds.). Gibbons, J. L., Hamby, B. A., & Dennis,
(1994). A grounded theory of (1978). Handbook of psychother- W. D. (1997). Researching gender-
counselors’ construal of success in apy and behavior change (2nd role ideologies internationally and
the initial session. Journal of ed.). New York: Wiley. cross-culturally. Psychology of Women
Counseling Psychology, 41, 492–499. Garfield, S. L., & Bergin, A. E. (Eds.). Quarterly, 21, 151–170.
Frost, R. O., Marten, P. A., Lahart, (1986). Handbook of psychother- Giorgi, A. (1970). Psychology as a
C., & Rosenblate, R. (1990). The apy and behavior change (3rd ed.). human science: A phenomenologi-
dimensions of perfectionism. Cog- New York: Wiley. cal based approach. New York:
nitive Therapy & Research, 14, Gelatt, H. B. (1989). Positive uncer- Harper & Row.
449–468. tainty: A new decision-making Giorgi, A. (1985). Sketch of a psycho-
Fuhriman, A., & Burlingame, G. M. framework for counseling. Jour- logical phenomenological method.
(1990). Consistency of matter: A nal of Counseling Psychology, 36, In A. Giorgi (Ed.), Phenomenology
comparative analysis of individual 252–256. and psychological research (pp. 8–22).
and group process variables. The Gelso, C. J. (1979). Research in coun- Pittsburgh, PA: Duquesne University
Counseling Psychologist, 18, 6–63. seling: Methodological and pro- Press.
Fukuyama, M. (1990). Taking a uni- fessional issues. The Counseling Glaser, B., & Strauss, A. (1967). The
versal approach to multicultural Psychologist, 8, 7–35. discovery of grounded theory:
counseling. Counselor Education Gelso, C. J. (1982). Editorial. Journal of Strategies for qualitative research.
& Supervision, 30, 6–17. Counseling Psychology, 29, 3–7. Chicago: Aldine.
Fuller, F., & Hill, C. E. (1985). Gelso, C. J. (1985). Rigor, relevance, Glaser, B. G. (1992). Basics of
Counselor and helpee perceptions and counseling research: On the grounded theory analysis: Emergence
of counselor intentions in relation need to maintain our course between vs. forcing. Mill Valley, CA: Sociol-
to outcome in a single counseling Scylla and Charybdis. Journal of ogy Press.
session. Journal of Counseling Counseling and Development, 63, Glass, G. V., Willson, V. L., &
Psychology, 32, 329–338. 551–553. Gottman, J. M. (1974). Design
Furlong, M. J., & Wampold, B. E. Gelso, C. J. (1993). On the making of and analysis of time-series experi-
(1982). Intervention effects and a scientist-practitioner: A theory ments. Boulder, CO: Colorado
relative variation as dimensions of research training in professional Associated University Press.
in experts’ use of visual inference. psychology. Professional Psychology: Gleitman, H. (1986). Psychology (2nd
Journal of Applied Behavior Anal- Research and Practice, 24, 468–476. ed.). New York: Norton.
ysis, 15, 415–421. Gelso, C. J. (1997). The making of a Glock, C. Y. (Ed.). (1967). Survey
Furnham, A., & Procter, E. (1988). scientist in applied psychology: An research in the social sciences.
The multi-dimensional Just World attribute by treatment conception. New York: Russell Sage Foundation.
Belief Scale. [Mimeograph] Lon- The Counseling Psychologist, 25, Goetz, J. P., & LeCompte, M. D.
don: London University. 307–320. (1984). Ethnography and qualita-
Furst, J. B., & Cooper, A. (1970). Gelso, C. J., Betz, N. E., Friedlander, tive design in educational research.
Combined use of imaginal and M. L., Helms, J. E., Hill, C. E., Patton, New York: Academic Press.
interoceptive stimuli in desensitiz- M. J., Super, D. E., & Wampold, Gold, R. L. (1958). Roles in socio-
ing fear of heart attacks. Journal of B. E. (1988). Research in counsel- logical field observations. Social
Behavior Therapy and Experimental ing psychology: Prospects and rec- Forces, 36, 217–223.
Psychiatry, 1, 87–89. ommendations. The Counseling Goldapple, K., Segal, Z., & Garson,
Gade, E., Fuqua, D., & Hurlburt, G. Psychologist, 16, 385–406. C. (2004). Modulation of cortical-
(1988). The relationship of Holland’s Gelso, C. J., & Fassinger, R. E. (1990). limbic pathways in major depres-
personality types to educational Counseling psychology: Theory sion. Archives of General Psy-
satisfaction with a native American and research on interventions. chiatry, 61, 34–41.
references 631
Goldman, L. (1976). A revolution in Gottman, J. M. (1979). Detecting Greene, J. C., Caracelli, V. J., &
counseling research. Journal of cyclicity in social interaction. Graham, W. F. (1989). Toward a
Counseling Psychology, 23, 543–552. Psychological Bulletin, 86, 338–348. conceptual framework for mixed
Goldman, L. (Ed.). (1978). Research Gottman, J. M., & Markman, H. J. method evaluation designs. Edu-
methods for counselors: Practical (1978). Experimental designs in cational Evaluation and Policy
approaches in field settings. New psychotherapy research. In S. L. Analysis, 11, 255–274.
York: Wiley. Garfield & A. E. Bergin (Eds.), Greenwald, A. G. (1968). Cognitive
Goldman, L. (1982). Defining non- Handbook of psychotherapy and learning, cognitive response to
traditional research. The Counseling behavior change (2nd ed., pp. persuasion, and attitude change.
Psychologist, 10, 87–89. 23–62). New York: Wiley. In A. G. Greenwald, T. C. Brock,
Goldstein, A. P., Heller, K., & Sechrest, Gottman, J. M., McFall, R. M., & & T. M. Ostrom (Eds.), Psycho-
L. B. (1966). Psychotherapy and the Barnett, J. T. (1969). Design and logical foundations of attitudes
psychology of behavior change. analysis of research using time (pp. 147–190). New York: Aca-
New York: Wiley. series. Psychological Bulletin, 72, demic Press.
Good, G. E., Gilbert, L. A., & Scher, 299–306. Grieger, I., & Ponterotto, J. G. (1995).
M. (1990). Gender aware therapy: Gottman, J. M., & Roy, A. K. (1990). In J. G. Ponterotto, J. M. Casas,
A synthesis of feminist therapy and Sequential analysis. Cambridge: L. A. Suzuki, & C. M. Alexander
knowledge about gender. Journal Cambridge University Press. (Eds.), Handbook of multicultural
of Counseling and Development, Gould, S. J. (1994). The geometer of counseling (pp. 357–374). Thousand
68, 376–380. race. Discover, 15, 65–69. Oaks, CA: Sage.
Good, G. E., Robertson, J. M., O’Neil, Graham, J. R. (1990). MMPI-2: Grummon, D. L., & Butler, J. M.
J. M., Fitzgerald, L. F., Stevens, M., Assessing personality and psy- (1953). Another failure to repli-
DeBord, K. A., Bartels, K. M., & chopathology. New York: Oxford cate Keet’s study, two verbal tech-
Braverman, D. G. (1995). Male University Press. niques in a miniature counseling
gender role conflict: Psychometric Greenbaum, T. L. (1998). The hand- situation. Journal of Abnormal
issues and relations to psychologi- book for focus group research and Social Psychology, 48, 597.
cal distress. Journal of Counseling (2nd ed.). Thousand Oaks, CA: Sage. Grundy, C. T., & Lambert, M. J.
Psychology, 42, 3–10. Greenberg, L. S. (1986a). Change (1994a, June). Assessing clinical
Good, G. E., Thoreson, R. W., & process research. Journal of Con- significance: Application to the
Shaughnessy, P. (1995). Substance sulting and Clinical Psychology, 54, Child Behavior Checklist. Paper
use, confrontation of impaired col- 4–9. presented at the annual meeting
leagues, and psychological function- Greenberg, L. S., & Foerster, F. S. of the Society for Psychotherapy
ing among counseling psychologists: (1996). Task analysis exemplified: Research, York, England.
A national survey. The Counseling The process of resolving unfinished Grundy, C. T., & Lambert, M. J. (1994b,
Psychologist, 23, 703–721. business. Journal of Consulting and June). Assessing clinical signifi-
Goodyear, R. K., & Benton, S. (1986). Clinical Psychology, 64, 439–446. cance: Application to the Hamilton
The roles of science and research Greenberg, L. S., & Newman, F. L. Rating Scale for Depression. Paper
in the counselor’s work. In A. J. (1996). An approach to psycho- presented at the annual meeting of
Palmo & W. J. Weikel (Eds.), therapy change process research: the Society for Psychotherapy Re-
Foundations of mental health Introduction to the special section. search, York, England.
counseling (pp. 287–308). Springfield, Journal of Consulting and Clinical Grzegorek, J. L., Slaney, R. B., Franze,
IL: Charles C Thomas. Psychology, 64, 435–438. S., & Rice, K. (2004). Self–
Goodyear, R. K., Tracey, T. J. G., Greenberg, L. S., & Pinsof, W. (Eds.). criticism, dependency, self-esteem
Claiborn, C. D., Lichtenberg, (1986). The psychotherapeutic and grade point average satisfac-
J. W., & Wampold, B. E. (2005). process: A research handbook. tion among clusters of perfection-
Ideographic concept mapping in New York: Guilford. ists and nonperfectionists. Journal
counseling psychology research: Greenberg, R. P., & Fisher, S. (1997). of Counseling Psychology, 51,
Conceptual overview, methodol- Mood-mending medicines: Probing 192–200.
ogy, and an illustration. Journal drugs, psychotherapy, and placebo Guba, E. G., & Lincoln, Y. S. (1994).
of Counseling Psychology, 52, solutions. In S. Fisher & R. P. Competing paradigms in qualita-
236–242. Greenberg (Eds.), From placebo to tive research. In N. K. Denzin &
Gottfredson, G. D., & Holland, J. L. panacea: Putting psychiatric drugs Y. S. Lincoln (Eds.), Handbook of
(1990). A longitudinal test of the to the test (pp. 115–172). New qualitative research (pp. 105–117).
influence of congruence: Job satis- York: Wiley. Thousand Oaks, CA: Sage.
faction, competency utilization, Greene, J. C., & Caracelli, V. J. (Eds.). Guba, E. G., & Lincoln, Y. S. (1998).
and counterproductive behavior. (1997). Advances in mixed- Competing paradigms in qualita-
Journal of Counseling Psychology, method evaluation: The challenges tive research. In N. K. Denzin &
37, 389–398. and benefits of integrating diverse Y. S. Lincoln (Eds.), The landscape
Gottman, J. M. (1973). N-of-one and paradigms (New Directions for of qualitative research: Theories
N-of-two research in psychotherapy. Evaluation, No. 74). San Francisco: and issues (pp. 156–184). Thousand
Psychological Bulletin, 80, 93–105. Jossey-Bass. Oaks, CA: Sage.
632 references
Gurin, N. K., Veroff, G. J., & Feld, S. Hardin, S. I., Subich, L. M., & Haupt, S. G. (1990). Client Christian
(1960). Americans view their mental Holvey, J. M. (1988). Expectancies belief issues in psychotherapy. Un-
health. New York: Basic Books. for counseling in relation to pre- published doctoral dissertation,
Gurman, A. S. (1977). The patient’s mature termination. Journal of University of Missouri, Columbia.
perception of the therapeutic rela- Counseling Psychology, 35, 37–40. Haverkamp, B. E., Morrow, S. L., &
tionship. In A. S. Gurman & Harding, S. (1991). Whose science? Ponterotto, J. G. (2005a). A time
A. M. Razin (Eds.), Effective psy- Whose knowledge? Thinking from and place for qualitative and
chotherapy: A handbook of research women’s lives. Ithaca, NY: Cornell mixed methods in counseling psy-
(pp. 503–543). New York: Pergamon University Press. chology research. Journal of Coun-
Press. Harmon, L. (1977). Career counseling seling Psychology, 52, 123–125.
Gurman, A. S., & Razin, A. M. (Eds.). for women. In E. Rawlings & D. Haverkamp, B. E., Morrow, S. L., &
(1977). Effective psychotherapy: Carter (Eds.), Psychotherapy for Ponterotto, J. G. (2005b). Knowledge
A handbook of research. New women (pp. 197–206). Springfield, IL: in context: Qualitative methods in
York: Pergamon Press. Charles C Thomas. counseling psychology research
Gusheu, G. V., & Carter, R. T. (2000). Harmon, L. (1982). Scientific affairs: [special issue]. Journal of Counseling
Remembering race: White racial The next decade. The Counseling Psychology, 52.
identity attitudes and two aspects Psychologist, 10, 31–38. Hayes, J. A., & Erkis, A. J. (2000).
of social memory. Journal of Harmon, L. W. (1981). The life and Therapist homophobia, client sex-
Counseling Psychology, 47, 199–210. career plans of young adult college ual orientation, and score of client
Gustav, A. (1962). Students’ attitudes women: A follow-up study. Journal HIV infection as predictor of ther-
toward compulsory participation of Counseling Psychology, 28, apist reaction to client with HIV.
in experiments. Journal of Psychology, 416–427. Journal of Counseling Psychology,
53, 119–125. Harmon, L. W. (1989). Changes in 47, 71–78.
Guthrie, R. V. (1998). Even the rat women’s career aspirations over Hayes, J. A., McCracken, J. E.,
was white: A historical view of time: Developmental or historical. McClananhan, M. K., Hill., C. E.,
psychology. Boston, MA: Allyn & Journal of Vocational Behavior, Harp, J. S., & Carozzoni, P. (1998).
Bacon. 33, 46–65. Therapist perspectives on coun-
Haase, R. F., Waechter, D. M., & Harre, R. (1970). The principles of sci- tertransference: Qualitative data
Solomon, G. S. (1982). How sig- entific thinking. Chicago: University in search of a theory. Journal
nificant is a significant-difference? of Chicago Press. of Counseling Psychology, 45,
Average effect size of research in Harre, R. (1972). Philosophies of science: 468–482.
counseling psychology. Journal of An introductory survey. Oxford, Hays, W. L. (1988). Statistics (4th
Counseling Psychology, 29, 58–65. England: Oxford University Press. ed.). New York: Holt, Rinehart &
Hackett, G. (1981). Survey research Harre, R. (1974). Blueprint for a Winston.
methods. Personnel and Guidance new science. In A. Nigel (Ed.), Hayton, J. C., Allen, D. G., &
Journal, 59, 599–604. Reconstructing social psychology Scarpello, V. (2004). Factor reten-
Hair, J. F., Jr., Anderson, R. E., & (pp. 11–64). Baltimore: Penguin tion decisions in exploratory fac-
Tatham, R. L. (1987). Multivariate Books. tor analysis: A tutorial on parallel
data analysis: With readings (2nd Harre, R. (1980). Social being: A analysis. Organizational Research
ed.). New York: Macmillan. theory of social psychology. Totowa, Methods, 7, 191–205.
Hair, J. K., & Black, W. C. (2000). NJ: Rowman & Littlefield. Hazaleus, S. L., & Deffenbacher, J. L.
Cluster analysis. In L. G. Grimm Hartley, D. E., & Strupp, H. H. (1986). Relaxation and cognitive
& P. R. Yarnold (Eds.), Reading (1983). The therapeutic alliance: treatments of anger. Journal of
and understanding more multi- Its relationship to outcome in Consulting and Clinical Psychology,
variate statistics (pp. 147–205). brief psychotherapy. In J. Masling 54, 222–226.
Washington, DC: American Psy- (Ed.), Empirical studies of psycho- Heesacker, M., Elliott, T. R., &
chological Association. analytic theories (Vol. 1, pp. Howe, L. A. (1988). Does the
Hall, S., Weinman, J., & Marteau, 1–37). Hillsdale, NJ: The Analytic Holland code predict job satisfac-
T. M. (2004). The motivating impact Press. tion and productivity in clothing
of informing women smokers of Harvey, B., Pallant, J., & Harvey, D. factory workers? Journal of Coun-
the link between smoking and cer- (2004). An evaluation of the fac- seling Psychology, 35, 144–148.
vical cancer: The role of coherence. tor structure of the Frost Multi- Heller, K. (1971). Laboratory inter-
Health Psychology, 23, 419–424. dimensional Perfectionism Scale. view research as an analogue to
Hanfling, O. (1981). Logical posi- Educational and Psychological Mea- treatment. In A. E. Bergin & S. L.
tivism. Oxford: Blackwell. surement, 64, 1007–1018. Garfield (Eds.), Handbook of psy-
Hanson, W. E., Creswell, J. W., Clark, Hathaway, S. R., & McKinley, J. C. chotherapy and behavior change
V. L., Petska, K. S., & Creswell, (1942). A multiphasic personality (pp. 126–153). New York: Wiley.
J. D. (2005). Mixed methods research schedule (Minnesota): III. The Helms, J. E. (1976). Comparison of
designs in counseling psychology. measurement of symptomatic de- two types of counseling analogue.
Journal of Counseling Psychology, pression. The Journal of Psychology, Journal of Counseling Psychology,
52, 224–235. 14, 73–84. 23, 422–427.
references 633
Helms, J. E. (1978). Counselor reac- Henwood, K., & Pidgeon, N. (2003). Heppner, P. P. (2006). The benefits
tions to female clients: Generalizing Grounded theory in psychology and challenges of becoming cross-
from analogue research to a coun- research. In P. M. Camic, J. E. culturally competent counseling
seling setting. Journal of Counseling Rhodes, & L. Yardley (Eds.), psychologists. The Counseling
Psychology, 25, 193–199. Qualitative research in psychol- Psychologist, 34, 147–172.
Helms, J. E. (1990). Black and white ogy: Expanding perspectives in Heppner, P. P., & Anderson, W. P.
racial identity: Theory, research, methodology and design (pp. (1985). The relationship between
and practice. Westport, CT: Green- 131–155). Washington, DC: Amer- problem-solving self-appraisal and
wood Press. ican Psychological Association. psychological adjustment. Cognitive
Helms, J. E. (1994). How multicultur- Heppner, M. J., & Hendricks, F. (1995). Therapy and Research, 9, 415–427.
alism obscures racial factors in the A process and outcome study Heppner, P. P., Baumgardner, A. H.,
therapy process: Comment on examining career indecision and Larson, L. M., & Petty, R. E. (1988).
Ridley et al. (1994), Sodowsky indecisiveness. Journal of Counseling The utility of problem-solving
et al. (1994), Ottavi et al. (1994), and Development, 73, 426–437. training that emphasizes self-man-
and Thompson et al. (1994). Heppner, M. J., & Heppner, P. P. agement principles. Counseling
Journal of Counseling Psychology, (2003). Identifying process variables Psychology Quarterly, 1, 129–143.
41, 162–165. in career counseling: A research Heppner, P. P., Carter, J., Claiborn, C. D.,
Helms, J. E., & Carter, R. T. (1990). agenda. Journal of Vocational Brooks, L., Gelso, C. J., Fassinger,
Development of the white racial Behavior, 62, 429–452. R. E., Holloway, E. L., Stone, G. L.,
identity inventory. In J. E. Helms Heppner, M. J., Multon, K. D., Wampold, B. E., & Galani, J. P.
(Ed.), Black and white racial Gysbers, N. C., Ellis, C., & Zook, (1992). A proposal to integrate
identity: Theory, research and C. E. (1998). Examining the rela- science and practice in counseling
practice (pp. 67–80). Westport, tionship of counselor self-efficacy psychology. The Counseling Psy-
CT: Greenwood. and selected client process and chologist, 20, 107–122.
Helms, J. E., & Cook, D. A. (1999). outcome measures in career coun- Heppner, P. P., Casas, J. M., Carter, J.,
Using race and culture in counsel- seling. Journal of Counseling Psy- & Stone, G. L. (2000). The matu-
ing and psychotherapy: Theory chology, 45, 393–402. ration of counseling psychology:
and process. Needham, MA: Allyn Heppner, M. J., Multon, K. D., & Multifaceted perspectives from
& Bacon. Johnston, J. A. (1994). Assessing 1978–1998. In S. D. Brown &
Helms, J. E., & Parham, T. A. (1996). psychological resources during ca- R. W. Lent (Eds.), Handbook of
The racial identity attitudes. In reer change: Development of the counseling psychology (3rd ed.,
R. L. Jones (Ed.), Handbook of tests career transitions inventory. Journal pp. 3–49). New York: Wiley.
and measurements for black popu- of Vocational Behavior, 44, 55–74. Heppner, P. P., & Claiborn, C. D. (1989).
lations (Vol. 1, pp. 53–72). Thousand Heppner, P. P. (1978a). A review of Social influence research in coun-
Oaks, CA: Sage. the problem-solving literature and seling: A review and critique
Helwig, A. A. (2004). A ten year lon- its relationship to the counseling [monograph]. Journal of Counseling
gitudinal study of the career devel- process. Journal of Counseling Psy- Psychology, 36, 365–387.
opment of students: Summary chology, 25, 366–375. Heppner, P. P., Cooper, C. C., Mul-
findings. Journal of Counseling and Heppner, P. P. (1978b). The clinical holland, A. M., & Wei, M. F.
Development, 82, 49–57. alteration of covert thoughts: A (2001). A brief, multidimensional,
Hembree, E. A., Street, G. P., & Riggs, critical review. Behavior Therapy, problem-solving based psychother-
D. S. (2004). Do assault-related 9, 717–734. apy outcome measure. Journal of
variables predict response to cog- Heppner, P. P. (1979). The effects of Counseling Psychology, 48, 330–343.
nitive behavioral treatment for client perceived need and coun- Heppner, P. P., & Dixon, D. N. (1981).
PTSD? Journal of Consulting and selor role on clients’ behaviors A review of the interpersonal
Clinical Psychology, 72, 531–534. (doctoral dissertation, University influence process in counseling.
Henry, W. P., Schacht, T. E., & Strupp, of Nebraska, 1979). Dissertation Personnel and Guidance Journal,
H. H. (1986). Structural analysis Abstracts International, 39, 5950A– 59, 542–550.
of social behavior: Application to 5951A. (University Microfilms Heppner, P. P, & Frazier, P. A. (1992).
a study of interpersonal processes No. 79-07,542) Social psychological processes in
in differential psychotherapeutic Heppner, P. P. (1989). Identifying the psychotherapy: Extrapolating basic
outcome. Journal of Consulting complexities within clients’ thinking research to counseling psychology.
and Clinical Psychology, 54, and decision-making. Journal of In S. D. Brown & R. W. Lent
27–31. Counseling Psychology, 36, 257–259. (Eds.), Handbook of counseling
Henry, W. P., Strupp, H. H., Butler, Heppner, P. P. (1995). On gender role psychology (2nd ed., pp. 141–175)
S. F., Schacht, T. E., & Binder, J. L. conflict in men: Future directions New York: Wiley.
(1993). Effects of training in time- and implications for counseling: Heppner, P. P., Gelso, C. J., &
limited dynamic psychotherapy: Comment on Good et al. (1995) Dolliver, R. H. (1987). Three
Changes in therapist behavior. and Cournoyer and Mahalik approaches to research training in
Journal of Consulting and Clinical (1995). Journal of Counseling Psy- counseling. Journal of Counseling
Psychology, 61, 434–440. chology, 42, 20–23. and Development, 66, 45–49.
634 references
Heppner, P. P, & Heppner, M. J. Hermansson, G. L., Webster, A. C., process and outcome of time-
(2004). Writing and publishing & McFarland, K. (1988). Coun- limited counseling. Journal of
your thesis, dissertation & research: selor deliberate postural lean and Counseling Psychology, 30, 3–18.
A guide for students in the help- communication of facilitative Hill, C. E., Greenwald, C., Reed, K. A.,
ing professions. Belmont, CA: conditions. Journal of Counsel- Charles, D., O’Farrell, M. K., &
Brooks/Cole. ing Psychology, 35, 149–153. Carter, J. A. (1981). Manual for the
Heppner, P. P., Heppner, M. J., Lee, Heron, J. & Reason, P. (1997). A partic- counselor and client verbal response
D-G., Wang, Y. W., Park, H-J., & ipatory inquiry paradigm. Qualitative category systems. Columbus, OH:
Wang, L-F. (2006). Development Inquiry, 3(3), 274–294. Marathon Consulting and Press.
and validation of a Collectivist Hersen, M., & Barlow, D. H. (1976). Hill, C. E., Helms, J. E., Spiegel, S. B., &
Coping Styles Inventory. Journal Single case experimental designs: Tichenor, V. (1988a). Development
of Counseling Psychology, 53, Strategies for studying behavior of a system for categorizing client
107–125. change. New York: Pergamon Press. reactions to therapist interventions.
Heppner, P. P., Kivlighan, D. M., Jr., Herskovits, M. J. (1955). Cultural Journal of Counseling Psychology,
Good, G., Roehlke, H. J., Hills, anthropology. New York: Knopf. 35, 27–36.
H. I., & Ashby, J. S. (1994). Presenting Highlen, P. S., & Hill, C. E. (1984). Hill, C. E., Helms, J. E., Tichenor, V.,
problems of university counseling Factors affecting client change in Spiegel, S. B., O’Grady, K. E., &
center clients: A snapshot and a counseling: Current status and Perry, E. S. (1988b). Effects of
multivariate classification scheme. theoretical speculations. In S. D. therapist response modes in brief
Journal of Counseling Psychology, Brown & R. W. Lent (Eds.), psychotherapy. Journal of Counseling
41, 315–324. Handbook of counseling psychol- Psychology, 35, 222–233.
Heppner, P. P., & Krauskopf, C. J. ogy (pp. 334–396). New York: Hill, C. E., Knox, S., Thompson, B. G.,
(1987). An information-processing Wiley. Williams, E. N., Hess, S. A., &
approach to personal problem Hill, C. E. (1982). Counseling process Ladany, N. (2005). Consensual
solving. The Counseling Psychologist, researcher: Philosophical and meth- qualitative research: An update.
15, 371–447. odological dilemmas. The Counsel- Journal of Counseling Psychology,
Heppner, P. P., & Neal, G. W. (1983). ing Psychologist 10, 7–20. 52(2), 196–205.
Holding up the mirror: Research Hill, C. E. (1984). A personal account Hill, C. E., & Lambert, M. J. (2004).
on the roles and functions of coun- of the process of becoming a coun- Methodological issues in studying
seling centers in higher education. seling process researcher. The psychotherapy processes and out-
The Counseling Psychologist, 11, Counseling Psychologist, 12, 99– comes. In M. J. Lambert (Ed.),
81–89. 109. Bergin and Garfield’s handbook of
Heppner, P. P., & Petersen, C. H. Hill, C. E. (1985). Manual for the psychotherapy and behavior change
(1982). The development and Hill Counselor Verbal Response (5th ed., pp. 84–135) New York:
implications of a personal problem- Modes Category System (rev. ed.). Wiley.
solving inventory. Journal of Coun- Unpublished manuscript, University Hill, C. E., & O’Grady, K. E. (1985).
seling Psychology, 29, 66–75. of Maryland. List of therapist intentions illus-
Heppner, P. P., & Roehlke, H. J. Hill, C. E. (1990). A review of trated in a case study and with
(1984). Differences among super- exploratory in-session process re- therapists of varying theoretical
visees at different levels of training: search. Journal of Consulting and orientations. Journal of Counseling
Implications for a developmental Clinical Psychology, 58, 288–294. Psychology, 32, 3–22.
model of supervision. Journal Hill, C. E. (1991). Almost everything Hill, C. E., Rochlen, A. B., Zack, J. S.,
of Counseling Psychology, 31, you ever wanted to know about McCready, T., & Dematatis, A.
76–90. how to do process research on (2003). Working with dreams using
Heppner, P. P., Rogers, M. E., & Lee, counseling and psychotherapy but the Hill cognitive-experimental
L. A. (1984). Carl Rogers: Re- didn’t know how to ask. In C. E. model: A comparison of computer-
flections on his life. Journal of Watkins, Jr. & L. J. Scheider assisted, therapy empathy, and
Counseling and Development, 63, (Eds.), Research in counseling (pp. therapist empathy + input conditions.
14–20. 85–118). Hillsdale, NJ: Lawrence Journal of Counseling Psychology,
Heppner, P. P., Rosenberg, J. I., & Erlbaum. 50, 211–220.
Hedgespeth J. (1992). Three meth- Hill, C. E. (1992). Research on thera- Hill, C. E., & Stephany, A. (1990).
ods in measuring the therapeutic pist techniques in brief individual Relation of nonverbal behavior to
process: Clients’ and counselors’ therapy: Implications for practi- client reactions. Journal of Coun-
constructions of the therapeutic tioners. The Counseling Psychologist, seling Psychology, 37, 22–26.
process versus actual therapeutic 20, 689–711. Hill, C. E., Thompson, B. J., &
events. Journal of Counseling Psy- Hill, C. E. (1997). The effects of my Williams, E. N. (1997). A guide to
chology, 39, 20–31. research training environment: conducting consensual qualitative
Herman, J., Morris, L., & Fitz- Where are my students now? The research. The Counseling Psychologist,
Gibbon, C. (1987). Evaluator’s Counseling Psychologist, 25, 74–81. 25, 517–572.
handbook. Newbury Park, CA: Hill, C. E., Carter, J. A., & O’Farrell, Hill, C. E., & Williams, E. N. (2000).
Sage. M. K. (1983). A case study of the The process of individual therapy.
references 635
In S. D. Brown & R. W. Lent Search (3rd ed.). Palo Alto, CA: paradoxical intervention with a
(Eds.), Handbook of counseling psy- Consulting Psychologists Press. couple: An interactional analysis.
chology (3rd ed., pp. 670–710). Holland, J. L. (1987). Manual supple- Journal of Family Psychology, 3,
New York: Wiley. ment for the Self-Directed Search. 385–402.
Hill, W. F. (1965). HIM: Hill Interaction Odessa, FL: Psychological Assess- Honos-Webb, L., Stiles, W. B., &
Matrix. Los Angeles: University of ment Resources. Greenberg, L. S. (2003). A method
Southern California, Youth Study Holland, J. L. (1992). Making voca- of rating assimilation in psy-
Center. tional choices: A theory of voca- chotherapy based on makers of
Hillenbrand-Gunn, T., Heppner, M. J., tional personalities and work change. Journal of Counseling
Mauch, P. A., & Park, H. J. (2004, environment (2nd ed.). Odessa, Psychology, 50, 189–198.
July). Sexual violence in high FL: Psychological Assessment Re- Hoshmand, L.T. (1994). Supervision
school males: Investigating social sources. of predoctoral graduate research:
norms theory. Paper presented at Holland, J. L., Daiger, D. C., & A proactive oriented approach.
the annual convention of the Power, P. G. (1980). Some diag- The Counseling Psychologist, 22,
American Psychological Association, nostic scales for research in decision- 147–161.
Honolulu, Hawaii. making and personality: Identity, Horan, J. J. (1979). Counseling for effec-
Hines, P. L., Stockton, R., & Morran, information, and barriers. Journal of tive decision making: A cognitive-
D. K. (1995). Self-talk of group Personality and Social Psychology, behavioral perspective. North Scituate,
therapists. Journal of Counseling 39, 1191–1200. MA: Duxbury Press.
Psychology, 42, 242–248. Holland, P. W. (1986). Statistics and Horowitz, L. M., Rosenberg, S. E.,
Hoch, P. H., & Zubin, J. (Eds.). causal inference. Journal of the Baer, B. A., Ureno, G., &
(1964). The evaluation of psychi- American Statistical Association, Villasenor, V. S. (1988). Inventory
atric treatment. New York: Grune 81, 945–960. of Interpersonal Problems: Psy-
& Stratton. Hollon, S. D. (1996). The efficacy and chometric properties and clinical
Hodder, I. (1994). The interpretation effectiveness of psychotherapy rel- applications. Journal of Consult-
of documents and material cul- ative to medications. American ing and Clinical Psychology, 56,
ture. In N. K. Denzin & Y. S. Psychologist, 51, 1025–1030. 885–892.
Lincoln (Eds.), Handbook of qual- Hollon, S. D., & Kendall, D. C. Horvath, A. O., & Greenberg, L. S.
itative research (pp. 393–402). (1980). Cognitive self-statements (1986). The development of the
Thousand Oaks, CA: Sage. in depression: Development of an Working Alliance Inventory. In
Hoffman, J., & Weiss, B. (1987). Automatic Thoughts Questionnaire. L. S. Greenberg & W. M. Pinsof
Family dynamics and presenting Cognitive Therapy and Research, (Eds.), The psychotherapeutic
problems in college students. 4, 383–395. process: A research handbook (pp.
Journal of Counseling Psychology, Holloway, E. L. (1987). Developmental 529–556). New York: Guilford
34, 157–163. models of supervision: Is it devel- Press.
Hoffman, M. A., & Carter, R. T. opment? Professional Psychology; Horvath, A. O., & Greenberg, L. S.
(2004). Counseling psychology Research and Practice, 18, 209–216. (1989). Development and valida-
and school counseling: A call to Holloway, E. L. (1992). Supervision: tion of the Working Alliance
collaboration. The Counseling Psy- A way of teaching and learning. In Inventory. Journal of Counseling
chologist, 32, 181–183. S. D. Brown & R. W. Lent (Eds.), Psychology, 36, 223–233.
Hogg, J. A., & Deffenbacher, J. L. Handbook of counseling psychol- Hoshmand, L. L. S. T. (1989).
(1988). A comparison of cognitive ogy (2nd ed., pp. 177–214). New Alternate research paradigms: A
and interpersonal-process group York: Wiley. review and teaching proposal.
therapies in the treatment of de- Holloway, E. L. (1995). Clinical The Counseling Psychologist, 17,
pression among college students. supervision: A systems approach. 3–79.
Journal of Counseling Psychology, Thousand Oaks, CA: Sage. Hoshmand, L. T. (1994). Supervision
35, 304–310. Holloway, E. L., Freund, R. D., of predoctoral graduate research:
Hohmann, A. A., & Parron, D. L. Gardner, S. L., Nelson, M. L., & A practice oriented approach. The
(1996). How the new NIH guide- Walker, B. R. (1989). Relation of Counseling Psychologist, 22, 147–161.
lines on inclusion of women and power and involvement to theoret- Hoshmand, L. T. (1997). The nor-
minorities apply: Efficacy trials, ical orientation in supervision: An mative context of research prac-
effectiveness trials, and validity. analysis of discourse. Journal of tice. The Counseling Psychologist,
Journal of Consulting and Clinical Counseling Psychology, 36, 88–102. 25, 599–605.
Psychology, 64, 851–855. Holloway, E. L., & Wampold, B. E. Hoshmand, L. T. (2005). Narratology,
Holland, J. L. (1985a). Making vo- (1986). Relation between concep- cultural psychology, and counseling
cational choices: A theory of vo- tual level and counseling-related research. Journal of Counseling
cational personalities and work tasks: A meta-analysis. Journal Psychology, 52, 178–186.
environments. Englewood Cliffs, of Counseling Psychology, 33, Howard, G. S. (1982). Improving
NJ: Prentice-Hall. 310–319. methodology via research on re-
Holland, J. L. (1985b). Professional Holloway, E. L., Wampold, B. E., & search methods. Journal of Coun-
manual for the Self-Directed Nelson, M. L. (1990). Use of a seling Psychology, 29, 318–326.
636 references
Howard, G. S. (1983). Toward meth- Ibrahim, F. A., & Kahn, H. (1987). Jensen, A. R. (1969). How much can
odological pluralism. Journal of Assessment of worldviews. Psy- we boost IQ and scholastic
Counseling Psychology, 30, 19–21. chological Reports, 60, 163–176. achievement? Harvard Educational
Howard, G. S. (1984). A modest pro- Ibrahim, F. A., & Owen, S. V. (1994). Review, 39, 1–123.
posal for a revision of strategies Factor analytic structure of the Jensen, A. R. (1985). The nature
in counseling research. Journal of Scale to Assess Worldview. Current of the black-white difference
Counseling Psychology, 31, 430–441. Psychology: Developmental, Learning, on various psychometric tests:
Howard, G. S. (1985). Can research Personality, Social, 13, 201–209. Spearman’s hypothesis. The behav-
in the human sciences become Ibrahim, F. A., Roysircar-Sodowsky, ioral and Brain Sciences, 8,
more relevant to practice? Journal G., & Ohnishi, H. (2001). World- 193–263.
of Counseling and Development, view: Recent developments and Johnson, C. V., & Hayes, J. A. (2003).
63, 539–544. needed directions. In J. G. Ponte- Troubled spirits: Prevalence and
Howard, K. I., Moras, K., Brill, P. L., rotto, J. M. Casas, L. A. Suzuki, & predictors of religious and spiri-
Martinovich, Z., & Lutz, W. C. A. Alexander (Eds.), Handbook tual concerns among university
(1996). Evaluation of psychother- of multicultural counseling. (2nd students and counseling center
apy. American Psychologist, 51, ed., pp. 425–456). Thousand clients. Journal of Counseling
1059–1064. Oaks, CA: Sage. Psychology, 50, 409–419.
Hoyle, R. H. (1995). The structural Ingram, R. (Ed.). (1986). Information pro- Johnson, R. F. Q. (1976). The experi-
equation modeling approach: Basic cessing approaches to clinical psy- menter attributes effect: A meth-
concepts and fundamental issues. chology. Orlando, FL: Academic Press. odological analysis. Psychological
In R. H. Hoyle (Ed.), Structural Ivey, A. E., Ivey, M. B., & Simek- Record, 26, 67–78.
equation modeling: Concepts, Morgan, L. (1997). Counseling Johnston, J. A., Buescher, K. L., &
issues, and applications (pp. 1–15). and psychotherapy: A multicul- Heppner, M. J. (1988). Computerized
Thousand Oaks, CA: Sage. tural perspective (4th ed.). Boston: career information and guidance
Hoyt, M. R., Marmar, C. R., Horo- Allyn & Bacon. systems: Caveat emptor. Journal
witz, M. J., & Alvarez, W. F. Jacobson, N. S. (1991). Behavioral of Counseling and Development,
(1981). The therapist actions scale versus insight-oriented marital 67, 39–41.
and the patient actions scale: therapy: Labels can be misleading. Joint Committee on Standards for
Instruments for the assessment of Journal of Consulting and Clinical Educational Evaluation. (1994).
activities during dynamic psy- Psychology, 59, 142–145. The program evaluation standards
chotherapy. Psychotherapy: Theory, Jacobson, N. S., & Christensen, A. (2nd ed.). Thousand Oaks, CA:
Research and Practice, 18, 109–116. (1996). Studying the effectiveness Sage Publications.
Hoyt, W. T. (1996). Antecedents and of psychotherapy. American Psy- Jones, A. S., & Gelso, C. J. (1988).
effects of perceived therapist cred- chologist, 51, 1031–1039. Differential effects of style of inter-
ibility: A meta-analysis. Journal of Jacobson, N. S., Dobson, K. S., & pretation: Another look. Journal
Counseling Psychology, 43, 430–447. Truax, P. A. (1996). A component of Counseling Psychology, 35,
Huberty, C. J., & Morris, J. D. (1989). analysis of cognitive-behavioral 363–369.
Multivariate analysis versus multiple treatment for depression. Journal of Jones, E. E. (1993). Introduction to
univariate analyses. Psychological Consulting and Clinical Psychology, special section: Single case research
Bulletin, 105, 302–308. 64, 295–304. in psychotherapy. Journal of Con-
Huck, S. W., & McLean, R. A. Jacobson, N. S., Follette, W. C., & sulting and Clinical Psychology, 61,
(1975). Using a repeated measures Revenstorf, D. (1984). Psychotherapy 371–372.
ANOVA to analyze the data from outcome research: Methods for Jones, E. E., Ghannam, J., Nigg, J. T.,
a pretest-posttest design: A poten- reporting variability and evaluat- & Dyer, J. F. P. (1993). A para-
tially confusing task. Psychological ing clinical significance. Behavior digm for single-case research: The
Bulletin, 82, 511–518. Therapy, 15, 336–352. time series study of a long-term
Hughes, E. C. (1952). Psychology: Jacobson, N. S., & Revenstorf, D. psychotherapy for depression. Jour-
Science and/or profession. American (1988). Statistics for assessing the nal of Consulting and Clinical Psy-
Psychologist, 7, 441–443. clinical significance of psychother- chology, 61, 381–394.
Hunt, D. W., Butler, L. F., Noy, J. E., apy techniques: Issues, problems Jones, H. G. (1956). The application
& Rosser, M. E. (1978). Assessing and new developments. Behavior of conditioning and learning
conceptual level by the paragraph Assessment, 10, 133–145. techniques to the treatment of a
completion method. Toronto, Can- Jacobson, N. S., & Truax, P. (1991). psychiatric patient. Journal of
ada: The Ontario Institute for Studies Clinical significance: A statistical Abnormal and Social Psychology,
in Education. approach to defining meaningful 52, 414–419.
Iberg, J. R. (1991). Applying statisti- change in psychotherapy research. Josephson, G. S., & Fong-Beyette,
cal control theory to bring together Journal of Consulting and Clinical M. L. (1987). Factors assisting
clinical supervision and psy- Psychology, 59, 12–19. female clients’ disclosure of incest
chotherapy research. Journal of Jayaratne, S., & Levy, R. L. (1979). during counseling. Journal of
Consulting and Clinical Psychol- Empirical clinical practice. New Counseling and Development,
ogy, 96, 575–586. York: Columbia University Press. 65, 475–478.
references 637
Juntunen, C. L., Barraclough, D. J., Kazdin, A. E. (2003). Research design Kiesler, D. J. (1984). Check List of
Broneck, C. L., Seibel, G. A., in clinical psychology. (4th ed.). Psychotherapy Transactions (CLOPT)
Winrow, S. A., & Morin, P. M. Boston: Allyn & Bacon. and Check List of Interpersonal
(2001). American Indian perspec- Keet, C. D. (1948). Two verbal tech- Transactions (CLOIT). Richmond:
tives on the career journey. Journal niques in a miniature counseling Virginia Commonwealth University.
of Counseling Psychology, 48, situation. Psychological Monographs, Kiesler, D. J. (1987). Research manual
274–285. 62(7, Whole No. 294). for the Impact Message Inventory.
Juntunen, C. L., & Wettersten, K. B. Keith-Spiegel, P., & Koocher, G. P. Palo Alto, CA: Consulting Psy-
(2006). Work hope: Development (1985). Ethics in psychology: Pro- chologists Press.
and initial validation of a measure. fessional standards and cases. Kiesler, D. J. (1988). Therapeutic meta-
Journal of Counseling Psychology, New York: Random House. communication. Palo Alto, CA:
53, 94–106. Kelerman, G., & Neu, C. (1976). A Consulting Psychologists Press.
Kagan, N. (1975). Influencing human manual for interpersonal treat- Kiesler, D. J., Klein, M. H., &
interaction: Eleven years with IPR. ment of depression. Unpublished Mathieu, P. L. (1965). Sampling
Canadian Counselor, 9, 44–51. manuscript, Yale University, New from the recorder therapy inter-
Kahn, J. H., & Gelso, C. J. (1997). Haven, CT. view: The problem of segment
Factor structure of the Research Keller, M. B. (2004). Remission versus location. Journal of Consulting
Training Environment Scale—Re- response: The new gold standard Psychology, 29, 337–344.
vised: Implications for research of antidepressant care. Journal of Kim, B. S. K., & Atkinson, D.
training in applied psychology. Clinical Psychiatry, 65, 53–59. (2002). Asian American client
The Counseling Psychologist, 25, Kelly, A. E., & Achter, J. A. (1995). adherence to Asian cultural
22–37. Self-concealment and attitudes values, counselor expression of
Kahn, J. H., & Scott, N. A. (1997). toward counseling in university cultural values, counselor ethnic-
Predictors of research productivity students. Journal of Counseling ity, and career counseling process.
and science-related career goals Psychology, 42, 40–46. Journal of Counseling Psyhology,
among counseling psychology doc- Kelly, A. E., Kahn, J. H., & Coulter, 49, 3–13.
toral students. The Counseling Psy- R. G. (1996). Client self-presentation Kim, B. S. K., Atkinson, D. R., &
chologist, 25, 38–67. at intake. Journal of Counseling Yang, P. H. (1999). The Asian
Kandel, D. K. (1973). Adolescent Psychology, 43, 300–309. values scale: Development, factor
marijuana use: Role of parents and Kelly, A. E., McKillop, K. J., & analysis, validation, and reliability.
peers. Science, 181, 1067–1070. Neimeyer, G. S. (1991). Effects of Journal of Counseling Psychology,
Kanfer, F. H., & Busemeyer, J. R. counselor as audience on the inter- 46, 342–352.
(1982). The use of problem solv- nalization of depressed and nonde- Kim, B. S. K., Brenner, B. R., Liang, C.
ing and decision making in behav- pressed self-presentation. Journal T. H., & Asay, P. A. (2003). A
ior therapy. Clinical Psychological of Counseling Psychology, 38, qualitative study of adaptation ex-
Review, 2, 239–266. 126–132. periences of 1.5-generation Asian
Kaul, T., & Bednar, R. L. (1986). Kelly, K. R., & Pulver, C. A. (2003). Americans. Cultural Diversity &
Experiential group research. In Refining measurement of career Ethnic Minority Psychology, 9,
S. L. Garfield & A. E. Bergin indecision types: A validity study. 156–170.
(Eds.), Handbook of psychother- Journal of Counseling and Devel- Kim, B. S. K, Hill, C. E., Gelso, C. J.,
apy and behavior change (3rd ed., opment, 81, 445–454. Goates, M. K., Asay, P. A., &
pp. 671–714). New York: Wiley. Kerlinger, F. N. (1986). Foundations Harbin, J. M. (2003). Counselor
Kazdin, A. D., & Kopel, S. A. (1975). of behavioral research (3rd ed.). self-disclosure, East Asian American
On resolving ambiguities of the New York: Holt, Rinehart & client adherence to Asian cultural
multiple-baseline design: Problems Winston. values, and counseling process.
and recommendations. Behavior Kerlinger, F. N., & Lee, H. B. Journal of Counseling Psychology,
Therapy, 6, 601–608. (2000). Foundations of behavioral 50, 324–332.
Kazdin, A. E. (1978). Methodology of research. Fort Worth, TX: Harcourt Kim, D. M., Wampold, B. E., & Bolt,
applied behavior analysis. In A. C. College. D. (2006). Therapist effects in
Catania & T. A. Brigham (Eds.), Kiesler, D. J. (1966). Some myths of psychotherapy: A random effects
Handbook of applied behavior psychotherapy research and the modeling of the NIMH TDCRP
analysis: Social and instructional search for a paradigm. Psychological data. Psychotherapy Research, 16,
processes (pp. 61–104). New York: Bulletin, 65, 110–136. 161–172.
Irvington Press/Halstead Press. Kiesler, D. J. (1971). Experimental Kim, U., & Choi, S-H. (1994).
Kazdin, A. E. (1980). Research design designs in psychotherapy research. Individualism, collectivism, and
in clinical psychology. New York: In A. E. Bergin & S. L. Garfield child development: A Korean per-
Harper & Row. (Eds.), Handbook of psychother- spective. In P. M. Greenfield &
Kazdin, A. E. (1982). Single-case apy and behavior change (pp. R. R. Cocking (Eds.), Cross-cultural
research designs: Methods for clin- 36–74). New York: Wiley. roots of minority child develop-
ical and applied settings. New Kiesler, D. J. (1973). The process of ment (pp. 227–257). Hillsdale, NJ:
York: Oxford University Press. psychotherapy. Chicago: Aldine. Erlbaum.
638 references
Kim, Y. C., & Cho, J. (2005). Now Kivlighan, D. M., Jr., & Lilly, R. L. self-disclosure in long-term therapy.
and forever: Portraits of qualitative (1997). Developmental changes in Journal of Counseling Psychology,
research in Korea. International group climate as they relate to 44, 274–283.
Journal of Qualitative Studies in therapeutic gain. Group Dynamics: Knox, S., Hess, S. A., Williams, E. N.,
Education, 18(3), 355–377. Theory, Research, and Practice, 1, & Hill, C. E. (2003). “Here’s a
Kirshner, T., Hoffman, M. A., & Hill, 208–221. little something for you”: How
C. E. (1994). A case study of the Kivlighan, D. M., Jr., Multon, K. M., therapists respond to client gifts.
process and outcome of career & Patton, M. J. (1996). Development Journal of Counseling Psychology,
counseling. Journal of Counseling of the Missouri Addressing Resis- 50, 199–210.
Psychology, 41, 216–226. tance Scale. Psychotherapy Research Koile, E. A., & Bird, D. J. (1956).
Kiselica, M. S., Baker, S. B., Thomas, 6, 291–308. Preferences for counselor help on
R. N., & Reedy, S. (1994). Effects Kivlighan, D. M., Jr., & Shapiro, freshman problems. Journal of
of stress inoculation training on R. M. (1987). Holland type as a Counseling Psychology, 3, 97–106.
anxiety, stress, and academic per- predictor of benefit from self-help Kokotovic, A. M., & Tracey, T. J.
formance among adolescents. career counseling. Journal of (1987). Premature termination at
Journal of Counseling Psychology, Counseling Psychology, 34, 326–329. a university counseling center.
41, 335–342. Kivlighan, D. M., Jr. & Shaughnessy, Journal of Counseling Psychology,
Kitchener, K., & Anderson, S. K. P. (1995). An analysis of the devel- 34, 80–82.
(2000). Ethical issues in counsel- opment of the working alliance Koocher, G. P., & Keith-Spiegel, P.
ing psychology: Old themes—new using hierarchical linear modeling. (1998). Ethics in psychology:
problems. In S. D. Brown & R. W. Journal of Counseling Psychology, Professional standards and cases
Lent (Eds.), Handbook of counseling 42, 338–349. (2nd ed.). New York: Oxford
psychology (3rd ed., pp. 50–82). Klare, G. R. (1974–1975). Assessing University Press.
New York: John Wiley & Sons. readability. Reading Research Quar- Koplik, E. K., & DeVito, A. J. (1986).
Kitchener, K. S. (1984). Intuition, crit- terly, 10, 62–102. Problems of freshmen: Comparison
ical evaluation and ethical princi- Klein, M., Mathieu, P., Kiesler, O., & of classes of 1976 and 1986.
ples: The foundation for ethical Gendlin, E. (1969). The experi- Journal of College Student Personnel,
decision in counseling psychology. encing scale. Madison: Wisconsin 27, 124–131.
The Counseling Psychologist, 12, Psychiatric Institute. Kraemer, H. C., & Thiemann, S.
43–55. Klein, M. H., Mathieu-Coughlan, P., (1987). How many subjects?:
Kivlighan, D. M., Jr. (1989). Changes & Kiesler, D. J. (1986). The expe- Statistical power analysis in re-
in counselor intentions and re- riencing scales. In L. Greenberg & search. Newbury Park, CA: Sage.
sponse modes and client reactions W. Pinsof (Eds.), The psychothera- Krause, A. A., & Allen, G. J. (1988).
and session evaluation following peutic process (pp. 21–77). New Perceptions of counselor supervi-
training. Journal of Counseling York: Guilford Press. sion: An examination of Stoltenberg’s
Psychology, 36, 471–476. Kleinke, C. L. (1986). Gaze and eye model from the perspectives of
Kivlighan, D. M., Jr. (1990). Relation contact: A research review. Psy- supervisor and supervisee. Journal
between counselors’ use of in- chological Bulletin, 100, 78–100. of Counseling Psychology, 35,
tentions and clients’ perception Kline, T. J. B. (2005). Psychological 77–80.
of working alliance. Journal of testing: A practical approach to Krebs, P. J., Smither, J. W., & Hurley,
Counseling Psychology, 37, 27–32. design and evaluation. Thousand R. B. (1991). Relationship of voca-
Kivlighan, D. M., Jr., & Angelone, Oaks, CA: Sage. tional personality and research
E. O. (1991). Helpee introversion, Klingelhofer, E. L. (1954). The rela- training environment to the re-
novice counselor intention use, and tionship of academic advisement search productivity of counseling
counseling session impact. Journal of to the scholastic performance of psychologists. Professional Psy-
Counseling Psychology, 38, 25–29. failing college students. Journal of chology: Research and Practice,
Kivlighan, D. M., Jr., Coleman, Counseling Psychology, 1, 125–131. 22, 362–367.
M. N., & Anderson, D. C. (2000). Klinger, E. (1971). Structure and func- Krivatsky, S. E., & Magoon, T. M.
Process, outcome, and methodol- tions of fantasy. New York: Wiley. (1976). Differential effects of three
ogy in group counseling research. Knox, S., Burkard, A. W., Johnson, vocational counseling treatments.
In S. D. Brown & R. W. Lent A. J., Suzuki, L. A., & Ponterotto, Journal of Counseling Psychology,
(Eds.), Handbook of counseling J. G. (2003). African American 23, 112–117.
psychology (3rd ed., pp. 767–796). and European American thera- Krop, H., & Krause, S. (1976). The
New York: Wiley. pists’ experiences of addressing elimination of shark phobia by
Kivlighan, D. M., Jr., Hageseth, J., race in cross-racial psychotherapy self-administered systematic desen-
Tipton, R., & McGovern, T. M. dyads. Journal of Counseling sitization: A case study. Journal of
(1981). The effects of matching Psychology, 50, 466–481. Behavior Therapy and Experimental
treatment approaches and person- Knox, S., Hess, S. A., Petersen, D. A., Psychiatry, 7, 293–294.
ality types in group vocational & Hill, C. E. (1997). A qualitative Krueger, R., & Casey, M. A. (2000).
counseling. Journal of Counseling analysis of client perceptions of Focus groups (3rd ed.). Thousands
Psychology, 28, 315–320. the effects of helpful therapist Oaks, CA: Sage.
references 639
Krumboltz, J. D. (1991). Career methodological considerations, and Leary, T. (1957). Interpersonal diag-
beliefs inventory. Palo Alto, CA: recommendations for future research. nosis of personality. New York:
Consulting Psychologist Press. Psychological Bulletin, 85, 467–489. Ronald Press.
Krumboltz, J. D., & Mitchell, L. K. Lambert, M. J., & Hill, C. E. (1994). Lee, D-G., Park, H-J., Shin, Y-J., &
(1979). Relevant rigorous research. Assessing psychotherapy outcomes Graham, D. (2005, August). Vali-
The Counseling Psychologist, 81, and processes. In A. E. Bergin & dation of the Frost Multidimen-
50–52. S. L. Garfield (Eds.), Handbook of sional Perfectionism Scale in South
Kruskal, W., & Mosteller, F. (1979). psychotherapy and behavior change Korea. Poster presented at the
Representative sampling III: The (4th ed., pp. 72–113). New York: annual meeting of the Asian
current statistical literature. Inter- Wiley. American Psychological Association,
national Statistical Review, 47, Lambert, M. J., & Ogles, B. M. Washington, DC.
245–265. (2004). The efficacy and effective- Lee, L. A., Heppner, P. P., Gagliardi,
Kuhn, T. S. (1970). The structure of ness of psychotherapy. In M. J. J., & Lee, J. S. (1987). Gender bias
scientific revolutions (2nd ed.). Lambert (Ed.), Bergin and Garfield’s in subject samples in counseling
Chicago: University of Chicago Press. handbook of psychotherapy and psychology. Journal of Counseling
Kushner, K. (1978). On the external behavior change (5th ed., pp. Psychology, 34, 73–76.
validity of two psychotherapy ana- 139–193). New York: Wiley. Lee, R. (2003). Do ethnic identity and
logues. Journal of Consulting and Lapan, R. T., Boggs, K. R., & Morrill, other-group orientation protect
Clinical Psychology, 46, 1394–1402. W. H. (1989). Self-efficacy as a me- against discrimination for Asian
Kvale, S. (1996). Interviews: An in- diation of investigative and real- Americans? Journal of Counseling
troduction to qualitative research istic general occupational themes Psychology, 50, 133–141.
interviewing. Thousand Oaks, CA: of the Strong-Campbell Interest Lee, R. M. (2005). Resilience against
Sage. Inventory. Journal of Counseling discrimination: Ethnic identity and
Ladany, N., O’Brien, K. M., Hill, Psychology, 36, 176–182. other-group orientation as protec-
C. E., Melincoff, D. S., Knox, S., Larson, L. M. (1998). The social cog- tive factors for Korean Americans.
& Petersen, D. A. (1997). Sexual nitive model of counselor training. Journal of Counseling Psychology,
attraction toward clients, use of The Counseling Psychologist, 26, 52, 36–44.
supervision, and prior training: A 219–273. Lee, R. M., Choe, J., Kim, G., & Ngo,
qualitative study of predoctoral Larson, L. M., & Besett-Alesch, T. M. V. (2000). Construction of the
psychology interns. Journal of (2000). Bolstering the scientist Asian American Family Conflicts
Counseling Psychology, 44, component in the training of Scale. Journal of Counseling Psy-
413–424. scientist-practitioners: One pro- chology, 47, 211–222.
Lakatos, I. (1970). Falsification and gram’s curriculum modifications. Lent, R. W., Brown, S. D., & Hackett,
the methodology of scientific re- The Counseling Psychologist, 28, G. (1994). Toward a unifying so-
search programmes. In I. Lakatos 873–896. cial cognitive theory of career and
& A. Musgrave (Eds.), Criticism Larson, L. M., & Daniels, J. A. (1998). academic interest, choice, and
and the growth of knowledge (pp. Review of the counseling self- performance. Journal of Voca-
91–196). Cambridge, England: efficacy literature. The Counseling tional Behavior, 45, 79–122.
Cambridge University Press. Psychologist, 26, 129–218. Lent, R. W., Brown, S. D., & Lankin,
Lambert, M. J., & Bergin, A. E. Larson, L. M., Heppner, P. P., Ham, K. C. (1987). Comparison of three
(1993). Achievements and limita- T., & Dugan, K. (1988). Inves- theoretically derived variables in
tions of psychotherapy research. tigating multiple subtypes of career predicting career and academic
In D. K. Freedheim (Ed.), History indecision through cluster anal- behavior: Self-efficacy, interest-
of psychotherapy (pp. 360–390). ysis. Journal of Counseling Psychol- consequence, and consequence
Washington, DC: American Psy- ogy, 35, 439–446. thinking. Journal of Counseling
chological Association. Larson, L. M., & Majors, M. S. Psychology, 34, 293–298.
Lambert, M. J., Bergin, A. E., & (1998). Application of the coping Lent, R. W., Russell, R. K., &
Collins, J. L. (1977). Therapist- with career indecision instrument Zamostny, K. P. (1981). Comparison
induced deterioration in psy- with adolescents. Journal of Ca- of cue-controlled desensitization,
chotherapy. In A. S. Gurman & A. reer Assessments, 6, 163–179. rational restructuring, and a credi-
M. Razin (Eds.), Effective psy- Larson, L. M., Suzuki, L. A., Gillespie, ble placebo in the treatment of
chotherapy: A handbook of re- K. N., Potenza, M. T., Bechtel, speech anxiety. Journal of Consulting
search (pp. 452–481). New York: M. A., & Toulouse, A. L. (1992). and Clinical Psychology, 49,
Pergamon Press. Development and validation of 608–610.
Lambert, M. J., Christensen, E. R., & the Counseling Self-Estimate Inven- Leong, F. T. L., & Ponterotto, J. G.
DeIulio, S. (1983). The assessment tory. Journal of Counseling Psy- (2003). A proposal for interna-
of psychotherapy outcome. New chology, 39, 105–210. tionalizing counseling psychology
York: Wiley. Lawless, E. J. (2001). Women escaping in the United States: Rationale,
Lambert, M. J., DeJulio, S. S., & violence: Empowerment through recommendations, and challenges.
Stein, D. M. (1978). Therapist inter- narrative. Columbia, MO: University The Counseling Psychologist, 31,
personal skills: Process, outcome, of Missouri Press. 381–395.
640 references
Leong, F. T. L., Wagner, N. S., & Tata, strategies in psychotherapy. Journal A handbook for clinical practice
S. P. (1995). Racial and ethnic vari- of Counseling Psychology, 50, (pp. 211–226). New York: Basic
ations in help seeking attitudes. In 267–275. Books.
J. G. Ponterotto, J. M. Casas, L. A. Liddle, B. J. (1996). Therapist sexual Luborsky, L., Diguer, L., Seligman,
Suzuki, & C. M. Alexander (Eds.), orientation, gender, and counseling D. A., Rosenthal, R., Krause, E. D.,
Handbook of multicultural coun- practices as they relate to ratings of Johnson, S., et al., (1999). The
seling (pp. 415–438). Thousand helpfulness by gay and lesbian cli- researcher’s own therapy alle-
Oaks, CA: Sage. ents. Journal of Counseling Psychol- giances: A “wild card” in com-
Levant, R. F. (2004). The empirically ogy, 43, 394–401. parisons of treatment efficacy.
validated treatment movement: A Lieberman, M. A., Yalom, I., & Miles, Clinical Psychology: Science and
practitioner/educator perspective. M. (1973). Encounter groups: Practice, 6(1), 95–106.
Clinical Psychology: Science and First facts. New York: Basic Books. Lucas, M. S., & Epperson, D. L.
Practice, 11, 219–224. Likert, R. (1932). A technique for the (1990). Types of vocational un-
Levin, J. R. & Wampold, B. E. (1999). measurement of attitudes. Archives decidedness: A replication and
Generalized single-case random- of Psychology, 140, 44–53. refinement. Journal of Counseling
ization tests: Flexible analyses for Lincoln, Y. S., & Guba, E. G. (1985). Psychology, 37, 382–388.
a variety of situations. School Psy- Naturalistic inquiry. Beverly Hills, Lundervold, D. A. & Belwood, M. F.
chology Quarterly, 14, 59–93. CA: Sage. (2000). The best kept secret in
Lewin, K. (1951). Formalization and Lincoln, Y. S., & Guba, E. G. (2000). counseling: Single-case (N = 1)
progress in psychology. In D. Paradigmatic controversies, contra- experimental designs. Journal of
Cartwright (Ed.), Field theory in dictions, and emerging confluences. Counseling and Development, 78,
social science (pp. 1–41). New In N. K. Denzin & Y. S. Lincoln 92–102.
York: Harper. (Eds.), Handbook of qualitative Mackay, H. C., Barkham, M., Rees,
Lewinsohn, P. M., Mischel, W., research (2nd ed., pp. 163–188). A., & Stiles, W. B. (2003). Ap-
Chapel, W., & Barton, R. (1980). Thousand Oaks, CA: Sage. praisal of published reviews of
Social competence and depression: Lindsey, R. T. (1984). Informed consent research on psychotherapy and
The role of illusory self-perceptions. and deception in psychotherapy re- counseling with adults 1990–1998.
Journal of Abnormal Psychology, search: An ethical analysis. The Journal of Consulting & Clinical
89, 203–212. Counseling Psychologist, 12, 79–86. Psychology, 71, 652–656.
Lewis, J. (2001). Career and personal Lipkus, I. (1991). The construction Madriz, E. (2000). Focus groups in
counseling: Comparing process and and preliminary validation of a feminist research. In N. K. Denzin
outcome. Journal of Employment Global Belief in a Just World Scale & Y. S. Lincoln (Eds.), Handbook
Counseling, 38, 82–90. and the exploratory analysis of the of qualitative research (2nd ed.,
Liang, C. T. H., Li, L. C., & Kim, Multidimensional Belief in a Just pp. 835–850). Thousand Oaks,
B. S. K. (2004). The Asian American World Scale. Personality and Indi- CA: Sage.
racism-related stress inventory: vidual Differences, 12, 1171–1178. Magoon, T. M., & Holland, J. L.
Development, factor analysis, reli- Locke, D. (1990). A not so provincial (1984). Research training and
ability, and validity. Journal of Coun- view of multicultural counseling. supervision. In S. D. Brown &
seling Psychology, 51, 103–114. Counseling Education & Supervision, R. W. Lent (Eds.), Handbook
Lichtenberg, J. W. (1984). Believing 30, 18–25. of counseling psychology (pp.
when the facts don’t fit. Journal of Locke, D. C., Myers, J. E., & Herr, 682–715). New York: Wiley.
Counseling and Development, 63, E. L. (2001). The handbook of coun- Mahalik, J. R., & Kivlighan,
10–11. seling. Thousand Oaks, CA: Sage. D. M., Jr. (1988). Research
Lichtenberg, J. W., & Heck, E. J. Loehlin, J. C. (1992). Latent variable training and supervision. In S.
(1983). Use of sequential analysis models: An introduction to factor, D. Brown & R. W. Lent (Eds.),
in counseling process research: A path, and structural analysis Handbook of counseling psy-
reply to Hill, Carter, and O’Farrell (2nd ed.). Hillsdale, NJ: Lawrence chology (pp. 682–715). New York:
and to Howard. Journal of Coun- Erlbaum Associates. Wiley.
seling Psychology, 30, 615–618. Lorr, M. (1983). Cluster analysis for Mahrer, A. R., Paterson, W. E.,
Lichtenberg, J. W., & Heck, E. J. (1986). social scientists. San Francisco: Theriault, A. T., Roessler, C., &
Analysis of sequence and pattern Jossey-Bass. Quenneville, A. (1986). How and
in process research. Journal of Luborsky, L. (1984). Principles of why to use a large number of clin-
Counseling Psychology, 33, 170–181. psychoanalytic psychotherapy: A ically sophisticated judges in psy-
Lichtenberg, J. W., & Hummel, T. J. manual for supportive-expressive chotherapy research. Voices: The
(1976). Counseling as a stochastic treatment. New York: Basic Books. Art and Science of Psychotherapy,
process: Fitting a Markov chain Luborsky, L., & Barber, J. P. (1993). 22, 57–66.
model to initial counseling inter- Benefits of adherence to treatment Maier, N. R. F. (1931). Reasoning in
views. Journal of Counseling Psy- manuals, and where to get them. humans: II. The solution of a prob-
chology, 23, 310–315. In N. Miller, L. Luborsky, J. P. lem and its appearance in con-
Lichtenberg, J. W., & Tracey, T. J. L. Barber, & J. P. Docherty (Eds.), sciousness. Journal of Comparative
(2003). Interaction rules and Psychodynamic treatment research: Psychology, 12, 181–194.
references 641
Malkiewich, L. E., & Merluzzi, T. V. Marotta S. A. & Garcia, J. G. (2003). its effect upon psychological data. In
(1980). Rational restructuring ver- Latinos in the United States 2000. D. Levine (Ed.), Nebraska Sympos-
sus desensitization with clients of Hispanic Journal of Behavioral Sci- ium on Motivation, 14, 67–103.
diverse conceptual levels: A test ences, 25, 13–34. Matsumoto, D. (1994). Cultural influ-
of client-treatment matching model. Marsden, G. (1965). Content-analysis ences on research methods and sta-
Journal of Counseling Psychol- studies of therapeutic interviews: tistics. Pacific Grove, CA: Brooks/
ogy, 27, 453–461. 1954–1964. Psychological Bulletin, Cole.
Mallinckrodt, B. (1989). Social sup- 63, 298–321. Matsumoto, D. (2000). Culture and
port and the effectiveness of group Marsden, G. (1971). Content analysis psychology: People around the
therapy. Journal of Counseling Psy- studies of psychotherapy: 1954 world. Belmont, CA: Wadsworth/
chology, 36, 170–175. through 1968. In A. E. Bergin & Thomson Learning.
Mallinckrodt, B., Gelso, C. J., & Royalty, S. L. Garfield (Eds.), Handbook of Maxwell, S. E., & Cole, D. A. (1995).
G. M. (1990). Impact of the research psychotherapy and behavior change Tips for writing (and reading) method-
training environment and counsel- (pp. 345–407). New York: Wiley. ological articles. Psychological Bul-
ing psychology students’ Holland Marteau, T. M., Rana, S., & Kubba, letin, 118, 193–198.
personality type on interest in A. (2002). Smoking and cervical McCarn, S. R., & Fassinger, R. E.
research. Professional Psychology: cancer: A qualitative study of the (1996). Revisioning sexual minority
Research and Practice, 21, 26–32. explanatory models of smokers with identity and its implications for
Mallinckrodt, B., & Helms, J. E. (1986). cervical abnormalities. Psychology, counseling and research. The Coun-
Effect of disabled counselor’s self- Health and Medicine, 7, 107–109. seling Psychologist, 24, 508–534.
disclosures on client perceptions of Martin, J. (1984). The cognitive medi- McCarthy, P. R., Shaw, T., &
the counselor. Journal of Counseling ational paradigm for research on Schmeck, R. R. (1986). Behavioral
Psychology, 33, 343–348. counseling. Journal of Counseling analysis of client learning style
Mallinckrodt, B., & Wang, C-C. Psychology, 31, 558–571. during counseling. Journal of
(2004). Quantitative methods for Martin, J. (1985). Measuring clients’ Counseling Psychology, 33, 249–254.
verifying semantic equivalence of cognitive competence in research on McCullough, L., & Farrell, A. D. (1983).
translated research instruments: A counseling. Journal of Counseling The Computerized Assessment for
Chinese version of the Experiences and Development, 63, 556–560. Psychotherapy Evaluation and
in Close Relationship Scale. Jour- Martin, J. (1987). Cognitive- Research [computer program].
nal of Counseling Psychology, 51, instructional counseling. London, New York: Beth Israel Medical
368–379. Ontario, Canada: Althouse Press. Center, Department of Psychiatry.
Manicas, P. T., & Secord, P. F. (1983). Martin, J., Martin, W., & Slemon, McCullough, L., Farrell, A. D., &
Implications for psychology of the A. G. (1989). Cognitive-mediational Longabaugh, R. (1986). The de-
new philosophy of science. Amer- models of action-act sequences in velopment of a microcomputer-
ican Psychologist, 38, 399–413. counseling. Journal of Counseling based mental health information
Markowitz, J. C., Kocsis, J. H., Psychology, 36, 8–16. system: A potential tool for bridg-
Bleiberg, K. L., Christos, P. J., & Martin, J., & Stelmaczonek, K. ing the scientist-practitioner gap.
Sacks, M. (2005). A comparative (1988). Participants’ identification American Psychologist, 41, 207–214.
trial of psychotherapy and phar- and recall of important events in McDaniel, M. A., Whetzel, D. L.,
macotherapy for “pure” dysthy- counseling. Journal of Counseling Schmidt, F. L., & Maurer, S.
mic patients. Journal of Affective Psychology, 35, 385–390. (1994). The validity of employ-
Disorders, 89, 167–175. Martin, J. S., Goodyear, R. K., & ment interview: A comprehensive
Markowitz, J. C., Kocsis, J. H., & Newton, F. B. (1987). Clinical view and meta-analysis. Journal of
Fishman, B. (1998). Treatment of supervision: An intensive case Applied Psychology, 79, 599–616.
depressive symptoms in human study. Professional Psychology: McGuire, W. J. (1985). Attitudes and
immunodeficiency virus-positive Research and Practice, 18, 225–235. attitude change. In G. Lindzey &
patients. Archives of General Psy- Maruyama, M. (1963). The second E. Aronson (Eds.), Handbook of
chiatry, 55, 452–457. cybernetics: Deviation-amplifying social psychology (3rd ed., Vol. 2,
Marmar, C. R. (1990). Psychotherapy mutual causal processes. American pp. 233–346). New York: Random
process research: Progress, dilem- Scientist, 51, 169–179. House.
mas, and future directions. Journal Marx, J. A., & Gelso, C. J. (1987). McKay, K. M., Imel, Z. E., &
of Consulting and Clinical Psy- Termination of individual coun- Wampold, B. E. (2006). Psychiatrist
chology, 58, 265–272. seling in a university counseling effects in the psychopharmacolo-
Marmar, C. R., Marziali, E., Horowitz, center. Journal of Counseling Psy- gical treatment of depression.
M. J., & Weiss, D. S. (1986). The chology, 34, 3–9. Journal of Affective Disorders, 92,
development of the therapeutic Mash, E. J., & Terdal, L. G. (1988). 287–290.
alliance rating system. In L. S. Behavioral assessment of child- McKinney, F. (1945). Four years of a
Greenberg & W. M. Pinsof (Eds.), hood disorders (2nd ed.). New college adjustment clinic: I.
The psychotherapeutic process: A York: Guilford Press. Organization of clinic and prob-
research handbook (pp. 367–390). Masling, J. (1966). Role-related behavior lems of counselors. Journal of
New York: Guilford Press. of the subject and psychologist and Consulting Psychology, 9, 203–212.
642 references
McLeod, J. (2001). Qualitative research ers: A new system based on the P. A. (1997). Questionnaire for eat-
in counseling and psychotherapy. MMPI. Beverly Hills, CA: Sage. ing disorder diagnosis: Reliability
Thousand Oaks, CA: Sage. Meichenbaum, D., Henshaw, D., & and validity of operationalizing
McNamara, K., & Horan, J. J. Himel, N. (1980). Coping with DSM-IV criteria diagnosed into a
(1986). Experimental construct stress as a problem-solving process. self-report format. Journal of
validity in the evaluation of cogni- In W. Krohne & L. Laux (Eds.), Counseling Psychology, 44, 63–79.
tive and behavioral treatments for Achievement stress and anxiety Mintz, L. B., & O’Neil, J. M. (1990).
depression. Journal of Counseling (pp. 127–142). Washington, DC: Gender roles, sex, and the process of
Psychology, 33, 23–30. Hemisphere. psychotherapy: Many questions and
Meara, N. M., & Day, J. D. (2003). Melby, J. N., Hoyt, W. T., & Bryant, few answers. Journal of Counseling
Possibilities and challenges for aca- C. M. (2003). A generalizability and Development, 68, 381–387.
demic psychology. American Behav- approach to assessing the effects of Miranda, J., Azocar, F., Organista,
ioral Scientist, 47(4), 459–478. ethnicity and training on observer K. C., Muñoz, R. F., & Lieberman, A.
Meara, N. M., & Patton, M. J. ratings of family interactions. (1996). Recruiting and retaining low-
(1986). Language use and social Journal of Social and Personal income Latinos in psychotherapy
influence in counseling. In F. J. Relationships, 20, 171–191. research. Journal of Consulting
Dorn (Ed.), The social influence Mercer, R. C., & Loesch, L. C. (1979). and Clinical Psychology, 64,
process in counseling and psy- Audio tape ratings: Comments and 868–874.
chotherapy (pp. 85–93). Springfield, guidelines. Psychotherapy: Theory, Mitchell, K. M., Bozarth, J. D., &
IL: Charles C Thomas. Research, and Practice, 16, 79–85. Kraft, C. C. (1977). A reappraisal
Meara, N. M., Schmidt, L. D., Merrill, R. M. (1952). On Keet’s of the therapeutic effectiveness of
Carrington, C. H., Davis, K. L., study, “Two verbal techniques in a accurate empathy, nonpossessive
Dixon, D. N., Fretz, B. R., Myers, miniature counseling situation.” warmth and genuineness. In A. S.
R. A., Ridley, C. R., & Suinn, Journal of Abnormal and Social Gurman & A. M. Razin (Eds.),
R. M. (1988). Training and accred- Psychology, 47, 722. Effective psychotherapy: A hand-
itation in counseling psychology. Merten, J., Anstadt, T., Ullrich, B., book of research (pp. 482–502).
The Counseling Psychologist, 16, Krause, R., & Buchheim, P. (1996). New York: Pergamon Press.
366–384. Emotional experience and facial Mohr, D. C., Shoham-Salomon, V.,
Meara, N. M., Schmidt, L. D., & Day, behavior during the psychothera- Engle, D., & Beutler, L. E. (1991).
J. D. (1996). Principles and virtues: peutic process and its relation to The expression of anger in psy-
A foundation for ethical decisions, treatment outcome: A pilot study. chotherapy for depression: Its role
policies, and character. The Coun- Psychotherapy Research, 6, 198–212. and measurement. Psychotherapy
seling Psychologist, 24, 4–77. Meyer, V. (1957). The treatment of Research, 1, 124–134.
Meara, N. M., Shannon, J. W., & two phobic patients on the basis of Mohr, J. J., Israel, T., & Sedlacek,
Pepinsky, H. B. (1979). Comparison learning principles. Journal of Ab- W. E. (2001). Counselor’s attitudes
of the stylistic complexity of the normal and Social Psychology, 55, regarding bisexuality as predictors
language of counselor and client 261–266. of counselor’s clinical responses: An
across three theoretical orientations. Meyers, L. S., Gamst, G., & Guarino, analogue study of a female bisexual
Journal of Counseling Psychology, A. J. (2006). Applied multivariate client. Journal of Counseling Psy-
28, 110–118. research: Design and interpreta- chology, 48, 212–222.
Meehl, P. E. (1971). A scientific, tion. Thousand Oaks, CA: Sage. Monte, C. F. (1980). Beneath the
scholarly nonresearch doctorate Mill, J. S. (1953). A system of logic, mask: An introduction to theories
for clinical practitioners. In R. R. Book VI; On the logic of the moral of personality (2nd ed.). New
Holt (Ed.), New horizon for psy- sciences. In P. P. Weiner (Ed.), York: Holt, Rinehart & Winston.
chotherapy: Autonomy as a pro- Readings in philosophy of science Mooney, R. L., & Gordon, L. V.
fession (pp. 37–81). New York: (pp. 255–281). New York: Scribner’s. (1950). Manual: The Mooney
International Universities Press. (Original work published 1843) problem checklists. New York:
Meehl, P. E. (1978). Theoretical risks Miller, A. (1981). Conceptual match- Psychological Corporation.
and tabular asterisks: Sir Karl, Sir ing models and interactional re- Moradi, B., & Hasan, N. T. (2004).
Ronald, and the slow progress of soft search in education. Review of Arab American persons’ reported
psychology. Journal of Consulting Educational Research, 51, 33–84. experiences of discrimination and
and Clinical Psychology, 46, 806–834. Mills, E. A. (1924). Rocky Mountain mental health: The mediating role
Meehl, P. E. (1987). Why summaries National Park. Garden City, NY: of personal control. Journal of
of research on a psychological Doubleday, Page, & Co. Counseling Psychology, 51, 418–428.
theory are often uninterpretable. Mintz, J., & Luborsky, L. (1971). Moradi, B., & Subich, L. M. (2003).
In R. Snow & D. E. Wiley (Eds.), Segments vs. whole sessions: Which A concomitant examination of the
Strategic thinking: A volume in is the better unit for psychotherapy relations of perceived racist and
honor of Lee J. Cronbach. San research? Journal of Abnormal sexist events to psychological dis-
Francisco: Jossey-Bass. Psychology, 78, 180–191. tress for African American women.
Megargee, E. I., & Bohn, M. J. Mintz, L. B., O’Halloran, S. M., The Counseling Psychologist, 31,
(1979). Classifying criminal offend- Mullholland, A. M., & Schneider, 451–469.
references 643
Moras, K., & Hill, C. E. (1991). Rater Multon, K. D., Ellis-Kalton, C. A., outcomes in academic training
selection in psychotherapy process Heppner, M. J., & Gysbers, N. C. programs in counseling psychol-
research: Observation on the state- (2003). Counselor verbal response ogy. The Counseling Psychologist,
of-the-art. Psychotherapy Research, modes and working alliance in career 33, 635–654.
1, 113–123. counseling. Career Development Neisser, U. (1976). Cognition and
Morgan, D. (1988). Focus groups as Quarterly, 51, 259–273. reality: Principles and implica-
qualitative research. Newbury Multon, K. D., Heppner, M. J., & Lapan, tions of cognitive psychology. San
Park, CA: Sage. R. T. (1995). An empirical derivation Francisco: Freeman.
Morgan, K. S., & Brown, L. S. of career decision subtypes in a high Nelson, C., Treichler, P. A., &
(1991). Lesbian career develop- school sample. Journal of Vocational Grossberg, L. (1992). Cultural
ment, work behavior, and voca- Behavior, 47, 76–92. studies: An introduction. In L.
tional counseling. The Counseling Multon, K. D., Wood, R., Heppner, Grossberg, C. Nelson, & P. A.
Psychologist, 19, 273–291. M. J., & Gysbers, N. C. (in press). Treichler (Eds.), Cultural studies
Morran, D. K., Kurpius, D. J., & A cluster-analytic investigation of (pp. 1–16). New York: Routledge.
Brack, G. (1989). Empirical inves- subtypes of adult career counsel- Neville, H., & Carter, R. (2005). Race
tigation of counselor self-talk cate- ing clients: Toward a taxonomy of and racism in counseling psychology
gories. Journal of Counseling Psy- career problems. Journal of Career research, training, and practice: A
chology, 36, 505–510. Assessment. critical review, current trends, and
Morrison, L. A., & Shapiro, D. A. Munley, P. H. (1974). A review of future directions. The Counseling
(1987). Expectancy and outcome counseling analogue research meth- Psychologist, 33, 413–418.
in prescription vs. exploratory psy- ods. Journal of Counseling Psy- Neville, H. A., Heppner, P. P., Ji, P., &
chotherapy. British Journal of chology, 21, 320–330. Thye, R. (2004). The relations
Clinical Psychology, 29, 54–60. Munley, P. H., Sharkin, B., & Gelso, among general and race-related
Morrow, S. L. (2005). Quality and C. J. (1988). Reviewer ratings and stressors and psychoeducational
trustworthiness in qualitative re- agreement on manuscripts reviewed adjustment in black students at-
search in counseling psychology. for the Journal of Counseling tending predominately white insti-
Journal of Counseling Psychology, Psychology. Journal of Counseling tutions. Journal of Black Studies,
52(2), 250–260. Psychology, 35, 198–202. 34, 599–618.
Morrow, S. L., Rakhasha, G., & Myers, J. E., & Sweeney, T. J. (2004). Neville, H. A., Lilly, R. L., Duran, G.,
Castañeda, C. L. (2001). Qual- Advocacy for the counseling pro- Lee, R. M., & Browne, L. (2000).
itative research methods for multi- fession: Results of a national Construction and initial validation
cultural counseling. In J. G. survey. Journal of Counseling and of the Color-Blind Racial Attitudes
Ponterotto, J. M. Casas, L. A. Development, 82, 466–471. Scale (CoBRAS). Journal of Coun-
Suzuki, & C. M. Alexander (Eds.), Nagel, D. P., Hoffman, M. A., & Hill, seling Psychology, 47, 59–70.
Handbook of multicultural coun- C. E. (1995). A comparison of verbal Newman, D., & Brown, R. (1996).
seling (2nd ed., pp. 575–603). response modes used by master’s Applied ethics for program evalua-
Thousand Oaks, CA: Sage. level career counselors and other tion. Beverly Hills, CA: Sage.
Morrow, S. L., & Smith, M. L. (1995). helpers. Journal of Counseling and Newman, I., & Benz, C. R. (1998).
Constructions of survivals and Development, 74, 101–104. Qualitative-quantitative research
coping by women who have survived Nagelberg, D. B., Pillsbury, E. C., & methodology: Exploring the in-
childhood sexual abuse. Journal of Balzor, D. M. (1983). The preva- teractive continuum. Carbondale:
Counseling Psychology, 42, 24–33. lence of depression as a function of University of Illinois Press.
Morrow, S. L., & Smith, M. L. gender and facility usage in college Nghe, L. T., & Mahalik, J. R. (2001).
(2000). Qualitative research for students. Journal of College Stu- Examining racial identity statuses
counseling psychology. In S. D. dent Personnel, 24, 525–529. as predictors of psychological de-
Brown & R. W. Lent (Eds.), National Institutes of Health. (1994). fenses in African American college
Handbook of counseling psychology NIH guidelines on the inclusion students. Journal of Counseling
(3rd ed., 199–230). New York: of women and minorities as sub- Psychology, 48, 10–16.
John Wiley & Sons. jects in clinical research (59 FR Nickerson, R. S. (2000). Null hypothesis
Moustakas, C. (1994). Phenomenological 14508–14513). Washington, DC: significance testing: A review of
research methods. Thousand Oaks, U.S. Department of Health and an old and continuing controversy.
CA: Sage. Human Services. Psychological Methods, 5, 241–301.
Mueller, R. O. (1996). Basic princi- Neimeyer, G., & Resnikoff, A. (1982). Nisbett, R. E., & Ross, L. (1980).
ples of structural equation model- Major contribution: Qualitative Human inference: Strategies and
ing: An introduction to LISREL strategies in counseling research. shortcomings of social judgment.
and EQS. New York: Springer. The Counseling Psychologist, 101(4), Englewood Cliffs, NJ: Prentice-Hall.
Mueller, W. J. (1969). Patterns of 75–85. Nisbett, R. E., & Wilson, T. D.
behavior and their reciprocal Neimeyer, G. J., Saferstein, J., & Rice, (1977). Telling more than we can
impact in the family and in psy- K. G. (2005). Does the model know: Verbal reports on mental
chotherapy [monograph]. Journal matter? The relationship between processes. Psychological Review,
of Counseling Psychology, 16, 1–25. science-practice emphasis and 84, 231–259.
644 references
Nocita, A., & Stiles, W. B. (1986). Educational Psychology, Counsel- Oyserman, D., Coon, H. M., & Kem-
Client introversion and counseling ing Psychology Program. (ERIC melmeier, M. (2002). Rethink-
session impact. Journal of Counseling Document Reproduction Service ing individualism and collectivism:
Psychology, 33, 235–241. No. ED275963). Evaluation of theoretical assump-
Noonan, B. M., Gallor, S. M., O’Neil, J. M., & Roberts Carroll, M. tions and meta-analyses. Psycho-
Hensler-McGinnis, N. F., Fassinger, (1988). A gender role workshop logical Bulletin, 128, 3–72.
R. E., Wang, S., & Goodman, J. focused on sexism, gender role Paivio, S. C., & Greenberg, L. S.
(2004). Challenge and success: A conflict, and the gender role jour- (1995). Resolving “unfinished
qualitative study of the career ney. Journal of Counseling and business”: Efficacy of experiential
development of highly achieving Development, 67, 193–197. therapy using empty-chair dia-
women with physical and sensory O’Neill, J., Small, B. B., & Strachan, logue. Journal of Consulting and
disabilities. Journal of Counseling J. (1999). The use of focus groups Clinical Psychology, 63, 419–425.
Psychology, 51, 68–80. within a participatory action Parham, T. A. (1989). Cycles of psy-
Norcross, J. C. (2001). Purposes, research environment. In M. chological nigresence. The Coun-
processes, and products of the Kopala & L. A. Suzuki (Eds.), seling Psychologist, 17, 187–226.
Task Force on Empirically Sup- Using qualitative methods in psy- Parham, T. A., & Helms, J. E. (1981).
ported Therapy Relationships. chology (pp. 199–209). Thousand The influence of black students’
Psychotherapy: Theory/Research/ Oaks, CA: Sage. racial identity attitudes on prefer-
Practice/Training, 38, 345–356. Orlinsky, D. E., Grawe, K., & Parks, ence for counselor’s race. Journal of
Norcross, J. C., Beutler, L. E., & B. K. (1994). Process and outcome Counseling Psychology, 28, 250–257.
Levant, R. F. (Eds.). (2006). in psychotherapy: Noch einmal. Parham, T. A., & Helms, J. E. (1985a).
Evidence-based practices in men- In A. E. Bergin & S. L. Garfield Attitudes of racial identity and self-
tal health: Debate and dialogue (Eds.), Handbook of psychother- esteem of black students: An
on the fundamental questions. apy and behavior change (4th ed., exploratory investigation. Journal of
Washington, DC: American Psy- pp. 270–376). New York: Wiley. College Student Personnel, 26,
chological Association. Orlinsky, D. E., & Howard, K. I. 143–147.
Norton, I. M., & Manson, S. M. (1975). Varieties of psychothera- Parham, T. A., & Helms, J. E. (1985b).
(1996). Research in American peutic experience: Multivariate The relationship of racial identity
Indian and Alaska Native com- analyses of patients’ and thera- attitudes to self-actualization and
munities: Navigating the cultural pists’ reports. New York: Teachers affective states of black students.
universe of values and process. College Press. Journal of Counseling Psychology,
Journal of Consulting and Clinical Orlinsky, D. E., & Howard, K. I. 32, 431–440.
Psychology, 64, 856–860. (1978). The relation of process to Parloff, M. B., Waskow, I. E., &
Nunnally, J. C. (1978). Psychometric outcome in psychotherapy. In S. L. Wolfe, B. E. (1978). Research on
theory. New York: McGraw-Hill. Garfield & A. E. Bergin (Eds.), therapist variables in relation to
O’Farrell, M. K., Hill, C. E., & Handbook of psychotherapy and process and outcome. In S. L.
Patton, S. M. (1986). A comparison behavior change (2nd ed., pp. Garfield & A. E. Bergin (Eds.),
of two cases of counseling with the 283–330). New York: Wiley. Handbook of psychotherapy and
same counselor. Journal of Counseling Orlinsky, D. E., & Howard, K. I. behavior change (2nd ed., pp.
and Development, 65, 141–145. (1986). The relation of process to 233–282). New York: Wiley.
Ogles, B. M., Lambert, M. J., & outcome in psychotherapy. In S. L. Parr, J., & Neimeyer, G. J. (1994).
Fields, S. A. (2002). Essentials of out- Garfield & A. E. Bergin (Eds.), Effects of gender, construct type,
come assessment. New York: Wiley. Handbook of psychotherapy and occupational information, and
Okazaki, S., & Kallivayalil, D. behavior change (3rd ed., pp. career relevance on vocational dif-
(2002). Cultural norms and sub- 311–381). New York: Wiley. ferentiation. Journal of Counseling
jective disability as predictors of Orlinsky, D. E., Ronnestad, M. H., & Psychology, 41, 27–33.
symptom reports among Asian Willutzki, U. (2004). Fifty years of Patten, M. L. (2001). Questionnaire
Americans and White Americans. psychotherapy process-outcome research: A practical guide (2nd
Journal of Cross-Cultural Psychology, research: Continuity and change. ed.). Los Angeles, CA: Pyrczak.
33, 482–491. In M. J. Lambert (Ed.), Bergin and Patterson, G. R., & Forgatch, M. S.
O’Neil, J. M., Helms, B., Gable, R., Garfield’s handbook of psychother- (1985). Therapist behavior as a
David, L., & Wrightsman, L. apy and behavior change (5th ed., determinant for client noncompli-
(1986). Gender Role Conflict pp. 307–389). New York: Wiley. ance: A paradox for the behavior
Scale: College men’s fear of femi- Osipow, S. H. (1979). Counseling modifier. Journal of Consulting
ninity. Sex Roles, 14, 335–350. researchers: Why they perish. The and Clinical Psychology, 53,
O’Neil, J. M., & Roberts Carroll, M. Counseling Psychologist, 8, 39–41. 846–851.
(1987). A six-day workshop on Osipow, S. H., Carney, C. G., Winer, Patton, M. J. (1984). Managing social
gender role conflict and strain: Helping J., Yanico, B., & Koschier, M. interaction in counseling: A contri-
adult men and women take the gender (1976). Career Decision Scale (3rd bution from the philosophy of
role journey. Storrs, CT: University rev.). Odessa, FL: Psychological science. Journal of Counseling Psy-
of Connecticut, Department of Assessment Resources. chology, 31, 442–456.
references 645
Patton, M. J., Kivlighan, D. M., Jr., Penman, R. (1980). Communication Phinney, J. (1996). When we talk
& Multon, K. D. (1997). The processes and relationships. Lon- about American ethnic groups,
Missouri psychoanalytic counsel- don: Academic Press. what do we mean? American Psy-
ing research project: Relation of Pepinsky, H. B. (1984). Language and chologist, 51, 98–927.
changes in counseling process the production and interpretation Pidgeon, N. (1996). Grounded theory:
to client outcome. Journal of of social interactions. In H. F. Theoretical background. In J. T. E.
Counseling Psychology, 44, 189–208. Fisher (Ed.), Language and logic Richardson (Ed.), Handbook of
Patton, M. Q. (1987). How to use in personality and society (pp. qualitative research methods for
qualitative methods in evaluation. 93–129). New York: Columbia psychology and the social sciences
Newbury Park, CA: Sage. University Press. (pp. 75–85). Leicester, UK: British
Patton, M. Q. (1990). Qualitative Pepinsky, H. B., & Pepinsky, P. N. Psychological Society Books.
evaluation and research methods (1954). Counseling theory and Pidgeon, N., & Henwood, K. (1996).
(2nd ed.). Newbury Park, CA: Sage. practice. New York: Ronald Press. Grounded theory: Practical imple-
Patton, M. Q. (1997). Utilization- Petersen, P. B., Draguns, J. G., Lonner, mentation. In J. T. E. Richardson
focused evaluation: The new cen- W. J. & Trimble, J. E. (in press) (Ed.), Handbook of qualitative
tury text (3rd ed.). Thousand (Eds.) Counseling across cultures research methods for psychology
Oaks, CA: Sage. (sixth ed.) Thousand Oaks, CA: Sage. and the social sciences (pp. 86–
Patton, M. Q. (2002). Qualitative Pett, M. A., Lackey, N. R., & 101). Leicester, UK: British Psychol-
evaluation and research methods Sullivan, J. J. (2003). Making ogical Society Books.
(3rd ed.). Thousand Oaks, CA: Sage. sense of factor analysis: The use of Plummer, D. L., & Slane, S. (1996).
Paul, G. L. (1967). Strategy of out- factor analysis for instrument devel- Patterns of coping in racially
come research in psychotherapy. opment in health care research. stressful situations. Journal of
Journal of Consulting Psychology, Thousand Oaks, CA: Sage. Black Psychology, 22, 89–95.
31, 109–118. Petty, R. E., & Cacioppo, J. T. (1977). Polkinghorne, D. (1983). Methodology
Paulhus, D. L. (1984). Two-compo- Forewarning, cognitive respond- for the human sciences: Systems of
nent models of socially desirable ing, and resistance to persuasion. inquiry. Albany: State University
responding. Journal of Personality Journal of Personality and a Social of New York.
and Social Psychology, 46, 598–609. Psychology, 35, 645–655. Polkinghorne, D. E. (1984). Further
Paulhus, D. L. (1991). Balanced in- Petty, R. E., & Cacioppo, J. T. (1981). extensions of methodological di-
ventory of desirable responding. In Attitudes and persuasion: Classic versity for counseling psychology.
J. P. Robinson, P. R. Shaver, & L. and contemporary approaches. Journal of Counseling Psychology,
S. Wrightsman (Eds.), Measures of Dubuque, IA: William C. Brown. 31, 416–429.
personality and social psychologi- Petty, R. E., & Cacioppo, J. T. (1986). Polkinghorne, D. E. (2005). Language
cal attitudes (pp. 37–41). San Communication and persuasion: and meaning, data collection in
Diego: Academic Press. Central and peripheral routes to atti- qualitative research. Journal of
Pedersen, P. (1988). A handbook for tude change. New York: Springer- Counseling Psychology, 52,
developing multicultural aware- Verlag. 137–145.
ness. Alexandria, VA: American Pfungst, O. (1911). A contribution to Ponce, F. Q., & Atkinson, D. R.
Association for Counseling and experimental, animal, and human (1989). Mexican-American accul-
Development. psychology. New York: Holt, turation, counselor ethnicity,
Pedersen, P. B. (1994). A handbook Rinehart & Winston. counseling style, and perceived
for developing multicultural aware- Phelps, A., Friedlander, M. L., & credibility. Journal of Counseling
ness (2nd ed.). Alexandria, VA: Enns, C. Z. (1997). Psychotherapy Psychology, 36, 203–208.
American Counseling Association. process variables associated with Ponterotto, J. G. (1988a). Racial con-
Pedersen, P. B. (1995). Culture-centered the retrieval of memories of child- sciousness development among
ethical guidelines for counselors. In hood sexual abuse: A qualitative white counselor trainees: A stage
J. G Ponterotto, J. M. Casas, L. A. study. Journal of Counseling Psy- model. Journal of Multicultural
Suzuki, & C. M. Alexander (Eds.), chology, 44, 321–332. Counseling and Development, 16,
Handbook of multicultural counsel- Phillips, J. C., & Russell, R. K. 146–156.
ing (pp. 34–49). Thousand Oaks, (1994). Research self-efficacy, the Ponterotto, J. G. (1988b). Racial/
CA: Sage. research training environment, ethnic minority research in the
Pedersen, P. B., Draguns, J. G., and research productivity among Journal of Counseling Psychology:
Lonner, W. J., & Trimble, J. E. graduate students in counseling A content analysis and methodo-
(Eds.). (1996). Counseling across psychology. The Counseling Psy- logical critique. Journal of Coun-
cultures (4th ed.). Thousand Oaks, chologist, 22, 628–641. seling Psychology, 35, 410–418.
CA: Sage. Phillips, S. D., Friedlander, M. L., Ponterotto, J. G. (1998). Charting
Pedhazur, E. (1982). Multiple regres- Pazienza, N. J., & Kost, P. P. a course for research in multi-
sion in behavioral research: Ex- (1985). A factor analytic investiga- cultural counseling training. The
planation and prediction (2nd tion of career decision-making Counseling Psychologist, 43, 26–68.
ed.). New York: Holt, Rinehart & styles. Journal of Vocational Be- Ponterotto, J. G. (2005a). Integrating
Winston. havior, 26, 106–115. qualitative research requirements
646 references
into professional psychology train- Journal of Counseling and Devel- Rappaport, D. (1960). The structure
ing programs in North America: opment, 72, 316–322. of psychoanalytic theory. Psycho-
Rationale and curriculum model. Porter, A. C., & Raudenbush, S. W. logical Issues, Monograph 6. New
Qualitative Research in Psychology, (1987). Analysis of covariance: Its York: International Universities
2, 97–116. model and use in psychological Press.
Ponterotto, J. G. (2005b). Qualitative research. Journal of Counseling Raudenbush, S. W., & Bryk, A. S. (2002).
research in counseling psychology: Psychology, 34, 383–392. Hierarchical linear models (2nd
A primer on research paradigms Powell, C. J. (1984). Ethical principles ed.). Thousand Oaks, CA: Sage.
and philosophy of science. Journal and issues of competence in coun- Raudenbush, S. W., & Chan, W. S.
of Counseling Psychology, 52, seling adolescents. The Counseling (1993). Application of a hierarchi-
126–136. Psychologist, 121(3), 57–68. cal linear model to the study of
Ponterotto, J. G., & Casas, J. M. Pressley, M., Raphael, L., Gallagher, adolescent deviance in an overlap-
(1991). Handbook of racial/ethnic J. D., & DiBella, J. (2004). ping cohort design. Journal of
minority counseling research. Providence-St. Mel school: How a Consulting and Clinical Psychology,
Springfield, IL: Charles C Thomas. school that works for African 61, 941–951.
Ponterotto, J. G., Casas, J. M., American students works. Journal Ravets, P. C. (1993). Group supervi-
Suzuki, L. A., & Alexander, C. A. of Educational Psychology, 96, sion: A multiple case study. Dis-
(Eds.). (1995). Handbook of mul- 216–235. sertation Abstracts International,
ticultural counseling. Thousand Price, D. J. (1963). Little science, big 54, 2768.
Oaks, CA: Sage. science. New York: Columbia Reising, G. N., & Daniels, M. H.
Ponterotto, J. G., Casas, J. M., University Press. (1983). A study of Hogan’s model
Suzuki, L. A., & Alexander, C. A. Purdon, C., Antony, M. M., & of counselor development and
(Eds.). (2001). Handbook of mul- Swinson, R. P. (1999). Psychometric supervision. Journal of Counseling
ticultural counseling. (2nd ed.). properties of the Frost Multidi- Psychology, 30, 235–244.
Thousand Oaks, CA: Sage. mensional Perfectionism Scale in a Rennie, D. L. (1994a). Clients’ ac-
Ponterotto, J. G., Fuertes, J. N., & clinical anxiety disorders sample. counts of resistance in counsel-
Chen, E. C. (2000). Models of Journal of Clinical Psychology, 55, ing: A qualitative analysis. Cana-
multicultural counseling. In S. D. 1271–1286. dian Journal of Counselling, 28,
Brown & R. W. Lent (Eds.), Putman, H. (1962). What theories are 43–57.
Handbook of counseling psychol- not. In E. P. Nagel, P. Suppes, & Rennie, D. L. (1994b). Client’s def-
ogy (3rd ed., pp. 639–669). New A. Tarski (Eds.), Logic, methodol- erence in psychotherapy. Journal
York: Wiley. ogy, and philosophy of science: of Counseling Psychology, 41,
Ponterotto, J. G., & Furlong, M. J. Proceedings of the 1960 interna- 427–437.
(1985). Evaluating counselor ef- tional congress (pp. 240–251). Rennie, D. L. (1994c). Storytelling on
fectiveness: A critical review of Stanford, CA: Stanford University psychotherapy: The client’s subjec-
rating scale instruments. Journal Press. tive experience. Psychotherapy,
of Counseling Psychology, 32, Quintana, S. M., Troyano, N., & 31, 234–243.
597–616. Gaylor, G. (2001). Cultural valid- Rennie, D. L. (1996). Fifteen years of
Ponterotto, J. G., & Grieger, I. (1999). ity and inherent challenges in doing qualitative research on psy-
Merging qualitative and quantita- quantitative methods for multi- chotherapy. British Journal of
tive perspectives in research iden- cultural research. In J. G. Ponte- Guidance & Counselling, 24(3),
tity. In M. Kopala & L. Suzuki rotto, J. M. Casas, L. A. Suzuki, & 317–327.
(Eds.), Using qualitative meth- C. A. Alexander (Eds.), Handbook Rennie, D. L. (2000). Grounded
ods in psychology (pp. 49–62). of multicultural counseling (pp. theory methodology as methodical
Thousand Oaks, CA: Sage. 604–630). Thousand Oaks, CA: hermeneutics. Theory & Psychology,
Ponterotto, J. G., & Grieger, I. (in Sage. 10(4), 481–502.
press). Guidelines and competen- Rabinowitz, F. E., Heppner, P. P., & Rennie, D. L., Watson, K. D., &
cies for cross-cultural counseling Roehlke, H. J. (1986). Descriptive Monteiro, A. M. (2002). The rise of
research. In P. B. Pedersen, J. G. study of process and outcome vari- qualitative research in psychology.
Draguns, W. J. Lonner, & J. E. ables of supervision over time. Canadian Psychology, 43, 179–189.
Trimble (Eds.), Counseling across Journal of Counseling Psychology, Resnikoff, A. (1978). Scientific affairs
cultures (6th ed.). Thousand Oaks, 33, 292–300. committee report, 1975–1977: A
CA: Sage. Rachman, S. L., & Wilson, G. T. discussion of methodology. The
Ponterotto, J. G., & Pedersen, P. B. (1980). The effects of psychologi- Counseling Psychologist, 71(4),
(1993). Preventing prejudice: A cal therapy. New York: Pergamon 67–71.
guide for counselors and educa- Press. Revicki, D. A., May, H. J., & Whitley,
tors. Newbury Park, CA: Sage. Raimy, V. (Ed.). (1950). Training in T. W. (1990). Reliability and valid-
Ponterotto, J. G., Rieger, B. P., Barrett, clinical psychology. New York: ity of the Work Related Strain
A., & Sparks, R. (1994). Assessing Prentice-Hall. Inventory among health profes-
multicultural counseling compe- Ramsey, P. (1970). The patient as person. sionals. Behavioral Medicine, 17,
tence: A review of instrumentation. New Haven, CT: Yale University. 111–120.
references 647
Reynolds, W. M. (1982). Development Rogers, C. R. (1955). Persons or sci- Artifacts and ethics in behavioral
of reliable and valid short forms of ence? A philosophical question. research. New York: Freeman.
the Marlowe-Crowne Social Desir- American Psychologist, 10, 267–278. Ross, R. R., & Altmaier, E. M.
ability Scale. Journal of Clinical Rogers, C. R. (1957). The necessary (1990). Job analysis of psychology
Psychology, 38, 119–125. and sufficient conditions of thera- internships in counseling center
Rice, L., & Kerr, G. (1986). Measures peutic personality change. Journal of settings. Journal of Counseling
of client and therapist vocal qual- Consulting Psychology, 21, 95–103. Psychology, 37, 459–464.
ity. In L. Greenberg & W. Pinsof Rogers, C. R. (1961). On becoming a Ross, W. D. (1930). The right and the
(Eds.), The psychotherapy process: A person. Boston: Houghton Mifflin. good. Oxford, England: Clarendon.
research handbook (pp. 73–106). Rogers, L. B. (1954). A comparison of Rossi, P., & Freeman, H. (1999).
New York: Guilford Press. two kinds of test interpretation Evaluation: A systematic approach.
Rice, L. N., Koke, L. J., Greenberg, interview. Journal of Counseling Thousand Oaks, CA: Sage.
L. S., & Wagstaff, A. K. (1979). Psychology, 1, 224–231. Rowe, D. C., Vazsonyi, A. T., &
Manual for client vocal quality Rojewski, J. W. (1994). Career indeci- Flannery, D. J. (1994). No more
(Vol. 1). Toronto: Counseling and sion types for rural adolescents than skin deep: Ethnic and racial
Development Center, York University. from disadvantaged and nondisad- similarity in developmental process.
Richardson, M. S., & Johnson, M. vantaged backgrounds. Journal of Psychological Review, 101, 396–413.
(1984). Counseling women. In Counseling Psychology, 41, 356–363. Rowe, W., & Atkinson, D. R. (1995).
S. R. Brown & R. W. Lent (Eds.), Rokeach, M. (1973). The nature of Misrepresentation and interpreta-
Handbook of counseling psychology human values. New York: Free Press. tion: Critical evaluation of white
(pp. 832–877). New York: Wiley. Romano, J. L., & Kachgal, M. M. racial identity development models.
Richie, B. S., Fassinger, R. E., Linn, (2004). Counseling psychology The Counseling Psychologist, 23,
S. G., Johnson, J., Prosser, J., & and school counseling: An under- 364–367.
Robinson, S. (1997). Persistence, utilized partnership. The Counseling Rowe, W., Bennett, S. K., & Atkinson,
connection, and passion: A quali- Psychologist, 32, 184–215. D. R. (1994). White racial identity
tative study of the career develop- Rosenbaum, M. (1982). Ethical prob- models: A critique and alternative
ment of highly achieving African lems of group psychotherapy. In proposal. The Counseling Psychol-
American-Black and White women. M. Rosenbaum (Ed.), Ethics and ogist, 22, 129–146.
Journal of Counseling Psychology, values in psychotherapy (pp. Royalty, G. M., Gelso, C. J., Mal-
44, 133–148. 237–257). New York: Free Press. linckrodt, B., & Garrett, K. D.
Ridley, C. R. (1995). Overcoming Rosenberg, M. (1965). Society and the (1986). The environment and the
unintentional racism in counseling adolescent self-image. Princeton, student in counseling psychology:
and therapy: A practitioner’s guide NJ: Princeton University Press. Does the research training environ-
to intentional intervention. Thousand Rosenthal, D., & Frank, J. D. (1956). ment influence graduate students’
Oaks, CA: Sage. Psychotherapy and the placebo attitude toward research? The
Ridley, C. R., Mendoza, D. W., & effect. Psychological Bulletin, 53, Counseling Psychologist, 14, 9–30.
Kanitz, B. E. (1994). Multicul- 294–302. Royalty, G. M., & Magoon, T. M.
tural training: Reexamination, Rosenthal, R. (1966). Experimenter (1985). Correlates of scholarly
operationalization, and integration. effects in behavioral research. productivity among counseling
The Counseling Psychologist, 22, New York: Meredith. psychologists. Journal of Counseling
227–289. Rosenthal, R. (1995). Writing meta- Psychology, 32, 458–461.
Ridley, C. R., Mendoza, D. W., analytic reviews. Psychological Royalty, G. M., & Reising, G. N.
Kanitz, B. E., Angermeier, L., & Bulletin, 118, 183–192. (1986). The research training of
Zenk, R. (1994). Cultural sensitiv- Rosenthal, R., & Jacobson, L. (1968). counseling psychologists: What the
ity in multicultural counseling: A Pygmalion in the classroom: professionals say. The Counseling
perceptual schema model. Journal Teacher expectations of the dis- Psychologist, 14, 49–60.
of Counseling Psychology, 41, advantaged. New York: Holt, Rubinton, N. (1980). Instruction in
125–136. Rinehart & Winston. career decision making and deci-
Riggio, R. E. (1989). Social Skills Rosenthal, R., & Rosnow, R. L. sion-making styles. Journal of
Inventory manual (research ed.). Palo (1969). The volunteer subject. In Counseling Psychology, 27, 581–588.
Alto, CA: Consulting Psychologists R. Rosenthal & R. L. Rosnow Ruch, F. L., & Zimbardo, P. G.
Press. (Eds.), Artifact in behavioral re- (1970). Psychology and life (8th
Robinson, F. P. (1950). Principles and search (pp. 61–118). New York: ed.). Glenview, IL: Scott, Foresman.
procedures in student counseling. Academic Press. Rumenik, D. K., Capasso, D. R., &
New York: Harper & Brothers. Rosnow, R. L., & Rosenthal, R. Hendrick, C. (1977). Experimenter
Robinson, T. L., & Howard- (1988). Focused tests of signifi- sex effects in behavioral research.
Hamilton, M. F. (2000). The con- cance and effect size estimation in Psychological Bulletin, 84, 852–877.
vergence of race, ethnicity, and counseling psychology. Journal of Rushton, J. P. (2000). Race, evolution,
gender: Multiple identities in Counseling Psychology, 35, 203–208. and behavior: A life history per-
counseling. Upper Saddle River, Rosnow, R. L., & Rosenthal, R. spective. Port Huron, MI: Charles
NJ: Prentice-Hall. (1997). People studying people: Darwin Research Institute.
648 references
Russel, D. E. (1986). The secrete Schmidt, L. D., & Strong, S. R. (1971). Serlin, R. C., Wampold, B. E., &
trauma: Incest in the lives of the Attractiveness and influence in Levin, J. R. (2003). Should pro-
girls and women. New York: Basic counseling. Journal of Counseling viders of treatment be regarded as
Books. Psychology, 18, 348–351. a random factor? If it ain’t broke,
Russell, R. K., & Lent, R. W. (1982). Schneidler, G. G., & Berdie, R. F. don’t “fix” it: A comment on
Cue-controlled relaxation and sys- (1942). Representativeness of col- Siemer and Joorman (2003). Psy-
tematic desensitization versus non- lege students who receive counseling chological Methods, 8, 524–534.
specific factors in treating test services. Journal of Educational Shadish, W. R., Cook, T. D., &
anxiety. Journal of Counseling Psychology, 33, 545–551. Campbell, D. T. (2002). Experimental
Psychology, 29, 100–103. Schofield, J. W. (1986). Causes and and quasi-experimental designs
Russell, R. K., & Sipich, J. F. (1973). consequences of the colorblind for generalized causal inference.
Cue-controlled relaxation in the perspective. In J. F. Dovidio & S. L. Boston: Houghton Mifflin.
treatment of test anxiety. Journal Gaertner (Eds.), Prejudice, discrim- Shaw, B. F., Elkin, I., Yamaguchi, J.,
of Behavior Therapy and Exper- ination, and racism (pp. 231–253). Olmsted, M., Vallis, T. M., Dobson,
imental Psychiatry, 4, 47–50. New York: Academic Press. K. S., et al., (1999). Therapist
Russell, R. L. (1987). Language in Schotte, D., & Clum, G. (1982). competence ratings in relation to
psychotherapy: Strategies of dis- Suicide ideation in a college popu- clinical outcome in cognitive ther-
covery. New York: Plenum. lation: A test of a model. Journal apy of depression. Journal of
Russell, R. L., & Stiles, W. B. (1979). of Consulting and Clinical Psy- Consulting and Clinical Psychol-
Categories for classifying language chology, 50, 690–696. ogy, 67, 837–846.
in psychotherapy. Psychological Schuller, R., Crits-Christoph, P., & Shlien, J. M., Mosak, H. H., &
Bulletin, 86, 404–419. Connolly, M. B. (1991). The Re- Dreikurs, R. (1962). Effects of
Rust, R. E., & Davie, J. S. (1961). The sistance Scale: Background and time limits: A comparison of two
personal problems of college students. psychometric properties. Psycho- psychotherapies. Journal of Coun-
Mental Hygiene, 45, 247–257. analytic Psychology, 8, 195–211. seling Psychology, 9, 31–34.
Sabnani, H. B., & Ponterotto, J. G. Schutz, A. (1964). Collected papers II: Silberschatz, G., Fretter, P. B., &
(1992). Racial/ethnic minority- Studies in social theory (A. Curtis, J. T. (1986). How do inter-
specific instrumentation in coun- Broderson, Ed.). The Hague, The pretations influence the process of
seling research: A review, critique, Netherlands: Martinus-Nijhoff. therapy? Journal of Consulting and
and recommendation. Measurement Schwandt, T. A. (2000). Three episte- Clinical Psychology, 54, 646–652.
& Evaluation in Counseling & mological stances for qualitative Sipich, J. F., Russell, R. K., & Tobias,
Development, 24, 161–187. inquiry: interpretivism, hermeneu- L. L. (1974). A comparison of
Salsman, N. L., & Linehan, M. M. tics, and social constructionism. In covert sensitization and “nonspe-
(2006). Dialectic-behavioral therapy N. K. Denzin & Y. S. Lincoln cific” treatment in the modification
for borderline personality disorder. (Eds.), Handbook of qualitative of smoking behavior. Journal of
Primary Psychiatry, 13, 51–58. research (pp. 189–213). Thousand Behavior, Therapy, and Experimental
Sax, G. (1989). Principles of educa- Oaks, CA: Sage. Psychiatry, 5, 201–203.
tional and psychological measure- Scriven, M. (1980). The logic of eval- Slaney, R. B., Mobley, M., Trippi, J.,
ment and evaluation (3rd ed.). uation. Inverness, CA: Edgepress. Ashby, J., & Johnson, D. G.
Belmont, CA: Wadsworth. Seeman, J. (1959). Organizing a thesis (1996). The Almost Perfect
Schechtman, Z., Gilat, I., Fos, L., & proposal. American Psychologist, Scale—Revised. Unpublished man-
Flasher, A. (1996). Brief group ther- 9, 794–797. uscript, Pennsylvania State Univer-
apy with low-achieving elementary Seeman, J. (1969). Deception in psy- sity, University Park campus.
school children. Journal of Counseling chological research. American Psy- Slaney, R. B., Rice, K. G., Mobley, M.
Psychology, 43, 376–382. chologist, 24, 1025–1028. Trippi, J., & Ashby, J. S. (2001).
Scher, M., & Good, G. E. (1990). Seggar, L., & Lambert, M. J. (1994, The Revised Almost Perfect Scale,
Gender and counseling in the June). Assessing clinical signifi- Measurement and Evaluation in
twenty-first century: What does the cance: Application to the Beck Counseling and Development, 34,
future hold? Journal of Counseling Depression Inventory. Paper pre- 130–145.
and Development, 68, 388–391. sented at the annual meeting of the Slate, J. R., & Jones, C. H. (1989).
Schmidt, F. L., & Hunter, J. E. (1996). Society for Psychotherapy Research, Can teaching of the WISC-R be
Measurement error in psychologi- York, England. improved? Quasi-experimental ex-
cal research: Lessons from 26 Serlin, R. C. (1987). Hypothesis test- ploration. Professional Psychology
research scenarios. Psychological ing, theory building, and the phi- Research and Practice, 20, 408–410.
Methods, 1, 199–223. losophy of science. Journal of Slusher, M. P., & Anderson, C. A. (1989).
Schmidt, L. D., & Meara, N. M. (1984). Counseling Psychology, 34, 365– Belief perseverance and self-
Ethical, professional, and legal 371. defeating behavior. In R. Curtis
issues in counseling psychology. In Serlin, R. C., & Lapsley, D. K. (1985). (Ed.), Self-defeating behaviors:
S. D. Brown & R. W. Lent (Eds.), Rationality in psychological re- Experimental research, clinical im-
Handbook of counseling psychology search: The good enough principle. pressions, and practical implications
(pp. 56–96). New York: Wiley. American Psychologist, 40, 73–83. (pp. 11–40). New York: Plenum.
references 649
Smith, M. L., & Glass, G. V. (1977). Spitzer, R. L., Gibbon, M., Skodol, of Counseling Psychology, 31,
Meta-analysis of psychotherapy A. E., Williams, J. B. W., & First, 108–110.
outcome studies. American Psy- M. B. (Eds.). (1994). DSM-IV case Strauss, A., & Corbin, J. (1990).
chologist, 32, 752–760. book. Washington, DC: American Basics of qualitative research:
Smith, M. L., Glass, G. V., & Miller, Psychiatric Press. Grounded theory procedures and
T. I. (1980). The benefits of psy- Stanton, J. M. (1998). An assessment techniques. Newbury Park, CA: Sage.
chotherapy. Baltimore: Johns Hop- of data collection using the internet. Strauss, A. L., & Corbin, J. (1998).
kins University Press. Personnel Psychology, 51, 709–725. Basics of qualitative research:
Smith, N. (Ed.). (1982). Communication Sternberg, R. J. (1988). The psycholo- Techniques and procedures for
strategies in evaluation. Beverly gist’s companion: A guide to sci- developing grounded theory (2nd
Hills, CA: Sage. entific writing for students and ed.). Thousand Oaks, CA: Sage.
Smith, N. G., & Ingram, K. M. researchers (2nd ed.). New York: Stricker, G. (1982). Ethical issues in
(2004). Workplace heterosexism Cambridge University Press. psychotherapy research. In M.
and adjustment among lesbian, Stewart, G. M., & Gregory, B. C. (1996). Rosenbaum (Ed.), Ethics and
gay, and bisexual individuals: The Themes of a long-term AIDS sup- values in psychotherapy: A guide-
role of unsupportive social interac- port group for gay men. The book (pp. 403–424). New York:
tion. Journal of Counseling Psy- Counseling Psychologist, 24, 285–303. Free Press.
chology, 51, 57–67. Stiles, W. B. (1980). Measurement of Strickland, O. L., Maloney, M. F.,
Smith, R. E., & Nye, S. L. (1989). the impact of psychotherapy ses- Dietrich, A. S., Myerburg, S.,
Comparison of induced effect and sions. Journal of Consulting and Cotsonis, G. A., & Johnson, R. V.
covert rehearsal in the acquisition Clinical Psychology, 48, 176–185. (2003). Measurement issues re-
of stress management coping Stiles, W. B. (1988). Psychotherapy lated to data collection on the
skills. Journal of Counseling Psy- process-outcome correlations may World Wide Web. Advances in
chology, 36, 17–23. be misleading. Psychotherapy, 25, Nurses Science, 26, 246–256.
Snijders, T., & Bosker, R. (1999). 27–35. Strohmer, D. C., & Blustein, D. L.
Multilevel analysis: An introduc- Stiles, W. B., Shapiro, D. A., & Firth- (1990). The adult problem solver
tion to basic and advanced multi- Cozens, J. A. (1988). Do sessions as personal scientist. Journal of
level modeling. London: Sage. of different treatments have different Cognitive Psychotherapy: An Inter-
Snyder, D., & Wills, R. M. (1989). impacts? Journal of Counseling national Quarterly, 4, 281–292.
Behavioral versus insight-oriented Psychology, 35, 391–396. Strohmer, D. C., & Newman, L. J.
marital therapy: Effects on indi- Stiles, W. B., & Snow, J. S. (1984). (1983). Counselor hypothesis-testing
vidual and interspousal function- Counseling session impact as strategies. Journal of Counseling
ing. Journal of Consulting and viewed by novice counselors and Psychology, 30, 557–565.
Clinical Psychology, 57, 39–46. their clients. Journal of Counseling Strong, S. R. (1968). Counseling: An
Snyder, D. K., Wills, R. M., & Grady- Psychology, 31, 13–21. interpersonal influence process.
Fletcher, A. (1991). Long-term Stiles, W. B., Startup, M., Hardy, Journal of Counseling Psychology,
effectiveness of behavioral versus G. E., Barkham, M., Rees, A., 15, 215–224.
insight oriented therapy: A 4-year Shapiro, D. A., & Reynolds, S. Strong, S. R. (1971). Experimental
follow-up. Journal of Consulting and (1996). Therapist session inten- laboratory research in counseling.
Clinical Psychology, 59, 138–141. tions in cognitive-behavioral and Journal of Counseling Psychology,
Spanierman, L. B., & Poteat, V. P. psychodynamic-interpersonal psy- 18, 106–110.
(2005). Moving beyond compla- chotherapy. Journal of Counseling Strong, S. R. (1984). Reflections on
cency to commitment: Multicultural Psychology, 43, 402–414. human nature, science, and progress
research in counseling. The Counseling Stipek, D. (1998). Differences between in counseling psychology. Journal
Psychologist, 33, 513–523. Americans and Chinese in the cir- of Counseling Psychology, 31,
Speight, S. L., & Vera, E. M. (2003). cumstances evoking pride, shame, 470–473.
Social justice agenda: Ready, or and guilt. Journal of Cross-Cultural Strong, S. R., & Dixon, D. N. (1971).
not? The Counseling Psychologist, Psychology, 29, 619–629. Expertness, attractiveness, and
32, 109–118. Stöber, J. (1998). The Frost Multi- influence in counseling. Journal of
Spengler, P. M., Strohmer, D. C., dimensional Perfectionism Scale Counseling Psychology, 18, 562–570.
Dixon, D. N., & Shivy, V. A. revisited: More perfect with four Strong, S. R., Hills, H. I., Kilmartin,
(1995). A scientist-practitioner (instead of six) dimensions. C. T., De Vries, H., Lanier, K.,
model of psychological assess- Personality and Individual Nelson, B., Strickland, D., &
ment: Implications for training, Differences, 24, 481–491. Meyer, C. W. (1988). The dynamic
practice, and research. The Counseling Stoltenberg, C. (1981). Approaching relations among interpersonal
Psychologist, 23, 506–534. supervision from a developmental behaviors: A test of complemen-
Spiegel, D., & Keith-Spiegel, P. perspective: The counselor com- tarity and autocomplementarity.
(1970). Assignment of publication plexity model. Journal of Coun- Journal of Personality and Social
credits: Ethics and practices of psy- seling Psychology, 28, 59–65. Psychology, 54, 798–810.
chologists. American Psychologist, Stone, G. L. (1984). Reaction: In Strong, S. R., & Matross, R. P. (1973).
25, 738–747. defense of the “artificial.” Journal Change processes in counseling
650 references
and psychotherapy. Journal of Sue, D. W. (2003). Overcoming our anxiety, and social desirability.
Counseling Psychology, 20, 25–37. racism: The journey to liberation. Journal of Consulting and Clinical
Strong, S. R., & Schmidt, L. D. San Francisco, CA: Wiley. Psychology, 54, 328–333.
(1970). Expertness and influence in Sue, D. W., & Sue, D. (1990). Counseling Task Force for the Promotion and
counseling. Journal of Counseling the culturally different: Theory Dissemination of Psychological
Psychology, 17, 81–87. and practice. New York: Wiley. Procedures. (1995). Training in
Strong, S. R., Welsh, J. A., Corcoran, Sue, D. W., & Sue, D. (2003). Counseling and dissemination of empirically-
J. L., & Hoyt, W. T. (1992). Social the culturally diverse: Theory and validated psychological treatment:
psychology and counseling psy- practice. New York: Wiley. Report and recommendations. The
chology: The history, products, Sue, S., Zane, N., & Young, K. Clinical Psychologist, 48, 2–23.
and promise of an interface. (1994). Research on psychother- Taussig, I. M. (1987). Comparative
Journal of Counseling Psychology, apy with culturally diverse popula- responses of Mexican Americans
39, 139–157. tions. In A. E. Bergin & S. L. and Anglo-Americans to early goal
Strupp, H. H. (1980a). Success and Garfield (Eds.), Handbook of psy- setting in a public mental health
failure in time-limited psychother- chotherapy and behavior change clinic. Journal of Counseling Psy-
apy: A systematic comparison of (4th ed., pp. 783–817). New York: chology, 34, 214–217.
two cases—Comparison 1. Archives John Wiley and Sons. Teixeira, M. A. P., & Gomes, W. B.
of General Psychiatry, 37, 595–603. Suen, H. K. (1988). Agreement, re- (2000). Autonomous career change
Strupp, H. H. (1980b). Success and liability, accuracy, and validity: among professionals: An empirical
failure in time-limited psychother- Toward a clarification. Behavioral phenomenological study. Journal
apy: A systematic comparison of Assessment, 10, 343–366. of Phenomenological Psychology,
two cases—Comparison 2. Archives Suh, C. S., O’Malley, S. S., Strupp, 31(1), 78–96.
of General Psychiatry, 37, 708–716. H. H., & Johnson, M. E. (1989). The Tesch, R. (1990). Qualitative research:
Strupp, H. H. (1980c). Success and Vanderbilt Psychotherapy Process Analysis types and software tools.
failure in time-limited psychotherapy: Scale (VPPS). Journal of Cognitive New York: Falmer.
Further evidence—Comparison 4. Psychotherapy, 3, 123–154. Thombs, D. L. (2000). A retrospective
Archives of General Psychiatry, Suppe, F. (1977). The structure of sci- study of DARE: Substantive
37, 947–954. entific theories (2nd ed.). Urbana: effects not detected in undergradu-
Strupp, H. H. (1981). Clinical University of Illinois Press. ates. Journal of Alcohol and Drug
research, practice, and the crisis of Swagler, M. A., & Ellis, M. V. (2003). Education, 46, 27–40.
confidence. Journal of Consulting Crossing the distance: Adjustment Thompson, A. S., & Super, D. E.
and Clinical Psychology, 49, of Taiwanese graduate students (Eds.). (1964). The professional
216–219. in the United States. Journal of preparation of counseling psychol-
Strupp, H. H., & Binder, J. L. (1984). Counseling Psychology, 50, 420–437. ogists. Report of the 1964 Grey-
Psychotherapy in a new key. New Swanson, J. (1995). The process and stone Conference. New York:
York: Basic Books. outcome of career counseling. In Bureau of Publications, Teachers
Strupp, H. H., & Howard, K. I. W. B. Walsh & S. H. Osipow College, Columbia University.
(1993). A brief history of psy- (Eds.), Handbook of vocational Thompson, B. (2004). Exploratory
chotherapy research. In D. K. psychology: Theory practice and and confirmatory factor analysis:
Freedheim (Ed.), History of psy- research (2nd ed., pp. 217–260). Understanding concepts and ap-
chotherapy (pp. 309–334). Wash- Mahwah, NJ: Lawrence Erlbaum plications. Washington, DC: Amer-
ington, DC: American psychologi- Associates. ican Psychological Association.
cal Association. Szarewski, A., & Cuzick, J. (1998). Thompson, B. J., & Hill, C. E. (1993).
Stumpf, H., & Parker, W. D. (2000). Smoking and cervical neoplasia: A Client perceptions of therapist
A hierarchical structural analysis review of the evidence. Journal of competence. Psychotherapy Research,
of perfectionism and its relation to Epidemiology and Biostatistics, 3, 3, 124–130.
other personality characteristics. 229–256. Thompson, C. E. (1994). Helms white
Personality and Individual Differ- Tabachnick, B. G., & Fidell, L. S. (1996). racial identity development (WRID)
ences, 28, 837–852. Using multivariate statistics. New theory: Another look. The Counseling
Suarez-Renaud, G. (2002). Exam- York: Harper Collins College. Psychologist, 22, 645–649.
ining the relationships of accultur- Tabachnick, B. G., & Fidell, L. S. (2001). Thompson, C. E., & Jenal, S. T.
ation, acculturative stress, problem Using multivariate statistics (4th (1994). Interracial and intraracial
solving appraisal and psychologi- ed.). Boston, MA: Allyn & Bacon. quasi-counseling interactions when
cal symptoms among Mexican Tashakkori, A., & Teddlie, C. (Eds.). counselors avoid discussing race.
immigrants in the Midwest. Un- (2003). Handbook on mixed meth- Journal of Counseling Psychology,
published doctoral dissertation, ods in the behavioral and social sci- 41, 484–491.
University of Missouri-Columbia. ences. Thousand Oaks, CA: Sage. Thompson, C. E., & Neville, H. A.
Sudman, S., & Bradburn, N. M. Tanaka-Matsumi, J., & Kameoka, (1999). Racism, mental health,
(1982). Asking questions: A prac- V. A. (1986). Reliabilities and and mental health practices. The
tical guide to questionnaire design. concurrent validities of popular Counseling Psychologist, 27, 155–
San Francisco: Jossey-Bass. self-report measures of depression, 223.
references 651
Thompson, C. E., Worthington, R., & Tinsley, H. E. A., Bowman, S. L., & Tracey, T. J., Glidden, C. E., &
Atkinson, D. R. (1994). Counselor Ray, S. B. (1988). Manipulation of Kokotovic, A. M. (1988). Factor
content orientation, counselor expectancies about counseling and structure of the Counselor Rating
race, and black women’s cultural psychotherapy: Review and analy- Form—Short. Journal of Counseling
mistrust and self-disclosures. Jour- sis of expectancy manipulation Psychology, 35, 330–335.
nal of Counseling Psychology, 41, strategies and results. Journal of Tracey, T. J., Hays, K. A., Malone, J.,
155–161. Counseling Psychology, 35, 99–108. & Herman, B. (1988). Changes in
Thompson, E. E., Neighbors, H. W., Tinsley, H. E. A., Roth, J. A., & counselor response as a function
Munday, C., & Jackson, J. S. Lease, S. H. (1989). Dimensions of of experience. Journal of Counseling
(1996). Recruitment and retention leadership and leadership style Psychology, 35, 119–126.
of African American patients for among group intervention special- Tracey, T. J., Leong, F. T. L., &
clinical research: An exploration ists. Journal of Counseling Psy- Glidden, C. (1986). Help seeking
of response rates in an urban chology, 36, 48–53. and problem perception among
psychiatric hospital. Journal of Tinsley, H. E. A., & Tinsley, D. J. Asian Americans. Journal of Coun-
Consulting and Clinical Psychology, (1987). Use of factor analysis in seling Psychology, 33, 331–336.
64, 861–867. counseling psychology research. Tracey, T. J., & Ray, P. B. (1984).
Thoreson, R. W., Budd, F. C., & Journal of Counseling Psychology, Stages of successful time-limited
Krauskopf, C. J. (1986). Alco- 34, 414–424. counseling: An interactional exam-
holism among psychologists: Tinsley, H. E. A., & Weiss, D. J. ination. Journal of Counseling Psy-
Factors in relapse and recovery. (1975). Interrater reliability and chology, 31, 13–27.
Professional Psychology: Research agreement of subjective judg- Triandis, H. C. (1972). The analysis of
and Practice, 17, 497–503. ments. Journal of Counseling Psy- subjective culture. New York: Wiley.
Thoreson, R. W., Kardash, K. A., chology, 22, 358–376. Truax, C. B., & Carkhuff, R. R.
Leuthold, D. A., & Morrow, K. A. Tinsley, H. E. A., Workman, K. R., & (1967). Toward effective counsel-
(1990). Gender differences in the Kass, R. A. (1980). Factor analysis ing and psychotherapy: Training
academic career. Research in of the domain of client expectan- and practice. Chicago: Aldine.
Higher Education, 3(2), 193–209. cies about counseling. Journal of Truax, C. B., & Wargo, D. G. (1966).
Thoreson, R. W., Miller, M., & Counseling Psychology, 27, 561–570. Psychotherapeutic encounters that
Krauskopf, C. J. (1989). The dis- Tobacco Advisory Group of the Royal change behavior: For better or for
tressed psychologist: Prevalence College of Physicians. (2000). Nic- worse. American Journal of Psy-
and treatment considerations. Pro- otine addiction in Britain. London: chotherapy, 20, 499–520.
fessional Psychology: Research and Royal College of Physicians of Tryon, W. W. (2001). Evaluating sta-
Practice, 20, 153–158. London. tistical difference, equivalence,
Tichenor, V., & Hill, C. E. (1989). A Toporek, R. L., Gerstein, L. H., and indeterminacy using inferen-
comparison of six measures of Fouad, N. A., Roysircar, G., & tial confidence intervals: An in-
working alliance. Psychotherapy, Israel, T. (Eds.). (2005). Handbook tegrated alternative method of
26, 195–199. for social justice in counseling psy- conducting null hypothesis statisti-
Tingey, R. C., Lambert, M. J., chology. Thousand Oaks, CA: Sage. cal tests. Psychological Methods,
Burlingame, G. M., & Hansen, Toporek, R. L., & Williams, R. 6, 371–386.
N. B. (1996). Assessing clinical (2005). Ethics and professional Turk, D. C., & Salovey, P. (Ed.).
significance: Proposed extensions to issues related to the practice of social (1988). Reasoning, inference, and
method. Psychotherapy Research, justice in counseling psychology. judgment in clinical psychology.
6, 109–123. In R. L. Toporek, L. H. Gerstein, New York: Free Press.
Tinsley, D. J., Tinsley, H. E. A., N. A. Fouad, G. Roysircar, & Turnbull, H. R., III (Ed.). (1977). Con-
Boone, S., & Shim-Li, C. (1993). T. Israel (Eds.), Handbook for sent handbook. Washington, DC:
Prediction of scientist-practitioner social justice in counseling psychol- American Association on Mental
behavior using personality scores ogy (pp. 17–36). Thousand Oaks, Deficiency.
obtained during graduate school. CA: Sage. Turner, P. R., Valtierra, M., Talken,
Journal of Counseling Psychology, Toukmanian, S. G., & Rennie, D. T. R., Miller, V. I., & DeAnda J. R.
40, 511–517. (1992). Psychotherapy process (1996). Effect of treatment on
Tinsley, H. E. A. (1992). Psychometric research: Paradigmatic and narra- treatment outcome for college stu-
theory and counseling psychology tive approaches. Newbury Park, dents in brief therapy. Journal of
research. In S. D. Brown & R. W. CA: Sage. Counseling Psychology, 43, 228–232.
Lent (Eds.), Handbook of counseling Toulmin, S. (1972). Human under- Tversky, A., & Kahneman, D. (1974).
psychology (2nd ed., pp. 37–70). standing: The collecting use and Judgment under uncertainty: Heu-
New York: John Wiley & Sons. evolution of concepts. Princeton, ristics and biases. Science, 185,
Tinsley, H. E. A. (1997). Synergistic NJ: Princeton University Press. 1124–1131.
analysis of structured essays: A Tracey, T. J. (1985). The N of 1 Tversky, A., & Kahneman, D. (1981).
large sample discovery oriented Markov chain design as a means of The framing of decisions and the
qualitative research approach. The studying the stages of psychother- psychology of choice. Science,
Counseling Psychologist, 25, 573–585. apy. Psychiatry, 48, 196–204. 211, 453–458.
652 references
Underwood, B. J. (1966). Experimental sulting and Clinical Psychology, 61, validity of clinical research. Jour-
psychology. New York: Appleton- 620–630. nal of Consulting and Clinical Psy-
Century-Crofts. Wampold, B. E. (1986a). State of the chology, 58, 360–367.
Valencia, R., & Suzuki, L.A. (Eds.). art in sequential analysis: Comment Wampold, B. E., & Drew, C. J.
(2001) Intelligence testing and on Lichtenberg and Heck. Journal (1990). Theory and application of
minority students: Foundations, of Counseling Psychology, 33, statistics. New York: McGraw-
performance factors, and assessment 182–185. Hill.
issues. Thousand Oaks, CA: Sage. Wampold, B. E. (1986b). Toward Wampold, B. E., & Freund, R. D.
Van de Vijver, F. J. R., & Leung, K. quality research in counseling psy- (1987). Use of multiple regression
(1997). Methods and data analysis chology: Current recommenda- in counseling psychology research:
for cross-cultural research. Thousand tions for design and analysis. The A flexible data-analytic strategy.
Oaks, CA: Sage. Counseling Psychologist, 14, 37–48. Journal of Counseling Psychology,
Vandenbos, G. R. (1996). Outcome Wampold, B. E. (1995). Analysis of 34, 372–382.
assessment of psychotherapy. Ameri- behavior sequences in psychother- Wampold, B. E., & Freund, R. D.
can Psychologist, 51, 1005–1006. apy. In J. Siegfried (Ed.), Therapeutic (1991). Statistical issues in clinical
Vaughn, S., Schumm, J. S., & Sinagub, and everyday discourse as behavior research. In M. Hersen, A. E.
J. M. (1996). Focus group inter- change: Towards a microanalysis Kazdin, & A. S. Bellack (Eds.),
views in education and psychol- in psychotherapy process research The clinical psychology handbook
ogy. Thousand Oaks, CA: Sage. (pp. 189–214). Norwood, NJ: Ablex. (2nd ed., pp. 313–326). Elmsford,
Vogt, W. P. (1993). Dictionary of sta- Wampold, B. E. (1997). Methodological NY: Pergamon Press.
tistics and methodology: A non- problems in identifying efficacious Wampold, B. E., & Furlong, M. J.
technical guide for the social psychotherapies. Psychotherapy (1981a). The heuristics of visual
sciences. Newbury Park, CA: Sage. Research, 7, 21–43. inference. Behavioral Assessment,
Vontress, C. E. (1970). Counseling Wampold, B. E. (1998). Necessary 3, 79–82.
blacks. Personnel and Guidance (but not sufficient) innovation: Wampold, B. E., & Furlong, M. J.
Journal, 48, 713–719. Comment on Fox and Jones (1981b). Randomization tests in
Vredenburg, K., O’Brien, E., & (1998), Koehly and Shivy (1998), single-subject designs: Illustrative
Krames, L. (1988). Depression in and Russell et al. (1998). Journal of examples. Journal of Behavioral
college students: Personality and Counseling Psychology, 45, 46–49. Assessment, 3, 329–341.
experiential factors. Journal of Wampold, B. E. (2001). The great Wampold, B. E., & Kim, K. H.
Counseling Psychology, 35, 419–425. psychotherapy debate: Models, (1989). Sequential analysis applied
Waehler, C. A., Kalodner, C. R., methods, and findings. Hillsdale, to counseling process and out-
Wampold, B. E., & Lichtenberg, NJ: Lawrence Erlbaum. comes: A case study revisited.
J. W. (2000). Empirically supported Wampold, B. E., Ankarlo, G., Journal of Counseling Psychology,
treatments (ESTs) in perspective: Mondin, G., Trinidad-Carrillo, 36, 357–364.
Implications for counseling psy- M., Baumler, B., & Prater, K. Wampold, B. E., Minami, T., Tierney,
chology training. The Counseling (1995). Social skills of and social S. C., Baskin, T. W., & Bhati, K. S.
Psychologist, 28, 657–671. environments produced by differ- (2005). The placebo is powerful:
Waldo, C. R. (1999). Working in the ent Holland types: A social per- Estimating placebo effects in med-
majority context: A structural spective on person-environment fit icine and psychotherapy from clin-
model of heterosexism as minority model. Journal of Counseling Psy- ical trials. Journal of Clinical
stress in the workplace. Journal of chology, 42, 365–379. Psychology, 61, 835–854.
Counseling Psychology, 46, 218–232. Wampold, B. E., & Bhati, K. S. (2004). Wampold, B. E., Mondin, G. W.,
Wallenstein, R. S. (1989). The psy- Attending to the omissions: A Moody, M., Stich, F., Benson, K.,
chotherapy research project of the historical examination of evidence- & Ahn, H. (1997). A meta-analy-
Menninger Foundation: An over- based practice movements. Pro- sis of outcome studies comparing
view. Journal of Consulting and Clin- fessional Psychology: Research bonafide psychotherapies: Empirically,
ical Psychology, 57, 195–205. and Practice, 35, 563–570. “all must have prizes.” Psychological
Walsh, W. B., & Heppner, M. J. Wampold, B. E., & Bolt, D. (2006). Bulletin, 122, 203–215.
(2006). Handbook of career coun- Therapist effects: Clever ways to Wampold, B. E., & Poulin, K. L. (1992).
seling for women. Hillsdale, NJ: make them (and everything else) Counseling research methods: Art
Lawrence Erlbaum. disappear. Psychotherapy Research, and artifact. In S. D. Brown &
Walsh, W. B., & Osipow, S. H. (Eds.). 16, 184–187. R. W. Lent (Eds.), Handbook of
(1994). Career counseling for Wampold, B. E., & Brown, G. S. counseling psychology (2nd ed.,
women. Hillsdale, NJ: Lawrence (2005). Estimating therapist vari- pp. 71–109). New York: Wiley.
Erlbaum. ability: A naturalistic study of out- Wampold, B. E., & Serlin, R. C.
Waltz, J., Addis, M. E., Koerner, K., comes in managed care. Journal of (2000). The consequences of ig-
& Jacobson, N. S. (1993). Testing Consulting and Clinical Psychology, noring a nested factor on mea-
the integrity of a psychotherapy 16, 184–187. sures of effect size in analysis of
protocol: Assessment of adherence Wampold, B. E., Davis, B., & Good, variance. Psychological Methods,
and competence. Journal of Con- R. H., III. (1990). Hypothesis 5, 425–433.
references 653
Wampold, B. E., & White, T. B. in the treatment of depression. Westen, D., Novotny, C. M., &
(1985). Research themes in coun- Journal of Consulting and Clinical Thompson-Brenner, H. (2004).
seling psychology: A cluster analy- Psychology, 71, 773–781. The empirical status of empiri-
sis of citations in the process and Watson, J. C., & Rennie, D. L. cally supported psychotherapies:
outcomes section of the Journal of (1994). Qualitative analysis of Assumptions, findings, and report-
Counseling Psychology. Journal of clients’ subjective experience of ing in controlled clinical trials. Psy-
Counseling Psychology, 32, 123–126. significant moments during the chological Bulletin, 130, 631–663.
Wampold, B. E., & Worsham, N. L. exploration of problematic reac- White, M. D., & White, C. A. (1981).
(1986). Randomization tests for tions. Journal of Counseling Psy- Involuntarily committed patients’
multiple-baseline designs. Behavioral chology, 41, 500–509. constituted right to refuse treat-
Assessment, 8, 135–143. Webb, E. J., Campbell, D. T., ment. American Psychologist, 36,
Wang, Y-W., Heppner, P. P., & Schwartz, R. C., & Sechrest, L. 953–962.
Heppner, M. J. (2004a). Qualitative (1966). Unobtrusive measures: White, O. R. (1974). The “split
methods: On the road to exploring Nonreactive research in the social middle”: A “quickie” method for
the unknown. In P. P. Heppner & sciences. Chicago: Rand McNally. trend estimation. Seattle: University
M. J. Heppner (Eds.), Writing and Webb, E. J., Campbell, D. T., of Washington, Experimental Edu-
publishing your thesis, dissertation Schwartz, R. D., Sechrest, L., & cation Unit, Child Development
& research: A guide for students Grove, J. B. (1981). Nonreactive and Mental Retardation Center.
in the helping profession (pp. measures in the social sciences. Whiteley, J. M. (1984). A historical
136–178). Pacific Grove, CA: Boston: Houghton Mifflin. perspective on the development of
Brooks/Cole. Webster-Stratton, C. (1988). Mothers’ counseling psychology as a profes-
Wang, Y-W., Heppner, P. P., & and fathers’ perceptions of child sion. In S. D. Brown & R. W. Lent
Heppner, M. J. (2004b). deviance: Roles of parent and (Eds.), Handbook of counseling psy-
Qualitative results: The meaning child behaviors and parent adjust- chology (pp. 3–55). New York: Wiley.
making process. In P. P. Heppner ment. Journal of Consulting and Wiley, M. O., & Ray, P. B. (1986).
& M. J. Heppner (Eds.), Writing Clinical Psychology, 56, 909–915. Counseling supervision by devel-
and publishing your thesis, disser- Wei, M., & Heppner, P. P. (2005). opmental level. Journal of Coun-
tation & research: A guide for stu- Counselor and client predictors of seling Psychology, 33, 439–445.
dents in the helping profession the initial working alliance: A Wilkinson, L., & Task Force on
(pp. 305–326). Pacific Grove, CA: replication and extension to Statistical Inference. (1999).
Brooks/Cole. Taiwanese client-counselor dyads. Statistical methods in psychology
Warchal, P., & Southern, S. (1986). The Counseling Psychologist, 33, journals. American Psychologist,
Perceived importance of counsel- 51–71. 54, 594–604.
ing needs among adult students. Wei, M., Heppner, P. P., & Mal- Willett, J. B. (1997). Measuring change:
Journal of College Student Per- linckrodt, B. (2003). Perceived What individual growth modeling
sonnel, 27, 43–48. coping as a mediator between buys you. In: E. Amsel & K. A.
Watkins, C. E., Jr. (1994). On hope, attachment and psychological Renninge (Eds.), Change and devel-
promise, and possibility in coun- distress: A structural equation mod- opment: Issues of theory, method,
seling psychology, or some simple, eling approach. Journal of Coun- and application (pp. 213–243).
but meaningful observations about seling Psychology, 50, 438–447. Mahwah, NJ: Erlbaum.
our specialty. The Counseling Psy- Wei, M., Vogel, D. L., Ku, T-Y., & Willett, J. B., Ayoub, C. C., &
chologist, 22, 315–334. Zakalik, R. A. (2005). Adult Robinson, D. (1991). Using
Watkins, C. E., Jr. (Ed.). (1997). attachment, affect regulation, growth modeling to examine sys-
Handbook of psychotherapy super- negative mood, and interpersonal tematic differences in growth: An
vision. New York: Wiley. problems: The mediating roles example of change in the function-
Watkins, C. E., Lopez, F. G., of emotional reactivity and ing of families at risk of maladap-
Campbell, V. L., & Himmell, emotional cutoff. Journal of Coun- tive parenting, child abuse, or
C. D. (1986). Contemporary counsel- seling Psychology, 52, 14–24. neglect. Journal of Consulting and
ing psychology: Results of a national Weisz, J. R., Rothbaum, F. M., & Clinical Psychology, 59, 38–47.
survey. Journal of Counseling Psy- Blackburn, T. C. (1984). Standing Willett, J. B., & Sayer, A. G. (1994).
chology, 33, 301–309. out and standing in: The psychol- Using covariance structure analysis
Watkins, C. E., & Terrell, F. (1988). ogy of control in America and to detect correlates and predictors
Mistrust level and its effects on Japan. American Psychologist, 39, of individual change over time. Psy-
counseling expectations in Black- 955–969. chological Bulletin, 116, 363–381.
White counselor relationships: Weiten, W. (1998). Psychology: Williams, E. N., Soeprapto, E., Like,
An analogue study. Journal of Themes and variations. Pacific K., Touradiji, P., Hess, S., & Hill,
Counseling Psychology, 35, 194–197. Grove, CA: Brooks/Cole. C. E. (1998). Perceptions of serendip-
Watson, J. C., Gordon, L. B., & Ster- Wertz, F. J. (2005). Phenomenological ity: Career paths of prominent
mac, L. (2003). Comparing the effec- research methods for counseling academic women in counseling
tiveness of process-experiential with psychology. Journal of Counseling psychology. Journal of Counseling
cognitive-behavioral psychotherapy Psychology, 52(2), 167–177. Psychology, 45, 379–389.
654 references
Williams, J. E. (1962). Changes in self World Values Survey. (n.d.). Retrieved Yalom, I. D. (1985). The theory and
and other perceptions following November 21, 2005, from http:// practice of group psychotherapy
brief educational-vocational coun- www.worldvaluessurvey.org (3rd ed.). New York: Basic Books.
seling. Journal of Counseling Psy- Worthington, E. L., Jr. (1984). Em- Yalom, I. D. (2005) The theory and
chology, 9, 18–30. pirical investigation of supervision practice of group psychotherapy.
Williams, K. E., & Chambless, D. L. of counselors as they gain experi- (5th ed.). New York: Basic Books.
(1994). The results of exposure- ence. Journal of Counseling Psy- Yates, A. J. (1958). The application of
based treatment in agoraphobia. chology, 31, 63–75. learning theory to the treatment of
In S. Friedman (Ed.), Anxiety dis- Worthington, E. L., Jr., & Roehlke, tics. Journal of Abnormal and
orders in African Americans (pp. H. J. (1979). Effective supervision as Social Psychology, 56, 175–182.
149–165). New York: Springer. perceived by beginning counselors- Yee, A. H., Fairchild, H. H.,
Wills, T. A. (1987). Help seeking as a in-training. Journal of Counseling Weizmann, F., & Wyatt, G. E.
coping mechanism. In C. R. Psychology, 26, 64–73. (1993). Addressing psychology’s
Snyder & C. E. Ford (Eds.), Worthington, E. L., & Stern, A. problem with race. American
Coping with negative life events: (1985). Effects of supervision Psychologist, 48, 1132–1140.
Clinical and psychological perspec- and supervisor degree level and Zane, N., Hall, G. C. N., Sue, S.,
tives (pp. 19–50). New York: Plenum. gender on supervisory relation- Young, K., & Nunez, J. (2004).
Wilson, L. S., & Ranft, V. A. (1993). ship. Journal of Counseling Psy- Overview, trends, and future
The state of ethical training for chology, 32, 252–262. issues. In M. J. Lambert (Ed.),
counseling psychology doctoral stu- Worthington, R. L. (2004). Sexual Bergin and Garfield’s handbook
dents. The Counseling Psychologist, identity, sexual orientation, reli- of psychotherapy and behavior
21, 445–456. gious identity, and change: Is it change (5th ed., pp. 767–804)
Winston, Roger B., Jr. (1985). A sug- possible to depolarize the debate? New York: Wiley.
gested procedure for determining The Counseling Psychologist, 32, Zane, N. W. S. (1989). Change mech-
order of authorship in research 741–749. anisms in placebo procedures:
publications. Journal of Counseling Worthington, R. L., & Atkinson, Effects of suggestion, social demand,
and Development, 63, 515–519. D. R. (1996). Effects of perceived and contingent success on improve-
Wolcott, H. F. (1992). Posturing in qual- etiology attribution similarity on ment in treatment. Journal of Coun-
itative inquiry. In M. D. LeCompte, client ratings of counselor credibility. seling Psychology, 36, 234–243.
W. L. Millroy, & J. Preissle (Eds.), Journal of Counseling Psychology, Zeidner, M., & Saklofske, D. (1996).
The handbook of qualitative re- 43, 423–429. Adaptive and maladaptive coping.
search in education (pp. 3–52). San Worthington, R. L., Dillon, F. R., & In M. Zeidner & N. S. Endler
Diego: Academic Press. Becker-Schutte, A. M. (2005). (Eds.), Handbook of coping: The-
Wolcott, H. F. (1994). Transforming Development, reliability, and va- ory, research, and applications
qualitative data: Description, analy- lidity of the Lesbian, Gay, and (pp. 505–531). New York: John
sis, and interpretation. Thousand Bisexual Knowledge and Attitudes Wiley.
Oaks, CA: Sage. Scale for Heterosexuals (LGB- Zimbardo, P. G., & Ebbesen, E. B.
Wolpe, J. (1969). The practice of KASH). Journal of Counseling (1970). Influencing attitudes and
behavior therapy. New York: Psychology, 52, 104–118. changing behavior. Reading, MA:
Pergamon Press. Wrenn, R. L. (1985). The evolution of Addison-Wesley.
Woodhouse, S. S., Schlosser, L. Z., Anne Roe. Journal of Counseling Zuckerman, M. (1990). Some dubious
Crook, R. E., Ligiero, D. P., & and Development, 63, 267–275. premises in research and theory on
Gelso, C. J. (2003). Client attach- Wundt, W. (1904). Principles of phys- racial differences: Scientific, social,
ment to therapist: Relations to iological psychology. New York: and ethical issues. American
transference and client recollections Macmillan. Psychologist, 45, 1297–1303.
of parental caregiving. Journal Wundt, W. (1916). Elements of folk
of Counseling Psychology, 50, psychology. London: Allen & Unwin.
395–408. (Original work published 1900.)
Author Index
Abramson, L. Y., 46, 542, 560 Atkinson, D. R., 30, 56, 58, 73, 299, Benjamin, L. S., 482
Adler, P., 283, 284 301–304, 308, 321, 361, 453, Benn, S. I., 111
Adler, P. A., 283, 284 502, 545, 547, 563 Bennum, I., 491
Albee, G. W., 24 Ayalon, L., 370 Berdie, R. F., 46
Aldenderfer, M. S., 240 Azibo, D. A., 375, 376 Berg, I. A., 451
Alexander, C. H., 496 Bergin, A. E., 3, 6, 202, 216, 218,
Alexander, L. B., 485 Babbie, E. R., 226, 339, 340 222, 426
Alloy, L. B., 542, 560 Bakeman, R., 463 Berkowitz, A. D., 176
Alvidrez, J., 361 Baldwin, S. A., 445 Bernard, J. M., 55
American Cancer Society, 161 Bamberger, M., 7 Bernstein, B. L., 25
American Counseling Association, Bangert, A. W., 198, 215 Betancourt, H., 367, 368, 371
109, 575 Barak, A., 235, 323, 483 Betz, N. E., 24, 36, 48, 53, 56,
American Psychological Association, Barber, J. P., 436, 437, 468, 489, 490 244, 359
16, 87, 109, 117, 118, 143, 232, Barber, T. X., 387, 390 Beutler, L. E., 361, 433
358, 540, 541, 556, 558, 615 Barkham, M., 53, 491 Bhaskar, R., 6, 14
American Psychological Association Barlow, D. H., 148, 200, 201, 221, Bishop, J. B., 226
Ethics Committee, 121, 143, 598 337, 364 Blashfield, R. K., 240
Andersen, B., 411 Baron, R. M., 250, 252, 253, Blos, P., 226
Anderson, C. A., 46 374, 375 Bohus, M., 427
Anderson, J. R., 46 Barrett, K. A., 345, 353 Bollen, K. A., 325
Anderson, M. Z., 56 Barrett-Lennard, G. T., 483, 491 Bordin, E. S., 405, 406, 407, 412,
Anderson, W. P., 24, 26, 46, Baskin, T. W., 432 422, 484, 486
48, 58 Baumrind, D., 138 Borenstein, M., 357
Anton, W. D., 430 Beauchamp, T. L., 108, 110–112 Borgen, F. H., 17, 18, 48, 53, 58,
Arean, P. A., , 355, 361 Beck, A. T., 7, 140, 190, 345, 429, 239, 240, 407
Argyris, C., 397 435, 437 Bracht, G. H., 83, 100
Arnold, C. L., 466 Beck, K. A., 7 Bradley, J. V., 355
Arredondo, P., 359 Bednar, R. L., 484, 485 Brennan, K. A., 506
Asner-Self, K. K., 247, 248 Behrens, J. T., 56 Breuer, J., 201, 217
Atkinson, D., 359, 360, 366, 368, Belar, C. D., 48 Bridgewater, C. A., 119
408–410 Bem, D. J., 541 Broad, W., 116
655
656 Author Index
Brock, T. C., 424, 475 Crosbie, J., 192 Exner, J. E., Jr., 340
Brophy, J. E., 552 Cross, W. E., 56 Eysenck, H. J., 201, 426
Brotemarkle, R. A., 226 Cummings, A. L., 456, 478
Brown, F., 110
Brown, S. D., 34, 36, 43, 239 Fabrigar, L. R., 234
Daniels, L. K., 202 Fagley, N. S., 358
Brown, W., 244
Danskin, D. G., 451 Farmer, H. S., 56
Bryk, A. S., 444, 446, 466
Dar, R., 14 Farrell, A. D., 55, 241
Buboltz, W. C., 494
Daus, J. A., 194 Fassinger, R., 41, 56
Buchler, J., 4
Dawis, R. V., 235, 494, 499, 509 Fassinger, R. E., 48, 53, 56, 263, 269,
Burgoon, J. K., 468, 471, 472
Day, S., 73 290, 325, 396
Burkard, A. W., 301
Deffenbacher, J. L., 74, 155, 189, Feldman, D. A., 394
Burns, B. D., 491
190, 305, 306, 396, 404 Fine, M., 275
Butler, S. F., 435 Delgado-Romero, E. A., 361, 363, 364
Byrne, B. M., 381 Fisher, A. R., 371
Dennin, M. K., 74, 199, 205, 214 Fiske, S. T., 46
Denzin, N., 258, 259, 263, 526 Fitzpatrick, J., 523, 532, 534
Campbell, A., 315 Department of Health and Human Flaherty, J. A., 505
Campbell, D. T., 67, 78, 83, 90 Services, 117, 124 Flaskerud, J. H., 382
Campbell, L. M., III., 202 DeProspero, A., 207 Foa, E. B., 433
Caple, R. B., 14 Derogatis, L. R., 231, 439, 440 Folger, R., 358
Carkhuff, R. R., 457 DeRubeis, R. J., 436 Folkman, S., 163
Carney, C. G., 226 DeSena, P. A., 226 Fong, M. L., 43, 226, 228
Carter, R, 366–368 DeVellis, R. F., 504 Fontana, A., 287, 289
Casas, J. M., 111, 396 Dhami, M. K., 301 Ford, D. H., 17, 18
Casas, M. J., 361 Diener, E., 108, 110, 124, 126, 128, 231 Forgy, E. W., 425, 426
Chang, P., 418, 419 Dillman, D. A., 505 Forum Qualitative Sozialforschung,
Chapman, L. J., 340 Dillon, F. R., 224, 235 259
Charmaz, K., 263–266 Dill-Standifond, T. J., 457 Fouad, N. A., 15, 235, 359, 362
Chartrand, J., 238 Dipboye, W. J., 450, 451, 468, 478 Fowler, F., 524
Chen, C., 499 Dixon, D. N., 2, 47, 55, 156, 411, Francis, D. J., 438, 466
Cheng, S. K., 497, 498 416, 420, 422, 483 Frank, G., 24
Chia, R. C., 499 Dixon, W. A., 141 Frank, J. D., 157, 158, 431, 485
Chickering, A. W., 226 Dolliver, R. H., 24, 53 Frankena, W. K., 110
Christensen, L. B., 214, 387, Dorn, F. J., 420, 492, 540, 542 Frankenberg, R., 503
396–399, 426 Doss, B. D., 444 Frazier, P. A., 46, 55, 250–253, 374,
Chwalisz, K., 3, 48 Douglas, J. D., 290 375, 483, 491
Claiborn, C. D., 24, 29, 46, 48, 50 Dowd, E. T., 394 Fremont, S., 58
Cobb, A. K., 294 Drane, D. F., 110 Fretz, B. R., 2, 16, 415
Cohen, J., 247, 248, 309, 355–357, Drew, C. F., 52, 57, 113, 114, 116, Friedlander, M. L., 205, 235, 291,
363 129, 206 338, 399, 456, 458, 459, 468,
Cohen, M. R., 5 Drew, C. J., 90, 154, 164, 165, 299, 484, 543, 561
Cohen, R. J., 508 347, 355, 363 Friesen, W. V., 470
Cokley, K., 36, 369, 374, 375 Frontman, K. C., 288
Cole, D. A., 244, 246, 325 Edgington, E., 207 Frost, R. O., 497
Coleman, H. L. K., 105, 361, 397, 400 Edgington, E. S., 207, 211, 219 Fuhriman, A., 485
Condon, K. M., 441 Elder, N. C., 294 Fukuyama, M., 366
Constantine, M. G., 58 Elkin, I., 350, 434, 442 Fuller, F., 479
Cook, E. P., 359 Elliott, E., 275, 294 Furlong, M. J., 207, 211, 323
Cook, T. D., 83, 103, 195, 220, 309 Elliott, R., 70, 240, 241, 291, 341, Furnham, A., 508
Cooke, R. A., 129 422, 449, 450, 457, 462, 467, Furst, J. B., 217
Coopersmith, S., 190 468, 472, 476, 483, 486, 491,
Corazzini, J., 219 492
Corning, A. F., 253 Ellis, A., 252 Gade, E., 244, 255
Corrigan, J. D., 55, 302, 323, 416, Ellis, J. V., 549 Gambrill, E., 46, 461
420, 422, 483 Ellis, M. V., 58, 61, 74, 199, 205, Garfield, S. L., 426
Cournoyer, R. J., 59, 359, 214, 215 Gelatt, H. B., 18
Creswell, J. W., 7, 76, 77, 260, 261, Enns, C. Z., 54, 458 Gelso, C. J., 17, 24, 25, 29–31, 36,
264, 266, 269, 270, 271 Erdur, O., 72 38–42, 48, 49, 53, 59, 63, 69,
Crews, J., 181 Erickson, C. D., 386 72, 75–78, 84, 93, 106, 212,
Crites, J. O., 167 Ericsson, K. A., 401 216, 223, 303, 304, 307–309,
Crits-Christoph, P., 196, 426, 436, Eugster, S. L., 362, 363, 457,468, 407, 412, 417, 421, 422, 546,
437, 489 477, 491 551, 564, 569
Author Index 657
Gibbons, J. L., 499 Helms, J. E., 56, 58, 77, 204, 239, Hoyt, W. T., 55, 316, 359, 420, 421,
Giorgi, A., 269, 272, 275 247, 366, 368, 369, 418, 422, 483
Glaser, B. G., 263, 264, 266, 275 453, 479, 480, 492 Huberty, C. J., 324
Glass, G. V., 83, 207, 426 Helwig, A. A., 226 Huck, S. W., 153
Gleitman, H., 148 Hembree, E. A., 433 Hughes, E. C., 24
Glock, C. Y., 226 Henry, W. P., 435, 438, 482 Hunt, D. W., 340
Goetz, J. P., 276 Henwood, K., 263–265
Gold, R. L., 283 Heppner, M. J., 3, 26, 42, 51, 54, 55,
Iberg, J. R., 436
Goldapple, K., 339 235, 238, 239, 253, 263, 294,
Ibrahim, F. A., 400
Goldman, L., 18, 24, 25, 49, 408 453, 454, 494, 510, 542,
Ingram, R., 46
Goldstein, A. P., 406, 407 547–551, 553, 554, 571, 572
Ivey, A. E., 400
Good, G. E., 59, 359 Heppner, P. P., 2, 15, 18, 20, 24–26,
Goodyear, R. K., 7, 24, 32, 46, 47, 35, 46, 48, 49, 51, 53–56, 58,
49, 55, 204, 231–233 156, 163, 175, 204, 223, 226, Jacobson, N. S., 426, 429, 434, 435,
Gottfredson, G. D., 244 239, 241, 242, 253, 263, 294, 436, 438–441
Gottman, J. M., 207, 463, 464 359, 364, 389, 475, 494, 510, Jayaratne, S., 207
Gould, S. J., 367 542, 547–551, 553, 554, 571, Jensen, A. R., 113
Graham, J. R., 334 572 Johnson, C. V., 58
Greenbaum, T. L., 291 Herman, J., 515, 527 Johnson, R. F. Q., 387
Greenberg, L. S., 205, 225, 254, 439, Hermansson, G. L., 169 Johnston, J. A., 3, 235
449, 457, 458, 467–469, 484, Heron, J., 7 Joint Committee on Standards for
485, 487, 491 Hersen, M., 148, 149, 200, 201, 221, Educational Evaluation, 534
Greenberg, R. P., 101 364 Jones, A. S., 304, 307–309
Greene, J. C., 77, 381 Herskovits, M. J., 368 Jones, E. E., 205, 464
Greenwald, A. G., 475 Highlen, P. S., 449, 451, 474, 481 Jones, H. G., 220
Grieger, I., 77, 111, 114, 115, 361, Hill, C. E., 34, 44, 73, 77, 203–205, Josephson, G. S., 394–395
396, 453 216, 221, 227, 239, 275–279, Juntunen, C. L., 277, 280–282, 504
Grummon, D. L., 406 282, 290, 301, 305, 309, 336,
Grundy, C. T., 441 338, 418, 437, 449–451, Kagan, N., 216, 288, 403
Grzegorek, J. L, 242 453–455, 457–464, 467, 468,
Kahn, J. H., 31, 36–39, 42–46, 398
Guba, E. G., 7, 14, 260, 261, 276, 472–474, 478–482, 485, 486,
Kandel, D. K., 402
291, 294 490, 492, 552, 553
Kanfer, F. H., 46
Gurin, N. K.,157, Hill, W. F., 477
Kasturirangan, 277
Gurman, A. S., 426, 449, 457, 483 Hillenbrand-Gunn, T., 175
Kaul, T., 485
Gushue, G. V., 56 Hines, P. L., 474–476 Kazdin, A. D., 214
Gustav, A., 397 Hoch, P. H., 202 Kazdin, A. E., 61, 67, 96, 135, 142,
Guthrie, R. V., 376 Hodder, I., 291
147, 153, 158, 162, 172, 176,
Hoffman, J., 245
179, 181, 198, 206, 207, 217,
Hoffman, M. A., 15, 453
Haase, R. F., 356 220, 324, 380, 389, 396, 407,
Hogg, J. A., 74, 189, 190, 396, 404
Hackett, G., 42, 225, 244, 366 412, 429, 437
Hohmann, A. A., 361
Hair, J. F., Jr., 234, 239, 240, Keet, C. D., 406
Holland, J. L., 25, 36, 37, 43, 45,
247, 249 Keith-Spiegel, P., 116, 118, 119,
160, 243, 244, 248
Hall, S., 151, 152 121–124, 128, 130, 140, 143
Holland, P. W., 36, 309, 310, 552,
Hanfling, O., 6 Kelerman, G., 435
Hanson, W. E., 7, 76 570 Keller, M. B., 438, 444
Hardin, S. I., 15 Hollon, S. D., 190, 426 Kelly, A. E., 58, 398
Harding, S., 275 Holloway, E. L., 55, 61, 338, 340, Kelly, K. R., 238
353, 360, 451, 473, 474
Harmon, L., 17, 56 Kerlinger, F. N., 5, 15, 59, 61, 68, 78,
Honos-Webb, L., 205
Harmon, L. W., 227 146, 147, 160, 170, 172, 173,
Horan, J. J., 2, 325
Harré, R., 6, 14 198, 230, 298, 310, 339, 350,
Hartley, D. E., 485 Horowitz, L. M., 439, 478 385, 403, 405, 412, 421, 422
Horvath, A. O., 458, 468, 484, 485,
Harvey, B., 497 Kiesler, D. J., 66, 67, 201, 202, 216,
Hathaway, S. R., 190 491 426, 449, 459, 467, 468, 483, 484
Hoshmand, L. T., 17, 46, 48, 261,
Haupt, S. G., 165 Kim, B. S. K., 30, 58, 73, 277, 301,
Haverkamp, B. E., 14, 76 266, 275 305, 359, 360, 408–410, 442, 502
Howard, G. S., 6, 7, 17, 18, 24, 25,
Hayes, J. A., 58, 227, 408, 409 Kim, D. M., 442, 445
Hays, W. L., 88, 327 49, 219, 246, 338, 426 Kim, K. H., 338, 204, 221, 451, 452,
Howard, K. I., 426, 449, 451, 452,
Hayton, J. C., 507 463, 464, 552, 553, 570, 571
480, 484, 490, 491
Hazaleus, S. L., 306 Kim, U., 498
Hoyle, R. H., 325
Heesacker, M., 341 Kim, Y. C., 259
Hoyt, M. R., 491
Heller, K., 406, 407 Kirshner, T., 453, 454
658 Author Index
Tracey, T. J., 7, 58, 207, 323, 333, 249, 283, 285, 289, 293, 299, Williams, E. N., 275, 277, 282, 449,
336, 3514, 352, 451 309, 321, 324, 338, 340, 347, 454, 457, 467
Triandis, H. C. 369 353, 355, 358, 360–363, 397, Williams, J. E., 149
Truax, C. B., 202, 438, 439 400, 426, 429, 431, 432, Williams, K. E, 352, 359
Tryon, W. W., 87 436–438, 442, 444–446, 451, Wills, T. A., 415
Turk, D. C., 46 452, 457, 463, 464, 468, 477, Wilson, L. S., 108
Turnbull, H. R., 128, 129 491, 545, 548, 549, 552, 553 Winston, R. B., Jr., 120
Turner, P. R., 430 Wang, Y.-W., 230, 237, 238, 263, Wolcott, H. F., 282, 291–293
Tversky, A., 18, 46 294, 505 Wolpe, J., 202
Warchal, P., 226 Woodhouse, S. S., 72, 93
Valencia, R., 381 Watkins, C. E., Jr., 23, 43, 48, 55, World Values Survey, 357
Van de Vijver, F. J. R., 505 387, 400, 434 Worthington, E. L., 61, 235, 468, 474
Vandenbos, G. R., 426 Watson, J. C., 269, 293, 446 Worthington, R. L, 73, 224, 235
Vogt, W. P., 501 Webb, E. J., 100, 205, 341 Wrenn, R. L., 52
Vontress, C. E., 453 Webster-Stratton, C., 312 Wundt, W., 219, 462
Vredenburg, K., 308, 309 Wei, M., 2, 20, 253
Weisz, J. R., 502 Yalom, I. D., 179, 208, 209,
Waehler, C. A., 48 Weiten, W., 372 213, 388
Waldo, C. R., 253 Wertz, F. J., 270, 271, 274 Yates, A. J., 220
Wallenstein, R. S., 71 Westen, D., 434 Yee, A. H., 361
Walsh, W. B., 56 White, M. D., 111
Waltz, J., 435–437 White, O. R., 207 Zane, N., 452, 453
Wampold, B. E., 3, 7, 24, 38, 43–45, Whiteley, J. M., 23 Zane, N. W. S., 341
48, 57, 83, 88, 90, 105, 149, Wiley, M. O., 59–61 Zeidner, M., 502
154, 158, 164, 165, 199, 204, Wilkinson, L., 87 Zimbardo, P. G., 5, 419
207, 210, 211, 221, 239, 247, Willett, J. B., 438, 444, 466, 511 Zuckerman, M., 367, 368
Subject Index
AB design, 207–209 Between-group designs, 146–159 Conjectures, 11, 13. See also
ACA code of ethics, 109, 575–595 examples of, 151, 155, 156 130, 133 Hypotheses
Alpha level, 349. See Errors, type I factorial, 159–162 Consensual qualitative research
Alternative explanations, 81, 90–92 historical overview, 148–149 (CQR), 256, 262, 274–282.
American Counseling Association, 109 posttest-only control group, See also Qualitative Research
American Psychological Association, 149–152 Consent, 128. See also Ethical Issues
16, 33, 87, 109, 117–118 pretest-posttest control group, capacity, 128
Analogue research, 405–423 152–155 information, 129
advantages, 410–411 Solomon four-group, 155–157 voluntariness, 29–30
client variables, 414–415 Bias, 385–386 Consent forms. See Ethical Issues
counseling process and setting participant bias, 396–397 Construct, 36–37, 44, 59–61, 66
variables, 417–418 investigator/experimenter bias, Construct validity, 81, 83–85, 97–98
counselor variables, 416–417 386–392 of putative causes and effects, 233
disadvantages, 412 double blind, 401 threats to construct validity, 97–98
examples of, 408–410 Bubble hypothesis, 63, 75, 76, 106 Content testing, 524–526
generalizability of, 412–414 Control groups, 148, 150, 154,
historical overview, 406–408 Causality, 10, 62, 68, 73, 213, 249, 157–159
Strong’s boundary conditions, 420 309–310. See also Internal matched, 158–159, 173
utility of, 421–422 validity no-treatment, 92, 158, 161, 299
Analysis of variance, 355, 430, 466 Ceiling effect, 89, 318 placebo, 158, 173, 215, 299
Association, 308 Checks and balances, 529 waiting list, 158, 173, 190. See also
Assumptions, violation of statistical, Clinical significance, 438, 439, 441, Between groups
87–88 442, 356 Convergent validity, 322, 325, 328,
Atlanta Conference, 23, 50 Cluster analysis, 238–240 496. See also Validity
Attenuation of correlations (and examples of, 240–243 Correlational research designs,
correcting for), 320, 325 Confidentiality and privacy, 133–136. 244–253
Attrition, 96–96, 150, 153, 546 See also Ethical issues Counseling process research.
Auditor, 279 Confound, 300–303, 551 See Process research
confounding variables, 66, 201, Criterion variable, 62, 247, 247–252.
Behavioral measures (or observa- 202, 339, 389, 397, 401 See also Dependent variable
tions), 325, 332, 337–342 construct confounding, 98–100 Critical theory, 13–14, 260
661
662 Subject Index
Crossover design, 166–170, 171, 173 scoring and recording, 317. Factorial designs, 148, 149, 159–160,
Culture, 12, 361, 368, 577, 582, 592 See also Reliability, scoring 245, 358–360
cultural differences, 373, , 509, and recording error example of, 161
512 specific. See Reliability, specific strengths, 160–161
culturally competent research error, 315 weaknesses, 161
studies, 381–382 transient. See Reliability, transient Feminist theory, 14
validity issues, 376–381 error, 315 Field notes, 258, 259, 267, 285, 286,
type I and II (and probability of), 531, 535. See also Qualitative
Data analyses, 228, 230, 260, 263, 86–87 Research
381, 460, 527, 530–532, 536, Error variance, 66, 68, 87, 89, 146, Floor effects, 89, 171
547, 530, 550 147, 152, 377, 385, 387, 392 Focus groups, 269, 288, 382, 502,
Data collection, 62, 214, 215, 228, Ethical issues. See also Scholarly work 524, 525
504,–505, 526–527 consent forms, 130–139, 499 Follow-up, 203, 228, 229, 230, 278,
methods, 332–341 code of ethics, 109, 575–595 402, 425
procedures, 70, 228, 505, 528, deception and debriefing, 131–133
530, 536 duplicate and piecemeal Gender, 19, 37, 52, 56, 59, 104, 106,
planning for,527 publication, 117 159, 160, 227, 245, 246, 359,
primary, 530–531 ethical principles of psychologists, 389
secondary, 531–532 109, 614–638 Generalizability, 8, 67, 68, 69,
Data reduction research designs, informed consent, 128–131, 137 103–104, 330, 350, 351, 358,
233– 243 participants confidentiality and 360. See also External validity;
Debriefing, 125, 131–138. See also privacy, 133–136 Population issues
Deception; Participants; plagiarism, 121–122 “Good enough” principle, 352
Ethical issues publication credit, 118–121 Greystone Conference, 23
deception and, 131–133 reporting the results, 115–117 Grounded theory, 258, 261, 262,
oral debriefing example, 141 responding to ethical dilemmas, 263–266
written debriefing example, 142 137–138 Growth curve modeling, 444, 466–467
Deception, 131–133. See also risks and benefits, 125–128
Debriefing; Participants; treatment issues, 136–137
Ethical issues Ethical principles, 109–112 Hermeneutics, 12, 261, 262, 263, 550
debriefing, 131–133 autonomy, 111, 123, 125, 129, Heterogeneity. See Population issues,
Deductive logic, 10 130, 131, 133 heterogeneity of
Demand characteristics, 386, beneficence, 110, 112, 113, 114, 125 Homogeneity. See Population issues,
397–399, 402 fidelity, 112, 123, 125, 128, 131, 133 homogeneity of
postexperimental inquiry, justice, 111–112, 118, 122 Hypotheses, 56, 57, 60, 62
402, 403 nonmaleficence, 110, 115, 123, alternative, 26, 86
Dependent samples designs, 163–166 125, 133 examples of, 87
Dependent variables, 62, 85–76, 247, Virtue ethics, 109, 112–113 guessing, 101
312–313, 332, 334 Ethnicity, 187, 354, 367, 368, 371, null, 10, 86, 87, 88 89
methods of data collections, 342, 374, 375, 380 research, 51, 57, 63, 545, 547, 548,
332–335 Ethnography, 258, 261 549, 563, 565
operationalization, 313 Existing materials, 282, 291–292 role of theory, 54
procedural considerations, 302–303 Expectancies, 172, 235 statistical, 10, 348
psychometric issues, 313–314 Experimental designs, 62, 67, 245,
reactivity, 313, 331–332 249, 254, 259, 300
Experimental field studies, 73–74 Independent variables, 62, 70, 73, 142,
Descriptive field studies, 71 245, 247, 249, 298, 299, 566
Descriptive laboratory studies, 70–71 Experimental laboratory studies, 72–73
Experimenter bias, 386, 404, 492 limiting differences, 299, 301
Descriptive statistics, 70, 371–373 salience of differences, 299,
Dialectics, 12–13. attributes, 367–388
expectancies, 389–392, 397 303–304, 308
Differential attrition. See Attrition status variables, 308–311, 345,
Discriminant validity, 328, 332, 496, procedures, 385, 387, 392, 393, 394
External validity, 67, 68, 69, 85, 151, 354, 361
507, 508. See also Validity Informed consent, 128–130, 137.
358, 405, 407, 412, 420, 422.
See also Populations and sam- See also Ethical Issues
Effect (as in cause and effect). ples; Threats to external validity elements of, 128–130
See Causality threats to, 74, 103–105 examples of, 132
Effect size, 90, 356, 357, 549, 567 risks and benefits, 124, 126–128
Error, 57, 85, 314–417. See also withholding treatment and, 136–137
Reliability Factor analysis, 234–238, 322, 482, Institutional review boards, 124, 127
compounding, 317 496–498, 506–507 Interaction effects, 179, 212, 300,
random response, 314–315 use with subscales, 322–324, 332 358–360, 361, 363
Subject Index 663
Internal validity, 68, 69, 70, 71, 73, Observational methods, 514, 523 versus applying research tools, 14, 19
90, 93, 95. See also Threats to Observations (as scores in a ways of knowing, 4, 5, 6, 65, 68
Internal validity population), 345–350 Physiological measures, 334, 338
Interpretation (in qualitative research) Observations (of participants), 523, 531 Pilot testing, 523, 529, 548
262, 263, 274, 279, 286, 291, complete observer, 283 Placebo control group, 158, 229.
293 complete participant, 283–284 See also Control Groups
Inter-rater agreement. See Reliability, degree of involvement, 283 Population issues, 344–365
inter-rater agreement drift, 96, 317 creating a participant pool, 334,
Interviews, 332, 336, 339 methods of, 284–286 349, 351–352
analyzing and presenting data, nature of, 282 defining the target population, 344,
291–293 observer-as-participant, 283 349, 350
group, 287, 288, 291 participant observer, 284, 530, 531 determining the number of partici-
methods of interviewing, 288–291 qualitative research, 256–295, 550 pants, 355–357
qualitative research, 256–295 One-shot pretest/posttest design, 90, 91 establishing validity in the absence
quantitative research, 7, 205, 259, Operational definitions, 51, 60–61 of random selection, 349,
260, 266, 339 examples of, 61 352–355
structured, 227, 287–288 Operationalization, 313, 314, 317, external validity and, 358–364
type of, 286–288 337, 545, 547, 548, 563, 565. heterogeneity of, 89, 246, 239
unstructured, 267–288 See also Constructs, homogeneity of, 276, 319
Intuition, 5, 271, 517, 518 operationalization of hypothetical populations, 346
Outcome research, 424–447 sampling theory, 344, 345–349
Key informants, 290 control groups, 430–432 selecting participants, 345, 349, 352
early studies, 451 Population parameter. See Parameter
measuring change, 438–446 Positivism. See Worldviews, positivism
Latent variables, 234, 244, 245, 325, methological issues in conducting, Postpositivism. See Worldviews, post-
327 433–438 positivism
Level of inquiry, 8 strategies for conducting, 427–433 Posttest-only control-group design,
treatment integrity, 434–435, 437, 149–152, 347, 348
Main effects, 161, 246, 359, 389, 550 438 example of, 151–152
Manipulation, 58, 62, 68, 298, 299, strengths, 150
311, 394, 406, 411 weaknesses, 151
checks of, 299, 298, 304–306, 307, Parameter, 347, 348, 430 Power (statistical), 87, 152, 170, 307,
308, 310 Participant assignment, 162–163. See 330, 355, 364, 365
MAXMINCON, 68, 78, 146, 147, 152, Random assignment Pretest-posttest control group design,
310, 350, 385, 403, 405, 421 Participant bias, 385–404 152–155
Mediating effects, 249 demand characteristics, 397–398 example of, 155
Meta-analysis, 360 introspective abilities, 386 strengths, 152–154
Meta-evaluation, 535 participant characteristics, 397, weaknesses, 154
Method variance, 325, 327–330 398–400 Primary data analyses, 530–531, 536,
Moderating effects, 250 strategies for reducing, 401–403 537
Mortality, 161, 279, 220. See Attrition Participant observation, 523, 531, Process research, 448–493
Multidimensional scaling, 475 537 allegiance, 302, 432, 433, 446, 447
Multiple-baseline design, 212–215, 222 Participant pool, 351–352. coding and analyzing data, 460–467
Multiple data sources, 269, 526, 531 See Population issues, creating current research findings, 452–454
Multiple measures, 190, 196, 204, a participant pool definitions, 449, 455
206, 207, 324–325, 329, 330 Participants, 344, 349–352, 355–358, design issues, 454–467
Multivariate analysis of variance, 542, 546–547. See also Role early research, 482
190, 466 of subjects/participants; ethnicity, 496
Population issues level of measurements, 468–490.
ethical issues, 123–142 See also Process research: rep-
Nonequivalent group posttest-only Passive designs, 255 resentative measures
design, 91 ex post facto design, 243–244 ancillary behaviors, global level
Nonequivalent groups designs, multiple regression, 246–253 471–472
180–191 Phenomenology, 269–274 ancillary behaviors, micro level,
example of, 186–187 Philosophy of science, 14–15 467–471
interpretable nonequivalent groups assumptions regarding human content, global level, 478
designs, 183–186 nature, 15 content, micro level, 477
uninterpretable nonequivalent goals of science, 15–17 covert behaviors, global level,
groups designs, 181–183 methodological diversity, 17–18 476–477
Nonintervention, 283 realist’s view, 6 covert behaviors, micro level,
Null results, 300, 307, 308 received view, 6, 7, 17 474–476
664 Subject Index
Process research (continued ) Professional writing, 540–555. themes, 239, 260, 262, 266,
interpersonal manner, global See also Research Report 272–273, 292–293
level, 483–484 abstract, 543 thick description, 291–292
interpersonal manner, micro discussion, 550–551 Qualitative research, 256–295
level, 482–483 literature review, 544 analyzing and presenting data,
quality, global level, 488–490 method, 546–548 291–293
quality, micro level, 486–488 results, 549–550 consensual qualitative research
strategies, global level, 481–482 Program evaluation, 511–537 (CQR), 256, 262, 274–28
strategies, micro level, 478–481 budget, 515, 527, 528, 529 field notes, 258, 259, 267, 285,
therapeutic relationship, global conceptualization of, 514, 520, 536 286, 531, 535
level, 485–486 definition of, 511–514 gathering data in, 282–291
therapeutic relationship, micro description, 518 philosophical foundation of,
level, 484–485 design, 514, 515, 516, 517, 260–261
verbal behaviors, global level, 521–526 myths and facts about, 259–260
473–474 evaluation trail, 511, 535 strategies of inquiry, 261–282
verbal behaviors, micro level, methods of, 518–528 themes, 239, 260, 262, 266,
472–473 plan of, 521–528 272–273, 292–293
measurement, 456–457, 467–468, phases of, 514–528 thick description, 291–292
479, 485 purpose, 515, 517–518 Quantitative descriptive designs,
perspective, 457–458 and research, 511, 513–514 224–255
representative instruments, 468 setting the boundaries, 515–518, 536 cluster analysis, 225, 234, 238–243
Arousal Scale, 468, 477 Program evaluation costs, 527–528 ex post facto, 243–244
Brief Structured Recall, 468, 476 financial, 527 factor analysis, 234–238
California Psychotherapy fixed costs, 527 passive design, 11, 255
Alliance Scales, 485, 491 per unit costs, 527 survey research, 225–233
Categories of Semantic Cohesion time, 527–528 Quasi–experimental designs, 158,
Analysis, 484 Program evaluation report, 529, 176–177, 178–180
Check List of Interpersonal 532–535 cohort designs, 187–191
Transactions, 468, 484 disseminating the report, 534–535 example of a cohort design,
Client Experiencing Scale, 468, executive summary, 532, 534, 536 189–191
487–488 recommendations, 532–534, 536 example of a nonequivalent-group
Client Reactions System, 468, 480 review by stakeholders, 534 design, 178
Client Vocal Quality, 468, 469, 487 sections of, 532 interpretable nonequivalent-group,
Comprehensive Scale of standards, 534–535 183–187
Psychotherapy Session Psychometric Issues, 313–330 nonequivalent group designs, 189,
Contructs, 468, 477, 491 calculating estimates of reliability, 196, 180–191
Counselor Intentions List, 468, 319–320 pretreatment-posttreatment cohort,
480, 481 compounding errors, 317 189
Counselor Rating Form, 483 effects of unreliability, 320–322 reversed treatment, 183, 185–186
Discussion Units, 468 generalizability, 330. See also time-series designs, 191–196
Emotional Facial Action Coding Generalizability uninterpretable nonequivalent-
System, 470 interpreting reliability estimates, group, 181–183
Hill Counselor Verbal Response 317–318
Modes Category System, 452, inter-rater disagreement, 316–317
463, 472, 478 method variance, 327–330. Race, 367
Hill Interaction Matrix, 468, 477 See also Method variance Racial identity, 35, 56, 247, 354, 361,
Impact Message Inventory, 484 reliability, 314–322. See also 364
Interpersonal Communication Reliability Random assignment, 147, 150, 154,
Rating Scale, 482 random response error, 314–315 187, 206, 211
Penn Adherence and Competence scoring and recording errors, 317 dependent samples, 163–166
Scale for Supportive-Expressive specific error, 315 Randomized posttest-only design,
Therapy, 489–490 transient error, 315–316 91–92
Penn Helping Alliance Rating validity, 322–330. See also Validity Random samples, 346, 352 364
Scale, 433 Random selection, 345–349,
Session Evaluation Questionnaire, 352–355, 364
485 Qualitative data analysis, 291–293, Raters, 168, 304, 310, 316, 343
Supervision Questionnaire, 474 460, 479 Ratings of other persons and events,
Therapist Session Intentions, 481 focus groups, 513, 524, 525, 530, 332, 335–336, 342
Thought-Listing Form, 468, 475 531 Reactivity, 151, 253, 303, 307,
Working Alliance Inventory, consensual qualitative research 313, 331
485, 491 (CQR), 256, 262, 274–28 Readability, 331, 503, 504
Subject Index 665
Unobtrusive measures, 307, 331, 332, multiple measures, 330, 204, 233 latin Square Designs, 168–169
340–341 structural equation modeling, 253, strengths and limitations,
325, 328, 329, 342 170–172
View of Human Nature, 18–19 Worldviews, 11, 259, 404
Validity, 81–105, 322–330 Virtue ethics, 109, 112–113. See also constructivism, 11–12,
construct validity, 85, 97–103, 322 Ethical Principles, virtue ethics 260–263
convergent validity, 322, 325, 328 critical theory, 13–14,
discriminant validity, 322, 328, 332 260
factor analysis, 322–324 Within-subjects designs, 166–172 positivism, 6, 7, 9–10, 20
method variance, 325, 327–330 crossover Designs, 166–168 postpositivism, 7, 10–11, 20
TO THE OWNER OF THIS BOOK:
I hope that you have found Research Design in Counseling, Third Edition, useful. So that this
book can be improved in a future edition, would you take the time to complete this sheet and
return it? Thank you.
Department:____________________________________________________________________
Instructor’s name:_______________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
5. Were all of the chapters of the book assigned for you to read?_________________________
6. In the space below, or on a separate sheet of paper, please write specific suggestions
for improving this book and anything else you’d care to share about your experience
in using this book.
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________
______________________________________________________________________________________________________