Strengths and weaknesses of different methods 1

Strengths and Weaknesses of Questionnaires

A questionnaire is a list of written questions that can be completed in one of two basic ways. Firstly,
respondents could be asked to complete the questionnaire with the researcher not present. This
is a postal questionnaire and (loosely) refers to any questionnaire that a respondent completes without
the aid of the researcher. Secondly, respondents could be asked to complete the questionnaire by verbally
responding to questions in the presence of the researcher. This variation is called a structured interview.

Although the two variations are similar (a postal questionnaire and a structured interview could contain
exactly the same questions), the difference between them is important. If, for example, we are concerned
with protecting the respondent’s anonymity then it might be more appropriate to use a postal
questionnaire than a structured interview.

Questionnaires are restricted to two basic types of question:

Closed-ended (or “closed question”) is a question for which a researcher provides a suitable list of
responses (e.g. Yes / No). This produces mainly quantitative data.
Open-ended (or “open question”) is a question where the researcher doesn’t provide the respondent with
a set answer from which to choose. Rather, the respondent is asked to answer "in their own words". This
produces mainly qualitative data.

Strengths:
Good for measuring attitudes and eliciting other content from research participants Inexpensive (especially mail
questionnaires and group-administered questionnaires)
Can administer to probability samples
Quick turnaround
Can be administered to groups
Perceived anonymity by respondents possibly high
Moderately high measurement validity for well-constructed and well-tested questionnaires
Low dross rate for closed-ended questionnaires
Ease of data analysis for closed-ended items

Weaknesses:
Need validation
Must be kept short
Might have missing data
Possible reactive effects (e.g., response sets, social desirability)
Non response to selective items
Response rate possibly low for mail questionnaires
Open-ended items possibly resulting in vague answers
Open-ended items possibly reflecting differences in verbal ability, obscuring the issues of interest
Data analysis sometimes time-consuming for open-ended items

Strengths and Weaknesses of Interviews

This technique is used to collect qualitative data by setting up a situation (the interview) that allows a respondent the time
and scope to talk about their opinions on a particular subject. The focus of the interview is decided by the researcher and
there may be areas the researcher is interested in exploring. The objective is to understand the respondent's point of view
rather than make generalisations about behaviour. It uses open-ended questions, some suggested by the researcher (“Tell

1
Kemper, E.A., Stringfield, S. & Teddlie, C. (2003) Mixed Methods Sampling Strategies in Social cience
Research in Tashakkori, A. & Teddlie, C. (Eds. 2003) Handbook of Mixed Methods in Social and
Behavioural Research, Sage, Thousand Oaks. Chapter Data Collection Strategies in Mixed Methods
Research Chapter 11. Tables 11.4-11.9.
me about…”) and some arise naturally during the interview (“You said a moment ago…can you tell me more?”).
The researcher tries to build a rapport with the respondent and the interview is like a conversation. Questions are asked
when the interviewer feels it is appropriate to ask them. They may be prepared
questions or questions that occur to the researcher during the interview. The wording of questions will not
necessarily be the same for all respondents.

Strengths:
Good for measuring attitudes and most other content of interest
Allow probing by the interviewer
Can provide in-depth information
Allow good interpretive validity
Low dross rate for closed-ended interviews
Very quick turnaround for telephone interviews
Moderately high measurement validity for well-constructed and well-tested interview protocols
Can use with probability sample
Relatively high response rates often attainable Useful for exploration and confirmation

Weaknesses:
In-person interviews expensive and time-consuming
Possible reactive and investigator effects
Perceived anonymity by respondents possibly low
Data analysis sometimes time-consuming for open-ended items
Measures in need of validation

Strengths and Weaknesses of Focus Groups

Focus groups are a method of group interviewing in which the interaction between the moderator and the group,
as well as the interaction between group members, serves to elicit information and insights in response to
carefully designed questions. The dynamic nature of the questions asked by the moderator and the group
process, produces a level of insight that is rarely derived from 'unidirectional' information collection devises
such as observation, surveys and less interactional interview techniques. Methods of recording and analyzing
information gathered during focus groups, and strategies for collecting unbiased information have helped focus
group research to gain credibility as an accurate and useful source of information collection.

Focus group methods gained popularity in marketing research. In the 1980s social scientists recognized the
value of focus groups for qualitative research and adapted the techniques accordingly. In the 1990s focus group
strategies have become widely researched and used in social sciences and human service organizations.

When to Use Focus Groups

Focus groups, like any other program evaluation method, are more appropriate for some situations than others.
Morgan & Krueger (1993) discuss instances when focus groups are beneficial:
• When the security provided by the group allows members who are lower in the 'power hierarchy' within an
organization to express feelings and experiences that they would not otherwise share.
• When the target audience is so different from decision makers that different terminology and points of view
• can be illuminated and understood (this information can be useful when constructing questionnaires for
those audiences).
• When desired information about behaviors and motivations is more complex than a questionnaire is likely
to
• reveal. Through a series of well designed questions, focus groups can often get at more honest and in depth
information.
• When one is interested in finding out the nature of consensus. While several respondents completing a
questionnaire may indicate that they 'agree' with an item, focus groups may reveal fundamental differences
among group members concerning the conditions of that agreement.
• When target audiences may not take questionnaires seriously or answer them honestly. Effective focus
groups
• will communicate a desire to obtain meaningful, honest information. Superficial or patronizing responses as
well
• as critical responses can be challenged and or put into an appropriate context.
• In situations where there is organizational conflict and or alienation, members of focus groups and their
 constituencies may feel 'listened to'. This may result in an honest and meaningful exchange of information.

Strengths:
Useful for exploring ideas Allow good interpretive validity
Can obtain in-depth information about exactly how people think about an issue Allow study of how participants
react to each other
Allow probing
Allow most content to be tapped Allow quick turnaround

Weaknesses:
Sometimes expensive
Possible reactive and investigator effects if participants feel that they are being watched May be dominated by
one or two participants
Very difficult to generalize if small unrepresentative samples are used Focus group moderator possibly biased
Might have high dross rate
Measurement validity possibly low
Usually should not be the only data collection method used in a study Data analysis sometimes time-consuming

Strengths and Weaknesses of Standardized Tests

Standardization 2 is a research process that includes careful selection of test items, administration of the items to
a representative sample drawn from a defined population, statistical analysis of results, establishment of age-
based norms, and development of instructions and response scoring procedures. A standardized test should be
self-contained in that all needed stimulus materials and recording forms should be included in it. The instrument
should allow for uniform administration and scoring of responses by different examiners in varied settings to
minimize any influence different examiners and settings may have on the skills being measured.

Most standardized tests also are norm-referenced. Norm-referenced tests are always standardized. Norm-
referenced tests help compare a client’s performance to the performance of another group of individuals called
the normative group. A norm is the performance measure of a normative group on a tested skill. The normative
group is usually a representative sample of individuals of the same age and possibly the same sex as the client.
A norm-referenced test should provide detailed information on the normative sample, including:

• The size of the sample; the American Psychological Association (1999) states that it should not be less than
100; meeting only this minimum requirement of a sample size, however, will result in a test of extremely
limited applicability.
• The ethnocultural and socioeconomic levels of the individuals selected for standardization; the greater the
variety of these variables and the more adequate the different subsamples selected, the higher the
applicability of the test.
• The geographic distribution of selected participants for standardization; once again, the greater the
geographic distribution of the samples, the higher the applicability of the test.
• Other relevant variables including the IQ or medical status of the sample; when children are the
participants, the manual should also specify the range of education and occupation of the parents.
• Descriptive statistics; the manual should include means and standard deviations for all groups on whom the
test items have been administered; additional statistical transformation of the raw scores (e.g., percentiles)
also may be specified.

Strengths:
Can provide good measures of many characteristics of people Strong psychometric properties (high
measurement validity) Availability of reference group data
Possibly can be administered to groups
May provide "hard" quantitative data
Allow comparability of common measures across research populations Instruments usually already developed
Wide range of tests available (with most content able to be tapped) Response rate high for group-administered
tests
Ease of data analysis
Researcher-designed tests possibly tailored to local needs

2
http://www.pluralpublishing.com/media/media_acdc_SamplePages.pdf pp 42,43
Weaknesses:
Can be expensive if test must be purchased for each research participant Possible reactive effects (e.g., response
sets)
Possibly not appropriate for a local population
Sometimes biased against certain groups of people
Potential non response error to selected items on the test
Might lack psychometric data
Psychometric data possibly do not apply to local population

Strengths and Weaknesses of Observational Data

What is participant observation?
Participant observation is a qualitative method with roots in traditional ethnographic research,whose objective is
to help researchers learn the perspectives held by study populations. As qualitative researchers, we presume that
there will be multiple perspectives within any given community. We are interested both in knowing what those
diverse perspectives are and in understanding the interplay among them.

Qualitative researchers accomplish this through observation alone or by both observing and participating, to
varying degrees, in the study community’s daily activities. Participant observation always takes place in
community settings, in locations believed to have some relevance to the research questions. The method is
distinctive because the researcher approaches participants in their own environment rather than having the
participants come to the researcher. Generally speaking, the researcher engaged in participant observation tries
to learn what life is like for an “insider” while remaining, inevitably, an “outsider.”

While in these community settings, researchers make careful, objective notes about what they see, recording all
accounts and observations as field notes in a field notebook. Informal conversation and interaction with
members of the study population are also important components of the method and should be recorded in the
field notes, in as much detail as possible. Information and messages communicated through mass media such as
radio or television may also be pertinent and thus desirable to document.

Also see the Book of Readings chapter on Observation methods 126-129.

Strengths:
Allow one to directly see what people do without having to rely on what they say they do
Allow relatively objective measurement of behavior
Can be used with participants with weak verbal skills
Good for description
Can give access to contextual factors operating in natural social settings
Moderate degree of realism (when done outside of the laboratory)

Weaknesses:
Reasons for behavior possibly unclear
Possible reactive and investigator effects when respondents know they are being observed
Possibility of observer being biased (e.g., selective perception)
Possibility of observer "going native" (i.e., over identifying with the group being studied)
Interpretive validity possibly low
Cannot observe large populations
Unable to observe some content of interest Dross rate possibly moderately high
More expensive to conduct than questionnaires and tests Data analysis sometimes time-consuming

Strengths and Weaknesses of Secondary Data

Strengths of documents and physical data:
Can provide insight into what people think and what they do Unobtrusive, making reactive and investigator
effects very unlikely
Can be collected for time periods occurring in the past (e.g., historical data) Personal documents often
unobtrusive
Useful for corroboration
Grounded in local setting
Useful for exploration
Strengths of archived research data:
Available on a wide variety of topics Inexpensive
Often reliable and valid (high measurement validity) Can study trends
Useful for exploration and confirmation Ease of data analysis
Sometimes based on large probability samples
Weaknesses of documents and physical data:
May be incomplete because of selective reporting or recording Access to some types of content possibly
difficult
Interpretive validity possibly low
Might not apply to general populations

Weaknesses of archived research data:

Might not apply to local population or to specific research need (content problem) Data possibly dated (e.g.,
ecological and temporal validity possibly low) Interpretive validity possibly low
Many of the most important findings possibly already have been mined from the data

Strengths and Weaknesses of Case Studies: 3

Most case study advocates point out that case studies produce much more detailed information than
what is available through a statistical analysis. Advocates will also hold that while statistical methods
might be able to deal with situations where behavior is homogeneous and routine, case studies are
needed to deal with creativity, innovation, and context. Detractors argue that case studies are difficult
to generalize because of inherent subjectivity and because they are based on qualitative subjective
data, generalizable only to a particular context.

Flexibility
The case study approach is a comparatively flexible method of scientific research. Because its project
designs seem to emphasize exploration rather than prescription or prediction, researchers are
comparatively freer to discover and address issues as they arise in their experiments. In addition, the
looser format of case studies allows researchers to begin with broad questions and narrow their focus
as their experiment progresses rather than attempt to predict every possible outcome before the
experiment is conducted.

Emphasis on Context
By seeking to understand as much as possible about a single subject or small group of subjects, case
studies specialize in "deep data," or "thick description"--information based on particular contexts that
can give research results a more human face. This emphasis can help bridge the gap between abstract
research and concrete practice by allowing researchers to compare their firsthand observations with
the quantitative results obtained through other methods of research.

Inherent Subjectivity
"The case study has long been stereotyped as the weak sibling among social science methods," and is
often criticized as being too subjective and even pseudo-scientific. Likewise, "investigators who do
case studies are often regarded as having deviated from their academic disciplines, and their
investigations as having insufficient precision (that is, quantification), objectivity and rigor" (Yin

3 Writing Center (2004), Colorado State University, Co. Retrieved 3/21/04 from

http://writing.colostate.edu/references/research/casestudy/pop2a.cfm.
1989). Opponents cite opportunities for subjectivity in the implementation, presentation, and
evaluation of case study research. The approach relies on personal interpretation of data and
inferences. Results may not be generalizable, are difficult to test for validity, and rarely offer a
problem-solving prescription. Simply put, relying on one or a few subjects as a basis for cognitive
extrapolations runs the risk of inferring too much from what might be circumstance.

High Investment
Case studies can involve learning more about the subjects being tested than most researchers would
care to know--their educational background, emotional background, perceptions of themselves and
their surroundings, their likes, dislikes, and so on. Because of its emphasis on "deep data," the case
study is out of reach for many large-scale research projects which look at a subject pool in the tens of
thousands. A budget request of $10,000 to examine 200 subjects sounds more efficient than a similar
request to examine four subjects.

Ethical Considerations
Researchers conducting case studies should consider certain ethical issues. For example, many
educational case studies are often financed by people who have, either directly or indirectly, power
over both those being studied and those conducting the investigation (1985). This conflict of interests
can hinder the credibility of the study.

The personal integrity, sensitivity, and possible prejudices and/or biases of the investigators need to
be taken into consideration as well. Personal biases can creep into how the research is conducted,
alternative research methods used, and the preparation of surveys and questionnaires.
A common complaint in case study research is that investigators change direction during the course of
the study unaware that their original research design was inadequate for the revised investigation.
Thus, the researchers leave unknown gaps and biases in the study. To avoid this, researchers should
report preliminary findings so that the likelihood of bias will be reduced.

Concerns about validity, reliability, and generalizability

Merriam (1985) offers several suggestions for how case study researchers might actively combat the
popular attacks on the validity, reliability, and generalizability of case studies.