METHODS

controversy, debate or a lack of clarity

exist.
However, the many applications and
descriptions available in the literature can

The Delphi method be confusing, and the Delphi method,

with a few exceptions, remains relatively
unexploited in psychological research
Susanne Iqbal and Laura Pipon-Young with a step-by-step guide (e.g. Graham & Milne, 2003; Haggard &
Haste, 1986; Haste et al., 2001; Jeffery et
al., 2000; Petry et al., 2007). To make the
Delphi method more accessible for
psychologists unfamiliar with this
method, this article provides a practical
step-by-step guide based on our
he Delphi survey method is popular are asked to identify the range of experiences of conducting Delphi studies

T in many disciplines. Originally

developed in the US as a means of
forecasting future scenarios, this method
salient issues.
I It collates ideas from Round 1 to
construct the survey instrument
in clinical psychology. It is not an
exhaustive account, and further guidance
is available elsewhere (e.g. Hackett et al.,
has been used to determine the range of distributed in subsequent rounds. 2006, Graham & Milne, 2003, Keeney et
opinions on particular matters, to test I It has an evaluation phase (third or al., 2006, Schneider & Dutton, 2002).
questions of policy or clinical relevance, further rounds) where panellists are
and to explore (or achieve) consensus on provided with the panel’s responses
disputed topics. Although there is and asked to re-evaluate their original What is the aim of your study?
considerable variation in how the method responses. The first step is to determine whether the
is applied, the Delphi method has its own I It is interested in the formation or study aims to measure the diversity of
distinct characteristics: exploration of consensus, often opinions on a topic or to steer a group
I It uses a group of participants (known defined as the number of panellists towards consensus. This is an important
as ‘panellists’) agreeing with each distinction in terms of the execution of
specially selected other on the Delphi. In general, if your study aims
for their particular questionnaire items. to generate consensus, three or more
expertise on a rounds are preferable. Ideally, the same
topic. The Delphi method is panel should be retained throughout and
I It is often particularly useful in high response rates are particularly
conducted across areas of limited important in order to determine the
a series of two or research, since survey impact of group feedback on panellists.
more sequential instruments and ideas On the other hand, if the Delphi
questionnaires are generated from a process is a means of measuring opinions,
known as ‘rounds’. knowledgeable fewer rounds are generally acceptable.
It employs an participant pool Having a complete dataset is less vital,
initial ‘idea (Hasson et al., 2000), and the panel can be expanded across
generation’ stage, Measuring diversity or steering and it is suited to rounds by inviting more panellists in
in which panellists towards consensus? explore areas where Round 2.

Strongly Strongly Your choice

1 2 3 4 5 6 7 8 9 10 Can’t say
disagree agree was

Overall 100%
0 0 0 4 0 0 4 13 29 50 0 10
percentages N=24

Figure 1: Fictitious example of an individualised feedback in Round 3 statement: Smoking should be banned in hospitals.

Beretta, R. (1996). A critical review of the technique: A critique. Journal of 105–112. Psychological Society, 39, 321–324.
references

Delphi technique. Nurse Researcher, Advanced Nursing, 12, 729–734. Hackett, S., Masson, H. & Philipps, S. Hardy, J.D., O’Brien, A.P., Gaskin, C.J. et
3(4), 79–89. Graham, L. & Milne, D. (2003). Developing (2006). Exploring consensus in al. (2004). Practical application of the
Braun, V. & Clarke, V. (2006). Using basic training programmes: A case practice with youth are sexually Delphi technique in a bicultural
thematic analysis in psychology. study illustration using the Delphi abusive: Findings from a Delphi study mental health nursing study in New
Qualitative Research in Psychology, 3, method in clinical psychology. Clinical of practitioner views in the United Zealand. Journal of Advanced Nursing,
77–101. Psychology and Psychotherapy, 10, Kingdom and the Republic of Ireland. 46(1), 95–109.
De Meyrick, J. (2003). The Delphi method 55–63. Child Maltreatment, 11(2), 146–156. Hasson F., Keeney S. & McKenna H.
and health research. Health Graneheim, U. & Lundman, B. (2004). Haggard, M. & Haste, H. (1986). One (2000). Research guidelines for the
Education, 103(1), 7–16 Qualitative content analysis in nursing generation after 1984: Psychology in Delphi survey technique. Journal of
Goodman, C.M. (1987). The Delphi research. Nurse Education Today, 24, the year 2010. Bulletin of the British Advanced Nursing, 32, 1008–1015.

598 vol 22 no 7 july 2009

 methods

Decide the structure

The next step is to decide the number of
rounds, to draw up a timeframe and to
The Delphi in use
construct study materials (e.g. letters to Iqbal et al. (in press) used the Delphi to explore and approach consensus in a study exploring
participants, consent forms, complete sexually inappropriate behaviours in children under the age of 10. Review of the pertinent
ethics procedures). literature had revealed that children’s sexual behaviours were judged differently by different
A two-round Delphi (e.g. Petry et al., professionals. The Delphi seemed appropriate to explore this sensitive topic as it is an ideal
2007) is most suitable when there is a tool to expose all the different positions, including arguments for and against these positions,
clear literature base from which to to generate consensus and to communicate this.
establish the survey instrument and if the The Delphi is a method for structuring a group communication process so that the
main aim is to take the temperature of process is effective in allowing a group of individuals, as a whole, to deal with a complex
opinion on a topic. Although quantitative problem’ (Linstone & Turoff, 2002, p.3), and based on the idea that it is possible and valuable
questionnaires have been used in the first to reach a consensus (Stone Fish & Busby, 2005). By feeding back percentages of all views
round, a qualitative first round is optimal, to each participant and inviting them to reflect on their responses in the light of these scores,
because the primary function of the consensus was achieved that children who display sexually inappropriate behaviours should
Delphi method is to explore an area of not be called ‘sex offenders’. No consensus was achieved with regard to many other sexual
future thinking that goes beyond the behaviours, particularly those considered ‘normal’.
currently known or believed. Also, the In line with the Delphi methodology, results were disseminated via participants
reliability and validity of the study may themselves, reports, and journal publications. ‘Divergence’ (when no consensus was
be improved if an initial group of experts achieved) was also fed back to highlight how little agreement existed amongst very
produces the items. experienced professionals with regard to what counts as normal sexual behaviours.
When exploring consensus, rounds This should improve practice and allow for further research to be carried out.
may continue until consensus is reached.
However, this approach can quickly
compromise panellists’ response rates and
enthusiasm. Three rounds, which would knowledge, qualifications and proven Delphi method sets out to generate new
typically take four months, often suffice track records in the field (Keeney et al., ideas, a quantitative Q1 seems to defy this
(Stone Fish & Busby, 2005). 2001), although of course expertise purpose.
comes in many guises and may include The Q1 is usually created following
those who are ‘experts by experience’ a detailed literature review, consultation
Selecting panellists (Hardy et al., 2004). In general, a varied with relevant individuals and
Panellists form the lynchpin of the panel is considered best in producing a consideration of the aims of your Delphi
Delphi, and clear inclusion criteria should credible questionnaire, and individuals study. Generally speaking, asking
be applied and outlined as a means of who might provide a minority or differing panellists to spend 30 minutes completing
evaluating the results and establishing the perspective should be actively recruited to the questionnaire is considered
study’s potential relevance to other the panel (Linstone & Turoff, 2002). reasonable, and pilot testing is essential
settings and populations. The number of With regard to the recruitment to determine timeframes as well as
panellists depends very much on the process itself, panellists are often readability and relevance of the questions.
topic area as well as the time and recruited via letter or e-mail. Recruitment Online surveys can be an efficient
resources at the researchers’ disposal. can be broadened through ‘snowballing’ alternative to posting questionnaires and
Although Delphi surveys have been (asking panellists to pass on invitations often appeal to panellists. Web services
conducted with as few as seven and as to other relevant individuals). (e.g. surveymonkey.com) can be a simple
many as 1000 panellists, Turoff (2002) way of constructing online
recommends panels between 10 and 50. questionnaires. Once the questionnaire
These numbers seem more appropriate, The questionnaires has been distributed, following up non-
given the amount of data and subsequent The more open-ended the Round 1 responders is recommended as high
analyses each panellist generates. questionnaire (Q1) the better, ideally response rates can improve the credibility
Researchers must also decide how to involving a series of open-ended of a study (Beretta, 1996). Ideally, a 70
conceptualise and define ‘expertise’. The questions inviting panellists to per cent response rate should be
method may be undermined if panellists brainstorm. A quantitative ‘tick-box’ style maintained (Sumsion, 1998). We found
are recruited who lack specialist format may also be used, but since the that regular contact, flexibility around

Haste, H., Hogan, A. & Zacharious, Y. interventions, service principles and (2006). Consulting the oracle: Ten from http://is.njit.edu/pubs/
(2001). Back (again) to the future. The service organisation for severe mental lessons from using the Delphi delphibook.
Psychologist, 14(1), 30–33. illness and substance misuse technique in nursing research. Petry, K., Maes, B. & Vlaskamp, C. (2007).
Iqbal, S. Hackett, S. & Callanan, M. (in problems. Journal of Mental Health, 9, Journal of Advanced Nursing, 53(2), Operationalizing quality of life for
press). ‘Normal’ and ‘inappropriate’ 371–384. 205–212. people with profound multiple
childhood sexual behaviours: Keeney S., Hasson F. & McKenna H.P. Likert, R. (1932). A technique for disabilities: A Delphi study. Journal of
Findings from a Delphi study of (2001). A critical review of the Delphi measurement of attitudes. Archives of Intellectual Disability Research, 51(1),
professionals in the UK. Journal of technique as a research methodology Psychology, 140, 44–53. 334–349.
Sexual Aggression. for nursing. International Journal of Linstone, H.A. & Turoff, M. (Eds.) (2002). Sackman, H. (1975). Delphi critique.
Jeffery, D., Ley, A., Bennun, I. & McLaren, Nursing, 38, 195–200. The Delphi method: Techniques and Boston: Lexington Books.
S. (2000). Delphi survey of opinion on Keeney, S., Hasson, F. & McKenna, H. applications. Retrieved 20 May 2009 Schneider, J. & Dutton, J. (2002).

read discuss contribute at www.thepsychologist.org.uk 599

 methods

deadlines and individual ‘thank you’ Table 1: Advantages and disadvantages of the Delphi survey method
messages increased response rates.
The Round 2 questionnaire is Advantages Disadvantages
constructed from the data gathered from
the Q1. Commonly, a quantitative, ‘tick- Very flexible methodology that can Method suffers from a lack of guidance
box’ style survey using Likert (1932) type accommodate many variations and and agreed standards regarding
agreement scales or ranking scales are applications. interpretation and analyses of results,
used. The construction of the Round 2 universally agreed definitions of
questionnaire (Q2) is often time- consensus, as well as criteria for how
consuming. The use of methodological panellists should be selected (Sackman,
tools such as qualitative content analysis 1975).
(e.g. Graneheim & Lundman, 2004) or
thematic analysis (e.g. Braun & Clarke, Draws together existing knowledge and Less efficient as a means of generating or
2006) is necessary to make the study pinpoints areas of agreement/ testing new knowledge and theories.
methodologically more robust. disagreement.
Furthermore, careful attention to
principles of questionnaire design is vital,
Enables a group communication that Generalisations are limited: another panel
and extended piloting may be necessary
otherwise might have been impossible may reach different conclusions, and it
to iron out ambiguous, repetitive or
due to geography, time or other cannot be concluded that the only or
inaccurate items.
constraints (Stone Fish & Osborn, correct issues have been identified.
On return of the Q2, descriptive data
1992).
analyses of the panel’s responses can
begin so that the Round 3 questionnaire Economical in terms of financial outlay High levels of commitment required from
(Q3) can be constructed. The purpose of and participant time. Potentially panellists; drop-out levels often high.
the Q3 is to invite panellists to consider rewarding research process for
their scores in the light of the group participants with multiple inbuilt
response and decide whether they want to opportunities for feedback.
change any of their responses. We suggest
feeding back percentages and providing Makes the potentially confounding May lack some of the richness and depth
individual round scores for every item interpersonal processes often found in ‘live’ groups.
(see Figure 1). This provides a visual occurring in ‘live’ groups less likely
means for the panellists of assessing the (e.g. conformity to the dominant view).
diversity of responses. It also allows them
to check that researchers have recorded Anonymity between panellists can Anonymity may produce less ‘ownership’ of
correct responses. encourage creativity, honesty and ideas. Delphi process assumes panellists
balanced consideration of ideas (De are willing or able to elucidate issues
Meyrick, 2003). individually and respond honestly.
Analyses and dissemination
Upon receipt of the completed Q3, you
need to check whether any changes have
been made, in which case the data need Finally, disseminate your findings Goodman (1987), and Sackman (1975).
to be re-analysed. Percentages, medians, (write a consensus report, article, present In our experience, the benefits outweigh
interquartile ranges, means and standard findings to services, etc.) amongst its drawbacks; and this method seems
deviations are commonly calculated. concerned parties, including your particularly relevant for psychology.
Results can be presented in various participants. Traditionally, there has been a divide
ways. This includes reporting only those between quantitative and qualitative
items that have reached a pre-agreed level methods. The Delphi method can straddle
of consensus (e.g. Petry et al., 2007), Strengths and weaknesses this divide. By virtue of its procedural
listing all items in order of consensus Like any other survey method, the Delphi structure (to incorporate both qualitative
magnitude (Hardy et al., 2004), or also method has strengths and weaknesses. and quantitative methods), it provides the
reporting those areas in which there is These are summarised in Table 1, and opportunity to achieve a more complete
debate amongst the panel. further critiques can be found in picture of the phenomenon under study.

Attitudes towards disabled staff and Delphi study. Family Relations, 41, I Susanne Iqbal is a Chartered Clinical
the effect of the national minimum 409–416. Psychologist at George MacKenzie House,
wage. Disability and Society, 3, 283–306. Sumsion, T. (1998). The Delphi technique.
Fulbourn Hospital, Cambridge
Stone Fish, L. & Busby, D. (2005). The British Journal of Occupational Therapy,
Delphi method. In D. Sprenkle & F. 61(4), 153–156.
Susanne.Iqbal@cpft.nhs.uk
Piercy (Eds.) Research methods in Turoff, M. (2002). The Policy Delphi. In H.
family therapy (2nd edn, pp.238–253). Linstone & M. Turoff (Eds.) The Delphi
New York: Guilford Press. method (pp.80–96). Retrieved 22 May I Laura Pipon-Young is a Chartered Clinical
Stone Fish, L. & Osborn, J. (1992). 2009 from Psychologist at the Women’ s Service,
Therapists’ views of family life: A http://is.njit.edu/pubs/delphibook. Secure & Forensic Services, Hellingly, East
Sussex

600 vol 22 no 7 july 2009

read discuss contribute at www.thepsychologist.org.uk 601