Appropriate Web Usability Evaluation
Appropriate Web Usability Evaluation
Appropriate Web Usability Evaluation
Software Engineering
Thesis no: MSE-2008-03
Feb 2008
School of Engineering
Blekinge Institute of Technology
Box 520
SE 372 25 Ronneby
Sweden
Contact Information:
Author(s):
Azeem Umar
E-mail: ifazeem@gmail.com
Kamran Khan Tatari
E-mail: kamrankhan.tatari@gmail.com
University advisor(s):
Dr. Mia Persson
Department of Software Engineering
School of Engineering
Blekinge Institute of Technology
Box 520
SE 372 25 Ronneby
Sweden
Internet
Phone
Fax
: www.bth.se/tek
: +46 457 38 50 00
: + 46 457 271 25
ii
ABSTRACT
Web development is different from traditional
software development. Like in all software applications,
usability is one of the core components of web
applications. Usability engineering and web engineering
are rapidly growing fields. Companies can improve
their market position by making their products and
services more accessible through usability engineering.
User testing is often skipped when approaching
deadline. This is very much true in case of web
application development. Achieving good usability is
one of the main concerns of web development. Several
methods have been proposed in literature for evaluating
web usability. There is not yet an agreement in the
software development community about which usability
evaluation method is more useful than another. Doing
extensive usability evaluation is usually not feasible in
case of web development. On the other hand unusable
website increases the total cost of ownership. Improved
usability is one of the major factors in achieving
satisfaction up to a sufficient level. It can be achieved
by utilizing appropriate usability evaluation method, but
cost-effective usability evaluation tools are still lacking.
In this thesis we study usability inspection and
usability testing methods. Furthermore, an effort has
been made in order to find appropriate usability
evaluation method for web applications during product
development and in this effort we propose appropriate
web usability evaluation method which is based on
observation of the common opinion of web industry.
ACKNOWLEDGEMENTS
I would like to heartily acknowledge my advisor Mia Persson for her excellent guidance
and encouragement which has lead to completion of my master thesis with in time. To my
father who is the core reason of what ever good thing I have achieved in my life.
Azeem Umar
I want to express my profound thanks and indebtedness to my supervisor for her patient
guidance, generous support and encouraging attitude during my thesis work. To my mother
whose prayers are chasing in every moment of my life.
Kamran Khan Tatari
ii
TABLE OF CONTENTS
ABSTRACT ...........................................................................................................................................I
LIST OF FIGURES............................................................................................................................. V
LIST OF TABLES..............................................................................................................................VI
1
INTRODUCTION ....................................................................................................................... 1
1.1
BACKGROUND........................................................................................................................ 1
1.2
PURPOSE ................................................................................................................................ 2
1.2.1 Conceptual Map of Thesis ................................................................................................ 2
1.3
AIMS AND OBJECTIVES .......................................................................................................... 3
1.4
RESEARCH QUESTIONS ........................................................................................................... 3
1.5
EXPECTED OUTCOMES............................................................................................................ 3
1.6
RESEARCH METHODOLOGY ................................................................................................... 3
USABILITY ................................................................................................................................. 5
2.1
USABILITY BACKGROUND ..................................................................................................... 6
2.1.1 Hardware and Software Usability .................................................................................... 6
2.1.2 Iterative Human-Centric Design....................................................................................... 6
2.2
ISO MODELS ON USABILITY .................................................................................................. 7
2.2.1 ISO 9126-1 Quality Model................................................................................................ 7
2.2.2 ISO 9241-11 Guidance of Usability.................................................................................. 9
2.2.3 Comparison between ISO 9126-1 and ISO 9241-11......................................................... 9
2.3
J.NIELSEN DEFINITION OF USABILITY .................................................................................. 10
2.4
BENEFITS OF USABILITY ...................................................................................................... 10
2.5
EFFECTS OF USER CHARACTERISTICS ON USABILITY ........................................................... 11
iii
7
COMPARISON OF WEB USABILITY EVALUATION METHODS WITH CARE
METHODOLOGY ............................................................................................................................. 35
7.1
CARE METHODOLOGY ......................................................................................................... 35
7.1.1 Reasons for selecting CARE ........................................................................................... 36
7.2
COMPARISON OF UEMS ON THE BASIS OF CARE ............................................................... 36
7.2.1 Comparison of web usability inspection methods........................................................... 36
7.2.2 Comparison between web usability testing methods....................................................... 36
7.3
FILTERED LIST OF UEMS .................................................................................................... 37
7.3.1 Combination of Web UEMs ............................................................................................ 37
8
DEMOGRAPHICS ................................................................................................................... 38
SURVEY METHOD, THE 100 DOLLAR TEST .......................................................................... 38
RESULTS .............................................................................................................................. 40
ANALYTICAL REVIEW .......................................................................................................... 41
10
REFERENCES........................................................................................................................... 44
iv
LIST OF FIGURES
Figur 1:Thesis Conceptual Map................................................................................................2
Figur 2:Iterative Human Centric Design Activities (ISO 1307)...............................................7
Figur 3:Software quality characteristics according to ISO 9126-1...........................................8
Figur 4:Usability sub-characteristics according to ISO 9126-1................................................8
Figur 5:Usability sub-characteristics according to ISO 9241-11..............................................9
Figur 6:Jakob Nielsens Definition of Usability.....................................................................10
Figur 7:Effects of user characteristics on usability.................................................................11
Figur 8:Characteristics of a Usable Website...........................................................................12
Figur 9: Web Design Process Phases......................................................................................13
Figur 10:Eight steps to web benchmarking ............................................................................14
Figur 11:Usability Pyramid for Websites ...............................................................................14
Figur 12:Relationship between interface design, evaluation and UEMs................................16
Figur 13:Derived A. Whitefield Model showing Classes of UEMs .......................................18
Figur 14:UEMs classification according to Adelman and Riedel ..........................................19
Figur 15:UEMs classification according to Y. Ivory and M. A. Hearst .................................20
Figur 16:Conceptual Visualization of Usability Evaluation Process......................................22
Figur 17:Questionnaire Process..............................................................................................32
Figur 18:Thesis Actual Map ...................................................................................................39
Figur 19:Summary of Rating E-Mail Survey in Graphical Form..........................................40
LIST OF TABLES
Table 1: Usability definitions according to three different standards .....................................5
Table 2: Hardware usability Vs Software usability..................................................................6
Table 3: Usability characteristics according to ISO 9241-11. ..................................................9
Table 4: Usability characteristics according to ISO 9241-11 ...................................................9
Table 5: ISO 9126-1 Vs ISO 9241-11 ......................................................................................9
Table 6: Mapped methods according to Wixon and Wilson Classification............................20
Table 7: Checklist by Brinck ..................................................................................................25
Table 8: Advantages and Disadvantages of Heuristic Evaluation ..........................................26
Table 9: Advantages and Disadvantages of Cognitive Walkthrough .....................................27
Table 10: Advantages and Disadvantages of Pluralistic Walkthrough...................................27
Table 11: Advantages and Disadvantages of Feature Inspection. ..........................................28
Table 12: Advantages and Disadvantages of Remote Usability Testing................................29
Table 13: Advantages and Disadvantages of Coaching Method ............................................29
Table 14: Advantages and Disadvantages of Co Discovery Method .....................................29
Table 15: Advantages and Disadvantages of Coaching Method ............................................30
Table 16: Advantages and Disadvantages of Think Aloud Protocol Method .......................30
Table 17: Interpretation from questionnaire results................................................................33
Table 18: UEMs practiced in web industry ............................................................................33
Table 19: UEMs usage in web development phases...............................................................33
Table 20: Usability Inspection Methods Comparison ............................................................36
Table 21: Usability Testing Methods Comparison.................................................................36
Table 22: Summary of Rating E- Mail Survey in Tabular form.............................................40
vi
INTRODUCTION
This chapter provides the background for this thesis, as well as the purpose, aims and
objectives of the thesis. The reader will also find the research questions along with the
research methodology.
1.1
Background
Despite of advancement in web technology, web software applications are still immature and
it posses significant risk to both industry and government [11]. But at the same time it also
represents an opportunity for software engineering researchers to extensively investigate in
this area [11]. Web application development is maturing from the experimental practice of
early years to a more professional discipline [12]. Quality is central to this maturing and it is
necessary to have a full understanding of the meaning of quality in the context of the ever
changing web applications [12]. The systematic and quantitative quality evaluation of web
applications are frequently neglected issue [16].
In literature, most work on web applications has been done on making them more powerful
but relatively little has been done to ensure their quality [13]. Important quality factors for
web applications include reliability, availability, usability and security [13].
Web site usability and accessibility continue to be a pressing problem [14]. An estimated
90% of web sites provide inadequate usability [15].
An ISO/IEC 9126-1 standard mentions six principle categories of quality characteristics.
They are functionality, reliability, usability, efficiency, maintainability and portability.
Usability represents one of the most important acceptance criteria for interactive software
applications in general and web applications in particular [17]. It is one of the most
important quality factors for web applications. Unusable web applications cause users to
reject them. A good usable web application is that one from which users can achieve their
goals effectively, efficiently and satisfactorily [5]. In order to design web applications two
things are needed to be considered. User needs and usage contexts. User needs point towards
functionally aspect of web application while usage contexts point towards usability aspect of
web application.
Usable Web sites are those which help users to accomplish a goal easily, quickly, and
pleasantly. Web usability is a core component of web quality. Without good usability
features the web quality will always be a question mark.
There is a sort of tug of war between web application content growth and need for more
usable web sites. In other words web sites are becoming more complex and at the same time
higher usability is also desired. In order to keep the balance between the two, appropriate
usability inspection and testing methods needs to be employed during product development
of websites. Web development is different from traditional software development. The main
objective of web development project is to create usable product in shortest possible time
while the main objective of software project is to create a quality product at lowest possible
cost [17]. Then web projects are of small duration about 3 to 6 months while average
software project duration is from 12 to 18 months [17]. In this situation usability inspection
and testing area are allocated with little time and this becomes one of primary reasons for the
failure of many websites [17]. There is no universally accepted web usability evaluation
method due to nature of World Wide Web domain. From literature [52, 54] and own
observation it is found that different web development companies follow different usability
evaluation methods. Many even do not follow any usability evaluation method during
product development. A Research is needed to investigate which usability evaluation
method can be appropriate for web industry during product development. This sort of
research can help web industry in improving usability of web application during product
1.2
Purpose
The purpose of this thesis is to investigate and propose an appropriate web usability
evaluation method during product development. The proposed method will be based partly
on a literature study and partly on the email survey that we will conduct in the web industry.
1.2.1
In Figure 1, a conceptual map of the thesis is shown. Web is the main focus in the map.
The conceptual map shows that Software Quality is one of the main branch of Software
Engineering field. Usability is one of the core attribute of Software Quality. There are certain
usability evaluation methods (UEMs for short) for evaluating usability of software.
Formative usability evaluation methods are those methods which are used during product
development [3]. Among the formative usability evaluation methods which method could be
the appropriate choice?
Software Quality
Usability
W
E
B
1.3
The aim of this thesis is to find out the best solution for evaluating usability of websites
during product development. By the best solution for evaluating web usability, we refer to a
usability evaluation method which is in line with the CARE methodology (See Section 7.1),
and which will be composed of one usability inspection method and one usability testing
method (See Section 5 for definitions).
The objectives of the thesis are as follows.
1. Identifying and understanding of important standards related to software usability
2. Investigation into the classification of UEMs in literature.
3. Investigating current practices of usability evaluation methods (UEMs) in web
industry
4. Analysis and comparison of formative web usability evaluation methods
We will achieve the aforementioned objectives stepwise. In order to compare formative
usability evaluation methods for the web industry, we need the knowledge of proposed
UEMs in literature. Furthermore, studying UEMs requires good understanding of the
software usability concept. We also need to know which UEMs are commonly found in
literature and the web industry. One will achieve a better understanding of software usability
by studying its standards and models.
1.4
Research questions
1.5
Expected outcomes
Expected outcome of the research paper will be a report that will contain
An understanding and an explanation of usability evaluation methods
A proposed conceptual usability evaluation process model, derived from the
classification of UEMs in literature
A rating email survey results, proposing an appropriate web usability evaluation
method for web industry during the product development stage
1.6
Research Methodology
Both quantitative and qualitative research methods will be used in our work. Three basic
research methods would be used. They are literature study, email survey and rating email
survey. A comprehensive literature study will be carried out by gathering material related to
software usability in general and usability evaluation methods in particular.
Questionnaire method will be used in email survey for data collection, in order to inquire
practices of usability evaluation methods in web industry during product development stage.
In the end a rating email survey will be carried out on the same web industry from which
data regarding usability evaluation methods will be collected. Hundred dollar test method
[85] would be used in rating email survey. The result of rating email survey will provide
authors with appropriate usability evaluation method for web industry.
USABILITY
Usability Definitions
The first two definitions in Table1 highly stresses on specified condition and context of use.
It refers to the environment where a product is going to be used.
2.1
Usability Background
History of usability can be traced back to the Second World War where it emerged as a result
of the intensive research into and use of more advanced technology [20]. It was realized that
the adaptation of machines to the human operator increased human-machine reaction, speed
and performance [20]. The science soon spread into the field of telecommunications and
finally computers [20]. Today usability became an integral concern of all major businesses
of world.
Traditionally, usability has been considered important in the professional field due to reasons
that range from safety, to annoyance, frustration, and factors of an economic nature that may
involve productivity or the sale of products [21]. Today following the mass introduction of
the personal computer and software into the home concepts such as "user friendly" has
become part of everyday language [21].
2.1.1
It is important to know the difference between hardware usability and software usability.
Table 2 summarizes the difference between the hardware and software usability. The
common concern of both hardware and software usability is ease of use for the users.
Table 2: Hardware usability Vs Software usability
Hardware Usability
Software Usability
The main hardware usability features are The main software usability features
volume, weight, cost, etc [22].
indicate the GUI (Graphical User
Interface) with its operatability and
structure, etc [22].
2.1.2
According to ISO 13407, iteration is a key principle in usability engineering for usability
evaluation. The cycle of analysis, design, implementation and evaluation is continued until
the iterative design has reached its usability objectives. Evaluation is an essential step in
human-centered design and should take place at all stages in system life cycle. It is important
to start evaluation as early as possible, because changes become more expensive to
implement as design and functionality [7]. Figure 2 briefs the activities involved during
iterative human centric design.
2.2
According to the ISO Standard there are three basic view points regarding quality i.e. User
View point, Developer View point, and Managers View point. Users always focus on the
external dimension of the quality which is quality in use and consider the run time quality of
a software product. On the other hand managers and developers focuses on the internal
quality of the software product which includes maintainability, cost effectiveness, portability
etc. for web sites. The evaluation has been done keeping the users view point, which is
external quality. Usability comes under external quality domain because it is measured
according to thinking of users. It is also a non functional requirement of the software
product.
Only a few software quality models have been designed to address usability aspects in a
detailed and structured way [5]. The major problem with the definition of usability is that it
is very difficult to specify characteristics and its attributes that should be considered
particular. The nature of the characteristics and required attributes depend on the context in
which the product is used [5].
The ISO has made standards related to usability but these standards do not support all
aspects of usability and they are not well integrated into current software engineering
practices due to lack of support [5].
The two major ISO standards related to usability are
ISO 9126 -1
ISO 9241-11
The definitions of both standards are written in table 1. Both standards are explained below.
2.2.1
ISO 9126-1 defines a quality model that describes six categories of software quality which
are relevant during product development. They are functionality, reliability, usability,
efficiency, maintainability and portability (See Fig 3).
Understandability
Operability
Learnability
Attractiveness
Usability
Attractiveness
Understandbility
Opertability
Learnability
Usability Characteristics
Description
Understandability
Operability
Learnability
Attractiveness
2.2.2
ISO 9241-11 explains the benefits of measuring usability in terms of user performance and
satisfaction. It emphasizes that visual display terminal usability is dependent on the context
of use and that the level of usability achieved will depend on the specific circumstances in
which a product is used. The context of use consists of the users, tasks and equipment [7].
1. Effectiveness
2. Efficiency
3. Satisfaction
Usability
Characteristics
Description
Effectiveness
Efficiency
Satisfaction
2.2.3
2.3
Despite ISO standards and definitions on usability, Jakob Nielsens (Web Usability Guru,
New York Times) definition on usability is widely accepted among usability experts [4, 32].
His definition also is very appropriate when it comes to web usability evaluation.
According to Nielsons definition, usability refers to following five components. They are as
follows [6].
1. Learnability
The system is easy to learn. Novice users are able to complete basic tasks in a short period of
time, with a minimum of training.
2. Efficiency
Experienced users are able to reach a steady state of productivity.
2.4
Benefits of Usability
10
Today, many leading corporations such as American Airlines, Apple Computer, Eastman
Kodak Company, Lotus Development Corporation, and Microsoft Corporation are
incorporating usability engineering into their product development cycles [24].
2.5
Comparatively stable
characteristics as
compare to experience
11
WEB USABILITY
This chapter is about web usability. The chapter describes the concept of web usability by
highlighting related work done in the field.
Paraphrasing the definition of usability supplied by ISO 9241, web usability is the efficient,
effective and satisfying completion of a specified task by any web user [5].
3.1
Related Work
Creating a usable website is not a trivial task. T.Brinck, D. Gergle, and S. D. Wood [10],
outlines the detail systematic process for creating usable websites. They present their
expertise gained through years of web usability and web design projects and practices. In
their book [10], they have given the concept of pervasive usability which says that usability
can be factored into every stage of the web site design process. According to [10], usable
website is a one that allow users to accomplish their goals quickly, efficiently, and easily.
According to them [10], characteristics (See Figure 8) of a usable website include following
factors.
Functional correctness
Efficient to use
Easy to learn
Easy to remember
Tolerant of error
Subjectively pleasing
Functional correctness
Easy to remember
Efficient to use
Subjective pleasing
Easy to learn
Tolerant to error
12
Requirements
Analysis
Conceptual Design
Evaluation
Mockups and
Prototypes
Launch
Production
Usability inspection
Group walk through
User testing
Usability inspection and User testing has been explained in Section 5. Group walk through is
very similar to usability inspection except that the evaluation is done by a group of
stakeholders [10].
K.Guenther [28] writes in his paper that it seems amazing how many websites score high
with regard to appearance but perform poorly when it comes to usability. The usability
expert Jacob Nielsen has shown rapidly in his studies that web user skim or scan the web
pages text rather than reading it [3]. Reading on web can be painful that is why Jacob
Nielsen and others advocate that web does require its own style of writing, a style which
facilitates scanning [28].
H. Shahizan and Li Feng [29] advocate the benchmarking approach for evaluating web
usability. Benchmarking is a technique performed by an organization to compare their web
site with its competitors. According to them, usability is a broad concept covering at least
seven factors. They are screen appearance, consistency, accessibility, and navigation, media
use, interactivity and content. It is up to organization if it wants to benchmark all seven
factors or some selected factors. Their [29] purpose of research was only to test applicability
of framework. Their [29] benchmarking approach consists of eight phases. It is shown in
Figure 10.
13
Marsico and Levialdi [30] mention three approaches currently used for evaluating web
usability. They are questionnaire, behavior assessment techniques and automatic tools
examine.
K.Guenther [31] advocates engaging users early one. He says that although web usability
has a significant priority for web development but there are very few organizations who take
time to formally test usability or engage potential users early enough in development stage of
project. He also writes that usability testing does not need to be sophisticated or expensive in
order to be successful.
E.Folmer and J.Bosch say that most usability issues do not depend on the interface but on
functionality [7].
Some Researches have proposed [73] the basic framework for cost-justifying usability
engineering on Web development projects.
A group of researchers [68] have done initial investigation into the website needs of the
Human Computer Interaction (HCI) community. Their research provides useful guidance for
website designers and developers for creating usable websites [68].
Human Factors International, Inc. (HFI) claims to be world leader in user-centered design
[90]. They offer a complete usability suit, consulting, training, and products to help
customers in creating intuitive, easy-to-use Web sites and applications [90]. They have made
10 principles of web usability (See Appendix 3).
There is a Web Usability Testing Institute in University WISCONSIN-STOUT USA. They
have made a good checklist for web usability evaluation (See Appendix 4).
3.2
On the basis to purpose and functionality websites can be categorized into three categories.
They are
Each type of website has its own design rules and design needs [93]. It would have
been an extensive investigation to find appropriate usability evaluation method for
each type of website. Due to this reason this thesis is only focusing on web
development in general.
Furthermore, we are of the opinion that it is a less professional approach to find
appropriate usability evaluation method for specific type of website without finding
appropriate usability evaluating method for all general websites.
We are interested in finding usability evaluation method appropriate for web
applications generally during its product development stage. Finding appropriate web
usability evaluation method for E commerce website or Information oriented
websites can be secondary research to our thesis.
15
4.1
The usability evaluation methods are as old as the term usability is. Before 1989 usability
evaluation process is thought to be an expensive process. In 1989 Jacob Nielson presented
his revolutionary research paper Discount Usability Methods, which latter known as
Guerilla HCI [88]. In his paper Nielson argued that good usability can be achieved with
little recourses. He has developed many usability methods specially usability inspection
methods during nineties.
Many of usability evaluation methods have their roots in psychology [1]. Examples are
experiments, questioners, interviews and incident diaries etc. Some methods have been
adapted for marketing. Examples are focus group and workshops etc. Some methods are
specifically developed for usability evaluation. Examples are co-discovery method, cognitive
walkthroughs and logging.
4.2
The interface design and evaluation have the same relationship what body has with head.
Without evaluation it is not possible to produce a professional interface design. The
relationship between usable design, usability evaluation and usability evaluation methods is
shown in Figure 12. The diagram shows that for making a good usable interface design,
evaluation is needed and evaluation process is guided by UEMs. The better the usability
evaluation method selected, better will be the software design and overall project.
Usability
Evaluation
Methods
User Centered Evaluation
16
4.2.1
In order to know software evaluation concept, it is very important to know the interface
design types. Design types have evolved over the past few decades. They can be categorized
into three generations. First generation design methods which are also known as product
oriented design methods focused on systems theory and software engineering [39]. Second
generation design methods which are also know as process oriented design methods
developed in 1970s, focused on user participation, communication and democracy in the
design process [39]. Third generation methods which are also known as use oriented design
methods focus on the actual use situation and assess the quality in use of the designed
system [39]. The concept of participation and evaluation finds its self an integral part of
software design industry.
Popular software design methods among the HCI and Software Engineering (SE) community
are participatory design, user-centered design, and interaction design.
4.2.2
Participatory design
There are many views about participatory design method (PD), but the common focus of
each approach is on users active participation and cooperation with designers in the design
process [38]. According to Kyng [40] participatory design (PD) method is a way for users
and designers to apply their knowledge and experience in designing computer systems.
Europe started using participatory design (PD) method in early 1970s [41]. North America
started using PD in late 1980s [42]. The Scandinavian approach to participatory design
stresses on the importance of active, creative, participation of potential end-users in the
design process [43].
4.2.3
User-Centered Design
Donald Norman states in his book The Design of Everyday Things [44] that user-centered
design (UCD) is a philosophy based on the needs and interests of the user, with an emphasis
on making products usable and understandable. According to Preece, Rogers and Sharp
[45] UCD is an approach that focuses on users and their goals, not just technology. The users
and their goals are the driving forces behind the development of a product [45]. Christine E.
Wania [38] states that PD and UCD are two similar approaches to design that are often
confused but Carroll [46] point out that in many UCD approaches users are involved but not
as full participants.
4.2.4
Interaction Design
According to Preece, Rogers, and Sharp [45] Interaction design is a method for designing
interactive products to support people in their everyday and working lives. There are three
key characteristics of interaction design [38]. They are
Focus on users
Iteration
Identification and documentation of specific usability and user experience goals
The usability evaluation methods are more concern with user centric design and interaction
design.
4.3
17
4.3.1
In 1990, Nielsen and Molich [8] divided usability evaluation into four categories:
Formal
Automatic
Empirical
Inspections
Formal methods are not much used in real software development projects because the
methods are tedious to apply [8]. Automatic evaluations, on the other hand are feasible only
to very primitive checks [8]. Therefore, empirical testing and usability inspection forms the
basis of usability evaluation in product development [8].
4.3.2
In 1991 Andy Whitefield [4] presented a model which divides usability evaluation methods
into four classes. They are Analytical Methods, User Report, Specialist Report and
Observational Methods.
Analytical Methods are also known as formal methods. They are used for usability
estimation purpose. These methods are used in scenarios when both user and system are not
real. User report methods are also known as usability inquiry methods. They are used for
feedback purpose. Specialist Methods are also known as usability inspection methods. They
are used in a scenario when system is real and users are absent. Observational Methods are
also known as usability testing methods. They are used in a scenario when real users and real
system are present. These methods are though to most effective and indispensable [1, 2].
Figure 13 shows a derived form of A. Whitefield classification of UEM according to thesis
design. Block a represents the original model and block b is derived version.
USER
a.
Representative
Representative
SYSTEM
Real
Real
Analytical
Methods
User Report
Specialist
Report
Observational
Methods
USER
b.
Representative
Representative
WEBSITE
Real
Real
Formal
Methods
Usability
Inquiry
Usability
inspection
Usability
Testing
18
4.3.3
Adelman and Riedel [33] identified three types of usability evaluation methods:
Heuristic
Subjective
Empirical
Figure 14 shows the diagram of Adelam and Riedel classification of usability methods.
Heuristic
Based on
Expert
Opinion
Subjective
Empirical
Based on
User
Opinion
Based on
User
Action
4.3.4
Wixon and Wilson studied usability methods in general. They name five dimensions that
characterize the methods: They are summarized below [8]:
Formative vs. summative methods: Formative methods are used to generate new
ideas, whereas summative methods are used to evaluate existing systems.
Discovery methods vs. decision methods: Discovery methods are sometimes also
called qualitative methods. They are used to discover how users work, behave or
think and what problems they have. Decision methods are used in selecting a design
among several alternatives or in picking elements of interface designs. These
methods are sometimes called quantitative methods.
Formalized methods vs. informal method: Many methods have been described
formally, but in practice, the evaluators adapt the methods to their needs, i.e., use
them informally.
Users are involved vs. users are not involved: Usability methods differ in the
extent to which users are involved in evaluation, analysis and design.
Complete methods vs. component methods: Some methods cover all the steps
needed to complete the usability design effort. Usability engineering as a whole is a
complete method. Most methods are component methods, so they represent only a
part of a complete usability process.
S. Riihiaho [8] has mapped (Table 6) some of usability evaluation methods according to
Wixon and Wilson Classification.
19
4.3.5
Y. Ivory and M. A. Hearst [71], researchers from University of California divided usability
evaluation methods in much detail manner. (See Fig 15)
20
4.3.6
Summative usability evaluation methods (SEMs) are used before and after product
development. These methods are used to assess overall quality of a finished interface.
Comparison of alternative designs and testing of definite performance requirements are the
main focus of SEMs. Formative evaluation methods (FEMs) are used during product
development. These methods help in improving interface design. Qualitative observations of
what happened and why something went wrong is the main focus of FEMs. In other words
formative UEMs are associated with qualitative usability data for example usability problem
identification [76]. The qualitative usability data is very vital for usability engineers,
managers and marketing people in order to identify convergence of a design to an acceptable
level of usability and to decide when to stop iterating the development process [76]. Some
researchers and practitioners [76] have gone so far in favor of FEMs that they consider
UEMs only about qualitative usability data. The focus of this thesis is formative usability
evaluation methods. The role of usability evaluator is very important during usability
evaluation process. Authors have mapped the role of evaluators with the classification of
usability evaluation methods in fig 16.
In figure 16, three product development process stages are shown. Before and after the
product development summative usability evaluation methods are used and usability inquiry
is the main method for evaluating usability in this stage. Before product development
requirement engineers gather requirements for the product and they interact with users for
this purpose. They normally use inquiry methods such as field observation, focus group, pro
active field study, interviews and questionnaire etc. After the product is developed and
released in market the user becomes the primary actor for evaluation the usability of a
product. Mostly questionnaire method is used for giving feedback regarding usability of
system. Web-based user interface evaluation with Questionnaires is popular means of
evaluating usability of a products next release. N. Claridge and J.Kirakowski [35] have
made a questionnaire tool called WAMMI (Website Analysis and Measurement Inventory)
for evaluating websites from users feedback. SUMI (Software Usability Measurement
Inventory) another web based questionnaire method [36] has been used for measuring
software quality from the end user's point of view. During product development usability
experts should evaluates the usability of a product. They use usability inspection and
usability testing methods. The usability experts also work with summative usability
evaluations but there they work as secondary actors.
We have drawn a model for classification of UEMs and named it as Conceptual
Visualization of Usability Evaluation Process (See Fig 16). In it, three primary actors are
mapped with three stages of product development (keeping web as a product in mind). They
are Usability Evaluators, Requirement Engineers and Users. Primary actor in usability
evaluation is a person who directly observes and reports usability problems. A person who
uses UEMs to evaluate the usability of interaction design is known as usability evaluator
[76]. The model has been drawn in order to assist authors research. The classification in the
model best suits the research of this thesis.
21
Before
Product
Development
Summative
Usability
Evaluation
Methods
During
Product
Development
Formative
Usability
Evaluation
Methods
After Product
Development
Summative
Usability
Evaluation
Methods
Usability
Inspection
Usability
Inquiry
Usability
Testing
Requirement
Engineers
Usability
Evaluators
Usability
Inquiry
Users
1 level breakup of
usability evaluation
methods
Primary actors in
usability evaluation
process
22
Since the model (Figure 16) is a conceptual one and is only drawn to provide authors with
clear direction of moving forward with their research there is neither a need nor desire to
validate this model. Furthermore, we are of the opinion that validation of this model will not
contribute anything related to aim and objectives of our thesis. It can be a secondary research
to validate this model in web industry, e.g. if web development companies start taking
interest in the drawn model.
23
This chapter is about usability inspection and usability testing methods. The UEMs which
were found common in literature and web industry are briefed.
5.1
Usability inspection is the generic name for a set of evaluation methods in which skilled
evaluators examine a user interface for finding usability problems [48]. It is a way of
evaluating user interface designs cheaper because testing with users is costly in terms of time
and resources [48].
5.1.1
Related Work
T.Hollingsed and D.Novick [9] throws light on the experience and practices of four
important usability inspection methods. According to them [9], Heuristic evaluation and the
cognitive walkthrough appear to be the most actively used and researched techniques. The
pluralistic walkthrough remains a recognized technique but it is not the subject of significant
further study. Formal usability inspections appear to have been incorporated into other
techniques or largely abandoned in practice.
According to J. McKirdy [50], one of the main problems in software development practice is
that both the development and evaluation of user interfaces (UI) are most often done by
developers, who are in general not dedicated usability experts. M. Schmettow [51] thinks
that Pattern Based Usability Inspection Method is appropriate for developers.
Z. Zhang, V. Basili, and B.Shneiderman [52] considers current usability inspection
techniques rather ineffective. They challenged Heuristic evaluation method (HE) which is
considered by many researchers and practitioners most effective usability evaluation
method. They compared Perspective-based Usability Inspection method (PUIM) with HE
and concluded PUIM better one.
Karat [2] has done a general comparison of usability testing and usability inspection method.
He mentions trade offs regarding inspection methods. According to him usability inspection
methods may be compared according to the following set of possible differences [2].
Method employs individuals or teams
Evaluator expertise
Prescribed tasks versus self guided exploration
Utility of guidelines
Data collection and analysis
Generation of recommendations
Role of debriefing session
A group of researchers [53] have developed MiLE (Milano-Lugano Evaluation method) for
web usability evaluation. It is the blend of Heuristic Evaluation (HE) and task-driven
techniques.
Another group of researchers [54] challenged the Heuristic Evaluation method. They valued
SUE (Systematic Usability Evaluation) a novel usability inspection technique in comparison
with HE [54].
24
Roger A. Grice [75] had done a comparison of usability inspection methods and concluded
that the combination of UEMs results in greater impact on assessing and improving the
usability of a product.
Table 7: Checklist by Brinck [10]
Web Usability Inspection Checklist
1
2
3
4
5
6
7
8
Brinck [10] advocates a checklist shown in table 7 as a part of web usability inspection.
We have identified following seven usability inspection methods in literature study [72, 79].
Heuristic evaluation
Cognitive Walkthroughs
Formal Usability Inspections
Pluralistic Walkthroughs
Feature Inspection
Consistency Inspection
Standards Inspection
From the authors questionnaire report (See Section 6), it was found that web industry has
the practical experience with following four usability inspection methods.
5.1.2
Heuristic Evaluation
In Heuristic Evaluation (HE) the evaluators judge whether each dialogue element conforms
to the heuristics or not in other word a small group of usability experts evaluate a user
interface using usability principles called the heuristics [2]. It is the most commonly used
usability inspection method in industry [4, 8, and 58]. It is based on experiences of Nielsen,
Molich and several usability guidelines [4]. It was proposed as a substitute for empirical user
testing.
After the introduction of this method, researchers began to compare the results of
heuristic evaluation to the results of other methods [9]. Many researchers and practitioners
consider that HE as one of the most popular inspection methods, due to its easiness,
cheapness and no need for advance planning [8, 9]. One study [55] compared the four bestknown usability evaluation methods empirical usability testing, heuristic evaluation, the
cognitive walkthrough, and software guidelines. The study [55] found that heuristic
evaluation reports more problems than any other evaluation method. The study [55, 57] also
25
concluded that usability testing revealed more severe problems, more recurring problems and
more global problems than heuristic evaluation. Another group of researchers [56] compared
HE with Cognitive Walkthrough (next section) with the condition of availability of usability
experts. They found out that heuristic evaluation found more problems than a cognitive
walkthrough.
Nielsen conducted number of experiments and concluded usability experts are more
effective in finding usability problems than the designers. He studied in dept the role of
expertise as a factor in the effectiveness of heuristic evaluation [58]. He compared evaluation
results from three distinct groups of usability experts: Novice evaluators, Regular experts
and Double experts. All had expertise both in usability and in the particular type of interface
being evaluated. The novice evaluators have little knowledge of usability evaluation
procedures, regular evaluators have sufficient knowledge of usability practices and doubles
experts have good knowledge of both usability evaluation practices and domain. Nielsen [59]
concluded that individual evaluators were mostly bad at doing heuristic evaluations and that
they can only found between 20% and 51% of the usability problems in the interfaces they
evaluated. He suggests two to three evaluators if double usability experts are employed [4].
He recommends three to five usability experts if regular usability experts are used and group
of fourteen evaluators if novice experts are used [8].
According to Nielsen [2, 59] HE is composed of following five steps
Advantages
Disadvantages
Several Evaluator experts are needed
Cheap
No requirement for advanced planning
Can be used early in the development Most of issues identified by HE are minor
process
Difficult to summarize the findings from
multiple evaluators as different evaluators
report problems differently and at different
levels
5.1.3
Cognitive Walkthrough
26
Wharton [60] originated this method in early nineties. The method came up several versions
[8]. A group of researchers showed the need for changes in the cognitive walkthrough
method because of difficulty in learning cognitive psychology terminologies by untrained
analysts [61]. In [61], they revised the cognitive walkthrough method to better suite their
needs and time schedule in projects. They called the revised version as cognitive jogthrough
[61]. The present version of this method [60] concentrates on user's motivation to select and
execute the correct sequence of actions. The versions of the cognitive walkthrough continue
to be developed [62]. Marilyn H. Blackmon and his team proposed Cognitive Walkthrough
for the Web (CWW) which they claim is superior for evaluating websites, support users
navigation and information search tasks.
While other usability inspection method evaluates the characteristics of the interface,
cognitive walkthrough method guides the analysts to consider users' mental processes in
detail [8]. The method can be used very early in design to evaluate designers' preliminary
design ideas and it is not necessary to have a running version of the system or detailed
layouts of displays [8].
According to Wharton [63] the process of cognitive walkthrough can be divided into
following five steps.
1. Define inputs to the walkthrough.
2. Find a group of analysts.
3. Walk through the tasks.
4. Record critical information.
5. Think of ways to fix the problems.
Table 9: Advantages and Disadvantages of Cognitive Walkthrough [65]
Advantages
Disadvantages
Does not require functioning model of the Does not provide guidelines about what
product
makes an action clearly available to a user
Rests on an acceptable cognitive model of Does not tell what types of actions are
user activity during the phase of exploratory considered by a broad range of users
learning
5.1.4
Advantages
Faster resolving of the usability issues
Disadvantages
Scheduling of group can be a problem
5.1.5
Feature inspection
Advantages
Disadvantages
Performs product inspection and usability Can not measure usability directly
inspection at a time
More significant in identifying problem
areas of websites
Gives information when no prototype or
previous versions of interface exist
5.2
Unlike usability inspection methods, there is relatively little research done on usability
testing methods. It seems that usability testing methods are not recognized as compare to
usability inspection methods.
In usability testing users are systematically observed as they perform tasks [48]. There is no
alternative of observing users directly and making notes while they perform their tasks.
Various methods of usability testing have been proposed in literature.
Coaching Method
Co-discovery Learning
Performance Measurement
Question-asking Protocol
Remote Testing
Retrospective Testing
Shadowing Method
Teaching Method
Thinking Aloud Protocol
From the authors questionnaire report (See Section 6) it was found that web industry has the
practical experience with following five usability testing methods.
28
5.2.1
The idea of conducting remote usability tests emerged ten years ago [69]. It is a relatively
distinct method with in other usability testing methods because user is not physically present
during testing. This method becomes an ultimate choice when users and usability experts are
sitting far away from each other.
Recently a group of researchers [69] have compared remote usability with conventional
usability testing. They [69] concluded that remote usability testing has the potential to cross
organizational and geographical boundaries and support new approaches to software
development such as outsourcing, global and open source software development.
Advantages
Disadvantages
Comparatively cheaper
5.2.2
Coaching Method
In Coaching Method usability expert works as a coach. Unlike other usability methods which
disallow questioning, users are encouraged to ask questions from usability expert in coaching
method [67]. The usability expert responds with appropriate instruction. By hearing typical
user questions, problems are identified and help documentation can be designed [67].
Table 13: Advantages and Disadvantages of Coaching Method [67]
Advantages
Disadvantages
5.2.3
Co discovery method
This method involves two participants that working together and verbalize their thoughts
while exploring a products interface [1]. This method also discovers how particular tasks
are done [1]. Pair of users helps each other through difficulties.
Table 14: Advantages and Disadvantages of Co Discovery Method [1]
Advantages
Disadvantages
29
5.2.4
Performance Measurement
In this method the quantitative data are obtained about the test participants performance
while performing task Quantitative data is very useful in doing comparative testing, or
testing against predefined benchmarks [72].
Advantages
Disadvantages
5.2.5
In Think Aloud Protocol method, users are asked to speak their thoughts as they perform a
task [67]. By thinking aloud while attempting to complete the task, users can explain their
method of attempting to complete the task. This will clarify any difficulties they encounter in
the process.
Table 16: Advantages and Disadvantages of Think Aloud Protocol Method [1]
Advantages
Disadvantages
30
6.1
Questionnaire Design
The authors designed a seven question questionnaire in Microsoft word document (See
Appendix 1). The questionnaire was structured in such a way that it provided all possible
answers to the evaluators. This was done in order to get quantitative data which is not
possible to obtain if questions are asked without possible answer parameters. The evaluators
just had to highlight the appropriate answers.
The design of questionnaire was made simple because it is said that the quality of giving
answers deteriorates with the passage of certain time [89, 91]. The designed questionnaire
seems to be answerable with in 20 minutes.
It is hard to motivate people to answer survey questions [91]. In [91], it is suggested that the
researcher will be able to increase the motivation by clearly state that the research that is
conducted will be relevant to them and furthermore, that their confidentiality will be
preserved.
The interest of web industry was motivated by persuading them that the research should be
relevant to them and their confidentiality will be preserved. This was done by sending them
brief emails which gave them the idea of research and its purpose. By web industry we refer
to selected sample of web development companies (See Section 6.2) for this research. The
questionnaire technique proved to be very simple and effective. From the feedback, most of
the respondents appreciated the easiness of questionnaire design.
In line with the results from [91], we have made an effort to develop neutral questions.
Neutral questions help in minimizing researchers bias [91], although researchers biasness is
one of main disadvantages of conducting survey [77].
Question one was a general one. It was designed to know whether web industry thinks that
usability evaluation is important or not for web development. Question two was specific to
usage of usability evaluation method. It was designed to know whether web industry is
following any usability evolution method during web development. Question three and four
are very specific to the needs of this thesis. Question three was designed to know familiarity
of industry with usability evaluation methods. Question four was designed to figure out
names of usability evaluation methods which web industry had applied or applying in their
projects. Question five was designed to know, in which phases of web development
companies deploy usability evaluation methods. Question six was designed to know about
the primary actor who normally evaluates the usability of web application. Primary actor is a
person who observes, and reports the usability problem during product development.
Question seven was designed to know the major constraints in a way of usability evaluation
methods becoming integral part of web development.
31
6.2
The questionnaire was send remotely to project managers of sixteen reputable web
development companies through email. The companies are situated in Pakistan and Gulf
region. Out of sixteen companies ten companies responded positively. From telephone
interviews it was made sure that those who will actually answer the questions would be
persons having at least 2 years of working experience with web design and its evaluation.
Three actors were involved in the process (See Figure 17). They are Authors, Project
mangers and Web Usability Evaluators. The Questionnaire was send to Project manager
who gave it to Web Usability Evaluators. Web usability evaluators were the people who had
been involved in evaluating usability of web application in previous projects. At the end
authors were able to get the answers from companies prospective.
Authors
Companys Prospective
Questionnaire
Project Managers
Questionnaire
Feedback
Web Usability
Evaluator
6.3
Questionnaire Results
Authors have divided the questionnaire results into three parts. They are described in tabular
form in table 17, 18 and 19 respectively.
In Table 17, the questionnaire data is summarized into four factors. The seven questions
questionnaire is divided into four factors. The four factors can be seen in table 17. The
answers given by ten web development companies are merged into percentage conclusions.
First two factors clearly shows that web industry agrees with the fact that usability is
essential for web application but only 20 % of industry are using usability evaluation
methods as a part of every web development project. This concludes that usability is not a
common practice. If we compare factor three and factor four we will come to know that 70%
of web industry are familiar with usability evaluation methods but only 40% of them have
practically used any of usability evaluation method in their projects. This means that 30% of
industrial personal have never used usability evaluation methods despite of its knowledge.
32
The results in Table 17 are concern with our general objective and that is to highlight that
web industry is facing the problem of adapting usability evaluation method.
Table 17: Interpretation from questionnaire results
Factors
Yes
Usability Evaluation is
important for web design
Usage of usability
evaluation methods is
always a part of web
development project
Familiarity with usability
evaluation methods
Practically worked with
usability evaluation
methods
No
100%
0%
20%
80%
70%
30%
40%
60%
The results of Table 18 are concern with the primary objective of this thesis. It provides the
filtered list of usability evaluation methods. We have identified sixteen usability evaluation
methods in literature [72, 79]. The questionnaire result ended up with nine usability
evaluation methods which were found common in literature and web industry.
Table 18: UEMs practiced in web industry
Web development
phases
Analysis
Design
Coding
Testing
33
There were also two secondary objectives of questionnaire (See Appendix 1). They were
To find primary actor (See Section 4.3.6 and Appendix 1), who is responsible for
usability evaluation of web applications.
To find two major constraints (See Appendix 1) that are liable for usability
evaluation method not becoming as an integral part of web development process.
From the questionnaire results, it was found that most of the web development companies
are not employing any usability specialist for evaluating usability of websites. It was also
found that the two major constraints in making usability evaluation method becoming an
integral part of web development process are lack of usability expert and fewer resources.
6.4
Results Validation
C. Wohlin [77] has proposed four types of validations for experiments in software
engineering. According to B. Kitchenham and S. Pfieeger [78], software engineering surveys
are weak in the area of validity and reliability [91]. A survey is reliable if we administer it
many times and get roughly the same distribution of results each time [78].
In order to check the validity of answers provided by the web development companies, the
questionnaire was resend to same web development companies. Authors changed the design
of questionnaire and reshuffled some questions. This was done to make sure that companies
will not be attempting the one hundred percent same questionnaire which they have
attempted before. The second time reshuffled questionnaire was resend to web development
companies after the gap of one week, they responded the first questionnaire. The results of
first questionnaire and second questionnaire were same. There was the probability that
companies might have changed their opinion but in this case it was not so.
6.4.1
Validity Threats
Some validity threats can be assessed by general assumptions. Since data was
collected from remote web development companies its impossible to physically
watch the conditions in the web development companies under consideration.
Obviously, not all the web development companies could have the same sort of
conditions and furthermore, we have no complete knowledge of the actual conditions
of web development companies. The inner and outer conditions of organization can
be major validity threat to our research. At the same time, the sincerity of persons
involved in answering the questions is accordingly not physically observable due to
the conditions of our study. Personal biasness of answering people to questionnaire
can be another validity threat to our research.
34
To best of our knowledge there is no standard criterion for comparison of UEMs. The
authors observation has been weighted by group of usability researchers and practitioners
from Virginia Tech University [76], who highlighted the reasons for why it is so hard to
make criteria for comparison of UEMs. According to them [76], UEMs can not be
evaluated and compared reliably because of lack of standard criteria for comparisons. Their
[76] observation is that its almost impossible to do an appropriate Meta comparison of
usability studies. They [76] believe that there are two reasons which contribute to the
challenge of comparing UEMs. The first reason is that the field of UEMs is young as
compared to social science discipline in which baseline studies are frequently performed.
The baseline comparative studies are almost non existence due to youth of UEMs. The
second reason is that UEMs themselves are not stable. The fact is that they continue to
change because of human computer systems, their interaction components and their
evaluation needs change rapidly. This change results in requirement for new kinds of UEMs
and need for constant improvement and modification to existing UEMs.
According to [76], researchers are finding it difficult to reliably compare UEMs due to lack
of
Since usability evaluation methods (UEMs) proposed in literature were analyzed with web
development companies so authors now have come with total nine usability evaluation
methods. Due to this reason authors considered these nine methods as web usability
evaluation methods.
The authors strategy was to filter out two usability inspection methods and two usability
testing methods. For this purpose CARE methodology has been used (See Section 7.2).
7.1
Care Methodology
CARE Methodology has been proposed by group of students from North Dakota University
[74]. The methodology has been made by combination of several past usability testing
methods. CARE stands for cheap, accurate, reliable and efficient method.
Authors have considered two parameters of CARE methodology in order to filter out two
usability inspection methods and two usability testing methods. The two parameters are
cheapness and efficiency. The reason of considering these two parameters is that, previous
research [72, 73] has been done for finding out relative cheapness and relative efficiency
among several UEMs. Furthermore the web projects are limited in terms of time and budget
[94] so cheapness parameter seems to valid criteria. Similarly the efficient usability
evaluation method will save the time. It was also one of the findings of our questionnaire
report that one of major constraint in making usability evaluation method becoming an
integral part of web development process is fewer resources. To best of our knowledge no
research has been done to find out relative reliability and relative accuracy among UEMs.
Ideally we would like to consider all four parameters of CARE methodology but it is beyond
the scope of our thesis.
35
7.1.1
CARE methodology [74], has been selected for comparison because of following reasons.
Relatively new research
Specific to web usability
IEEE publication
7.2
Two comparisons were done. One was among the four usability inspection
methods and the other was among five usability testing methods.
7.2.1
The comparison of derived four web usability inspection methods in the light of previous
research [72, 73] has been summarized in table 20. From the table 20 it can be seen that
Heuristic Evaluation (HE) and Feature inspection are the two methods which qualifies for
rating survey. The efficiency parameter has been compared earlier in previous research [72].
HE and Feature inspection are cheaper method then cognitive walkthroughs and pluralistic
usability walkthrough (PUW) [9, 72]. It is because in CW and PUW, usability expert also
need to have good knowledge of human psychology. Developers and novice level usability
professional can adapt HE. Feature inspection work like a dual bladed sword from which
usability is indirectly checked by inspecting all the features of website.
Table 20: Usability Inspection Methods Comparison
Sr. Web Usability Inspection Methods
Relative
Relative
Efficiency Cheapness
1
Heuristic Evaluation (HE)
2
Cognitive Walkthrough (CW)
x
x
3
Pluralistic Usability Walkthrough
x
x
4
Feature Inspection
x
7.2.2
The comparison of derived four web usability testing methods in the light of previous
research [72, 73] has been summarized in table 21. The efficiency factor between these five
web usability testing methods has been compared in previous research [72]. Remote usability
testing is cheaper then the remaining four web usability testing methods because customer
physical presence is not needed and no special lab are required. From the table 21, it can be
seen that Remote usability testing and Performance measurement are two web usability
testing methods which qualifies for next round of rating survey.
Table 21: Usability Testing Methods Comparison
Sr. Web Usability Testing Methods
Relative
Relative
Efficiency Cheapness
1
Remote Usability Testing
2
Coaching Method
x
x
3
Co-discovery Learning
x
x
4
Performance Measurement
x
5
Think Aloud Protocol
x
x
36
7.3
After the comparison (See Section 7.2), the two filtered web usability inspection methods
and two web usability testing methods are following.
7.3.1
Heuristic Evaluation
Feature Inspection
Remote Usability Testing
Performance Measurement
In line with the authors strategy (See Section 7) following four UEMs have been made.
Each method is composed of one usability inspection method and one usability testing
method.
1.
2.
3.
4.
Heuristic Evaluation +
Heuristic Evaluation +
Feature Inspection +
Feature Inspection +
37
According to a group of researchers [80], survey is useful when control of the independent
and dependent variables is not possible or not desirable. They [80] further describe in
their research that, in software engineering field survey usually means to poll a set of data
from an event that has accorded to determine how the population will react to a particular
method.
Since the field of web usability evaluation is relatively new so our research has been more
exploratory rather then confirmatory. The survey was conducted because the control of the
independent and dependent variable were not desirable. E-Mail survey was conducted
because it was more feasible for the research as compare to interview survey.
The survey was performed in a typical way of software engineering field i.e. opinion polls of
the industrial practitioners (See Section 8.1) who have the experiences with usability
evaluation of web applications.
The strategy used in email survey was to combine one usability inspection method with one
usability testing method (See Section 7.3.1) and use 100 Dollar test (See Section 8.2). The
rating survey was conducted on the combination of web usability evaluation methods (See
Section 7.3.1).
8.1
Demographics
One of the findings of questionnaire report (Section 6) was that there were seven web
development companies who had the familiarity with UEMs. We decided to collect more
information regarding these seven development companies in order to conduct rating email
survey with more visibility. We come to know that those who have evaluated our
questionnaire were all at least graduates in computer science or software engineering. Four
of the companys evaluators were master degree holders while other three companys
evaluators had bachelor degrees. Two of the companies evaluators had more then 5 years of
experience, while remaining five companies evaluators had experience between one to three
years in web designing and evaluation.
In short average qualification of evaluators was bachelors degree, average experience was 3
years and average age was 28 years. These seven web development companies were reapproached for rating email survey. Five companies were Pakistan based and two were from
Gulf region.
8.2
The 100 dollar [81] test method has been used in rating email survey (See Appendix 2). The
100 dollar test is commonly known as Cumulative Voting (CV). This method has been used
widely in political elections [82]. Analytical Hierarchy Process (AHP) and CV are widely
used techniques in software engineering for ratio scale requirement prioritization. The
Hundred Dollar has not been reported as popular as Analytical Hierarchy Process (AHP), but
its usage has been increased during last few years especially in the area of software
requirement prioritization [83] and prioritization of process improvement [84].
By ratio scale prioritization, appropriate UEM can be proposed. The 100 Dollar Test and
AHP are the two main choices for ratio scale prioritization. The 100 dollar Test has been
used in research because it seemed more feasible then AHP. The comparison between AHP
and CV on the basis of results has shown [85] that CV is much easier to apply, faster,
providing more accurate results, more scalable and considered overall better then AHP.
Another reason for using it was that the research was intended to figure out the opinion of
the small sample of web industry (Section 8.1) regarding most appropriate combination of
38
usability evolution method during product development. For small sample, The Hundred
Dollar test is a better choice.
In 100 dollar test, users are asked to distribute 100 points among the various choices. From
the combined opinion, ratio scale result is obtained. In our case it was four combinations of
usability evaluation methods. The web usability evaluators were asked to distribute 100
points among four UEMs (Section 7.3.1).
Figure 18 shows diagram of overall research. UEMs were identified from literature and
compared its practice in web industry. Filtration among selected web UEMs was done on the
basis of CARE methodology (See Section 7.2). The filtered web UEMs formed the basis of
a good rating email survey. Rating email survey provided our research with appropriate web
UEM according to opinion of web industry. Note that the double arrow between the UEMs
and Web Industrial Practices represents the comparison of UEMs in literature with Web
Industry.
Usability Evaluation
Methods
Web Industrial
Practices
Analysis and
Comparison with
CARE Methodology
Rating
Survey
Hundred Dollar
Test
Web Usability
Evaluators
39
8.3
Results
The whole rating survey process took two weeks. We got response from all the seven
companies. Each of company had 100 points to distribute. So the seven companies had 700
total points to distribute. The results of rating survey are combined into a single table. It is
summarized in Table 22. The table total points are the points given by seven companies
among each of the four combinations of web usability evaluation methods (See Section
7.3.1).
Table 22: Summary of Rating E- Mail Survey in Tabular form
Sr.
Combinational UEMs
Feature Inspection
Total
Points
170
240
160
130
According to survey the most appropriate web usability evaluation method is the
combination of Heuristic Evaluation and Performance Measurement. The following ratio
scale results were concluded from the mail survey.
1.
2.
3.
4.
40
8.4
Analytical Review
The participants of the rating email survey have considered the combination of Heuristic
Evaluation (HE) and Performance Measurement (PM) method most appropriate for web
development. It is prevalent from the results that HE has been considered as a clear choice in
comparison to feature inspection.
We think it is probably because web industry wants to deploy easiest method early in web
development stage and at latter stages it wants to work with quantifiable data
The quantifiable data collection process takes more time. So people involved in web
development project wants to make sure that they should do all other tasks before working
with quantifiable data which is specific to usability. The quantifiable data is useful for
making future decisions but it seems that PM takes more time then Remote Usability Testing
(RUT). It looks that participants seems to consider that initially a cheaper and easiest method
to work so that they can start work. In the testing stage when there is some ready module for
example when a web page or prototype is available then they should work with quantifiable
data collection technique such as PM.
The two combinations with feature inspection have been rated low by the participants,
probably because feature inspection indirectly serves usability problems. Feature inspection
method can not be deployed until features are ready. Although web industry could have
considered that feature inspection distinct ability to serve two main objectives of product at a
time. They are functionality and usability. It can be concluded from result that web industry
seems to feel more comfortable with specialized usability evaluation methods, means those
methods which directly find usability problems like HE.
41
This chapter consists of conclusion and identification of areas where future research could be
valuable for web industry.
9.1
Conclusion
The importance of usability evaluation has dramatically increased due to extremely fast
growth in Internet technology. The website design is directly related to the purpose of the
website. Website with poor usability can easily destroy the purpose of website. Various
usability evaluation methods have been proposed in literature to identify usability problems.
There is no standard on classification of usability evaluation methods. The contribution of
this thesis is the development of a conceptual process model (See Figure 16) on
classification of usability evaluation methods. The model has been made in order to assist
authors research work. We generally studied the literature and drawn the conceptual process
model.
There is no standard framework or mechanism of selecting usability evaluation method for
software development. In the context of web development projects where time and budget
are more limited than traditional software development projects, it becomes even harder to
select appropriate usability evaluation method. Certainly it is not feasible for any web
development project to utilize multiple usability inspection method and multiple usability
testing methods during product development. The good choice can be the combinational
method composed of one usability inspection method and one usability testing method. The
thesis has contributed by identifying those usability evaluation methods which are common
in literature and current web industry.
Despite of recognizing the value of usability evaluation methods for web application
development, the usability evaluation methods are not consider as an integral part of web
development projects. The thesis has contributed in filling some part of the gap between
companies thinking and their actual practices by proposing appropriate combination of
usability inspection and usability testing method. This can be seen as a step towards finding
the best solution for web usability evaluation.
Different strategies have been adopted in the research focusing on filtration and combination
of web usability evaluation methods. In the process of finding an appropriate web usability
evaluation method, some filtration has been done on the basis of two parameters which are
method cheapness and method efficiency. These two parameters are in line with CARE
methodology. The purpose of filtration was to conduct a better and more effective rating
email survey which was the basic technique for finding the appropriate web usability
evaluation method.
Although this thesis has proposed an appropriate usability evaluation method for web
development companies, but the degree of subjectivity and expert judgment still plays an
important role in overall effectiveness of utilizing usability evaluation methods.
42
9.2
Future Work
During the research some interesting areas were found for future reaseach. They are as
follows.
9.2.1
There is no standard framework for UEM selection and utilization in a software development
project. Standard selection framework of UEMs can benefit software industry in developing
good usable software.
9.2.2
Web development is different from traditional software development. The birth of Web
Engineering as an independent field from Software Engineering field is a clear cut example.
There are no specific UEMs for Web domain. The traditional software UEMs are used for
analyzing web usability. A research is needed to develop new UEMs specific to web domain
because web interfaces are more dynamic and they need constant change as compare to
traditional software interfaces.
9.2.3
To best of our knowledge there is no research done on finding out relative reliability among
UEMs. Such research will help in development of standard model for selection of UEMs
during product development.
9.2.4
Like finding of relative reliability among UEMs, the finding of relative accuracy among
UEMs will also help in development of a standard model for selection of UEMs during
product development.
9.2.5
The purpose of using usability evauation methods is to idenfity usability problems which is
usability data. Converting usability data into usability information is an area where lot of
reseach is needed [86]. The usability data is useless if not converted in to usability
information. There has been some reaseach done [86] on this area, but still it needs more
attention from reserachers to develop some stardard framwork or process models of
transforming usability data into usability information.
43
10
REFERENCES
44
[21] L. Gamberini, E. Valentini, Web Usability Today: Theories, Approach and Methods,
IOS Press, 2003
[22] N. Hosono, K. Suzuki Usability study of the pit (portable information terminal)
platform, SICE Annual Conference, IEEE, 1996.
[24] D.Pieratti, Xerox company document, available from Internet,
www.stcsig.org/usability/resources/toolkit/brochur4.doc, October 2007.
[25] D. Wixon, C. Wilson, The Usability Engineering Framework for Product Design and
Evaluation, Handbook of Human-Computer Interaction, M.Helander (Ed.), Amsterdam,
1997, pp. 665
[26]A. Abran, A. Khelifi, W. Suryn, A.Seffah, Usability Meanings and Interpretations in
ISO Standards, Software Quality Journal 11(4), 2003 pp. 325-338.
[27] H. Banati, P.Bedi, P. Grover, Evaluating Web Usability from the Users Perspective,
Journal of Computer Science 2 (4), Science Publications, 2006 pp. 314-317.
[28] K. Guenther, Web Site Management, ABI/INFORM Global, July- August 2004, pp
46.
[29] H. Shahizan, L. Feng, Evaluating the Usability and Content Usefulness of Web Sites:
A Benchmarking approach, Journal of Electronic Commerce in Organizations,
ABI/INFORM Global, Apr-Jun 2005, pp. 46.
[30] K. Guenther, Assessing Web Site Usability, Vol. 27 Issue 2, Ebsco, 2003, pp 65.
[31] M. Marsico, S.Levialdi, Evaluating web sites: exploiting user's expectations,
International Journal of Human-Computer Studies, Volume 60, Issue 3, pp381-416 Science
Direct, March 2004.
[32] A.Holzinger, Usability Evaluation Methods
Communications of ACM Vol. 48, No. 1, Jan 2005.
for
Software
Developers,
45
[51] M.Schmettow, Towards a Pattern Based Usability Inspection Method for Industrial
Practitioners, Fraunhofer IESE, Fraunhofer-Platz, 2005.
46
47
[68] T. Brinck and A. Hand, What Do Users Want in an HCI Website? A survey by
Foraker Design, 1998.
[69] M. Andresen , H. Nielsen , S. Schrder , J.Stage, What happened to remote usability
testing?: an empirical study of three methods, Conference on Human Factors in Computing
Systems, Proceedings of the SIGCHI conference on Human factors in computing systems,
ACM Press, 2007.
[70] internotredici.com, available form Internet,
http://www.internotredici.com/article/remoteusabilitytesting, 13 November 2007
[71] M. Y. Ivory and M. A. Hearst, The State of the Art in Automating Usability Evaluation
of User Interfaces, ACM Computing Surveys (CSUR), Volume 33 , Issue 4 , 2001, pp. 470
516.
[72] http://usabilityhome.com/, available from Internet, 14th November 2007.
[73] R. Bias, D.Mayhew, Cost Justifying Usability, 2nd edition, Morgan Kaufmann, 2005.
[74] A.Anandhan, S.Dhandapani, H.Reza, K.Namasivayam, Web usability testing CARE
methodology, IEEE, 2006.
[75] R.A. Grice, Comparison of Cost and Effectiveness of Several Different Usability
Evaluation Methods: A Classroom Case Study, IEEE, 2003.
[76] H.H. Rex, A. S.Terence, Williges, C.Robert, Criteria For Evaluating Usability
Evaluation Methods, International Journal of Human-Computer Interaction, 2003.
[77] C.Wohlin, P. Runeson and M.Hst , Experimentation in software engineering: an
introduction, Kluwer, cop, Boston 2000.
[78] B.Kitchenham and S. Pfieeger, Principles of survey research part 4: questionnaire
evaluation, ACM SIGSOFT Software Engineering Notes, Volume 27, Issue 3, May 2002,
pp. 20 23.
[79] J. Horn Usability Evaluation Method Toolbox,
Available from Internet,
http://jthom.best.vwh.net/usability/, 29 November 2007.
[80] D. Sjoberg, T. Dyba, M. Jorgensen, The Future of Empirical Methods in Software
Engineering Research, In Future of Software Engineering (FOSE '07), IEEE Computer
Society, May 2007, pp. 358-378.
[81] Leffingwell and D. Winder, Product Management, Fourth International Edition,
McGraw hill, New York, 2005.
[82] Investopedia.com, available from Internet,
http://www.investopedia.com/terms/c/cumulativevoting.asp, 5 December 2007.
48
http://www.humanfactors.com/downloads/documents/HFI_brochure.pdf, Oct
2007.
49
APPENDIX 1: QUESTIONNAIRE
Instructions: The purpose of questionnaire is to assist authors research work. The
questionnaire is meant for elicitation of the basic information about web usability
evaluation practices in industry during product development. Highlight (In Yellow Colour)
the answers which are applicable. The information will be kept confidential.
Name
Organization Name
Designation
Contact
Q 1: Is usability evaluation important for web design?
Yes
No
Cant say
Q 2: Do you use usability evaluation methods (UEMs) during development of every web
project?
Yes
No
Cant say
Q 3: How many UEMs you are familiar with from the following list? Highlight (In
Yellow Colour) all which are applicable.
Usability Testing
Coaching Method
Co-discovery Learning
Performance Measurement
Question-asking Protocol
Remote Testing
Retrospective Testing
Shadowing Method
Teaching Method
Thinking Aloud Protocol
Usability Inspection
Heuristic evaluation
Cognitive Walkthroughs
Formal Usability Inspections
Pluralistic Walkthroughs
Feature Inspection
Consistency Inspection
Standards Inspection
50
Q 4: Which of the following usability evaluation methods you have used practically in
your projects? Highlight (In Yellow Color) all which are applicable
Usability Testing
Coaching Method
Co-discovery Learning
Performance Measurement
Question-asking Protocol
Remote Testing
Retrospective Testing
Shadowing Method
Teaching Method
Thinking Aloud Protocol
Usability Inspection
Heuristic evaluation
Cognitive Walkthroughs
Formal Usability Inspections
Pluralistic Walkthroughs
Feature Inspection
Consistency Inspection
Standards Inspection
Q 5: At which stages of web development life cycle you use UEMs? Tick all appropriate
Analysis
Design
Coding
Testing
Q 6: Who evaluates the usability of web applications in your company?
Developers
Testers
Usability Experts
Project Manager
If others specify
Q7: What are two major constraints in making usability evaluation methods as an
integral part of web development projects?
Less time
Less budget
Lack of Usability Expert
Too much conservative management
If others specify
Comments:
51
Designation
Contact
Heuristic Evaluation
Heuristic Evaluation
Feature Inspection
Feature Inspection
Points
+
+
+
+
52
Motivate
Design your site to meet specific user needs and goals. Use motivators
to draw different user personae into specific parts of your site.
User taskflow
Who are your users? What are their tasks and online environment?
For a site to be usable, page flow must match workflow.
Replicate
Why reinvent the wheel? Use ergonomically designed
templates for the most common 812 page types.
Multimedia be discriminating
Good animation attracts attention to specific information, then stops.
Too much movement distracts reading and slows comprehension.
10
53
Characteristics
Load Time
Checks
54
55