The Utility of Digitally Supported Manual Interactive Mockups

The Wizard-of-Oz (WOz) method allows researchers and practitioners to conduct prototyping, including interactive evaluations, without programming. The aim of the present account is to display the many ways digital, but manually controlled, interactive mockups can be used. It is based on a longitudinal study of one WOz supporting tool, Ozlab. The discussions here are not made in contrast to any existing general theory of users-in-the-development-loop or Participatory Design. Rather, they demonstrate resemblance with many statements or observations by others, even if the recurrent line of argument in the present paper is that (1) human-machine interaction is a very important part in prototype construction and not only in prototype evaluation – this argument is generally not emphasized by participatory designers – and that (2) plasticity in the defining process of an interactive design is well worth a specific tool.

The plethora of tools addressing designers, allowing them to create mockups and add interactivity, do not offer functions for negotiations in the user interface. That is, such tools are not means for negotiations in the medium of the user interface. They are merely means for assessment of interaction ideas. Surely, they allow for iterative refinements, but they do not allow situational exploration in the same way as WOz does, and they are not as cost-effective as WOz.

Other ways of finding good requirements for interaction design also lack the immediate substantiation in use allowing use-based evaluation of new ideas as these ideas take form. For instance, simply parsing user reviews to gather requirements is not cost-effective without combination with WOz, as announced by Abad et al. (2017; “Learn more – pay less!”). The essential ingredient in WOz is the possibility to form the interaction design as the interaction with users takes place. Such plasticity is demonstrated when humans are interacting with humans but it is often not in focus in HCI; that is why (1) was highlighted above. This plasticity in the formative phase of an interaction design development is not discussed in three conceptual studies of the notion of ‘interactivity’ or ‘interaction’ that all appeared in 2017 after a rising interest in HCI circles for the concept at the very core of the discipline, the ‘I’ in ‘HCI’ (Barry and Doherty 2017; Hornbaek and Oulasvirta 2017; Janlert and Stolterman 2017; the same holds for Hornbaek et al. 2019). Thus, it can be in place to have a look at what interaction via the user interface with users means in the process of defining interaction automation.

The account given here of uses of our tool for concept development through concept substantiation (that is, concept realization in interaction), spans 2000–2019 and is chronologically ordered as the extension of the types of use of Ozlab came gradually as did the valuation (2) of it. Historically, WOz was becoming a frequently used method in the ’80s for faking Natural Language Processing (NLP). NLP had in the ’70s appeared to provide an alternative to the command-line horror. However, by the turn of the century, GUI ruled human-computer interaction which explains the idea behind Ozlab inaugurated in 2001: quick implementation of GUI elements, movable elements, less emphasis on support for generating sentences (e.g. Pettersson and Wik 2015).

An account of the varying employments of wizard-controlled interactive mockups through twenty years reveals a two-pronged conceptual development: tool functions and methods, respectively. This paper, in all its brevity, will emphasize method, in particular arrangements for user interface interaction between designers and various kinds of users. Of course, the tool features bias the material base that makes possible the negotiation in the GUI. The “material without qualities” (Löwgren and Stolterman 2004) needs machinery to substantiate the interaction designers’ design. Thus, the highlights made below are not in any sense unbiased, but this account provides for a rich picture of what it means to be designing by interacting in the very user interfaces of prospective digital artefacts (i.e., of mockups of the prospective artefacts – these mockups may range from empty slates to detailed proposals).

The Ozlab concept of a general-purpose, mainly GUI Wizard-of-Oz system has had two major implementations; one from 2001 based on the multimedia production tool Director, and one web-based implementation from 2013 which is still developed. In contrast to so many other attempts to make general WOz systems, Ozlab was not an off-spring of any particular project and thus did not have the limited scope of such other systems (Pettersson and Wik 2015). Experiences were drawn from a range of applications, especially where these were guiding new users in introductory sessions (whether for the use of a program or for explaining a subject content; examples included both adult multimedia and preliterate children), that is, when the expressive recourses of the GUI were used to their limits.

This resulted in a detailed list of what a system for Wizard of Oz should be able to support a test leader with, some directed to functions which always should be in place during test sessions, some to be available in preparation of a mockup: GUI widgets and placeholders for texts and images that were inserted into the screens of a prototype if meant to be accessible there during a session with a “user”.

The examples following in Sect. 3 will show how different employments developed this setup. “Shell” is our word for the prototypes built in Ozlab as they are empty, i.e. mainly without automatic functionality (but full of widgets, texts, and images). Another note on terminology is TL for Test Leader (wizard) and TP for Test Person (participant, often a prospective “user” of the prototyped system). The abbreviations are easy to use, even if we nowadays in some instances regret the connotation to “test” rather than to “co-design”.

This section highlights insights gained from the ever expanding range of uses of Ozlab. The highlights are numbered H1, H2 etc. From the above-mentioned systematic review of graphical and auditory means of communicating with a PC user, an idea formed, H1: Potentially, a general WOz setup would let others than professional interaction designers trial layout and responses with peers or clients.

2001 Development.

A low-pace pilot study ran for several months concurrently with the development of Ozlab. Real-life conditions for non-professional designers: the three test wizards from a Department of Special Education had no reduction in their ordinary workload. They were inexperienced in programming and even in simple multimedia tools. H2: Great enthusiasms through the whole process. Often noticeable excitement during the test runs. H3: A programmer or designer is needed who acts TP to raise awareness of designing computer responses also for aberrant inputs.

2002 Portable Lab.

Continued experimentation within special education. A wizard laptop with Ozlab was used for easy installation of TP files on “any” computer at the site of experimentation. This meant also that no dedicated test room could be used, and wizard voice had to be pre-recorded. A similar setup had already been used for discussion among Ozlab users: the “mini-Ozlab” where TL and TP computers are positioned close to each other. H4: A portable package and mini-Ozlab emphasized the notion of Ozlab as a computer system rather than as a type of experimental setup.

2002–2003 Multimedia Students in HCI Classes.

It took a year before we understood how to teach early user testing to students already familiar with GUI design.

Spring 2002: students did not take advantage of the possibility of testing during the design phase. Spring 2003: more time was spent on explaining Wizard-of-Oz experimenting and Ozlab. Within two weeks, eleven teams produced 22 usability tested proposals. One of the participants later used her team’s proposals in a user interface programming course she taught for schoolteachers. H5: Students who have learnt multimedia production are not mentally ready to rest implementations until after usability testing. Later this also showed to be the case for programmers in large R&D projects – proposition (1) is very hard to prove to programmers until it is too late.

2002–2003 Student Thesis Projects.

Student projects were conducted where Ozlab was used or was the object of study. The portable package was utilized in some cases, so students clearly got the idea that usability work can be conducted outside a lab. On the other hand, students are often new to the idea of Wizard of Oz and to perform usability tests and, consequently, their real-time exploration of interaction details was limited. H6: Students using Ozlab often lack a drive to explore the design space – this does not provide a basis for rich GUI dialogue.

To even out response time in comparison tests and other fidelity differences between existing software and students’ improvements, also existing software was sometimes mocked up in Ozlab. H7: Comparing existing software with a new design can be facilitated by mockups built on screen captures of the existing software. This evens out response time and other fidelity differences between the comparanda. This made us more aware of the unimportance of instantaneous responses until interaction flow has been settled.

2003–2004 Robot Surgery.

Some 60 operations had been made where operators complained about ambiguities in the work sequence and about details of the user interface of the robot arm. To redevelop the touch screen and trackball user interface, an Ozlab shell with archived X-rays was made for the user interface. There was also some smaller development of user interfaces for the nurses who were to pull the robot from storage into the operating theatre and set it up for use. H8: Usability tests demonstrate that even if users are participating in re-design sessions they may later stumble on their own design – this shows that users do not always anticipate the problems they will have. H9: Ozlab can be used as a training tool before implementation is finished. A supervisor mediating the user-machine interaction allows a gradual withdrawal of the support that comes from outside the user interface (cf. Mavrikis and Gutierrez-Santos 2010).

The flexibility of Ozlab makes tests into co-design sessions: surgeon (TP) “gives some suggestion for improvements [and] is eager to see the results of the suggestion and asks if it will take long to implement. The alteration is made in approximately ten minutes and it is then possible to perform a new test run.” (Larsson and Molin 2006, p. 367) H10: Immediate feedback can contribute to the willingness to continue to participate in a long development process.

Moreover, as envisioned originally, in one test session one of the surgeons acted as a wizard (TL) with another surgeon as TP. H11: If a designer prepares the infrastructure, user dialogues via the user interface is possible even for highly complicated and specialized products.

2004–2006 Daily Use in Larger Project.

For a project with complicated concepts about anonymity and pseudonymity, Ozlab prototyping was intensively used in the start-up phase, partly to see how various concepts could be presented to ordinary Internet users, partly to facilitate communication between developers in Germany and usability evaluators in Sweden. Laptop-to-laptop variant of the mini-Ozlab was sometimes used for internal project demos as this European Union project involved much travelling. H12: No need to hide wizards: contents walk-throughs with experts and also real usability tests were made by simple mini-Ozlab settings (TPs seem to believe that test leaders simply monitor the trials and make notes).

H13: Re-use of TP input in output (temporary stored in Ozlab’s “hidden fields”). Includes labels of checkboxes, radio buttons etc. This facilitates mockups of many kinds of web sites for ordinary testing, but from 2014 it also proved valuable in more co-designed oriented uses as written inputs (also by TL) can appear in various places.

Mockups were exploited for making videos, so-called “user interface animations”. Example of employment: in evaluations with larger audiences giving answers on paper questionnaires. But also as introduction to group co-design sessions to show problematic interaction before WOz is used to try out alternatives ways or layouts. H14: Video extends WOz demonstration in several directions as “users” can in addition to watch also experiment with new interaction flows in the same prototypes.

2009 Code Quality.

This study was based on a post-hoc analysis of a major three-year re-development of a software package. Code quality was assessed when the software “was either programmed based on mock-upped and user-tested designs, initially made from perceived needs by real users, or programmed only according to perceived needs by real users” (Pettersson and Nilsson 2011, p. 502) H15: Programmers reported that code was less difficult to maintain if Ozlab tests had preceded the programming.

In one case, the content expert had seen pictures of the new design for her module and approved it but had not interacted with the WOz mockup. Later user tests, after programming was completed, generated 24 new requirements from this expert. We see this as an indication of how essential the interactive demonstration is. It illustrates the factors behind H15 and can be phrased as, H16: user interface interaction matters: “late” requirements can be spotted early if all user groups are included in the pre-testing; real users (incl. expert users) are not experts enough to make go/no-go decisions from detailed GUI pictures. This underpins both proposition (1) and (2) in Sect. 1.

2003–2012 Course Assignments.

There were always some multimedia students who tried inserting programming code in the Director file that constituted their shell. H17: We realized that a little bit of AI was good to make the WOz method stand out: a board game with simple but hard to program rules makes the Wizard necessary. It still is our standard introductory assignment for undergraduates.

The multimedia production tool in which wizards composed their Ozlab shells, Macromedia’s Director, became increasingly outdated as the producer was not updating it for new versions of operation systems (Flash took over during this decade). H18: Software does not last very long: update, update, update. (Pettersson and Wik 2015).

2013–2015 (−2019) Web-Based Ozlab.

Booking lab time or handing out pairwise connected laptops (TL + TP) was now replaced by providing a web address to each student group. For us to be able to examine students and give advice, they had to run one pilot and also the final test sessions in the lab which also allowed us to evaluate Ozlab. H19: Acting as wizard can come with heavy cognitive load and stress if wizards have the feeling that they have to produce a response “as fast as a computer”. Web lag adds to this, unfortunately. H20: On the positive side, the time needed to learn the wizard-role is surprisingly short (at least if the wizards are supported like they are in Ozlab).

2014–2015 Embedded Websites.

H21: Use of embedded web sites (embedded Ozlab shells) makes it possible for two TLs acting on independent areas in the same shell; also makes possible borrowing from external web sites but TP actions in such areas are not effectible by TL. H22: Screen sharing were used for sharing the entire TP screen to TL – this is to go outside Ozlab but even if it is hard to pursue in some environments, it makes sense for co-design (Wik and Pettersson 2019).

2015–2019 Remote Co-design.

As accessing Ozlab now was very easy for a TP even if TL did not travel to this TP, we trialed slow-paced long-distance interaction evaluation based on Ozlab and phone (or Skype, GoToMeeting) in EU projects (Italy, German, Norway, but also internally in Sweden. (Pettersson et al. 2018; and with South Africa, see Wik and Khumalo 2020). One noticeable web defect was the following. H23: Need to pacify TP in long-distance WOz tests to counteract lag.

However, by further removing TL and TP from a testing mindset, an elaboration started of immediate co-designing, that is, acting together in the user interface when designing it. H24: GUI interactive interviews (GUI-ii) names a “remote” Participatory Design method combining design and evaluation, both within a session and between sessions. Co-design in this way can be done one-by-one: the repeated confrontations with co-designers provide an opportunity for a designer to cross-trial suggestions to mitigate the risk of inbreeding when contributors test their own design. H25: GUI-ii TPs in their own office can add supplementary material from their computers and directly reference paper materials stacked in the book shelves of their offices.

2017 Two-Device TP.

In two-factor authentication it is quite common to use a smartphone for verifying a web user. Using assistants to carry out some of the experiments was facilitated by letting one TL run two Ozlab prototypes while TP handled one laptop and one phone (Karegar et al. 2018). H26: Serial use of two TP devices is not more demanding for TL than single shell setups.

2017–2018 Measuring Co-design.

Student research project on co-designers’ verbal productivity when co-designing in four different ways including GUI-ii (Boodaghian-Asl 2020). H27: Need to develop measurements for activity and contribution in co-design in order to gauge the impact of the material basis on the design session.

2018–… .

Wizard of Oz may be hard when developing designs for mobile apps. Students assignments, student thesis, and research studies on high school students and hospital visitors ran on the topic of way finding. WOz and GUI-ii as methods evaluated. The promise of GUI-ii to overcome distance in co-design is stretched to the limit by engaging ambulant TPs. (Report will follow in 2021).

To illustrate (1) and (2) in the Introduction, our Wizard-of-Oz method evolved from a testing method which also users-as-interaction-designers should be able to utilize, to a designers’ support for discussions with content experts or with team members, and further via slow-paced walkthroughs and immediate redesign to a digital co-design method for interactive GUI interviews over distances where design activities and user’s walkthroughs take turns (Pettersson et al. 2018; Wik and Khumalo 2020). The original thoughts on a system to support wizards’ GUI articulation had come to encompass several types of “GUI dialogue” and purposes:

1. 1.

   Professional developers/designers utilise test subjects (to test/explore)

2. 2.

   Users test on peers

3. 3.

   Users test on clients

4. 4.

   Users test on developer

5. 5.

   Developers and users test/discuss together

6. 6.

   Developers and content experts test/discuss together

The first point is exemplified by several project uses although the development of Ozlab was not primarily aimed at this constellation but rather at point 3 (and the students as semi-professional designers). It can be seen that some studies like the initial pilot, the user interface for robot orthopaedic, and the GUI-ii ones, have been more focused on participatory aspects than others even if every study per definition includes prospective users of some mocked up system. Other studies have been trials and evaluations of WOz in HCI courses, but likewise they may also be useful in participatory-design settings. For instance, the tendency among some IT students to strive to insert code in a Wizard-of-Oz mockup may of course also occur in a professional project where developers are let to do the early prototypes. Noteworthy, the web-based Ozlab by remote co-design and testing captures some of the advantages of programmed prototypes, namely that they can be distributed for review. Certainly, some problems might arise with remote modi operandi. The Nielsen Norman Group recently published an article on “Tools for Remote UX Workshops” (Fessenden 2020). Three “major difficulties raised by remote workshops” are mentioned:

• People are not wholly engaged or feel like their input does not matter

• New tools can be intimidating and decrease participation

• People do not have time to do a long workshop or meeting

We all recognise these problems from big projects’ meetings and it is reasonable to fear the same for participatory design pursued remotely. However, for the interactive GUI interviews the first and third of Fessenden’s difficulties are negligible, while the real problem for the second difficulty can be that company restrictions do not allow access to Ozlab or conferencing software. Then TP can be asked to use her own laptop with mobile access to Internet or join a session before she goes to work (Pettersson et al. 2018). The one-to-one setting makes for trusted relation between TL and TP.

Of course, it has to be admitted that for the roles digital tools play in interaction design work and in workshops with users, there are also other aspects to treat than the ones discussed in this brief paper. The relationship between the prototypes and the final system is affected by the notion of what the prototypes are representing. Even for the single system treated here there are noticeable shifts in how the notion of the tool and the mockups changes over time depending often on small changes in their implementation or on a fortuitous use of the WOz technique. But this has to be treated in other studies.