Multiscreen Patterns – Interactions Across the Borders of Devices

2.1 Derivation of the Library

Throughout our project we identified 25 patterns in total. We derived them by an intense literature survey as well as throughout our practical work with our partnering companies, 3M5. Media GmbH,^[1] AMERIA GmbH,^[2] and CAS SOFTWARE AG.^[3] Together with them we realized two demonstrator scenarios for the multiscreen context. One scenario supported a futuristic configuration and ordering of a new car, the other a planning process in a production floor setting. Both application scenarios had several multiscreen gestures incorporated as exemplified in Figure 1 for the car configuration and ordering scenario. The pattern library evolved during the subsequent elaboration of these scenarios. We started with a set of identified pattern candidates mainly derived from literature and further refined the description during the practical experience with the interaction design and implementation of the demonstrator scenarios according to the feedback we got from our partners.

Figure 1

Customer journey of a car configuration in a future concept store implemented as demonstrator scenario for a multiscreen context.

2.2 Structure of the Library

Each pattern is named after a gesture (e. g., swipe, bump, shake) and a reaction of the devices as a consequence of the gesture (e. g., give, take, connect). ‘Bump to Exchange’ for example, describes a gesture of bumping two devices against each other and thereby exchanging data between the two devices. A possible use case for this pattern could be the exchange of business cards.

Each pattern describes the detailed interaction between user and devices and provides guidance for implementation. Therefore, the contribution of our pattern library is twofold: We support interaction designers as well as developers. An overview of all identified patterns is provided in Table 1. All comprehensive pattern descriptions including references to literature, examples and links to the GitHub repository are accessible via the web page: http://multiscreen-patterns.uxid.de/.

A study conducted by Google [6] revealed that people use devices in the multiscreen context in two main modes: either simultaneously or sequentially. This finding is also reflected in the structure of the library. We categorized our patterns in five groups as shown in the titles of the rows of Table 1. Three of them specifying the aforementioned actions ‘Give’, ‘Take’ and ‘Exchange’ for a bidirectional data transfer between devices. Gestures of these three categories support a sequential task completion in solitary use as well as in cooperative settings, e. g. sending a document via swipe to smartphones of participants of a meeting sitting around a table. Pattern in the category ‘Extend’ allows to combine different devices in a simultaneous mode, which could be extending pictures above screens of adjacent smartphones. Gestures that allow connecting or disconnecting devices to and from each other are summarized in the fifth category named ‘Connect’. They can be combined with gestures of all other categories.

2.3 Theoretical Foundation

To reach the aforementioned goal of intuitive gestures between devices, we built on previous work in the area of tangible and natural interactions. As such we used the conceptual framework of blended interactions and reality based interactions provided by Jetter and colleagues [12]. This framework takes into consideration that humans can combines experience gained by interacting with physical, real world objects or through the communication with people in social interactions with the experience of digital interactions. It refers to the concept of ‘embodied cognition’ and integration of ‘concepts through blends’ [11]. During our work we focused on these gestures because rationalizing them in terms of the blended interaction framework makes their intuitiveness more evident. For each pattern we refer to the according image scheme [10], which defines the generic space and specify the input space according to the reality based interaction scheme in addition. Furthermore our pattern descriptions use the terminology provided by Terrenghi et al. [16] to define display types. They also specify the distance of interactions by using Halls classification of proxemics [13]. As a more comprehensive explanation of the theoretical foundation of our work would exploit the size of this article we refer to our web page for more details.

3.1 What

Figure 2

Swipte to Give.

The first and shortest section of each of our patterns aims to give an overview. It proposes a general problem that could be solved using the interaction described in the pattern document. The solution to the problem is described briefly to allow people to quickly decide if the pattern is applicable to their specific use case. A sequence of images showing the interaction further eases the decision making process (Figure 2).

In the case of Swipe to Give the problem posed is that a user wants to share a piece of data (e. g. an image) with a second device or user.

3.2 How

This section describes the interaction in more detail by presenting all user inputs with the corresponding reactions from all involved devices. This part aims to help designers to come up with adequate inputs and feedback to provide a functional, comprehensive and satisfying interaction for the user. In order to achieve this we broke down the interaction into its smallest components. On the first level we describe the user’s actions and the systems reactions for the sender and receiver device.

In the case of ‘Swipe to Give’ this means:

1. User action: The ‘swipe’ on the sender device towards the border of the sender device with a specified speed.

2. Sender device reaction: the movement of the data towards the boarder of this device following the touch and move gesture of the user.

3. Receiver device reaction: the appearance of the data on the receiver device, entering the device from the direction of the sender device.

In the example of ‘Swipe to Give’ we identify four Atomic Interactions:

1. The moment the user touches the display (Touch)

2. The movement of the finger (Move)

3. The moment the user lifts his finger from the display (Release)

4. The moment the data is received by the secondary device (Receive)

Our patterns further simplify this process by providing a decomposition of the Atomic Interactions identified for the given multiscreen interaction (see Table 2). The message that the feedback needs to convey to the user is described explicitly in the rules. In addition to that a comprehensive example for adequate feedback is given. Generally, feedback serves the purpose to subtly inform the user what they have done by interacting with the application in that manner and what they need to do next to accomplish their task.

We suggest to give the feedback a tangible feel to further improve the user experience of the interaction. This can be accomplished by applying certain parts from Google’s Material Design library [7] and Disney’s 12 animation principles [5] that serve from real world metaphors. Swipe to Give’s ‘Move’ animation will serve as an example. The user moves his finger after touching an image (the data object) which follows it until he lifts up his finger from the screen. Through applying Squash and Stretch, according to Frank Thomas (Long-time Disney animator) one of the most important techniques in animation, the user can feel the acceleration of the image he is about to release. Not only does the animation look more appealing but it also encourages the user to use a sufficient swipe velocity. The user knows that throwing a real world object in the real world requires a certain velocity. Feeling and seeing the acceleration and velocity on the device guides users in a very unintrusive way to swipe correctly.

This and other principles can be applied to lots of feedback animations to improve the user experience of the entire application.

By this very detailed contemplation the multiscreen patterns provide a framework for laying out a best case scenario, that leads to an interaction design with a high intuitiveness when implemented with all the aspects. Designers are welcome to pick and choose which Atomic Interactions they want to concentrate on.

3.3 When

This section provides all the information needed to decide whether or not a pattern is appropriate for the given context and use case. In order to achieve this, we give positive and negative examples along with checkboxed properties of the recommended context, one of which being if the interaction is for a single user or rather for collaborating with others. In the case of Swipe to Give ‘Single User’ and ‘Collaboration’ would be checked since the gesture could be used to send something to a bigger, stationary device as well as to another user.

3.4 Why

The questions answered in this section are ‘What makes the gesture appropriate?’ It covers the rational for intuitiveness, but also refers to research on the pattern and usage of this pattern in products.

Swipe to Give for example refers to the metaphor of “throwing an object through the air.”. Also it is shown in several videos featuring products or visions, they are also listed in this section.

3.5 Technical Details

Figure 3

Swipe Recognition (used for the Swipe to Give Pattern).

Each pattern description provides developers implementing multiscreen interactions with general guidance concerning the technical details of the pattern as well as an overall frame of reference for applications that combine multiple patterns.

Conducting a review of research papers that describe systems with multiscreen interactions, a general model was extracted that can be utilized to describe all identified multiscreen patterns. It consists of an abstract lifecycle with four lifecycle events: (1) Connecting the involved devices, (2) Selecting a piece of data to send, (3) Transferring that piece of data and ultimately (4) Disconnecting the devices.

Each multiscreen pattern describes a gesture that can be used to trigger one or more of these events. This is a similar approach to how Dachselt and Bucholz [3] implemented their multiscreen application. Rekimoto et al. [14] and Holmquist et al. [9] also implement aspects of the lifecycle and presented early examples of gesture-based interactions.

The recognition algorithm for each gesture is described in a state diagram acting on sensor events. Figure 3 exemplarily shows how to implement a Swipe gesture by monitoring ‘down’, ‘up’ and ‘move’ events on a Touchscreen, which correspond to our Atomic Interactions from above, and fire a Swipe Event on successful recognition. Threshold velocity, swipe length and direction can further restrict the recognition of Swipe gesture, as it is part of the Swipe to Give pattern of Figure 2 and explained in the sections before.

Generally, two types of patterns can be distinguished: Simple Gestures, like Swipe to Give and Synchronous Gestures like Stitch to Extend. The pattern Stich to Extend extends the screen of two adjacent smartphones or tablets to one combined screen by moving the finger with a touch across the borders of the first screen to the second screen. Figure 4 illustrates this interaction. Simple Gestures are implemented by monitoring one or more sensors on a single device, while Synchronous Gestures can only be recognized by communicating with another device to check if the sensor data aligns. Stitch to Extend is only a valid gesture, if the gesture is detected at both devices with a defined very short time delay and the gesture on the second device enters from the border of the device being adjoined to the first screen. The nature of different gestures constrains the possibilities of their application in the lifecycle: Simple Gestures can be utilized for all lifecycle events, Synchronous Gestures only work when a connection between two devices exists. This is necessary to ensure the aforementioned check of sensor data between the two devices.

Figure 4

Stitch to Extend (a variation also described by Hinckley et al. [8]).

Synchronous Gestures can typically be described as a combination of two or more Simple Gestures, as illustrated in the following example.

In the ‘Stitch to Extend’ gesture, two tablets are connected two a webserver. Each tablet performs a simple Stitch recognition and sends its Stitch events to the webserver. An additional constraint check comparing the direction and duration attributes of the Stitch events allows for the recognition of a synchronous Stitch gesture, which for example can be used to transfer files from one device to another. These technical descriptions are not tied to a specific technology and can be applied to all devices that publish the required touchscreen events. This is especially important because in the multiscreen context devices with different operating systems can be present. We did not want to limit our contribution to a dedicated operating system. In addition to those generic instructions each pattern links to the project’s GitHub repository,^[4] where developers can consult the Android-specific Java implementation as an example and find further, more specific information.