Collaborative Tangible Interface (CoTI) for Complex Decision Support Systems

In complex systems, multiple stakeholders from a variety of backgrounds need to interact collaboratively to make informative decisions. This collaboration is crucial especially in the context of urban planning where stakeholders use simulations for complex systems of city infrastructures that are interrelated, which generate a System of Systems (SoS). Simulating the behaviors of these systems and their interdependencies as well as assisting stakeholders in predicting future scenarios is challenging. Nevertheless, technologies have evolved through the years from web-based tools to tangible user interfaces (TUI) to support co-located collaborative decision making. Moreover, with the proliferation of multi-touch surfaces in decision support systems, a new form of TUIs has emerged to ease the decision making process by facilitating interactivity. Web-based decision support systems for infrastructure planning have been shown to be effective in supporting remote decision making [1]. However, web-based interaction inadequately supported joint scenarios/projections of interrelated decision dependencies. In recent years, TUIs such as multi-touch surfaces have been shown to support collaborative decision-making by revealing interdependencies between the tangibles ‘physical objects’ and the interaction surface’s users [5, 6].

In this paper we present (CoTI), a Collaborative Tangible Interface for complex systems that provides multi-touch interactive capabilities with analytical and visualization components to facilitate the decision making process. In CoTI, stakeholders can interact with the 3D objects that we called smart blocks (a more elaborate description in Sect. 3) and the multi-touch surface to get an immediate feedback for the impact of their decisions not only on the system under study but also on other related systems affected by those decisions. This adds another dimension to the thinking process, which enhances the users’ experience and enable them to make more informed decisions by understanding the implication of their decisions on other systems. The objective of CoTI is to support the decision making in complex systems. Therefore, an integration with a simulation engine that performs real model analysis is essential. In our case study of urban planning for example, a simulation engine for analyzing the data of the urban, transportation, energy and water systems of the city has been integrated into the platform.

This paper is structured as follows. The following section describes related work in tangible user interfaces and urban planning tools. Following that, we present an overview of CoTI system architecture. Next, we present the CoTI tangible interaction surface configurations and the user interface design considerations. An in-depth description of CoTI decision support system is provided. We conclude with a case study showcasing the CoTI in the context of urban planning.

In recent years, interactive technologies for supporting co-located and remote collaborative decision making have been designed to address the complexity and scalability of design challenges in complex system. Collaborative decision making was facilitated by direct manipulation tools such as 2D and 3D interfaces for information visualization and scenario projections [7, 12].

2.1 Tangible User Interfaces

Embedding digital information in tangible objects has led to the emergence of opportunities for designing intuitive interaction platforms in urban modeling. The context of urban planning in particular has experienced a trend in experimenting with different augmented reality and mixed reality tools in the past two decades [11, 13, 14]. More recently, tangible objects in dynamic models of urban areas have facilitated more flexibility in the design of hybrid physical and virtual interfaces.

2.2 Urban Design and Human-Computer Interaction (HCI)

Different TUIs have been developed to facilitate communication and collaboration for urban planners in decision making [9, 10]. One of the urban planning TUIs is the ‘Mark IV’ prototype developed by the Media lab, at Massachusetts Institute of Technology; which is an interactive collaborative tangible interface for urban planners [8]. The model has a set of pre-defined user interactions; for example, urban planners can move tangible objects (buildings/amenities), and check the result of their interactions on a screen. The interface does not include a control panel for stakeholders to make changes to the model or the decision variables. One issue often cited as problematic with urban planning tools is the limited scope of application domains. For example, some of the TUIs tools focus either on energy or mobility. Comprehensive tools that examine the interdependencies are emerging for sustainable design and in the area of complex engineering. One example, ‘UMI’ is an urban modeling interface that integrates different models (walkability, energy, and daylight) in one tool to examine the interplay of decision variables on urban models [3]. Scalability and extendibility are important design consideration in systems engineering.

CoTI is a collaborative tangible interactive tool that utilizes City Schema Decision Support System [2]. The system was designed in iterative design cycles with stakeholders and target users of the complex system involved as design informants. Stakeholders were engaged at different phases of the system’s development process for eliciting feedback on high-fidelity prototypes. In earlier phases of the system development, we applied City Schema DSS on the Scout table [8], a simple TUI table for urban planning that allows for rapid prototyping with limited interactivity and limited number of simulations. We then applied the same DSS on the MARK IV [8]; an intuitive TUI table with more interactive capabilities but with constrained control over decision variables. CoTI is built to overcome the limitations of Scout and MARK IV tables and give the users more interactivity and control over the system. CoTI’s architecture is comprised of three main layers: First, the tangible interaction surface which represent CoTI user interface and consists of user and user interface management components. While the user management component handles the authorization and conflicts between users, the user interface takes care of the projection of two main components: decision variables ‘DVs’ which are the inputs of the users to the simulation, and key performance indicators ‘KPIs’ which are the results that are produced by the simulation engine. Each one of these DVs and KPIs differ based on the user’s location and expertise. Second, the CoTI decision support system which in turn consists of three main components: A detector of physical artifacts to detect any change that occurs in the physical surface (such as an object’s position); a translator that maps and reflects detected changes on the physical surface with their digital representation, and a controller that connects both physical and digital artifacts together through an interactive interface for various simulation purposes. The third layer is the computational environment, which is a multi-modular simulation engine that provides various sophisticated simulations related to the study under which CoTI used such as operational energy, daylight and walkability. Figure 1 shows the high level architecture of CoTI.

The CoTI high level architecture

Full size image

The framework was developed on the context of urban modeling; nevertheless, the workflow can be applied to different application domains. In the following sections, we describe in more detail the system design components, hardware and software.

3.1 Tangible Interaction Surface

The interaction surface includes three main components: A multi-touch table, tangible objects, and display and simulation.

i. The Multi-Touch Table

In this section we describe CoTI’s multi-touch table configuration with a description of hardware and software components. The system is shown in Fig. 2. The touch-surface is mounted horizontally as a table and it consists of a transparent acrylic glass (117 cm, 117 cm). We used 6 mm acrylic glass that is permeable to the infrared IR light. The outer perimeter of the surface is lined with tracing paper to allow rear projection onto the acrylic glass as well as to diffuse the direct projector lamp beam beneath the table. Eight lasers with a near IR range (850 nm) were mounted on the edges of the table. A line generator is attached at the end of each laser to distribute the beam covering an area with an angle of 120 degrees from the laser source. These lasers produce a plane of infrared light across the entire surface of the table. When a finger touches this plane of light, the light illuminates it. The sensing apparatus consists of a camera placed underneath the table. In our setup, a PS3 Eye camera is used. This camera is modified by attaching filter to it that only allows infrared light through. The camera sees the “blobs” of infrared light and tracks these points.

CoTI multi-touch table configuration

Full size image

ii. Smart Blocks: Tangible Objects

In order to detect tangible objects on the proposed multi-touch table, a unique fiducial is attached onto each smart block [4]. These fiducials help in identifying smart block’s type (e.g. Tree, Building) and location on the interactive surface. A cross-platform computer vision framework reacTIVision [4] is used to allocate the x and y coordinates of the smart block with the use of Logitech HD Pro Webcam C920. ReacTIVision is an open source camera based two dimensional fiducial (marker) tracking system, which has the ability to track a large number of varied size fiducials with faster than real-time performance. ReacTIVision uses only cameras and projectors, which is usually required as part of the TUI implementation. In order to have a fast and accurate capturing of fiducials, the light source is placed next to the camera to enhance the camera’s vision and therefore improve detection.

iii. Display/Simulation

Due to the high complexity of CoTI, its user interface was carefully designed to ease the user-system interaction and decision making process. In order to design an interactive tangible user interface system that allows novice users to understand a concept as complex as urban planning in an engaging and informative way, we followed a multidisciplinary approach, combining human-computer interaction in tangible user interfaces, and psychology/human sciences. The CoTI TUI includes two main components: First, the interactive area where users are able to interact with the physical objects. And second, the decision variables where users are able to define and modify the physical objects’ specifications. System results are presented in multiple ways to enhance the user experience and the level of understanding. Figure 3 shows the CoTI tangible interaction interface and its components.

The CoTI user interface

Full size image

3.2 Decision Support System

In [1], we introduced a decision support system platform ‘City Schema’ that is designed to connect a tangible interface with a simulation engine to support collaborative city planning. This DSS connects the physical and the digital model together into one integrated system. The physical model is a tangible user interface with 3D representation of a city or neighborhood within the city that supports interactive and collaborative activities from multiple users. The simulation engine integrated into the platform is an urban modeling design tool that has several simulation modules such as operational energy, mobility, daylight, transportation and others [3]. City Schema DSS consists of: database management component, simulation modules management component, and an interactive user interface with dynamic visualization and decision scenario simulations. City Schema DSS is embedded in our CoTI platform with Fig. 4 highlighting the flow of events used to ensure the connectivity between the tangible interaction surface and the computational environment. The controller is to be considered the brain of CoTI as it receives commands from users, and coordinates the execution of each command among other components. After the controller scans and detects any physical modifications on the multi touch interaction surface, it projects the simulation results back to users as heat maps, statistics, and charts. The database management component can be used as a logical link to connect physical and digital artifacts together and it is responsible for the data storage and retrieval.

The flow of events on CoTI DSS

Full size image

The workflow suggested in CoTI for the mapping between physical and digital model artifacts allows for direct manipulation on the interactive surface. CoTI utilizes reacTIVision framework to recognize and detect the physical artifacts by using fiducials to distinguish one object from another. Once the objects have been detected, a script in processing, an open source development environment used mainly for rapid vision-based prototypes, is used to keep track of each object’s position, angle, and ID (fiducial ID). For each physical object, we distinguish between two types of attributes: user input such as name and type; and automatically detected attributes such as position and angle. All objects’ attributes are saved within CoTI database for later analysis. In order to generate the graphical digital representations as a direct reflection of the physical artifacts in the TUI model, we utilized the CoTI database to create geometries that match an object’s attributes. Finally, various simulations are to be run on the digital model that is generated, and reflected back on the physical model.

3.3 Computational Environment

The Computational Environment is a multi-modular simulation engine that gives stakeholders the ability to test the efficiency, sustainability and livability of cities and neighborhoods. It provides users with a choice to perform one of four possible simulations. The four available simulations supported by UMI are as follows: Operational Energy, Mobility, Daylight and Embodied Energy. First, the Operational Energy simulation, where users can create a detailed building template and assign them to different blocks. This simulation produces sophisticated analysis of the energy consumption for each building within the simulated area. Second, the Mobility simulation allows users to introduce amenities (e.g. banks, schools or restaurants) to the area under study and modify the road network. Two simulation results are computed for walkability and bike-ability which are calculated based on the accessibility and distance of each building to the different amenities available within the areas. Third, the Daylight simulation, which evaluates the design of buildings by calculating the solar radiation on the facades of buildings. Last, the Embodied Energy simulation, which calculates the consumption cost of the lifetime of each building within the area.

In the next few years and within the Riyadh center development plan, there will be significant changes and development in the Al-Dhahira district. For instance, new buildings will be built to meet the demand of the natural population growth in this area. Moreover, commercial buildings will be renovated, and a couple of new skyscrapers are planned to be introduced in this area. The different parts of the district have been designed with diverse building topologies in order to respond to different uses, as well as to reflect the specific location within the district. The central square of this district will be surrounded by three iconic skyscrapers with curved shapes and glazed façades which are protected by shading panels with the shape of palm leaves. Along King Fahad road side there is set of tall glazed buildings expected to serve business purposes. Figure 5 shows the old and new urban plan for the district.

Al-Dhahira district old (right) and new (left) urban plans

Full size image

We chose Al-Dhahira district as a case study because it is recognized as a high-intervention urban development zone, and it is in the early stages of the development. In modeling this urban area, we started by building the current state of Al-Dhahira district using LEGOs, and in which various computational simulations were executed. Results of these early computations were used as a benchmark for the new proposed plans. Every new session starts with the current state of Al-Dhahira district and allows the decision makers to propose their new plan by adding or removing buildings and/or changing buildings’ attributes (e.g. number of floors, building type). With this setting, the decision makers can compare between different proposed plans and evaluate the performance of each one. All this is constructed around an interactive multi touch table that is intended to support collaboration, and reveal interdependencies between simulations’ variables.

CoTI is tested in the context of urban planning in two sessions where the objective is to propose a new plan for Al-Dhahira district which provide a better performance than the existing plan. The decision variables included are related to accessibility (e.g. amenity’ type and location) and energy efficiency (e.g. building’s construction materials, cooling and lighting schedule,). The KPIs are heat-maps for walkability and energy efficiency projected for each building, an accumulative score (percentage) per simulation, and charts that inform users about their current performance per simulation compared to previous plans. In each session, users collaborate to generate different plans to assess how their decisions impact the future state of the district. The first session included two decision makers with urban planning background. The second session included four decision makers; the two urban planners from the first session with two energy planners. The two proposed plans achieved the objective of the session with different efficiency levels. While the first session showed a really good accessibility score (Walking\Biking), the second session showed a balance between accessibility and energy efficiency scores due to the better collaboration between decision makers from different background.

The CoTI described in this paper is a system that contributes to the urban planning domain with tangible collaborative tools designed for a wide spectrum of users. CoTI was designed with an emphasis on usability and enhancing the user experience of stakeholders involved in the decision making process. Insights from the design and development of CoTI have led to design implications for the context of urban and city planning systems. Design considerations for extending the interactivity to remote interaction and novel interaction modalities are areas for further research and development.