research-article

Spreadsheets on Interactive Surfaces: Breaking through the Grid with the Pen

Authors:

Vincent Cavez,

Caroline Appert,

Emmanuel PietrigaAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 31, Issue 2

Article No.: 16, Pages 1 - 33

https://doi.org/10.1145/3630097

Published: 29 January 2024 Publication History

Get Access

Abstract

Spreadsheet programs for interactive surfaces have limited manipulations capabilities and are often frustrating to use. One key reason is that the spreadsheet grid creates a layer that intercepts most user input events, making it difficult to reach the cell values that lie underneath. We conduct an analysis of commercial spreadsheet programs and an elicitation study to understand what users can do and what they would like to do with spreadsheets on interactive surfaces. Informed by these, we design interaction techniques that leverage the precision of the pen to mitigate friction between the different layers. These enable more operations by direct manipulation on and through the grid, targeting not only cells and groups of cells, but values and substrings within and across cells as well. We prototype these interaction techniques and conduct a qualitative study with information workers who perform a variety of spreadsheet operations on their own data.

Appendices

A Elicitation Study

A.1 List of Referents

Table 2 lists questions for the 28 referents considered in the elicitation study. Some questions were actually a bit more detailed to give context (e.g., the name of the columns to merge for \(GM_{1}\)). Study material is available as supplemental material.

Table 2.

Scope	Action Type	Question
Value-level	Selection	( \(VS_{1}\)) How would you select the first character of a string in a cell?
		( \(VS_{2}\)) How would you select the comma (and only the comma character) in a cell?
		( \(VS_{3}\)) How would you select the last character of a string in a cell?
		( \(VS_{4}\)) How would you select the left part of a string in a cell?
		( \(VS_{5}\)) How would you select the sequence of characters “, NY” (and only that sequence) in a cell?
		( \(VS_{6}\)) How would you select the right part of a string in a cell?
		( \(VS_{7}\)) How would you generalize a sub-cell selection to its parent column?
	Manipulation	( \(VM_{1}\)) How would you move a selection within a cell?
		( \(VM_{2}\)) How would you delete part of the content of a cell?
		( \(VM_{3}\)) How would split a column into two columns?
Grid-level	Selection	( \(GS_{1}\)) How would you select a cell?
		( \(GS_{2}\)) How would you select a range of cells?
		( \(GS_{3}\)) How would you select a column?
		( \(GS_{4}\)) How would you select a range of columns?
		( \(GS_{5}\)) How would you select a set of columns?
		( \(GS_{6}\)) How would you select a row?
		( \(GS_{7}\)) How would you select a range of rows?
		( \(GS_{8}\)) How would you select a set of rows?
		( \(GS_{9}\)) How would you select the set of cells that have the same value in a column?
		( \(GS_{10}\)) How would you select the set of rows that have the same value for a specific cell?
	Manipulation	( \(GM_{1}\)) How would you merge two columns into one?
		( \(GM_{2}\)) How would you move a column?
		( \(GM_{3}\)) How would you move a row?
		( \(GM_{4}\)) How would you clear a cell?
		( \(GM_{5}\)) How would you delete a column?
		( \(GM_{6}\)) How would you delete a row?
		( \(GM_{7}\)) How would you sort a column?
		( \(GM_{8}\)) How would you fill up a column following the pattern of selected values?

Table 2. Referents Considered in the Elicitation Study

A.2 Definition of a Sign

We define a sign as a series of events that is described along the following dimensions:

•

The input modality, which can be Pen tip, Pen eraser, Single Touch, Multi-touch or Pen + Touch.

•

The start and end locations of input, which can be a Column header, a Row header, a Cell, somewhere Inside-a-Cell, the Select-All button, a Column separator, the Background. We use Inside-a-Cell when the location within the cell itself carries information (e.g., the participant draws a line between two specific characters of the value string).

•

The input event type. We use four types of discrete events: Tap, Double Tap, Dwell and Flick. For continuous events, if the trace’s trajectory does not bear meaningful information, we classify it as Drag. For other continuous events, we use the following five categories: Vertical Line, Horizontal Line, Diagonal Line, Enclose or ZigZag. A few traces do not fall in any of those categories and rather correspond to custom-shape gestures that we categorize into one of the following shapes: Circle, Arrow, Equal sign, Parallel sign, Less-than sign, V, Loops.

An event is defined as a combination of these dimensions, and a sign can be either a single event or a combination of atomic events. Our definition of a sign is quite specific not only regarding the description of an event but also regarding the transition between consecutive events. In particular, when a sign involves a couple of events that have the same modality, we make a distinction between the case where the input device remains in contact with the screen during the transition, and the case where it is lifted up between the two events. For example, a Dwell immediately followed by a Drag without lifting the pen up is different from a Dwell + Drag sequence where the user lifts the pen up after the Dwell. For the coarser modality-based classification, a participant’s proposal is simply described as the combination of its events’ modalities.

B Implementation Details

B.1 Prototype Implementation

The Web-based prototype depicted in Figure 15 and used for the semi-structured qualitative study implements all interaction techniques from Section 5. It is developed entirely in JavaScript and D3 [4], and runs on the client side. Spreadsheet elements and interface widgets are all rendered in SVG. User pen and touch input events are handled with the W3C Pointer API [6].

The prototype is made available as supplemental material, and has been tested extensively with the Chromium Web browser on a Windows 10 PC connected to a Wacom Cintiq Pro. It also runs for instance on a Microsoft Surface Studio 2+, although some interactions that involve two simultaneous contact points are not supported so far because of input event API compatibility issues (the level of support for the W3C pointer API varies significantly across Web browsers and operating systems).

B.2 Generalizing Subcell Selections

Algorithm 1 below details how generalization works for subcell selections that include the cell’s first character. Informally, priority is given to special characters such as dash, comma, and the like, falling back to different alphanumeric transitions (including juxtapositions of uppercase and lower case letters in either order) if no such character could be found. Other cases work similarly but are not detailed for the sake of conciseness: selections that include the last character use a mirror of the algorithm below; selections that include neither the first nor the last character use a combination of both algorithms; selections of the latter category consisting of a single character are generalized based on the transition from the previous character rather than the next one, consistent with the reading direction.

References

[1]

Caroline Appert and Shumin Zhai. 2009. Using strokes as command shortcuts: Cognitive benefits and toolkit support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). Association for Computing Machinery, New York, NY, USA, 2289–2298.

Abstract

A Elicitation Study

A.1 List of Referents

A.2 Definition of a Sign

B Implementation Details

B.1 Prototype Implementation

B.2 Generalizing Subcell Selections

References

Cited By

Index Terms

Recommendations

Bimanual Interaction with Interscopic Multi-Touch Surfaces

Exploring pen and paper interaction with high-resolution wall displays

Mode switching techniques through pen and device profiles

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Full Text

Share

Share this Publication link

Share on social media

Affiliations