research-article

Beyond Applications: Interaction Substrates and Instruments

Author:

Michel Beaudouin-LafonAuthors Info & Claims

IHM '23: Proceedings of the 34th Conference on l'Interaction Humain-Machine

Article No.: 7, Pages 1 - 15

https://doi.org/10.1145/3583961.3583968

Published: 29 May 2023 Publication History

Abstract

This paper introduces a new interaction model based on the concepts of interaction substrate for organizing digital information and interaction instruments for manipulating these substrates. This approach makes the concept of application unnecessary. Instead, it leads to flexible and extensible environments in which users can combine content at will and choose the tools they need to manipulate it. We present Stratify, a proof-of-concept implementation that combines a data-reactive approach to specify relationships among digital objects with a functional-reactive approach to handle interaction. This combination enables the creation of rich information substrates that can be freely inspected and modified, as well as interaction instruments that are decoupled from the objects they interact with, making it possible to use instruments with objects they were not designed for. We illustrate the flexibility of the approach with several examples and present directions for future work.

References

[1]

Michel Beaudouin-Lafon. 2000. Instrumental Interaction: An Interaction Model for Designing post-WIMP User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands) (CHI ’00). ACM, New York, NY, USA, 446–453. https://doi.org/10.1145/332040.332473

Digital Library

[2]

Michel Beaudouin-Lafon. 2017. Towards Unified Principles of Interaction. In Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (Cagliari, Italy) (CHItaly ’17). ACM, New York, NY, USA, Article 1, 2 pages. https://doi.org/10.1145/3125571.3125602

Digital Library

[3]

Michel Beaudouin-Lafon, Susanne Bødker, and Wendy E. Mackay. 2021. Generative Theories of Interaction. ACM Transactions on Computer-Human Interaction 28, 6, Article 45 (11 2021), 54 pages. https://doi.org/10.1145/3468505

Digital Library

[4]

Michel Beaudouin-Lafon and Wendy E. Mackay. 2000. Reification, Polymorphism and Reuse: Three Principles for Designing Visual Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (Palermo, Italy) (AVI ’00). ACM, New York, NY, USA, 102–109. https://doi.org/10.1145/345513.345267

Digital Library

[5]

Pascal Béguin and Pierre Rabardel. 2000. Designing for instrument-mediated activity. Scandinavian Journal of information Systems 12 (2000), 173–190.

[6]

Stuart K. Card, Jock D. Mackinlay, and Ben Shneiderman (Eds.). 1999. Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Digital Library

[7]

Marianela Ciolfi Felice, Nolwenn Maudet, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2016. Beyond Snapping: Persistent, Tweakable Alignment and Distribution with StickyLines. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). ACM, New York, NY, USA, 133–144. https://doi.org/10.1145/2984511.2984577

Digital Library

[8]

Dave Curbow and Elizabeth Dykstra-Erickson. 1997. Designing the OpenDoc Human Interface. In Proceedings of the 2nd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (Amsterdam, The Netherlands) (DIS ’97). ACM, New York, NY, USA, 83–95. https://doi.org/10.1145/263552.263581

Digital Library

[9]

Alan Dix. 2007. Designing for Appropriation. In Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK 21. British Computer Society, London, UK, 1–4. https://doi.org/10.14236/ewic/HCI2007.53

[10]

James R. Eagan, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2011. Cracking the Cocoa Nut: User Interface Programming at Runtime. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). ACM, New York, NY, USA, 225–234. https://doi.org/10.1145/2047196.2047226

Digital Library

[11]

Jérémie Garcia, Theophanis Tsandilas, Carlos Agon, and Wendy Mackay. 2012. Interactive Paper Substrates to Support Musical Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). ACM, New York, NY, USA, 1825–1828. https://doi.org/10.1145/2207676.2208316

Digital Library

[12]

William W. Gaver. 1991. Technology Affordances. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New Orleans, Louisiana, USA) (CHI ’91). ACM, New York, NY, USA, 79–84. https://doi.org/10.1145/108844.108856

Digital Library

[13]

Eleanor Jack Gibson. 1969. Principles of Perceptual Learning and Development.Appleton & Co.

[14]

James J Gibson. 1986. The Ecological Approach to Visual Perception. Lawrence Erlbaum.

[15]

Camille Gobert and Michel Beaudouin-Lafon. 2022. I-LaTeX: Manipulating Transitional Representations between LaTeX Code and Generated Documents. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). ACM, New York, NY, USA, Article 554, 16 pages. https://doi.org/10.1145/3491102.3517494

Digital Library

[16]

Camille Gobert and Michel Beaudouin-Lafon. 2022. Représentations Intermédiaires Interactives Pour La Manipulation de Code LaTeX: Interactive Intermediate Representations for LaTeX Code Manipulation. In Proceedings of the 32nd Conference on l’Interaction Homme-Machine (Virtual Event, France) (IHM ’21). ACM, New York, NY, USA, Article 10, 11 pages. https://doi.org/10.1145/3450522.3451325

Digital Library

[17]

Adele Goldberg and David Robson. 1983. Smalltalk-80: The Language and Its Implementation. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[18]

Julien Gori, Han L. Han, and Michel Beaudouin-Lafon. 2020. FileWeaver: Flexible File Management with Automatic Dependency Tracking. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). ACM, New York, NY, USA, 22–34. https://doi.org/10.1145/3379337.3415830

Digital Library

[19]

Viktor Gustafsson, Benjamin Holme, and Wendy E. Mackay. 2020. Narrative Substrates: Reifying and Managing Emergent Narratives in Persistent Game Worlds. In International Conference on the Foundations of Digital Games (Bugibba, Malta) (FDG ’20). ACM, New York, NY, USA, Article 46, 12 pages. https://doi.org/10.1145/3402942.3403015

Digital Library

[20]

Han L. Han, Junhang Yu, Raphael Bournet, Alexandre Ciorascu, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2022. Passages: Interacting with Text Across Documents. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). ACM, New York, NY, USA, Article 338, 17 pages. https://doi.org/10.1145/3491102.3502052

Digital Library

[21]

Patrick J. Hayes. 1979. The Naive Physics Manifesto. In Expert Systems in the Electronic Age, Donald Michie (Ed.). Edinburgh University Press, Edinburgh, Scotland, 242–270.

[22]

Daniel Ingalls, Tim Felgentreff, Robert Hirschfeld, Robert Krahn, Jens Lincke, Marko Röder, Antero Taivalsaari, and Tommi Mikkonen. 2016. A World of Active Objects for Work and Play: The First Ten Years of Lively. Proceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (2016), 238–249. https://doi.org/10.1145/2986012.2986029

Digital Library

[23]

Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based Interaction: A Framework for post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). ACM, New York, NY, USA, 201–210. https://doi.org/10.1145/1357054.1357089

Digital Library

[24]

Ghita Jalal, Nolwenn Maudet, and Wendy E. Mackay. 2015. Color Portraits: From Color Picking to Interacting with Color. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 4207–4216. https://doi.org/10.1145/2702123.2702173

Digital Library

[25]

Jeff Johnson and Austin Henderson. 2011. Conceptual Models: Core to Good Design. Morgan Claypool.

[26]

Jeff Johnson, Teresa L. Roberts, William Verplank, David Canfield Smith, Charles H. Irby, Marian Beard, and Kevin Mackey. 1989. The Xerox Star: A Retrospective. Computer 22, 9 (1989), 11–26.

Digital Library

[27]

Alan C. Kay. 1984. Computer Software. Scientific American251 (1984), 53–59.

[28]

Roberta L Klatzky, Brian MacWhinney, and Marlene Behrmann. 2012. Embodiment, Ego-Space, and Action. Psychology Press.

[29]

Clemens N. Klokmose, James R. Eagan, Siemen Baader, Wendy Mackay, and Michel Beaudouin-Lafon. 2015. Webstrates: Shareable Dynamic Media. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). ACM, New York, NY, USA, 280–290. https://doi.org/10.1145/2807442.2807446

Digital Library

[30]

Glenn E. Krasner and Stephen T. Pope. 1988. A Cookbook for Using the Model-view Controller User Interface Paradigm in Smalltalk-80. J. Object Oriented Program. 1, 3 (Aug. 1988), 26–49. http://dl.acm.org/citation.cfm?id=50757.50759

Digital Library

[31]

Henry Lieberman, Fabio Paternò, and Volker Wulf (Eds.). 2006. Springer Netherlands, Dordrecht. https://doi.org/10.1007/1-4020-5386-X_1

[32]

Wendy E. Mackay. 2000. Responding to cognitive overload: Co-adaptation between users and technology. Intellectica 30 (Jan. 2000), 177–193.

[33]

Allan MacLean, Kathleen Carter, Lennart Lövstrand, and Thomas Moran. 1990. User-tailorable Systems: Pressing the Issues with Buttons. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). ACM, New York, NY, USA, 175–182. https://doi.org/10.1145/97243.97271

Digital Library

[34]

Nolwenn Maudet, Ghita Jalal, Philip Tchernavskij, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2017. Beyond Grids: Interactive Graphical Substrates to Structure Digital Layout. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 5053–5064. https://doi.org/10.1145/3025453.3025718

Digital Library

[35]

Don Norman. 1998. The Design of Everyday Things. Doubleday.

Digital Library

[36]

Dan R. Olsen, Jr., Scott E. Hudson, Thom Verratti, Jeremy M. Heiner, and Matt Phelps. 1999. Implementing Interface Attachments Based on Surface Representations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). ACM, New York, NY, USA, 191–198. https://doi.org/10.1145/302979.303038

Digital Library

[37]

François Osiurak. 2010. What Neuropsychology Tells Us about Human Tool Use? The Four Constraints Theory (4CT): Mechanics, Space, Time and Effort. Neurosychology Review 24, 2 (2010), 88–115. https://doi.org/10.1007/s11065-014-9260-y

[38]

François Osiurak and Arnaud Badets. 2016. Tool Use and Affordance: Manipulation-Based Versus Reasoning-Based Approaches. Psychological Review 123, 5 (2016), 35 pages. https://doi.org/10.1037/rev0000027

[39]

François Osiurak, Christophe Jarry, and Didier Le Gall. 2010. Grasping the Affordances, Understanding the Reasoning: Toward a Dialectical Theory of Human Tool Use. Psychological Review 117, 2 (2010), 517–540. https://doi.org/10.1037/a0019004

[40]

Pierre Rabardel. 1995. Les hommes et les technologies; approche cognitive des instruments contemporains. Armand Colin.

[41]

Miguel A. Renom. 2022. Theoretical bases of human tool use in digital environments. Theses. Université Paris-Saclay. https://theses.hal.science/tel-03675933

[42]

Miguel A. Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon. 2022. Exploring Technical Reasoning in Digital Tool Use. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). ACM, New York, NY, USA, Article 579, 17 pages. https://doi.org/10.1145/3491102.3501877

Digital Library

[43]

Miguel A. Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon. 2023. Interaction Knowledge: Understanding the ‘Mechanics’ of Digital Tools. In CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). ACM, New York, NY, USA. In press.

[44]

Alex Repenning. 1994. Programming Substrates to Create Interactive Learning Environments. Interactive Learning Environments 4, 1 (1994), 45–74.

[45]

Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. Conflict-Free Replicated Data Types. In Symposium on Self-Stabilizing Systems. Springer, 386–400.

[46]

Randall B. Smith. 1987. Experiences with the Alternate Reality Kit: An Example of the Tension Between Literalism and Magic. Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (1987), 61–67. https://doi.org/10.1145/29933.30861

Digital Library

[47]

Paul S. Strauss. 1993. IRIS Inventor, a 3D Graphics Toolkit. SIGPLAN Not. 28, 10 (Oct. 1993), 192–200. https://doi.org/10.1145/167962.165889

Digital Library

[48]

Brygg Ullmer, Hiroshi Ishii, and Robert J. K. Jacob. 2005. Token+Constraint Systems for Tangible Interaction with Digital Information. ACM Trans. Comput.-Hum. Interact. 12, 1 (March 2005), 81–118. https://doi.org/10.1145/1057237.1057242

Digital Library

Cited By

Tricaud MBeaudouin-Lafon M(2023)Revisiting creative behaviour as an epistemic process: lessons from 12 computational artists and designersProceedings of the 35th Australian Computer-Human Interaction Conference10.1145/3638380.3638395(175-190)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3638380.3638395

Index Terms

Beyond Applications: Interaction Substrates and Instruments
1. Human-centered computing
  1. Human computer interaction (HCI)
  2. Interaction design
    1. Systems and tools for interaction design

Recommendations

Instrumental interaction: an interaction model for designing post-WIMP user interfaces
CHI '00: Proceedings of the SIGCHI conference on Human Factors in Computing Systems

This article introduces a new interaction model called Instrumental Interaction that extends and generalizes the principles of direct manipulation. It covers existing interaction styles, including traditional WIMP interfaces, as well as new interaction ...
Designing interaction, not interfaces
AVI '04: Proceedings of the working conference on Advanced visual interfaces

Although the power of personal computers has increased 1000-fold over the past 20 years, user interfaces remain essentially the same. Innovations in HCI research, particularly novel interaction techniques, are rarely incorporated into products. In this ...
An Interaction Model for Grasp-Aware Tangibles on Interactive Surfaces
ITS '14: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces

Tangibles on interactive surfaces enable users to physically manipulate digital content by placing, manipulating, or removing a tangible object. However, the information whether and how a user grasps these objects has not been mapped out for tangibles ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

IHM '23: Proceedings of the 34th Conference on l'Interaction Humain-Machine

April 2023

288 pages

ISBN:9781450398244

DOI:10.1145/3583961

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

H2020 European Research Council

Conference

IHM '23

IHM '23: 34th International Francophone Conference on Human-Computer Interaction

April 3 - 6, 2023

TROYES, France

Acceptance Rates

Overall Acceptance Rate 103 of 199 submissions, 52%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
121
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)12

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tricaud MBeaudouin-Lafon M(2023)Revisiting creative behaviour as an epistemic process: lessons from 12 computational artists and designersProceedings of the 35th Australian Computer-Human Interaction Conference10.1145/3638380.3638395(175-190)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3638380.3638395

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents