research-article

User-defined semantics for the design of IoT systems enabling smart interactive experiences

Authors:

Carmelo Ardito,

Giuseppe Desolda,

Rosa Lanzilotti,

Alessio Malizia,

Maristella Matera,

Antonio PiccinnoAuthors Info & Claims

Personal and Ubiquitous Computing, Volume 24, Issue 6

Pages 781 - 796

https://doi.org/10.1007/s00779-020-01457-5

Published: 26 September 2020 Publication History

Abstract

Automation in computing systems has always been considered a valuable solution to unburden the user. Internet of Things (IoT) technology best suits automation in different domains, such as home automation, retail, industry, and transportation, to name but a few. While these domains are strongly characterized by implicit user interaction, more recently, automation has been adopted also for the provision of interactive and immersive experiences that actively involve the users. IoT technology thus becomes the key for Smart Interactive Experiences (SIEs), i.e., immersive automated experiences created by orchestrating different devices to enable smart environments to fluidly react to the final users’ behavior. There are domains, e.g., cultural heritage, where these systems and the SIEs can support and provide several benefits. However, experts of such domains, while intrigued by the opportunity to induce SIEs, are facing tough challenges in their everyday work activities when they are required to automate and orchestrate IoT devices without the necessary coding skills. This paper presents a design approach that tries to overcome these difficulties thanks to the adoption of ontologies for defining Event-Condition-Action rules. More specifically, the approach enables domain experts to identify and specify properties of IoT devices through a user-defined semantics that, being closer to the domain experts’ background, facilitates them in automating the IoT devices behavior. We also present a study comparing three different interaction paradigms conceived to support the specification of user-defined semantics through a “transparent” use of ontologies. Based on the results of this study, we work out some lessons learned on how the proposed paradigms help domain experts express their semantics, which in turn facilitates the creation of interactive applications enabling SIEs.

References

[1]

Cidoc CRM. Retrieved from http://www.cidoc-crm.org/ Last Access June 10, 2019

[2]

Akiki PA, Bandara AK, and Yu Y Visual simple transformations: empowering end-users to wire Internet of Things objects ACM Transaction on Computer-Human Interaction 2017 24 2 1-43

[3]

Ambe AH, Brereton M, Soro A, Chai MZ, Buys L, Roe P (2019) Older people inventing their personal internet of things with the IoT un-kit experience. In Proc. of the Conference on Human Factors in Computing Systems (CHI '19). ACM, paper 322

[4]

Ardito C, Buono P, Desolda G, and Matera M From smart objects to smart experiences: an end-user development approach International Journal of Human-Computer Studies 2017 114 51-68

[5]

Ardito C, Desolda G, Lanzilotti R, Malizia A, Matera M (2017) Advanced interaction paradigms to define smart visit experiences in the Internet of Things era in Proc. of the Conference on the Italian SIGCHI Chapter (CHItaly '17), CEUR Workshop Proceedings, Vol-1910, 1148-1152

[6]

Ardito C, Desolda G, Lanzilotti R, Malizia A, and Matera M Analysing trade-offs in frameworks for the design of smart environments Behav Inform Technol 2019 39 1 47-71

[7]

Balducci F, Buono P, Desolda G, Impedovo D, and Piccinno A Improving smart interactive experiences in cultural heritage through pattern recognition techniques Pattern Recogn Lett 2020 131 142-149

[8]

Bosca A, Bonino D, Pellegrino P (2005) OntoSphere: more than a 3D ontology visualization tool. In Proc. of the Italian Semantic Web Workshop (SWAP '05). CEUR workshop proceedings

[9]

Braun V and Clarke V Using thematic analysis in psychology Qual Res Psychol 2006 3 2 77-101

[10]

Brich J, Walch M, Rietzler M, Weber M, and Schaub F Exploring end user programming needs in home automation ACM Transaction on Computer-Human Interaction 2017 24 2 1-35 11 (2017)

[11]

Budweg S, Draxler S, Lohmann S, Rashid A, Stevens G (2010) Open design spaces: socially crafting interactive experiences. In Proc. of the ACM Conference on Designing Interactive Systems (DIS '10). ACM, New York, pp 442–443

[12]

Caivano D, Fogli D, Lanzilotti R, Piccinno A, and Cassano F Supporting end users to control their smart home: design implications from a literature review and an empirical investigation J Syst Softw 2018 144 295-313

[13]

Cappiello C, Matera M, and Picozzi M A UI-centric approach for the end-user development of multidevice mashups ACM Transaction Web 2015 9 3 1-40

[14]

Cappiello C, Matera M, Picozzi M, Sprega G, Barbagallo D, and Francalanci C Auer S, Díaz O, and Papadopoulos G DashMash: a mashup environment for end user development Web Engineering - ICWE 2011. Lecture Notes in Computer Science 2011 Heidelberg Springer Berlin 152-166

[15]

Catala A, Jaen J, Dijk Bv, Jordà S (2012) Exploring tabletops as an effective tool to foster creativity traits. In Proc. of the International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, New York, NY, USA, 143–150

[16]

Corno F, De Russis L, and Monge Roffarello A A high-level semantic approach to end-user development in the Internet of Things International Journal of Human-Computer Studies 2019 125 41-54

[17]

Corno F, De Russis L, Roffarello AM (2017) A high-level approach towards end user development in the IoT. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems - Extended Abstracts (CHI '17). ACM, New York, pp 1546–1552

[18]

Corno F, Russis LD, and Roffarello AM A semantic web approach to simplifying trigger-action programming in the IoT Computer 2017 50 11 18-24

[19]

Corno F, Russis LD, Roffarello AM (2019) RecRules: recommending IF-THEN rules for end-user development. ACM Trans. Intell. Syst. Technol., Article 58

[20]

Coronado M and Iglesias CA Task automation services: automation for the masses IEEE Internet Comput 2016 20 1 52-58

[21]

Crosser. Crosser. Retrieved from https://crosser.io/platform/crosser-node/. Accessed 28 Feb 2020

[22]

Delprino F, Piva C, Tommasi G, Gelsomini M, Izzo N, Matera M (2018). ABBOT: a smart toy motivating children to become outdoor explorers. In Proc. of the Conference on Advanced Visual Interfaces (AVI '18). ACM, New York, pp 1–9

[23]

Desolda G, Ardito C, Matera M (2017) Empowering end users to customize their smart environments: model, composition paradigms and domain-specific tools. ACM Transactions on Computer-Human Interaction 24(2), Article 12 (April 2017), 52 pages

[24]

Desolda G, Matera M, and Lanzilotti R Metamorphic data sources: a user-centric paradigm to consume linked data in interactive workspaces Futur Gener Comput Syst 2020 102 992-1015

[25]

Doering T, Beckhaus S, Schmidt A (2009) Towards a sensible integration of paper-based tangible user interfaces into creative work processes. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems - Extended Abstracts (CHI '09). ACM, New York, pp 4627–4632

[26]

Eklund P, Roberts NP, Green S (2002) Ontorama: browsing an rdf ontology using a hyperbolic-like browser. In Proc. of the First International Symposium on CyberWorlds (CW '02), 405–411

[27]

Fröhlich P, Baldauf M, Meneweger T, Erickson I, Tscheligi M, Gable T, Ruyter BD, Paternò F (2019) Everyday automation experience: non-expert users encountering ubiquitous automated systems. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems - Extended Abstracts (CHI '19). ACM, New York, pp 1–8

[28]

Garzotto F, Beccaluva E, Giannotti M, Riccardi F (2020) Interactive multisensory environments for primary school children. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI '20). in print

[29]

Gennari R, Matera M, Melonio A, Roumelioti E (2019) A board-game for co-designing smart nature environments in workshops with children. In Proc. of the International Symposium on End User Development (IS-EUD '19). Springer international publishing, New York, pp 132–148

[30]

Ghiani G, Manca M, Paternò F, Santoro C (2017) Personalization of context-dependent applications through trigger-action rules. ACM Transaction on Computer-Human Interaction, 24(2), Article 14 (April 2017), 33 pages

[31]

Gilman E, Davidyuk O, Su X, and Riekki J Towards interactive smart spaces Journal of Ambient Intelligence and Smart Environments (JAISE) 2013 5 1 5-22

[32]

Gruber TR A translation approach to portable ontology specifications Knowledge Acquisition - Special issue: Current issues in knowledge 1993 5 2 199-220

[33]

Guo B, Zhang D, Wang Z, Yu Z, and Zhou X Opportunistic IoT: exploring the harmonious interaction between human and the Internet of Things J Netw Comput Appl 2013 36 6 1531-1539

[34]

Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits and atoms. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI '97). ACM, New York, pp 234–241

[35]

Janssen CP, Donker SF, Brumby DP, and Kun AL History and future of human-automation interaction International Journal of Human-Computer Studies 2019 131 99-107

[36]

Jeong C.-S, Pang A (1998) Reconfigurable disc trees for visualizing large hierarchical information space. In Proc. of the IEEE Symposium on Information Visualization (INFOVIS '98). IEEE computer society, Washington, DC, pp 19–25

[37]

JS Foundation. Node-RED. Retrieved from http://nodered.org/ Last Access February 28th, 2020

[38]

Katifori A, Halatsis C, Lepouras G, Vassilakis C, and Giannopoulou E Ontology visualization methods—a survey ACM Comput Surv 2007 39 4 10

[39]

Kim MJ and Maher ML The impact of tangible user interfaces on spatial cognition during collaborative design Des Stud 2008 29 3 222-253

[40]

Lamm H and Trommsdorff G Group versus individual performance on tasks requiring ideational proficiency (brainstorming): a review European Journal of Social Psychology (EJSP) 1973 3 4 361-388

[41]

Leifer L The design of implicit interactions: making interactive systems less obnoxious Des Issues 2008 24 72-84

[42]

Li Z, Barbara RB, and Claudia I Nik B A meta-design model for creative distributed collaborative design Development of distributed systems from design to application and maintenance 2013 Hershey IGI Global 252-268

[43]

Liebig T, Noppens O (2004) Ontotrack: combining browsing and editing with reasoning and explaining for owl lite ontologies. In Proc. of the International Semantic Web Conference (ISWC '04). Springer, ‎Berlin‎ ‎Heidelberg, pp 244–258

[44]

Manca M, Fabio P, Santoro C, and Corcella L Supporting end-user debugging of trigger-action rules for IoT applications International Journal of Human-Computer Studies 2019 123 56-69

[45]

Markopoulos P, Nichols J, Paternò F, and Pipek V Editorial: end-user development for the Internet of Things ACM Transactions on Computer-Human Interaction 2017 24 2 1-3

[46]

Metaxas G and Markopoulos P Natural contextual reasoning for end users ACM Transaction on Computer-Human Interaction 2017 24 2 1-36 Article 13

[47]

Microsoft. Microsoft Flow. Retrieved from https://flow.microsoft.com/. Accessed 28 Feb 2020

[48]

Miller GA WordNet: a lexical database for English Commun ACM 1995 38 11 39-41

[49]

Mohanraj I, Ashokumar K, and Naren J Field monitoring and automation using IOT in agriculture domain Procedia Computer Science 2016 93 931-939

[50]

Mora S, Gianni F, Divitini M (2017) Tiles: a card-based ideation toolkit for the Internet of Things. In Proc. of the Conference on Designing Interactive Systems (DIS '17). ACM, New York, pp 587–598

[51]

Noy NF, Fergerson RW, Musen MA (2000) The knowledge model of Protégé-2000: combining interoperability and flexibility. In Proc. of the International Conference on Knowledge Engineering and Knowledge Management (EKAW '00). Springer Berlin Heidelberg, 17–32

[52]

Oppezzo M and Schwartz DL Give your ideas some legs: the positive effect of walking on creative thinking Journal of Experimental Psychology: Learning, Memory, and Cognition (PeePs) 2014 40 4 1142

[53]

Patil AA, Oundhakar SA, Sheth AP, Verma K (2004) Meteor-s web service annotation framework. In Proc. of the International Conference on World Wide Web (WWW '04). ACM, New York, pp 553–562

[54]

Petrelli D, Lechner M (2014) The meSch project – material EncounterS with digital cultural heritage: reusing existing digital resources in the creation of novel forms of visitor’s experiences. In Proc. of the International Committee for Documentation of ICOM (CIDOC '14)

[55]

Petrelli D, Marshall MT, O’Brien S, McEntaggart P, and Gwilt I Tangible data souvenirs as a bridge between a physical museum visit and online digital experience Pers Ubiquit Comput 2017 21 2 281-295

[56]

Plaisant C, Grosjean J, Bederson BB (2002). SpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation. In Proc. of the IEEE Symposium on Information Visualization (INFOVIS 2002), 57–64

[57]

Gennari R, Matera M, Melonio A, Roumelioti E, Delprino F, Filaferro M, Gennaioli G, Manzi V(2019). Interactive nature: designing smart devices for nature exploration by children. In Proc. of the Biannual Conference of the Italian Chapter of SIGCHI 2019 - Adjunct Proceedings, CLEUP (CHItaly '19), 67–70

[58]

Robertson, G.G., Mackinlay, J.D. and Card, S.K. (1991). Cone trees: animated 3D visualizations of hierarchical information. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91). ACM, New York, pp 189–194

[59]

Rodríguez-García MÁ, Valencia-García R, García-Sánchez F, and Samper-Zapater JJ Ontology-based annotation and retrieval of services in the cloud Knowl-Based Syst 2014 56 15-25

[60]

Rosson MB and Carroll JM Julie AJ and Andrew S Scenario-based design The human-computer interaction handbook 2003 Mahwah L. Erlbaum associates Inc. 1032-1050

[61]

Saquicela V, Vilches-Blázquez LM, Corcho Ó (2010). Semantic annotation of RESTful services using external resources. In Proc. of the International Conference on Web Engineering (ICWE '10). Springer Berlin Heidelberg, pp 266–276

[62]

Shneiderman B Creativity support tools: accelerating discovery and innovation Commun ACM 2007 50 12 20-32

[63]

Shneiderman B Redondo M, Bravo C, and Ortega M Creativity support tools: a grand challenge for HCI researchers Engineering the user interface: from research to practice 2009 London Springer London 1-9

[64]

Sivakumar R, Arivoli P, and Sri A Ontology visualization PROTÉGÉ tools–a review International Journal of Advanced Information Technology 2011 1 4 1-11

[65]

Sure Y, Angele J, Staab S (2002). OntoEdit: guiding ontology development by methodology and inferencing. In Proc. of the OTM Confederated International Conferences "On the Move to Meaningful Internet Systems" (OTM '02). Springer Berlin Heidelberg, 1205–1222

[66]

Tutenel T, Bidarra R, Smelik RM, and Kraker KJD The role of semantics in games and simulations Comput Entertain 2008 6 4 1-35

[67]

Unger R, Chandler C (2012). A project guide to UX design: for user experience designers in the field or in the making. New Riders

[68]

Ur B, Ho MPY, Brawner S, Lee J, Mennicken S, Picard N, Schulze D, Littman ML (2016). Trigger-action programming in the wild: an analysis of 200,000 IFTTT recipes. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, pp 3227–3231

[69]

Warr A, O'Neill E (2005). Understanding design as a social creative process. In Proc. of the Conference on Creativity & cognition (C&C '05). ACM, New York, pp 118–127

[70]

Zancanaro M, Not E, Petrelli D, Marshall M, van Dijk T, Risseeuw M, van Dijk D, Venturini A, Cavada D, Kubitza T (2015). Recipes for tangible and embodied visit experiences. In Proc. of the Museums and the Web conference (MW '15)

[71]

Zhong S, Tian L, Li C, Storch K-F, Wong WH (2004). Comparative analysis of gene sets in the gene ontology space under the multiple hypothesis testing framework. In Proc. of the IEEE Computational Systems Bioinformatics (CSB '04). IEEE, 425–435

[72]

Zobel J, Dart P (1996). Phonetic string matching: lessons from information retrieval. In Proc. of the ACM SIGIR conference on Research and development in information retrieval (SIGIR '96). ACM, New York, pp 166–172

Cited By

Nusir MRekik M(2024)Systematic review of co-design in digital health for COVID-19 researchUniversal Access in the Information Society10.1007/s10209-022-00964-x23:2(637-651)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10209-022-00964-x
Al Muhander BWiese JRana OPerera C(2023)Interactive Privacy Management: Toward Enhancing Privacy Awareness and Control in the Internet of ThingsACM Transactions on Internet of Things10.1145/36000964:3(1-34)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3600096
Liu XShi YYu CGao CYang TLiang CShi Y(2023)Understanding In-Situ Programming for Smart Home AutomationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962547:2(1-31)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596254
Show More Cited By

Index Terms

User-defined semantics for the design of IoT systems enabling smart interactive experiences
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Software and its engineering

Index terms have been assigned to the content through auto-classification.

Recommendations

A systematic literature review on semantic models for IoT-enabled smart campus

Smart communities have recently gained much attention. Researchers have been trying to tackle a number of challenges faced by smart communities. Interoperability is one key challenge that occurs due to different systems using different knowledge ...
Enabling End-Users to Configure Smart Environments
SAINT '11: Proceedings of the 2011 IEEE/IPSJ International Symposium on Applications and the Internet

Our everyday living environments are inhabited by a huge and constantly increasing number of electronic devices. Smart Environment is a common name for a physical place where these various devices interact with each other in order to provide useful ...
Pervasive Smart Contracts for Blockchains in IoT Systems
ICBTA '18: Proceedings of the 2018 International Conference on Blockchain Technology and Application

Thanks to its decentralized structure and immutability, blockchain technology has the potential to address relevant security and privacy challenges in the Internet of Things (IoT). In particular, by hosting and executing smart contracts, blockchain ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Personal and Ubiquitous Computing

Personal and Ubiquitous Computing Volume 24, Issue 6

Dec 2020

178 pages

ISSN:1617-4909

Issue’s Table of Contents

© The Author(s) 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 26 September 2020

Accepted: 05 September 2020

Received: 16 December 2019

Author Tags

Qualifiers

Research-article

Funding Sources

Ministero dell’Istruzione, dell’Università e della Ricerca

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
2
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nusir MRekik M(2024)Systematic review of co-design in digital health for COVID-19 researchUniversal Access in the Information Society10.1007/s10209-022-00964-x23:2(637-651)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10209-022-00964-x
Al Muhander BWiese JRana OPerera C(2023)Interactive Privacy Management: Toward Enhancing Privacy Awareness and Control in the Internet of ThingsACM Transactions on Internet of Things10.1145/36000964:3(1-34)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3600096
Liu XShi YYu CGao CYang TLiang CShi Y(2023)Understanding In-Situ Programming for Smart Home AutomationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962547:2(1-31)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596254
Bowen JTurner J(2023)Interactive System Modelling for the Internet of ThingsProceedings of the ACM on Human-Computer Interaction10.1145/35932337:EICS(1-19)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593233
Zheng W(2023)RETRACTED ARTICLE: Cloud service and interactive IoT system application in the service management mode of logistics enterprisesSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-08166-027:13(9051-9064)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.1007/s00500-023-08166-0
Breve BDesolda GGreco FDeufemia V(2023)Democratizing Cybersecurity in Smart Environments: Investigating the Mental Models of Novices and ExpertsEnd-User Development10.1007/978-3-031-34433-6_9(145-161)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-34433-6_9
Andrao MBalducci FBreve BCena FDesolda GDeufemia VGena CMatera MMattioli APaternò FSantoro CTreccani BVernero FZancanaro M(2023)Understanding Concepts, Methods and Tools for End-User Control of Automations in Ecosystems of Smart Objects and ServicesEnd-User Development10.1007/978-3-031-34433-6_7(104-124)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-34433-6_7
Breve BCimino GDesolda GDeufemia VElefante A(2023)On the User Perception of Security Risks of TAP Rules: A User StudyEnd-User Development10.1007/978-3-031-34433-6_10(162-179)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-34433-6_10
Balducci FBreve BCena FMattioli ARizvi MBottoni PPanizzi E(2022)EMPATHY: 3rd International Workshop on Empowering People in Dealing with Internet of Things EcosystemsProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3535251(1-4)Online publication date: 6-Jun-2022
https://dl.acm.org/doi/10.1145/3531073.3535251
Desolda GLanzilotti RPiccinno ARossano V(2021)A System to Support Children in Speech Therapies at HomeProceedings of the 14th Biannual Conference of the Italian SIGCHI Chapter10.1145/3464385.3464745(1-5)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3464385.3464745
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents