research-article

Open access

Applications across Co-located Devices: User Interface Distribution, State Management and Collaboration

Authors:

Pedro Albuquerque Santos,

Rui Neves Madeira,

Nuno CorreiaAuthors Info & Claims

iiWAS2021: The 23rd International Conference on Information Integration and Web Intelligence

Pages 602 - 613

https://doi.org/10.1145/3487664.3487748

Published: 30 December 2021 Publication History

All formats PDF

Abstract

We live surrounded by computing devices, but we are not yet combining them into a natural unified user experience. Therefore, we have designed and implemented a framework to generalize and support the development of multi-device applications, which distribute their User Interfaces (UI) across co-located devices. The framework is composed of: authentication and authorization services; a broker to sync information across application instances; background services to gather the capabilities of the devices; an integration module with an indoor positioning system to determine when to associate devices. It provides a library to determine the components to be displayed based on UI requirements and the capabilities of the available devices. It also includes elements to manage the distribution of UI components and application states that can be shared to enable collaboration. We present an application prototype and a user study which assesses how multi-device applications are perceived by users. The results are very positive, indicating that we should keep working on our research since the users received very well the prototype and the concept, which also validates the support provided by the framework.

Supplementary Material

MP4 File (juxtboard-demo.mp4)

Demo video

Download
11.49 MB

References

[1]

P. Bahl and V.N. Padmanabhan. 2000. RADAR: an in-building RF-based user location and tracking system. In Proc. IEEE INFOCOM 2000. Conf. Comput. Commun. Ninet. Annu. Jt. Conf. IEEE Comput. Commun. Soc. (Cat. No.00CH37064), Vol. 2. IEEE, 775–784. https://doi.org/10.1109/INFCOM.2000.832252

[2]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Stud. 4(2009), 114–123. http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_May2009.pdf

Digital Library

[3]

Ramon F. Brena, Juan Pablo García-Vázquez, Carlos E. Galván-Tejada, David Muñoz-Rodriguez, Cesar Vargas-Rosales, and James Fangmeyer. 2017. Evolution of Indoor Positioning Technologies: A Survey. Journal of Sensors 2017 (March 2017), 1–21. https://doi.org/10.1155/2017/2630413

[4]

John Brooke. 1996. SUS - A quick and dirty usability scale. In Usability Eval. Ind.(1st ed.), Patrick W. Jordan, B. Thomas, Ian Lyall McClelland, and Bernard Weerdmeester (Eds.). CRC Press, Chapter 21, 189–194.

[5]

Pedro Centieiro, Rui Neves Madeira, Teresa Romão, A. Eduardo Dias, and Nuno Correia. 2015. In sync with fair play!: delivering a synchronized and cheat-preventing second screen gaming experience. In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology - ACE ’15. ACM Press, New York, New York, USA, 1–11. https://doi.org/10.1145/2832932.2832953

Digital Library

[6]

Marco Cruz, Teresa Romão, Pedro Centieiro, and A. Eduardo Dias. 2017. Exploring the use of second screen devices during live sports broadcasts to promote social interaction. In Proceedings of the 14th Conference on Advances in Computer Entertainment Technology - ACE ’17, Vol. 10714 LNCS. Springer, Cham, 318–338. https://doi.org/10.1007/978-3-319-76270-8_23

[7]

Ed D. Hardt. 2012. The OAuth 2.0 Authorization Framework. Technical Report. Microsoft. https://doi.org/10.17487/rfc6749

[8]

David Dearman and Jeffery S. Pierce. 2008. It’s on my other computer!: computing with multiple devices. In Proceeding twenty-sixth Annu. CHI Conf. Hum. factors Comput. Syst. - CHI ’08. ACM Press, New York, New York, USA, 767. https://doi.org/10.1145/1357054.1357177

[9]

Mario Di Nardo and Haoxuan Yu. 2021. Special Issue “Industry 5.0: The Prelude to the Sixth Industrial Revolution”. Applied System Innovation 4, 3 (2021). https://doi.org/10.3390/asi4030045

[10]

Tao Dong, Elizabeth F. Churchill, and Jeffrey Nichols. 2016. Understanding the challenges of designing and developing multi-device experiences. In DIS 2016 - Proc. 2016 ACM Conf. Des. Interact. Syst. Fuse. ACM Press, New York, New York, USA, 62–72. https://doi.org/10.1145/2901790.2901851

Digital Library

[11]

Andrea Gallidabino and Cesare Pautasso. 2016. The Liquid.js Framework for Migrating and Cloning Stateful Web Components across Multiple Devices. In Proc. 25th Int. Conf. Companion World Wide Web - WWW ’16 Companion. ACM Press, New York, New York, USA, 183–186. https://doi.org/10.1145/2872518.2890538

Digital Library

[12]

Andrea Gallidabino, Cesare Pautasso, Ville Ilvonen, Tommi Mikkonen, Kari Systa, Jari-Pekka Voutilainen, and Antero Taivalsaari. 2016. On the Architecture of Liquid Software: Technology Alternatives and Design Space. In 2016 13th Work. IEEE/IFIP Conf. Softw. Archit.IEEE, 122–127. https://doi.org/10.1109/WICSA.2016.14

[13]

Hans Gellersen, Carl Fischer, Dominique Guinard, Roswitha Gostner, Gerd Kortuem, Christian Kray, Enrico Rukzio, and Sara Streng. 2008. Supporting device discovery and spontaneous interaction with spatial references. Pers. Ubiquitous Comput. 13, 4 (July 2008), 255–264. https://doi.org/10.1007/s00779-008-0206-3

[14]

Fabio Giglietto and Donatella Selva. 2014. Second Screen and Participation: A Content Analysis on a Full Season Dataset of Tweets. J. Commun. 64, 2 (April 2014), 260–277. https://doi.org/10.1111/jcom.12085

[15]

Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic interactions: the new ubicomp?interactions 18, 1 (Jan. 2011), 42. https://doi.org/10.1145/1897239.1897250

Digital Library

[16]

Yanying Gu, Anthony Lo, and Ignas Niemegeers. 2009. A survey of indoor positioning systems for wireless personal networks. IEEE Commun. Surv. Tutorials 11, 1 (2009), 13–32. https://doi.org/10.1109/SURV.2009.090103

Digital Library

[17]

Edward Twitchell Hall. 1990. The Hidden Dimension(27 ed.). Anchor Books. 217 pages. https://books.google.com/books?id=HlmqAAAACAAJ&pgis=1

[18]

John Hartman, Udi Manber, and L Peterson. 1996. Liquid software: A new paradigm for networked systems. Technical Report. Tucson, AZ, USA.

[19]

Suining He and S.-H. Gary Chan. 2016. Wi-Fi Fingerprint-Based Indoor Positioning: Recent Advances and Comparisons. IEEE Communications Surveys & Tutorials 18, 1 (2016), 466–490. https://doi.org/10.1109/COMST.2015.2464084

Digital Library

[20]

Tom Horak, Andreas Mathisen, Clemens N. Klokmose, Raimund Dachselt, and Niklas Elmqvist. 2019. Vistribute: Distributing Interactive Visualizations in Dynamic Multi-Device Setups. In Proc. 2019 CHI Conf. Hum. Factors Comput. Syst. - CHI ’19. ACM Press, New York, New York, USA, 1–13. https://doi.org/10.1145/3290605.3300846

Digital Library

[21]

Maria Husmann, Michael Spiegel, Alfonso Murolo, and Moira C. Norrie. 2016. UI Testing Cross-Device Applications. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS ’16). Association for Computing Machinery, New York, NY, USA, 179–188. https://doi.org/10.1145/2992154.2992177

Digital Library

[22]

Kantar TNS Germany. 2019. The Connected Consumer. https://www.google.com/publicdata/explore?ds=dg8d1eetcqsb1_

[23]

Ali Khalajmehrabadi, Nikolaos Gatsis, and David Akopian. 2017. Modern WLAN Fingerprinting Indoor Positioning Methods and Deployment Challenges. IEEE Communications Surveys & Tutorials 19, 3 (2017), 1974–2002. https://doi.org/10.1109/COMST.2017.2671454

Digital Library

[24]

Clemens N. Klokmose, James R. Eagan, Siemen Baader, Wendy Mackay, and Michel Beaudouin-Lafon. 2015. Webstrates: Shareable Dynamic Media. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST ’15. ACM Press, New York, New York, USA, 280–290. https://doi.org/10.1145/2807442.2807446

Digital Library

[25]

James Lin and James A. Landay. 2008. Employing Patterns and Layers for Early-Stage Design and Prototyping of Cross-Device User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1313–1322. https://doi.org/10.1145/1357054.1357260

Digital Library

[26]

Hui Liu, Houshang Darabi, Pat Banerjee, and Jing Liu. 2007. Survey of Wireless Indoor Positioning Techniques and Systems. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) 37, 6 (Nov. 2007), 1067–1080. https://doi.org/10.1109/TSMCC.2007.905750

Digital Library

[27]

Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In Proc. 2012 ACM Int. Conf. Interact. tabletops surfaces - ITS ’12. ACM Press, New York, New York, USA, 31. https://doi.org/10.1145/2396636.2396642

Digital Library

[28]

Nicolai Marquardt, Robert Diaz-Marino, Sebastian Boring, and Saul Greenberg. 2011. The Proximity Toolkit: Prototyping Proxemic Interactions in Ubiquitous Computing Ecologies. In Proc. 24th Annu. ACM Symp. User interface Softw. Technol. - UIST ’11. ACM Press, New York, New York, USA, 315. https://doi.org/10.1145/2047196.2047238

Digital Library

[29]

Nicolai Marquardt, Ken Hinckley, and Saul Greenberg. 2012. Cross-device interaction via micro-mobility and f-formations. In Proc. 25th Annu. ACM Symp. User interface Softw. Technol. - UIST ’12. ACM Press, New York, New York, USA, 13. https://doi.org/10.1145/2380116.2380121

Digital Library

[30]

Asier Marzo. 2016. CustomConsole: A framework for supporting cross-device videogames. In Conf. Hum. Factors Comput. Syst. - Proc., Vol. 07-12-May-. ACM Press, New York, New York, USA, 1952–1958. https://doi.org/10.1145/2851581.2892374

Digital Library

[31]

Tommi Mikkonen, Kari Systä, and Cesare Pautasso. 2015. Towards Liquid Web Applications. In 15th Int. Conf. Web Eng. ICWE 2015, Vol. 9114. Springer, Cham, Rotterdam, the Netherlands, 134–143. https://doi.org/10.1007/978-3-319-19890-3_10

[32]

Partha Mukherjee and Bernard J. Jansen. 2015. Correlation of Brand Mentions in Social Media and Web Searching Before and After Real Life Events: Phase Analysis of Social Media and Search Data for Super Bowl 2015 Commercials. In 2015 IEEE Int. Conf. Data Min. Work.IEEE, 21–26. https://doi.org/10.1109/ICDMW.2015.60

[33]

Miguel A. Nacenta, Dzmitry Aliakseyeu, Sriram Subramanian, and Carl Gutwin. 2005. A comparison of techniques for multi-display reaching. In Proc. SIGCHI Conf. Hum. factors Comput. Syst. - CHI ’05. ACM Press, New York, New York, USA, 371. https://doi.org/10.1145/1054972.1055024

Digital Library

[34]

Timothy Neate, Matt Jones, and Michael Evans. 2017. Cross-device media: a review of second screening and multi-device television. Pers. Ubiquitous Comput. 21, 2 (April 2017), 391–405. https://doi.org/10.1007/s00779-017-1016-2

Digital Library

[35]

Michael Nebeling. 2017. XDBrowser 2.0: Semi-Automatic Generation of Cross-Device Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM Press, New York, New York, USA, 4574–4584. https://doi.org/10.1145/3025453.3025547

Digital Library

[36]

Michael Nebeling, Theano Mintsi, Maria Husmann, and Moira Norrie. 2014. Interactive Development of Cross-Device User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2793–2802. https://doi.org/10.1145/2556288.2556980

Digital Library

[37]

Tobias G. Oberstein and Alexander Goedde. 2021. The Web Application Messaging Protocol. https://wamp-proto.org/_static/gen/wamp_latest.html

[38]

Katie O’Leary, Tao Dong, Julia Katherine Haines, Michael Gilbert, Elizabeth F. Churchill, and Jeffrey Nichols. 2017. The Moving Context Kit: Designing for context shifts in multi-device experiences. In DIS 2017 - Proc. 2017 ACM Conf. Des. Interact. Syst.ACM, New York, NY, USA, 309–320. https://doi.org/10.1145/3064663.3064768

Digital Library

[39]

F. Orujov, R. Maskeliūnas, R. Damaševičius, Wei Wei, and Ye Li. 2018. Smartphone based intelligent indoor positioning using fuzzy logic. Future Generation Computer Systems 89 (dec 2018), 335–348. https://doi.org/10.1016/j.future.2018.06.030

[40]

Tim Paek, Maneesh Agrawala, Sumit Basu, Steve Drucker, Trausti Kristjansson, Ron Logan, Kentaro Toyama, and Andy Wilson. 2004. Toward universal mobile interaction for shared displays. In Proc. 2004 ACM Conf. Comput. Support. Coop. Work - CSCW ’04. ACM Press, New York, New York, USA, 266. https://doi.org/10.1145/1031607.1031649

Digital Library

[41]

Seonwook Park, Christoph Gebhardt, Roman Rädle, Anna Feit, Hana Vrzakova, Niraj Dayama, Hui-Shyong Yeo, Clemens Klokmose, Aaron Quigley, Antti Oulasvirta, and Otmar Hilliges. 2018. AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time. In Proc. 2018 CHI Conf. Hum. Factors Comput. Syst. - CHI ’18. ACM Press, New York, New York, USA, 1–14. https://doi.org/10.1145/3173574.3173758 arxiv:1803.01166

Digital Library

[42]

Fabio Paternò. 2019. Concepts and design space for a better understanding of multi-device user interfaces. Univers. Access Inf. Soc. 19, 2 (June 2019), 409–432. https://doi.org/10.1007/s10209-019-00650-5

Digital Library

[43]

Fabio Paternò and Carmen Santoro. 2012. A logical framework for multi-device user interfaces. In Proc. 4th ACM SIGCHI Symp. Eng. Interact. Comput. Syst. - EICS ’12(EICS ’12). ACM Press, New York, New York, USA, 45. https://doi.org/10.1145/2305484.2305494

Digital Library

[44]

Mohamed Er Rida, Fuqiang Liu, Yassine Jadi, Amgad Ali Abdullah Algawhari, and Ahmed Askourih. 2015. Indoor Location Position Based on Bluetooth Signal Strength. In 2015 2nd International Conference on Information Science and Control Engineering. IEEE, 769–773. https://doi.org/10.1109/ICISCE.2015.177

[45]

Tom Rodden. 2008. Living in a ubiquitous world. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366, 1881 (2008), 3837–3838. https://doi.org/10.1098/rsta.2008.0146

[46]

Nat Sakimura, John Bradley, Michael B. Jones, Breno de Medeiros, and Chuck Mortimore. 2014. OpenID Connect Core 1.0 incorporating errata set 1. Technical Report. http://openid.net/specs/openid-connect-core-1_0.html

[47]

Pedro Albuquerque Santos, Rui Neves Madeira, and Nuno Correia. 2013. FCT4U - When Private Mobile Displays Meet Public Situated Displays to Enhance the User Experience. In 2013 IEEE 10th Int. Conf. Ubiquitous Intell. Comput. 2013 IEEE 10th Int. Conf. Auton. Trust. Comput.IEEE, 186–193. https://doi.org/10.1109/UIC-ATC.2013.112

Digital Library

[48]

Pedro Albuquerque Santos, Rui Neves Madeira, and Nuno Correia. 2018. Designing a Framework to Support the Development of Smart Cross-device Applications. In Proc. 17th Int. Conf. Mob. Ubiquitous Multimed. - MUM 2018. ACM Press, New York, New York, USA, 367–374. https://doi.org/10.1145/3282894.3289727

Digital Library

[49]

Pedro Albuquerque Santos, Rui Neves Madeira, and Nuno Correia. 2019. YanuX - Pervasive distribution of the user interface by co-located devices. In Proc. 16th EAI Int. Conf. Mob. Ubiquitous Syst. Comput. Netw. Serv.ACM, New York, NY, USA, 368–377. https://doi.org/10.1145/3360774.3360832

[50]

Stephanie Santosa and Daniel Wigdor. 2013. A field study of multi-device workflows in distributed workspaces. In Proc. 2013 ACM Int. Jt. Conf. Pervasive ubiquitous Comput. - UbiComp ’13. ACM Press, New York, New York, USA, 63. https://doi.org/10.1145/2493432.2493476

Digital Library

[51]

Henrik Sørensen, Dimitrios Raptis, Jesper Kjeldskov, and Mikael B Skov. 2014. The 4C framework: Principles ofinteraction in digitalecosystems. In UbiComp 2014 - Proc. 2014 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput.(UbiComp ’14). ACM Press, New York, New York, USA, 87–97. https://doi.org/10.1145/2632048.2636089

[52]

Antero Taivalsaari, Tommi Mikkonen, and Kari Systa. 2014. Liquid Software Manifesto: The Era of Multiple Device Ownership and Its Implications for Software Architecture. In 2014 IEEE 38th Annu. Comput. Softw. Appl. Conf.IEEE, 338–343. https://doi.org/10.1109/COMPSAC.2014.56

Digital Library

[53]

David Thevenin and Joëlle Coutaz. 1999. Plasticity of User Interfaces: Framework and Research Agenda. In Human-Computer Interact. INTERACT ’99 IFIP TC. 13, M. Angela Sasse and Chris Johnson (Eds.). IOS Press, Amsterdam, The Netherlands, 110–116. http://iihm.imag.fr/publs/1999/interact99_plasticite.pdf

[54]

Mark Weiser. 1991. The Computer for the 21st Century. Sci. Am. 265(1991), 94–105. https://doi.org/10.2307/24938718

[55]

Martin Werner. 2014. Indoor Location-Based Services. Springer International Publishing, Cham. 1–233 pages. https://doi.org/10.1007/978-3-319-10699-1

Digital Library

[56]

Jishuo Yang and Daniel Wigdor. 2014. Panelrama: enabling easy specification of cross-device web applications. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14. ACM Press, New York, New York, USA, 2783–2792. https://doi.org/10.1145/2556288.2557199

Digital Library

[57]

Moustafa Youssef and Ashok Agrawala. 2005. The Horus WLAN location determination system. In Proc. 3rd Int. Conf. Mob. Syst. Appl. Serv. - MobiSys ’05. ACM Press, New York, New York, USA, 205. https://doi.org/10.1145/1067170.1067193

Digital Library

Cited By

Li J(2024)Big Data-driven Decision Support: Enhancing Information Integration and User Experience with Mobile Integrated TechnologyJournal of Information Systems Engineering and Management10.55267/iadt.07.147479:2(24148)Online publication date: 2024
https://doi.org/10.55267/iadt.07.14747
Li QTian HQiao WHan QMa M(2022)Co-located Collaboration Analysis Based on Group Traces2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD54268.2022.9776203(956-961)Online publication date: 4-May-2022
https://doi.org/10.1109/CSCWD54268.2022.9776203

Recommendations

Designing Proxemic-aware Cross-Device Applications: A Feasibility Study
MUM '21: Proceedings of the 20th International Conference on Mobile and Ubiquitous Multimedia

We live surrounded by the most varied computing devices, which may give us the opportunity to combine them to form a unified and richer user experience. Considering this opportunity, we created the UnaxY Framework to support the development of ...
Computational Framework to Support Development of Applications Running on Multiple Co-located Devices
EICS '21: Companion of the 2021 ACM SIGCHI Symposium on Engineering Interactive Computing Systems

We designed a framework to generalize the development of applications with UI elements distributed across co-located devices. The framework is comprised of diverse components in order to deal with the complexity of such a task, including: ...
YanuX: pervasive distribution of the user interface by co-located devices
MobiQuitous '19: Proceedings of the 16th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

We currently live surrounded by many different computing devices. Therefore, it is important to take better advantage of those devices by coming up with smart ways of integrating and combining them. We have been exploring the possibility of building ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

iiWAS2021: The 23rd International Conference on Information Integration and Web Intelligence

November 2021

658 pages

ISBN:9781450395564

DOI:10.1145/3487664

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Fundação para a Ciência e Tecnologia (NOVA LINCS)
Fundação para a Ciência e Tecnologia

Conference

iiWAS2021

iiWAS2021: The 23rd International Conference on Information Integration and Web Intelligence

November 29 - December 1, 2021

Linz, Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
245
Total Downloads

Downloads (Last 12 months)112
Downloads (Last 6 weeks)19

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li J(2024)Big Data-driven Decision Support: Enhancing Information Integration and User Experience with Mobile Integrated TechnologyJournal of Information Systems Engineering and Management10.55267/iadt.07.147479:2(24148)Online publication date: 2024
https://doi.org/10.55267/iadt.07.14747
Li QTian HQiao WHan QMa M(2022)Co-located Collaboration Analysis Based on Group Traces2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD54268.2022.9776203(956-961)Online publication date: 4-May-2022
https://doi.org/10.1109/CSCWD54268.2022.9776203

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents