research-article

Open access

Navigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR

Authors:

Richard Cloete,

Jatinder SinghAuthors Info & Claims

FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

Pages 1418 - 1431

https://doi.org/10.1145/3593013.3594090

Published: 12 June 2023 Publication History

All formats PDF

Abstract

“Extended reality” (XR) systems work to blend the physical and digital worlds. This means that XR is highly contextual: its functionality, operation and therefore consequences are driven by a tight, run-time coupling of the technology, the user, and their physical environment. It follows that XR brings particular challenges regarding transparency and accountability, given that it can be difficult to foresee and mitigate all potential issues that might arise from using such systems, given their many potential contexts of use. Further, the physicality of XR can directly result in injury, property damage, or worse, in addition to the more traditionally discussed harms arising from algorithmic systems. Therefore the ability to audit the operation of XR systems is paramount – where information revealing and enabling some reconstruction of an XR system’s use, run-time behaviour, and surrounding context is important for understanding and scrutinising what happens/happened, and why.

Towards this, we present a framework to support those involved in developing XR systems to make them more auditable. The framework focuses on supporting the building and instrumentation of an XR system for transparency aims, elaborating key considerations regarding the capture and management of audit data during system operation. We demonstrate the framework’s efficacy with expert XR developers, who indicate the utility and need for such in practice. In all, we provide practical ways forward on, as well as seek to draw attention to, XR transparency and accountability.

References

[1]

Surin Ahn, Maria Gorlatova, Parinaz Naghizadeh, Mung Chiang, and Prateek Mittal. 2018. Adaptive Fog-Based Output Security for Augmented Reality. In Proceedings of the 2018 Morning Workshop on Virtual Reality and Augmented Reality Network (Budapest, Hungary) (VR/AR Network ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3229625.3229626

Digital Library

[2]

Karan Ahuja, Eyal Ofek, Mar Gonzalez-Franco, Christian Holz, and Andrew D. Wilson. 2021. CoolMoves: User Motion Accentuation in Virtual Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2, Article 52 (June 2021). https://doi.org/10.1145/3463499

Digital Library

[3]

Hasaneen Fathy Al Janabi, Abdullatif Aydin, Sharanya Palaneer, Nicola Macchione, Ahmed Al-Jabir, Muhammad Shamim Khan, Prokar Dasgupta, and Kamran Ahmed. 2020. Effectiveness of the HoloLens Mixed-Reality Headset in Minimally Invasive Surgery: A Simulation-Based Feasibility Study. Surgical Endoscopy 34, 3 (2020), 1143–1149.

[4]

Mike Ananny and Kate Crawford. 2018. Seeing Without Knowing: Limitations of the Transparency Ideal and its Application to Algorithmic Accountability. New Media and Society 20, 3 (2018), 973–989. https://doi.org/10.1177/1461444816676645

[5]

McKane Andrus and Sarah Villeneuve. 2022. Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 1709–1721. https://doi.org/10.1145/3531146.3533226

Digital Library

[6]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and Depth-First Exploration for Systematic Testing of Android Apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. 641–660.

Digital Library

[7]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of usability studies 4, 3 (2009), 114–123.

Digital Library

[8]

BBC Worldwide. 2017. Bear Island. https://www.oculus.com/experiences/rift/1228652173894654. Accessed: 2023-05-09.

[9]

Mafkereseb Kassahun Bekele, Roberto Pierdicca, Emanuele Frontoni, Eva Savina Malinverni, and James Gain. 2018. A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. Journal on Computing and Cultural Heritage (JOCCH) 11, 2 (2018), 1–36.

Digital Library

[10]

Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2020. Explainable Machine Learning in Deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 648–657. https://doi.org/10.1145/3351095.3375624

Digital Library

[11]

Lindsay Blackwell, Nicole Ellison, Natasha Elliott-Deflo, and Raz Schwartz. 2019. Harassment in Social Virtual Reality: Challenges for Platform Governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–25.

Digital Library

[12]

Boeing. 2018. Boeing Tests Augmented Reality in the Factory. https://webcache.googleusercontent.com/search?q=cache:GJxEUfXolpgJ:https://www.boeing.com/features/2018/01/augmented-reality-01-18.page. Accessed: 2023-05-09.

[13]

Mark Bovens. 2006. Analysing and Assessing Public Accountability. A Conceptual Framework. European Governance PapersC-06-012006 (2006).

[14]

Mark Bovens. 2010. Two Concepts of Accountability: Accountability as a Virtue and as a Mechanism. West European Politics 33, 5 (2010), 946–967. https://doi.org/10.1080/01402382.2010.486119

[15]

Doug A Bowman and Ryan P McMahan. 2007. Virtual Reality: How Much Immersion is Enough?Computer 40, 7 (2007), 36–43.

Digital Library

[16]

Karen Boyd. 2022. Designing Up with Value-Sensitive Design: Building a Field Guide for Ethical ML Development. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 2069–2082. https://doi.org/10.1145/3531146.3534626

Digital Library

[17]

Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101.

[18]

John Brooke. 1996. SUS-A Quick and Dirty Usability Scale. Usability evaluation in industry 189, 194 (1996), 4–7.

[19]

Chu Cao, Zhenjiang Li, Pengfei Zhou, and Mo Li. 2018. Amateur: Augmented Reality Based Vehicle Navigation System. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 155 (Dec. 2018). https://doi.org/10.1145/3287033

Digital Library

[20]

Po-Hsuan Cameron Chen, Krishna Gadepalli, Robert MacDonald, Yun Liu, Shiro Kadowaki, Kunal Nagpal, Timo Kohlberger, Jeffrey Dean, Greg S Corrado, Jason D Hipp, 2019. An Augmented Reality Microscope with Real-Time Artificial Intelligence Integration for Cancer Diagnosis. Nature medicine 25, 9 (2019), 1453–1457.

[21]

Yan Chen, Steve Oney, and Walter S. Lasecki. 2016. Towards Providing On-Demand Expert Support for Software Developers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3192–3203. https://doi.org/10.1145/2858036.2858512

Digital Library

[22]

Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI Testing of Android Apps with Minimal Restart and Approximate Learning. Acm Sigplan Notices 48, 10 (2013), 623–640.

Digital Library

[23]

Bonnie Christian. 2017. HoloLens Trial Gives Doctors ‘X-Ray Vision’ to Allow Them to Peer inside Patients during Surgery. Wired UK (March 2017).

[24]

Richard Cloete, Chris Norval, and Jatinder Singh. 2020. A Call for Auditable Virtual, Augmented and Mixed Reality. In 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, Canada) (VRST ’20). Association for Computing Machinery, New York, NY, USA, Article 16. https://doi.org/10.1145/3385956.3418960

Digital Library

[25]

Richard Cloete, Chris Norval, and Jatinder Singh. 2022. Auditable Augmented/Mixed/Virtual Reality: The Practicalities of Mobile System Transparency. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 149 (Dec 2022). https://doi.org/10.1145/3495001

Digital Library

[26]

Jennifer Cobbe, Michelle Seng Ah Lee, and Jatinder Singh. 2021. Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 598–609. https://doi.org/10.1145/3442188.3445921

Digital Library

[27]

Amanda Coston, Alan Mishler, Edward H. Kennedy, and Alexandra Chouldechova. 2020. Counterfactual Risk Assessments, Evaluation, and Fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 582–593. https://doi.org/10.1145/3351095.3372851

Digital Library

[28]

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 120–128. https://doi.org/10.1145/3287560.3287572

Digital Library

[29]

Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, and Haiyi Zhu. 2022. Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 473–484. https://doi.org/10.1145/3531146.3533113

Digital Library

[30]

Nicholas Diakopoulos. 2016. Accountability in Algorithmic Decision Making. Commun. ACM 59, 2 (Jan. 2016), 56–62. https://doi.org/10.1145/2844110

Digital Library

[31]

European Commission. 2021. Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act). https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence-artificial-intelligence Accessed: 2023-05-09.

[32]

European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union L119 (4 May 2016), 1–88.

[33]

European Union. 2017. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on Medical Devices, Amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and Repealing Council Directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union L117 (5 Apr 2017), 1–228.

[34]

Felix & Paul Studios. 2017. The People’s House. https://www.oculus.com/experiences/rift/1258599840927577. Accessed: 2023-05-09.

[35]

Simson Garfinkel, Jeanna Matthews, Stuart S. Shapiro, and Jonathan M. Smith. 2017. Toward Algorithmic Transparency and Accountability. Commun. ACM 60, 9 (Aug 2017), 5. https://doi.org/10.1145/3125780

Digital Library

[36]

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datasets. Commun. ACM 64, 12, 86–92. https://doi.org/10.1145/3458723

Digital Library

[37]

Trystan S. Goetze. 2022. Mind the Gap: Autonomous Systems, the Responsibility Gap, and Moral Entanglement. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 390–400. https://doi.org/10.1145/3531146.3533106

Digital Library

[38]

Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and Touch-Sensitive Record and Replay for Android. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 72–81.

[39]

Google Maps. 2019. Introducing Live View, the New Augmented Reality Feature in Google Maps. https://support.google.com/maps/thread/11554255?hl=en. Accessed: 2023-05-09.

[40]

Alex Hadwick. 2019. VRX Industry Insight Report 2019-2020. https://s3.amazonaws.com/media.mediapost.com/uploads/VRXindustryreport.pdf. Accessed: 2023-05-09.

[41]

Matthew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: Cross-Platform User-Interaction Record and Replay for the Fragmented Android Ecosystem. In 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 215–224.

[42]

Jude D Hemanth, Utku Kose, Omer Deperlioglu, and Victor Hugo C de Albuquerque. 2020. An Augmented Reality-Supported Mobile Application for Diagnosis of Heart Diseases. The Journal of Supercomputing 76, 2 (2020), 1242–1267.

Digital Library

[43]

Melanie Herschel, Ralf Diestelkämper, and Houssem Ben Lahmar. 2017. A Survey on Provenance: What For? What Form? What From?The VLDB Journal 26 (2017), 881–906.

[44]

HM Government. 2019. Online Harms White Paper. Technical Report. 1–102 pages. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/793360/Online_Harms_White_Paper.pdf

[45]

HTC. 2021. HTC Vive Health and Safety. https://www.htc.com/eu/sunrise-safety-guide/. Accessed: 2023-05-09.

[46]

Gang Hu, Xinhao Yuan, Yang Tang, and Junfeng Yang. 2014. Efficiently, Effectively Detecting Mobile App Bugs with Appdoctor. In Proceedings of the Ninth European Conference on Computer Systems. 1–15.

Digital Library

[47]

Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile Yet Lightweight Record-and-Replay for Android. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. 349–366.

Digital Library

[48]

Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 560–575.

Digital Library

[49]

XR Safety Initiative. 2020. The XRSI Privacy Framework, version 1.0. https://xrsi.org/wp-content/uploads/2020/09/XRSI-Privacy-Framework-v1_003.pdf. Accessed: 2023-05-09.

[50]

XR Safety Initiative. 2021. An Imperative: Developing Standards for Safety and Security in XR Environments. https://xrsi.org/wp-content/uploads/2021/02/An-Imperative-Emteq-XRSI-Whitepaper-on-Standards-for-safety-and-Security.pdf. Accessed: 2023-05-09.

[51]

Shubham Jain, Carlo Borgiattino, Yanzhi Ren, Marco Gruteser, and Yingying Chen. 2014. On the Limits of Positioning-Based Pedestrian Risk Awareness. In Proceedings of the 2014 workshop on Mobile augmented reality and robotic technology-based systems. 23–28.

Digital Library

[52]

Suman Jana, David Molnar, Alexander Moshchuk, Alan Dunn, Benjamin Livshits, Helen J. Wang, and Eyal Ofek. 2013. Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers. In 22nd USENIX Security Symposium (USENIX Security 13). USENIX Association, Washington, D.C., 415–430. https://www.usenix.org/conference/usenixsecurity13/technical-sessions/presentation/jana

Digital Library

[53]

Seyyed Ahmad Javadi, Richard Cloete, Jennifer Cobbe, Michelle Seng Ah Lee, and Jatinder Singh. 2020. Monitoring Misuse for Accountable ‘Artificial Intelligence as a Service’. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 300–306.

Digital Library

[54]

Seyyed Ahmad Javadi, Chris Norval, Richard Cloete, and Jatinder Singh. 2021. Monitoring AI Services for Misuse. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 597–607.

Digital Library

[55]

Sarah Jones, Alexander Ball, and Çuna Ekmekcioglu. 2008. The Data Audit Framework: A First Step in the Data Management Challenge. International Journal of Digital Curation 3, 2 (2008), 112–120.

[56]

Sarah Jones, Seamus Ross, and Raivo Ruusalepp. 2018. The Data Audit Framework: A Toolkit to Identify Research Assets and Improve Data Management in Research Led Institutions. In iPRES 2008.

[57]

S Jones, S Ross, R Ruusalepp, and M Dobreva. 2009. Data Audit Framework Methodology. https://www.data-audit.eu/DAF_Methodology.pdf., 70 pages. Accessed: 2023-05-09.

[58]

J. Jung, H. Lee, J. Choi, A. Nanda, U. Gruenefeld, T. Stratmann, and W. Heuten. 2018. Ensuring Safety in Augmented Reality from Trade-off Between Immersion and Situation Awareness. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 70–79. https://doi.org/10.1109/ISMAR.2018.00032

[59]

Ryan K. L. Ko, Bu Sung Lee, and Siani Pearson. 2011. Towards Achieving Accountability, Auditability and Trust in Cloud Computing. In Advances in Computing and Communications, Ajith Abraham, Jaime Lloret Mauri, John F. Buford, Junichi Suzuki, and Sabu M. Thampi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 432–444.

[60]

P. M. Krafft, Meg Young, Michael Katell, Jennifer E. Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, Corinne Bintz, Daniella Raz, Pa Ousman Jobe, Franziska Putz, Brian Robick, and Bissan Barghouti. 2021. An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 772–781. https://doi.org/10.1145/3442188.3445938

Digital Library

[61]

Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165, 3 (Feb. 2017), 633–705.

[62]

Joshua A Kroll. 2021. Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 758–771.

Digital Library

[63]

Kiron Lebeck, Tadayoshi Kohno, and Franziska Roesner. 2016. How to Safely Augment Reality: Challenges and Directions. In Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications. 45–50.

Digital Library

[64]

Kiron Lebeck, Kimberly Ruth, Tadayoshi Kohno, and Franziska Roesner. 2017. Securing Augmented Reality Output. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE, San Jose, CA, USA, 320–337. https://doi.org/10.1109/SP.2017.13

[65]

Kiron Lebeck, Kimberly Ruth, Tadayoshi Kohno, and Franziska Roesner. 2018. Towards Security and Privacy for Multi-User Augmented Reality: Foundations with End Users. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 392–408.

[66]

Michelle Seng Ah Lee and Jat Singh. 2021. The Landscape and Gaps in Open Source Fairness Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 699, 13 pages. https://doi.org/10.1145/3411764.3445261

Digital Library

[67]

Michael Xieyang Liu, Aniket Kittur, and Brad A. Myers. 2022. Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 68. https://doi.org/10.1145/3491102.3501968

Digital Library

[68]

Eduardo E. Lovo, Juan C. Quintana, Manuel C. Puebla, Gonzalo Torrealba, José L. Santos, Isidro H. Lira, and Patricio Tagle. 2007. A Novel, Inexpensive Method of Image Coregistration for Applications in Image-guided Surgery Using Augmented Reality. Operative Neurosurgery 60, 4 (Apr 2007), 366–372. https://doi.org/10.1227/01.NEU.0000255360.32689.FA

[69]

A. Maiti, M. Jadliwala, and C. Weber. 2017. Preventing Shoulder Surfing Using Randomized Augmented Reality Keyboards. In 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). 630–635. https://doi.org/10.1109/PERCOMW.2017.7917636

[70]

Frank Marcinkowski, Kimon Kieslich, Christopher Starke, and Marco Lünich. 2020. Implications of AI (Un-)Fairness in Higher Education Admissions: The Effects of Perceived AI (Un-)Fairness on Exit, Voice and Organizational Reputation. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 122–130. https://doi.org/10.1145/3351095.3372867

Digital Library

[71]

Philippe Massonet, Syed Naqvi, Christophe Ponsard, Joseph Latanicki, Benny Rochwerger, and Massimo Villari. 2011. A Monitoring and Audit Logging Architecture for Data Location Compliance in Federated Cloud Infrastructures. In 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum. IEEE, Anchorage, AK, USA, 1510–1517. https://doi.org/10.1109/IPDPS.2011.304

Digital Library

[72]

Anay Mehrotra, Bary S. R. Pradelski, and Nisheeth K. Vishnoi. 2022. Selection in the Presence of Implicit Bias: The Advantage of Intersectional Constraints. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 599–609. https://doi.org/10.1145/3531146.3533124

Digital Library

[73]

Meta. 2019. Explore VR Gaming Experiences. https://www.oculus.com/experiences. Accessed: 2023-05-09.

[74]

Microsoft. 2021. Safety Information. https://support.microsoft.com/en-gb/help/4023454/safety-information. Accessed: 2023-05-09.

[75]

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). ACM, New York, USA, 220–229. https://doi.org/10.1145/3287560.3287596

Digital Library

[76]

NASA’s Jet Propulsion Laboratory/Caltech. 2018. NASA’s Exoplanet Excursions. https://www.oculus.com/experiences/rift/2136785793017883. Accessed: 2023-05-09.

[77]

National Geographic. 2019. National Geographic Explore VR. https://www.oculus.com/experiences/quest/2046607608728563. Accessed: 2023-05-09.

[78]

Terrence Neumann, Maria De-Arteaga, and Sina Fazelpour. 2022. Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 1504–1515. https://doi.org/10.1145/3531146.3533205

Digital Library

[79]

Chris Norval, Jennifer Cobbe, and Jatinder Singh. 2021. Towards an Accountable Internet of Things: A Call for ‘Reviewability’. Privacy by Design for the Internet of Things. The Institution of Engineering and Technology. (2021).

[80]

Chris Norval, Kristin Cornelius, Jennifer Cobbe, and Jatinder Singh. 2022. Disclosure by Design: Designing Information Disclosures to Support Meaningful Transparency and Accountability. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 679–690. https://doi.org/10.1145/3531146.3533133

Digital Library

[81]

Oculus. 2021. Oculus Health and Safety Warnings. https://securecdn.oculus.com/sr/oculusquest-warning-english. Accessed: 2023-05-09.

[82]

Royal Courts of Justice. 2020. R (Bridges) v Chief Constable of South Wales Police & Information Commissioner [2020] EWCA Civ 1058.

[83]

Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020. Bias in Word Embeddings. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 446–457. https://doi.org/10.1145/3351095.3372843

Digital Library

[84]

Simon Parkin, George Danezis, Isabel Lopez-Neira, and Trupti Patel. 2018. Implications of the Internet of Things (IoT) on Victims of Gender-Based Domestic Violence and Abuse. Technical Report. https://www.ucl.ac.uk/research/domains/sites/research_domains/files/br_ssp_2017-18_tanczer_internet_of_things_and_gender_based_violence.pdf

[85]

Lindsay Poirier. 2022. Accountable Data: The Politics and Pragmatics of Disclosure Datasets. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 1446–1456. https://doi.org/10.1145/3531146.3533201

Digital Library

[86]

Pokémon Go. 2019. Pokémon Go. https://www.pokemongo.com/en-us/. Accessed: 2023-05-09.

[87]

Dawid Połap, Karolina Kęsik, Kamil Książek, and Marcin Woźniak. 2017. Obstacle Detection as a Safety Alert in Augmented Reality Models by the Use of Deep Learning Techniques. Sensors 17, 12 (2017), 1–16.

[88]

Philip Pratt, Matthew Ives, Graham Lawton, Jonathan Simmons, Nasko Radev, Liana Spyropoulou, and Dimitri Amiras. 2018. Through the HoloLensTM Looking Glass: Augmented Reality for Extremity Reconstruction Surgery using 3D Vascular Models with Perforating Vessels. European radiology experimental 2 (2018), 1–7.

[89]

Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 1776–1826. https://doi.org/10.1145/3531146.3533231

Digital Library

[90]

Zhengrui Qin, Yutao Tang, Ed Novak, and Qun Li. 2016. Mobiplay: A Remote Execution Based Record-and-Replay Tool for Mobile Applications. In Proceedings of the 38th International Conference on Software Engineering. 571–582.

Digital Library

[91]

Fazlay Rabbi, Taiwoo Park, Biyi Fang, Mi Zhang, and Youngki Lee. 2018. When Virtual Reality Meets Internet of Things in the Gym: Enabling Immersive Interactive Machine Exercises. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 2, Article 78 (July 2018). https://doi.org/10.1145/3214281

Digital Library

[92]

Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 33–44. https://doi.org/10.1145/3351095.3372873

Digital Library

[93]

Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where Responsible AI Meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–23.

Digital Library

[94]

Toni Robertson and Jesper Simonsen. 2012. Challenges and Opportunities in Contemporary Participatory Design. Design Issues 28, 3 (2012), 3–9.

[95]

Javier Sánchez-Monedero, Lina Dencik, and Lilian Edwards. 2020. What Does It Mean to ‘Solve’ the Problem of Discrimination in Hiring? Social, Technical and Legal Perspectives from the UK on Automated Hiring Systems. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 458–468. https://doi.org/10.1145/3351095.3372849

Digital Library

[96]

Qijia Shao, Amy Sniffen, Julien Blanchet, Megan E. Hillis, Xinyu Shi, Themistoklis K. Haris, Jason Liu, Jason Lamberton, Melissa Malzkuhn, Lorna C. Quandt, James Mahoney, David J. M. Kraemer, Xia Zhou, and Devin Balkcom. 2020. Teaching American Sign Language in Mixed Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 152 (Dec. 2020). https://doi.org/10.1145/3432211

Digital Library

[97]

J. Singh, J. Cobbe, and C. Norval. 2019. Decision Provenance: Harnessing Data Flow for Accountable Systems. IEEE Access 7 (2019), 6562–6574. https://doi.org/10.1109/ACCESS.2018.2887201

[98]

Jatinder Singh, Thomas Pasquier, Jean Bacon, Julia Powles, Raluca Diaconu, and David Eyers. 2016. Big Ideas Paper: Policy-Driven Middleware for a Legally-Compliant Internet of Things. In Proceedings of the 17th International Middleware Conference (Trento, Italy) (Middleware ’16). ACM, New York, NY, USA, Article 13, 13:1–13:15 pages. https://doi.org/10.1145/2988336.2988349

Digital Library

[99]

Jatinder Singh, Julia Powles, Thomas Pasquier, and Jean Bacon. 2015. Data Flow Management and Compliance in Cloud Computing. IEEE Cloud Computing 2, 4 (July 2015), 24–32. https://doi.org/10.1109/MCC.2015.69

[100]

Sean Sirur, Jason R.C. Nurse, and Helena Webb. 2018. Are We There Yet? Understanding the Challenges Faced in Complying with the General Data Protection Regulation (GDPR). In Proceedings of the 2nd International Workshop on Multimedia Privacy and Security (Toronto, Canada) (MPS ’18). Association for Computing Machinery, New York, NY, USA, 88–95. https://doi.org/10.1145/3267357.3267368

Digital Library

[101]

Ivo Sluganovic, Matej Serbec, Ante Derek, and Ivan Martinovic. 2017. HoloPair: Securing Shared Augmented Reality Using Microsoft HoloLens. In Proceedings of the 33rd Annual Computer Security Applications Conference (Orlando, FL, USA) (ACSAC 2017). Association for Computing Machinery, New York, NY, USA, 250–261. https://doi.org/10.1145/3134600.3134625

Digital Library

[102]

Kacper Sokol and Peter Flach. 2020. Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 56–67. https://doi.org/10.1145/3351095.3372870

Digital Library

[103]

Sólfar Studios. 2017. Everest VR. https://www.oculus.com/experiences/rift/1043021355789504. Accessed: 2023-05-09.

[104]

Steam. 2019. Virtual Reality Titles. https://store.steampowered.com/vr/. Accessed: 2023-05-09.

[105]

Cynthia Stohl, Michael Stohl, and Paul M Leonardi. 2016. Digital Age | Managing Opacity: Information Visibility and the Paradox of Transparency in the Digital Age. International Journal of Communication 10 (2016), 123–137.

[106]

Leonie Tanczer, Isabel Lopez-Neira, Simon Parkin, Trupti Patel, and George Danezis. 2018. The Rise of the Internet of Things and Implications for Technology-Facilitated Abuse. (2018). https://www.ucl.ac.uk/steapp/sites/steapp/files/giot-report.pdf

[107]

Michael Carl Tschantz. 2022. What is Proxy Discrimination?. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 1993–2003. https://doi.org/10.1145/3531146.3533242

Digital Library

[108]

Lachlan D. Urquhart and Peter J. Craigon. 2021. The Moral-IT Deck: A Tool for Ethics by Design. Journal of Responsible Innovation 8, 1 (2021), 94–126. https://doi.org/10.1080/23299460.2021.1880112

[109]

Ben Wagner, Krisztina Rozgonyi, Marie-Therese Sekwenz, Jennifer Cobbe, and Jatinder Singh. 2020. Regulating Transparency? Facebook, Twitter and the German Network Enforcement Act. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 261–271. https://doi.org/10.1145/3351095.3372856

Digital Library

[110]

Qi Wang, Wajih Ul Hassan, Adam Bates, and Carl Gunter. 2018. Fear and Logging in the Internet of Things. In Network and Distributed Systems Symposium.

[111]

Chris Warin and Delphine Reinhardt. 2022. Vision: Usable Privacy for XR in the Era of the Metaverse. In Proceedings of the 2022 European Symposium on Usable Security (Karlsruhe, Germany) (EuroUSEC ’22). Association for Computing Machinery, New York, NY, USA, 111–116. https://doi.org/10.1145/3549015.3554212

Digital Library

[112]

Peter Washington, Catalin Voss, Aaron Kline, Nick Haber, Jena Daniels, Azar Fazel, Titas De, Carl Feinstein, Terry Winograd, and Dennis Wall. 2017. SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 112 (Sept. 2017). https://doi.org/10.1145/3130977

Digital Library

[113]

Tan Kiat Wee, Eduardo Cuervo, and Rajesh Balan. 2018. FocusVR: Effective 8 Usable VR Display Power Management. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–25.

Digital Library

[114]

Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks Posed by Language Models. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 214–229. https://doi.org/10.1145/3531146.3533088

Digital Library

[115]

Maranke Wieringa. 2020. What to Account for When Accounting for Algorithms: A Systematic Literature Review on Algorithmic Accountability. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3351095.3372833

Digital Library

[116]

Rebecca Williams, Richard Cloete, Jennifer Cobbe, Caitlin Cottrill, Peter Edwards, Milan Markovic, Iman Naja, Frances Ryan, Jatinder Singh, Wei Pang, and et al.2022. From Transparency to Accountability of Intelligent Systems: Moving Beyond Aspirations. Data & Policy 4 (2022). https://doi.org/10.1017/dap.2021.37

[117]

Kanghyeok Yang, Sepi Aria, Changbum R. Ahn, and Terry L. Stentz. 2014. Automated Detection of Near-Miss Fall Incidents in Iron Workers Using Inertial Measurement Units. In Construction Research Congress 2014. American Society of Civil Engineers, Atlanta, Georgia, 935–944. https://doi.org/10.1061/9780784413517.096

Cited By

Heemsbergen L(2024)Data-Mediated Environments: Reality after the MetaverseNavigating the Metaverse - A Comprehensive Guide to the Future of Digital Interaction [Working Title]10.5772/intechopen.1006622Online publication date: 10-Sep-2024
https://doi.org/10.5772/intechopen.1006622
Terzis PVeale MGaumann N(2024)Law and the Emerging Political Economy of Algorithmic AuditsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658970(1255-1267)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658970
Lourimi NElkefi STounsi ALayeb S(2024)Exploring Healthcare Providers' Perspectives on Extended Reality Technologies for Sustainable Healthcare: A Twitter Analysis2024 ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS)10.1109/ICETSIS61505.2024.10459388(1053-1059)Online publication date: 28-Jan-2024
https://doi.org/10.1109/ICETSIS61505.2024.10459388

Index Terms

Navigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods
    2. Ubiquitous and mobile computing systems and tools
2. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. System management
        Technology audits

Recommendations

What is XR? Towards a Framework for Augmented and Virtual Reality
Abstract
Augmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and ...
Highlights
- XR should not be used as an abbreviation for “Extended Reality”; X should be the placeholder for “all” new reality formats.
A Call for Auditable Virtual, Augmented and Mixed Reality
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

XR (Virtual, Augmented and Mixed Reality) technologies are growing in prominence. However, they are increasingly being used in sectors and in situations that can result in harms. As such, this paper argues the need for auditability to become a key ...
Garden: A Mixed Reality Experience Combining Virtual Reality and 3D Reconstruction
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Garden is a Mixed Reality (MR) experience that combines both Virtual Reality (VR) and Augmented Reality (AR), and lets players transform their environment into a virtual garden they can play in. This is done by doing both stereoscopic rendering and 3D ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

June 2023

1929 pages

ISBN:9798400701924

DOI:10.1145/3593013

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Microsoft
UK Research and Innovation
The Alan Turing Institute

Conference

FAccT '23

FAccT '23: the 2023 ACM Conference on Fairness, Accountability, and Transparency

June 12 - 15, 2023

IL, Chicago, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
566
Total Downloads

Downloads (Last 12 months)425
Downloads (Last 6 weeks)71

Reflects downloads up to 02 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Heemsbergen L(2024)Data-Mediated Environments: Reality after the MetaverseNavigating the Metaverse - A Comprehensive Guide to the Future of Digital Interaction [Working Title]10.5772/intechopen.1006622Online publication date: 10-Sep-2024
https://doi.org/10.5772/intechopen.1006622
Terzis PVeale MGaumann N(2024)Law and the Emerging Political Economy of Algorithmic AuditsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658970(1255-1267)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658970
Lourimi NElkefi STounsi ALayeb S(2024)Exploring Healthcare Providers' Perspectives on Extended Reality Technologies for Sustainable Healthcare: A Twitter Analysis2024 ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS)10.1109/ICETSIS61505.2024.10459388(1053-1059)Online publication date: 28-Jan-2024
https://doi.org/10.1109/ICETSIS61505.2024.10459388

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents