research-article

Avoiding the intrinsic unfairness of the trolley problem

Authors:

Tobias Holstein,

Gordana Dodig-CrnkovicAuthors Info & Claims

FairWare '18: Proceedings of the International Workshop on Software Fairness

Pages 32 - 37

https://doi.org/10.1145/3194770.3194772

Published: 29 May 2018 Publication History

Abstract

As an envisaged future of transportation, self-driving cars are being discussed from various perspectives, including social, economical, engineering, computer science, design, and ethical aspects. On the one hand, self-driving cars present new engineering problems that are being gradually successfully solved. On the other hand, social and ethical problems have up to now being presented in the form of an idealized unsolvable decision-making problem, the so-called "trolley problem", which is built on the assumptions that are neither technically nor ethically justifiable. The intrinsic unfairness of the trolley problem comes from the assumption that lives of different people have different values.

In this paper, techno-social arguments are used to show the infeasibility of the trolley problem when addressing the ethics of self-driving cars. We argue that different components can contribute to an "unfair" behaviour and features, which requires ethical analysis on multiple levels and stages of the development process. Instead of an idealized and intrinsically unfair thought experiment, we present real-life techno-social challenges relevant for the domain of software fairness in the context of self-driving cars.

References

[1]

Moral Machine. http://moralmachine.mit.edu, 2016.

[2]

Ethics commission on automated driving presents report: First guidelines in the world for self-driving computers. Technical report, Federal Ministry of Transport and Digital Infrastructure, 2017.

[3]

J. Achenbach. Driverless cars are colliding with the creepy Trolley Problem. https://www.washingtonpost.com/news/innovations/wp/2015/12/29/will-self-driving-cars-ever-solve-the-famous-and-creepy-trolley-problem/, December 2015.

[4]

E. Ackerman. People Want Driverless Cars with Utilitarian Ethics, Unless They're a Passenger. https://spectrum.ieee.org/cars-that-think/transportation/self-driving/people-want-driverless-cars-with-utilitarian-ethics-unless-theyre-a-passenger, June 2016.

[5]

H. S. Alavi, F. Bahrami, H. Verma, and D. Lalanne. Is driverless car another weiserian mistake? In Proceedings of the 2017 ACM Conference Companion Publication on Designing Interactive Systems, DIS '17 Companion, pages 249--253, New York, NY, USA, 2017. ACM.

Digital Library

[6]

S. Applin. Autonomous vehicle ethics: Stock or custom? IEEE Consumer Electronics Magazine, 6(3):108--110, July 2017.

[7]

A. Bleske-Rechek, L. Nelson, J. P. Baker, M. Remiker, and S. J. Brandt. Evolution and the trolley problem: People save five over one unless the one is young, genetically related, or a romantic partner. 4:115--127, 01 2010.

[8]

J.-F. Bonnefon, A. Shariff, and I. Rahwan. The social dilemma of autonomous vehicles. Science, 352(6293):1573--1576, 2016.

[9]

M. Broy, I. H. Kruger, A. Pretschner, and C. Salzmann. Engineering Automotive Software. Proceedings of the IEEE, 95(2):356--373, feb 2007.

[10]

I. Coca-Vila. Self-driving cars in dilemmatic situations: An approach based on the theory of justification in criminal law. Criminal Law and Philosophy, Jan 2017.

[11]

Coodriver GmbH. Schutzranzen App - anonymous, safe, commercial-free - makes it easier to see children near roads, September 2017. https://www.schutzranzen.com/files/2115/0478/2378/PM_Schutzranzen_IAA_2017_english.pdf.

[12]

K. Deamer. What the First Driverless Car Fatality Means for Self-Driving Tech. https://www.scientificamerican.com/article/what-the-first-driverless-car-fatality-means-for-self-driving-tech/, 2016.

[13]

L. Dennis, M. Fisher, M. Slavkovik, and M. Webster. Ethical Choice in Unforeseen Circumstances, pages 433--445. Springer Berlin Heidelberg, Berlin, Heidelberg, 2014.

[14]

L. Dennis, M. Fisher, M. Slavkovik, and M. Webster. Formal verification of ethical choices in autonomous systems. Robotics and Autonomous Systems, 77(Supplement C):1 -- 14, 2016.

Digital Library

[15]

D. Dolgov. Google self-driving car project - monthly report - september 2016 - on the road. Technical report, Google, 2016.

[16]

J. Dunbar and J. E. Gilbert. The human element in autonomous vehicles. In D. Harris, editor, Engineering Psychology and Cognitive Ergonomics: Cognition and Design, pages 339--362, Cham, 2017. Springer International Publishing.

[17]

S. I. Earth Imaging Journal (EIJ): Remote Sensing, Satellite Images. Lidar boosts brain power for self-driving cars, 2012.

[18]

Ethics Commission. Automated and connected driving. Technical report, Federal Ministry of Transport and Digital Infrastructure, 2017.

[19]

A. K. Faulhaber, A. Dittmer, F. Blind, M. A. Wächter, S. Timm, L. R. Sütfeld, A. Stephan, G. Pipa, and P. König. Human decisions in moral dilemmas are largely described by utilitarianism: Virtual car driving study provides guidelines for autonomous driving vehicles. Science and Engineering Ethics, Jan 2018.

[20]

P. Foot. The problem of abortion and the doctrine of double effect. Oxford Review, 5, 1967.

[21]

A.-K. Frison, P. Wintersberger, and A. Riener. First person trolley problem: Evaluation of drivers' ethical decisions in a driving simulator. In Adjunct Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI '16 Adjunct, pages 117--122, New York, NY, USA, 2016. ACM.

Digital Library

[22]

S. Galhotra, Y. Brun, and A. Meliou. Fairness testing: Testing software for discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 498--510, New York, NY, USA, 2017. ACM.

Digital Library

[23]

General Motors. GM Self-Driving Safety Report, January 2018. https://www.gm.com/content/dam/gm/en_us/english/selfdriving/gmsafetyreport.pdf.

[24]

G. Ghisio. Challenges for the Automotive Platform of the Future, 2016.

[25]

N. J. Goodall. Vehicle automation and the duty to act. In Proceedings of the 21st world congress on intelligent transport systems, pages 7--11, 2014.

[26]

N. J. Goodall. Can you program ethics into a self-driving car? IEEE Spectrum, 53(6):28--58, June 2016.

Digital Library

[27]

B. Goodman and S. Flaxman. European Union regulations on algorithmic decision-making and a "right to explanation". ArXiv e-prints, June 2016.

[28]

Google. Google self-driving car project, 2016.

[29]

J. D. Greene. Our driverless dilemma. Science, 352(6293):1514--1515, 2016.

[30]

L. Greenemeier. Driverless Cars Will Face Moral Dilemmas. https://www.scientificamerican.com/article/driverless-cars-will-face-moral-dilemmas/, 2016.

[31]

F. Gustafsson. Automotive safety systems. IEEE Signal Processing Magazine, 26(4):32--47, July 2009.

[32]

T. Holstein, G. Dodig-Crnkovic, and P. Pelliccione. Ethical and Social Aspects of Self-Driving Cars. ArXiv e-prints, Feb. 2018.

[33]

N. Kalra and S. M. Paddock. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94(Supplement C):182 -- 193, 2016.

[34]

K. Kirkpatrick. The moral challenges of driverless cars. Commun. ACM, 58(8):19--20, July 2015.

Digital Library

[35]

S. Kuchinskas. Crash Course: Training the Brain of a Driverless Car. https://www.scientificamerican.com/article/autonomous-driverless-car-brain/, 2013.

[36]

B. MacKinnon. Ethics: Theory and Contemporary Issues, Concise Edition. Cengage Learning, 2012.

[37]

N. McBride. The ethics of driverless cars. SIGCAS Comput. Soc., 45(3):179--184, Jan. 2016.

Digital Library

[38]

C. Metz. Former Apple Engineers Working on New Eyes for Driverless Cars, September 2017. https://www.nytimes.com/2017/09/20/technology/former-apple-engineers-driverless-cars.html.

[39]

C. Metz. What Virtual Reality Can Teach a Driverless Car, October 2017. https://www.nytimes.com/2017/10/29/business/virtual-reality-driverless-cars.html.

[40]

C. Mooney. Save the driver or save the crowd? Scientists wonder how driverless cars will 'choose'. https://www.washingtonpost.com/news/energy-environment/wp/2016/06/23/save-the-driver-or-save-the-crowd-scientists-wonder-how-driverless-cars-will-choose/, 2016.

[41]

National Highway Traffic Safety Administration (NHTSA). Federal automated vehicles policy - accelerating the next revolution in roadway safety. Technical report, U.S. Department of Transportation, 2016.

[42]

N. H. T. S. A. (NHTSA). "dot/nhtsa policy statement concerning automated vehicles" 2016 update to "preliminary statement of policy concerning automated vehicles". Technical report, National Highway Traffic Safety Administration (NHTSA).

[43]

P. Oliver, B. Jan, and K. Sören. Automated driving on public roads: Experiences in real traffic, 2015.

[44]

G. Orwell. 1984. Secker & Warburg, 1949.

[45]

P. Pelliccione, E. Knauss, R. Heldal, S. M. Ågren, P. Mallozzi, A. Alminger, and D. Borgentun. Automotive architecture framework: The experience of volvo cars. Journal of Systems Architecture, 77(Supplement C):83 -- 100, 2017.

[46]

M. Persson and S. Elfström. Volvo Car Group's first self-driving Autopilot cars test on public roads around Gothenburg, 2014.

[47]

S. Pillath. Briefing: Automated vehicles in the EU. European Parliamentary Research Service (EPRS), (January):12, 2016.

[48]

A. Riener, M. P. Jeon, I. Alvarez, B. Pfleging, A. Mirnig, M. Tscheligi, and L. Chuang. 1st workshop on ethically inspired user interfaces for automated driving. In Adjunct Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI '16 Adjunct, pages 217--220, New York, NY, USA, 2016. ACM.

Digital Library

[49]

SAE. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Global Ground Vehicle Standards, (J3016):30, 2016.

[50]

A. Shashkevich. Stanford professors discuss ethics involving driverless cars. https://news.stanford.edu/2017/05/22/stanford-scholars-researchers-discuss-key-ethical-questions-self-driving-cars-present/, may 2017.

[51]

J. D. Stoll. Gm executive credits silicon valley for accelerating development of self-driving cars, 2016.

[52]

L. R. Sütfeld, R. Gast, P. König, and G. Pipa. Using virtual reality to assess ethical decisions in road traffic scenarios: Applicability of value-of-life-based models and influences of time pressure. Frontiers in Behavioral Neuroscience, 11:122, 2017.

[53]

Tesla. A tragic loss | tesla deutschland, 2016.

[54]

Tesla. Upgrading Autopilot: Seeing the World in Radar | Tesla Deutschland, 2016.

[55]

Toyota. New toyota test vehicle paves the way for commercialization of automated highway driving technologies | toyota global newsroom, 2015.

[56]

D. WAKABAYASHI. WaymoâĂ&Zacute;s Autonomous Cars Cut Out Human Drivers in Road Tests, November 2017. https://www.nytimes.com/2017/11/07/technology/waymo-autonomous-cars.html.

[57]

M. M. Waldrop. Autonomous vehicles: No drivers required. Nature, 518:20--3, 2015.

[58]

Waymo. Waymo - Navigating city streets, December 2016. https://youtu.be/fbWeKhAPMig?t=10.

[59]

Waymo. Technology - Waymo, 2017. https://waymo.com/tech/.

[60]

Waymo. Waymo, September 2017. https://waymo.com.

[61]

Waymo. Waymo Self-Driving Safety Report, 2017. https://storage.googleapis.com/sdc-prod/v1/safety-report/waymo-safety-report-2017.pdf.

[62]

Waymo. Waymo - On the Road, February 2018. https://waymo.com/ontheroad/.

[63]

P. Wintersberger, A. K. Prison, A. Riener, and S. Hasirlioglu. The experience of ethics: Evaluation of self harm risks in automated vehicles. In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 385--391, June 2017.

Cited By

Fossa FFossa F(2023)Unavoidable Collisions. The Automation of Moral JudgmentEthics of Driving Automation10.1007/978-3-031-22982-4_4(65-94)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-031-22982-4_4
Zheng HChen ZDu TZhang XCheng YJi SWang JYu YChen JDwyer MDamian DZeller A(2022)NeuronFairProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510123(1519-1531)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510123
Holstein TDodig-Crnkovic GPelliccione P(2021)Steps Toward Real-World Ethics for Self-Driving CarsMachine Law, Ethics, and Morality in the Age of Artificial Intelligence10.4018/978-1-7998-4894-3.ch006(85-107)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-4894-3.ch006
Show More Cited By

Index Terms

Avoiding the intrinsic unfairness of the trolley problem
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Trolled by the Trolley Problem: On What Matters for Ethical Decision Making in Automated Vehicles
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Automated vehicles have to make decisions, such as driving maneuvers or rerouting, based on environment data and decision algorithms. There is a question whether ethical aspects should be considered in these algorithms. When all available decisions ...
Real-world ethics for self-driving cars
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings

Ethical and social problems of the emerging technology of self-driving cars can best be addressed through an applied engineering ethical approach. However, currently social and ethical problems are typically being presented in terms of an idealized ...
First Person Trolley Problem: Evaluation of Drivers' Ethical Decisions in a Driving Simulator
AutomotiveUI '16 Adjunct: Adjunct Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

No matter how safe automated driving will be in the future, there will remain situations where an accident is unavoidable and vehicles will have to decide between options that all potentially result in lethal outcome. Empirical ethics is one method that ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

FairWare '18: Proceedings of the International Workshop on Software Fairness

May 2018

56 pages

ISBN:9781450357463

DOI:10.1145/3194770

Conference Chairs:
Yuriy Brun
University of Massachusetts, Amherst, MA
,
Brittany Johnson
University of Massachusetts, Amherst, MA
,
Alexandra Meliou
University of Massachusetts, Amherst, MA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 29, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
938
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)6

Reflects downloads up to 09 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fossa FFossa F(2023)Unavoidable Collisions. The Automation of Moral JudgmentEthics of Driving Automation10.1007/978-3-031-22982-4_4(65-94)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-031-22982-4_4
Zheng HChen ZDu TZhang XCheng YJi SWang JYu YChen JDwyer MDamian DZeller A(2022)NeuronFairProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510123(1519-1531)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510123
Holstein TDodig-Crnkovic GPelliccione P(2021)Steps Toward Real-World Ethics for Self-Driving CarsMachine Law, Ethics, and Morality in the Age of Artificial Intelligence10.4018/978-1-7998-4894-3.ch006(85-107)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-4894-3.ch006
Balayn ALofi CHouben G(2021)Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systemsThe VLDB Journal10.1007/s00778-021-00671-830:5(739-768)Online publication date: 5-May-2021
https://doi.org/10.1007/s00778-021-00671-8
Kallioinen NPershina MZeiser JNosrat Nezami FPipa GStephan AKönig P(2019)Moral Judgements on the Actions of Self-Driving Cars and Human Drivers in Dilemma Situations From Different PerspectivesFrontiers in Psychology10.3389/fpsyg.2019.0241510Online publication date: 1-Nov-2019
https://doi.org/10.3389/fpsyg.2019.02415
Mirnig AMeschtscherjakov ABrewster SFitzpatrick GCox AKostakos V(2019)Trolled by the Trolley ProblemProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300739(1-10)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300739
Aliman NKester L(2019)Extending Socio-Technological Reality for Ethics in Artificial Intelligent Systems2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)10.1109/AIVR46125.2019.00064(275-2757)Online publication date: Dec-2019
https://doi.org/10.1109/AIVR46125.2019.00064
Davnall R(2019)Solving the Single-Vehicle Self-Driving Car Trolley Problem Using Risk Theory and Vehicle DynamicsScience and Engineering Ethics10.1007/s11948-019-00102-6Online publication date: 1-Apr-2019
https://doi.org/10.1007/s11948-019-00102-6
McGrath JGupta A(2018)Writing a Moral Code: Algorithms for Ethical Reasoning by Humans and MachinesReligions10.3390/rel90802409:8(240)Online publication date: 9-Aug-2018
https://doi.org/10.3390/rel9080240
Luzuriaga MHeras AKunze O(undefined)Hurting Others vs. Hurting Myself, a Dilemma for our Autonomous VehicleSSRN Electronic Journal10.2139/ssrn.3345141
https://doi.org/10.2139/ssrn.3345141

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents