research-article

Classification of global catastrophic risks connected with artificial intelligence

Authors:

Alexey Turchin,

David DenkenbergerAuthors Info & Claims

AI & SOCIETY, Volume 35, Issue 1

Pages 147 - 163

https://doi.org/10.1007/s00146-018-0845-5

Published: 01 March 2020 Publication History

Abstract

A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of AI development, namely, (1) before it starts self-improvement, (2) during its takeoff, when it uses various instruments to escape its initial confinement, or (3) after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level.

References

[1]

Alexander S (2016) Ascended economy? Star Slate Codex. http://slatestarcodex.com/2016/05/30/ascended-economy/. Accessed 27 Apr 2018

[2]

Anderson M (2017) RethinkX: self-driving electric cars will dominate roads by 2030. In: IEEE Spectrum: technology, engineering and science news. http://spectrum.ieee.org/cars-that-think/transportation/self-driving/rethinkx-selfdriving-electric-cars-will-dominate-roads-by-2030. Accessed 17 Jul 2017

[3]

Angel.co (2017) Artificial intelligence startups. https://angel.co/artificial-intelligence. Accessed 27 Apr 2018

[4]

Armstrong S (2017) Good and safe uses of AI Oracles. ArXiv171105541 Cs

[5]

Auerbach D (2014) The Most Terrifying Thought Experiment of All Time. In: Slate. http://www.slate.com/articles/technology/bitwise/2014/07/roko_s_basilisk_the_most_terrifying_thought_experiment_of_all_time.html. Accessed 27 Apr 2018

[6]

Baker BH The gray matter: the forgotten story of the telephone 2000 Kent, WA Telepress

[7]

Bardi U (2008) The Universal Mining Machine. http://europe.theoildrum.com/node/3451. Accessed 27 Apr 2018

[8]

Barrett AM and Baum SD A model of pathways to artificial superintelligence catastrophe for risk and decision analysis J Exp Theor Artif Intell 2017 29 397-414

[9]

BBC (2017) Cyber-attack: europol says it was unprecedented in scale—BBC News. http://www.bbc.com/news/world-europe-39907965. Accessed 17 Jul 2017

[10]

Bender J (2014) Russia may still have an automated nuclear launch system aimed across the northern hemisphere. In: Bus. Insid. https://www.businessinsider.com.au/russias-dead-hand-system-may-still-be-active-2014-9. Accessed 17 Jul 2017

[11]

Blair BG The logic of accidental nuclear war 2011 Washington, DC Brookings Institution Press

[12]

Boles KS, Kannan K, Gill J, et al. Digital-to-biological converter for on-demand production of biologics Nat Biotechnol 2017 35 672-675

[13]

Bostrom N Existential risks: analyzing human extinction scenarios and related hazards J Evol Technol 2002 9 1 1-30

[14]

Bostrom N Astronomical waste: The opportunity cost of delayed technological development Utilitas 2003 15 308-314

[15]

Bostrom N Are you living in a computer simulation? Publ Philos Q 2003 53 211 243-255

[16]

Bostrom N What is a singleton Linguist Philos Investig 2006 5 48-54

[17]

Bostrom NPascal’s muggingAnalysis2009693443-4452545385

[18]

Bostrom N (2011) Infinite ethics. Anal Metaphys 9–59

[19]

Bostrom N Superintelligence 2014 Oxford Oxford University Press

[20]

Bradbury RJ (2001) Matrioshka brains. preprint. http://www.aeiveos.com/bradbury/MatrioshkaBrains/MatrioshkaBrains.html

[21]

Carrigan RA Jr Do potential SETI signals need to be decontaminated? Acta Astronaut 2006 58 112-117

[22]

Chalmers DJ Gendler T and Hawthorne J Does conceivability entail possibility?Conceivability possibility 2002 New York Oxford University Press 145-200

[23]

Chiew KL, Yong KSC, and Tan CL A survey of phishing attacks: their types, vectors and technical approaches Expert Syst Appl 2018 106 1-20

[24]

Christiano P (2016) Prosaic AI alignment. https://ai-alignment.com/prosaic-ai-control-b959644d79c2. Accessed 27 Apr 2018

[25]

Clavero M and García-Berthou E Invasive species are a leading cause of animal extinctions Trends Ecol Evol 2005 20 110

[26]

Cole DD, Denkenberger D, Griswold M et al (2016) Feeding everyone if industry is disabled. In: Proceedings of the 6th international disaster and risk conference. Davos, Switzerland

[27]

Critch A (2017) Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making (arXiv:1701.01302)

[28]

Daniel M (2017) S-risks: why they are the worst existential risks, and how to prevent them (EAG Boston 2017). https://foundational-research.org/s-risks-talk-eag-boston-2017/. Accessed 27 Apr 2018

[29]

Dennett DC Why you can’t make a computer that feels pain Synthese 1978 38 415-456

[30]

Ellison H I have no mouth, and i must scream 1967 New York Galaxy Publishing Corp

[31]

Enserink M (2011) Scientists brace for media storm around controversial flu studies. In: Sciencemag. http://www.sciencemag.org/news/2011/11/scientists-brace-media-storm-around-controversial-flu-studies. Accessed 27 Apr 2018

[32]

Freitas R (2000) Some limits to global ecophagy by biovorous nanoreplicators, with public policy recommendations. Foresight Institute Technical Report

[33]

Future of Life Institute (2016) Accidental nuclear war: a timeline of close calls. https://futureoflife.org/background/nuclear-close-calls-a-timeline/. Accessed 4 Nov 2017

[34]

Futureworld (2013) Airplane “crashes” as hacker gets control. In: Futureworld. http://www.futureworld.org/PublicZone/MindBullets/MindBulletsDetails.aspx?MindBulletID=498. Accessed 27 Apr 2018

[35]

Gildert S (2011) Why “computronium” is really “unobtanium” IO9. http://io9.gizmodo.com/5758349/why-computronium-is-really-unobtanium. Accessed 27 Apr 2018

[36]

Goertzel B Should humanity build a global ai nanny to delay the singularity until it’s better understood? J Conscious Stud 2012 19 1–2 96-111

[37]

Grace K, Salvatier J, Dafoe A et al (2017) When will AI exceed human performance? evidence from AI experts. (arXiv:1705.08807 [cs.AI])

[38]

Granoff J (2016) Donald trump is an existential threat to America and the world. Time

[39]

Gwern (2016) Why tool AIs want to be agent AIs. https://www.gwern.net/Tool-AI

[40]

Hanson R Bostrom N and Cirkovic MM Catastrophe, social collapse, and human extinction Global catastrophic risks 2008 Oxford Oxford University Press 554

[41]

Hanson R The age of Em: work, love, and life when robots rule the earth 2016 Oxford Oxford University Press

[42]

Hines N (2016) Neural implants could let hackers hijack your brain. In: Inverse. https://www.inverse.com/article/19148-neural-implants-could-let-hackers-hijack-your-brain. Accessed 17 Jul 2017

[43]

Hume D A treatise of human nature 1739 London, UK Oxford: Clarendon Press

[44]

Hutter M (2000) A theory of universal artificial intelligence based on algorithmic complexity. ArXiv Prepr Cs0004001

[45]

Jenkins A Uber may not be to blame for self-driving car death in Arizona 2018 New York Fortune

[46]

Joy B Why the future doesn’t need us 2000 San Francisco, CA Wired

[47]

Kahn H On thermonuclear war 1959 Princeton Princeton University Press

[48]

Kardashev NS On the inevitability and the possible structures of supercivilizations 1985 Dordrecht Reidel Publishing Co. 497-504

[49]

Karpathy A (2015) The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy Blog. http://karpathy.github.io/2015/05/21/rnn-effectiveness/

[50]

Kushner D The real story of stuxnet IEEE Spectr 2013 50 48-53

[51]

LaVictoire P, Fallenstein B, Yudkowsky E et al (2014) Program equilibrium in the prisoner’s dilemma via Löb’s theorem. MIRI

[52]

LaVictorie P An Introduction to Löb’s Theorem in MIRI Research 2015 San Francisco CA MIRI

[53]

Lem S Return from the stars 1961 Boston, US Houghton Mifflin Harcourt

[54]

Lem S Summa technologiae 1963 Berlin, Germany Suhrkamp

[55]

Lem S The Invincible: science fiction 1973 London, UK Sidgwick & Jackson

[56]

Lenat DB and Brown JS Why AM and EURISKO appear to work Artif Intell 1984 23 269-294

[57]

LoPucki LM Algorithmic ENTITIES 2017 Rochester Social Science Research Network

[58]

Lubin G (2016) Data reveals the 20 most popular TV shows of 2016. Business Insider

[59]

Mennen A (2017) Existential risk from AI without an intelligence explosion. http://lesswrong.com/lw/p28/existential_risk_from_ai_without_an_intelligence/

[60]

Menzel C Zalta EN Actualism The stanford encyclopedia of philosophy 2017 2014 Stanford Metaphysics Research Lab, Stanford University

[61]

Meuhlhauser L (2014) How big is the field of artificial intelligence? (initial findings). https://intelligence.org/2014/01/28/how-big-is-ai/. Accessed 27 Apr 2018

[62]

Muehlhauser L (2011) Intelligence explosion FAQ. https://intelligence.org/ie-faq/. Accessed 27 Apr 2018

[63]

Mullin G (2017) What is the Blue Whale suicide challenge, how many deaths has the game been linked to so far and is it in the UK? TheSun

[64]

Oberhaus D (2017) Watch ‘Slaughterbots’, a warning about the future of killer bots. In: Motherboard. https://motherboard.vice.com/en_us/article/9kqmy5/slaughterbots-autonomous-weapons-future-of-life. Accessed 17 Dec 2017

[65]

Omohundro S Wang P, Goertzel B, and Franklin S The basic AI drives Proceedings of the 2008 conference on Artificial General Intelligence 2008: proceedings of the First AGI Conference 2008 The Netherlands IOS Press Amsterdam

[66]

Orwell G 1984 1948 Boston, US Houghton Mifflin Harcourt

[67]

Pinker S The better angels of our nature: The decline of violence in history and its causes 2011 London Penguin

[68]

Reason J Human error: models and management BMJ 2000 320 768-770

[69]

Russell S (2017) 3 principles for creating safer AI. https://www.youtube.com/watch?v=EBK-a94IFHY. Accessed 27 Apr 2018

[70]

Saito T and Angles J Hikikomori: adolescence without end 2013 Minnesota Univesity Of Minnesota Press

[71]

Sarma GP, Hay NJ (2016) Mammalian value systems. (arXiv:1607.08289 [cs.AI])

[72]

Schneier B Perspective | The next ransomware attack will be worse than WannaCry 2017 Post Wash

[73]

Shakirov V (2016) Review of state-of-the-arts in artificial intelligence with application to AI safety problem. (ArXiv Prepr ArXiv160504232)

[74]

Shulman C (2010) Omohundro’s “basic AI drives” and catastrophic risks. http://intelligence.org/files/BasicAIDrives.pdf. Accessed 27 Apr 2018

[75]

Shulman C Arms races and intelligence explosions Singularity Hypotheses 2011 New York Springer

[76]

Sotala K (2016) Decisive strategic advantage without a hard takeoff. http://kajsotala.fi/2016/04/decisive-strategic-advantage-without-a-hard-takeoff/#comments. Accessed 27 Apr 2018

[77]

Sotala K (2017) Disjunctive AI scenarios: Individual or collective takeoff? http://kajsotala.fi/2017/01/disjunctive-ai-scenarios-individual-or-collective-takeoff/. Accessed 27 Apr 2018

[78]

Sotala K and Yampolskiy R Responses to catastrophic AGI risk: a survey Phys Scr 2014 90 018001

[79]

Srugatsky N and Strugatsky B The time wanderers 1985 New York, US Richardson & Steirman

[80]

Strugatsky A and Strugatsky B The final circle of paradise, Translated by Leonid Renen 1976 New York DAW

[81]

Taylor A (2017) Flying around the world in a solar powered plane—the Atlantic

[82]

The Telegraph (2009) Russian spacecraft landed on moon hours before Americans. The telegraph. http://www.telegraph.co.uk:80/science/space/5737854/Russian-spacecraft-landed-on-moon-hours-before-Americans.html. Accessed 27 Apr 2018

[83]

Torres P (2014) Why running simulations may mean the end is near. https://ieet.org/index.php/IEET2/more/torres20141103. Accessed 27 Apr 2018

[84]

Torres P (2016) Problems with defining an existential risk. IEET. https://ieet.org/index.php/IEET2/more/torres20150121. Accessed 27 Apr 2018

[85]

Turchin A (2018) The risks connected with possibility of finding alien AI code during SETI. Rev J Br Interplanet Soc. Manuscript, https://philpapers.org/rec/TURCSW

[86]

Turchin A, Denkenberger D (2017) Levels of self-improvement. Manuscript, University of Louisville, TN

[87]

Turchin A and Denkenberger D Yampolskiy R Military AI as convergent goal of the self-improving AI Artificial intelligence safety and security 2018 Baca Raton CRC Press

[88]

Turchin A, Denkenberger D (2018b) Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons. Manuscript

[89]

Turchin A, Green B, Denkenberger D (2017) multiple simultaneous pandemics as most dangerous global catastrophic risk connected with bioweapons and synthetic biology. Rev Health Secur

[90]

Turing AMOn computable numbers, with an application to the EntscheidungsproblemProc Lond Math Soc19372230-265157703062.1059.03

[91]

Velicovich B I could kill you with a consumer drone 2017 Washington, DC Defense one

[92]

Watkins J (2016) “Shut up and dance”—“Black mirror” series

[93]

Wei D (2013) Outside view(s) and MIRI’s FAI endgame. http://lesswrong.com/lw/ig9/outside_views_and_miris_fai_endgame/. Accessed 27 Apr 2018

[94]

Wootson J Elon Musk doesn’t think we’re prepared to face humanity’s biggest threat: artificial intelligence 2017 Post Wash

[95]

Yampolskiy Roman V. Utility function security in artificially intelligent agents Journal of Experimental & Theoretical Artificial Intelligence 2014 26 3 373-389

[96]

Yampolskiy R Artificial superintelligence: a futuristic approach 2015 Boca Raton CRC Press

[97]

Yampolskiy R (2015b) Taxonomy of pathways to dangerous AI. (ArXiv Prepr ArXiv151103246)

[98]

Yampolskiy R, Spellchecker M (2016) artificial intelligence safety and cybersecurity: a timeline of AI failures. (ArXiv Prepr ArXiv161007997)

[99]

Yudkowsky E Creating friendly AI 1.0: the analysis and design of benevolent goal architectures 2001 San Francisco, CA MIRI 1-282

[100]

Yudkowsky E (2002) The AI-Box Experiment. http://yudkowsky.net/singularity/aibox. Accessed 27 Apr 2018

[101]

Yudkowsky E (2003) HUMOR: friendly AI critical failure table. http://www.sl4.org/archive/0310/7163.html. Accessed 27 Apr 2018

[102]

Yudkowsky E (2004) Coherent extrapolated volition. http://intelligence.org/files/CEV.pdf. Accessed 27 Apr 2018

[103]

Yudkowsky E Artificial intelligence as a positive and negative factor in global risk, in global catastrophic risks 2008 Oxford Oxford University Press

[104]

Yudkowsky E From AI to zombies 2015 San Francisco, CA MIRI

[105]

Yudkowsky E (2017) Comment on paper clip maximiser scenario. http://www.jefftk.com/p/examples-of-superintelligence-risk#fb-886930452142_886983450932. Accessed 27 Apr 2018

[106]

Yudkowsky E, Hanson R (2008) The Hanson-Yudkowsky AI-foom debate. In: MIRI Technical report

[107]

Yudkowsky E, Herreshoff M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle. Early Draft MIRI

Cited By

Ofte HKatsikas S(2024)Paralyzed or Compromised: A Case Study of Decisions in Cyber-Physical SystemsHCI for Cybersecurity, Privacy and Trust10.1007/978-3-031-61382-1_9(134-152)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61382-1_9
Carayannis EDraper J(2023)Optimising peace through a Universal Global Peace Treaty to constrain the risk of war from a militarised artificial superintelligenceAI & Society10.1007/s00146-021-01382-y38:6(2679-2692)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00146-021-01382-y
Clarke SWhittlestone JConitzer VTasioulas JScheutz MCalo RMara MZimmermann A(2022)A Survey of the Potential Long-term Impacts of AIProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3514094.3534131(192-202)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3514094.3534131
Show More Cited By

Recommendations

Artificial intelligence and ambient intelligence

Ambient intelligence (AmI) is intrinsically and thoroughly connected with artificial intelligence (AI). Some even say that it is, in essence, AI in the environment. AI, on the other hand, owes its success to the phenomenal development of the information ...
Risk management standards and the active management of malicious intent in artificial superintelligence
Abstract
The likely near future creation of artificial superintelligence carries significant risks to humanity. These risks are difficult to conceptualise and quantify, but malicious use of existing artificial intelligence by criminals and state actors is ...
Artificial intelligence
Abstract
Artificial intelligence (AI) is the Science and Engineering domain concerned with the theory and practice of developing systems that exhibit the characteristics we associate with intelligence in human behavior. Starting with a brief history of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image AI & Society

AI & Society Volume 35, Issue 1

Mar 2020

274 pages

ISSN:0951-5666

Issue’s Table of Contents

© Springer-Verlag London Ltd., part of Springer Nature 2018.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 March 2020

Accepted: 23 April 2018

Received: 18 January 2018

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ofte HKatsikas S(2024)Paralyzed or Compromised: A Case Study of Decisions in Cyber-Physical SystemsHCI for Cybersecurity, Privacy and Trust10.1007/978-3-031-61382-1_9(134-152)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61382-1_9
Carayannis EDraper J(2023)Optimising peace through a Universal Global Peace Treaty to constrain the risk of war from a militarised artificial superintelligenceAI & Society10.1007/s00146-021-01382-y38:6(2679-2692)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00146-021-01382-y
Clarke SWhittlestone JConitzer VTasioulas JScheutz MCalo RMara MZimmermann A(2022)A Survey of the Potential Long-term Impacts of AIProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3514094.3534131(192-202)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3514094.3534131
Maisonobe M(2022)The future of urban models in the Big Data and AI era: a bibliometric analysis (2000–2019)AI & Society10.1007/s00146-021-01166-437:1(177-194)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s00146-021-01166-4

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents