Artificial Moral Agents: A Survey of the Current Status

• Abbass, H. A., Petraki, E., Merrick, K., Harvey, J., & Barlow, M. (2016). Trusted autonomy and cognitive cyber symbiosis: Open challenges. Cognitive Computation,8(3), 385–408.

  Article  Google Scholar 

• Alaieri, F., & Vellino, A. (2016). Ethical decision making in robots: Autonomy, trust and responsibility. In International conference on social robotics (pp. 159–168). Cham: Springer.

• Allen, C., Smit, I., & Wallach, W. (2005). Artificial morality: Top-down, bottom-up, and hybrid approaches. Ethics and Information Technology,7(3), 149–155.

  Article  Google Scholar 

• Amstutz, M. R. (2013). International ethics: Concepts, theories, and cases in global politics. New York: Rowman & Littlefield Publishers.

  Google Scholar 

• Anderson, M., & Anderson, S. L. (2007a). Machine ethics: Creating an ethical intelligent agent. AI Magazine,28(4), 15–26.

  Google Scholar 

• Anderson, M., & Anderson, S. L. (2007b). The status of machine ethics: A report from the AAAI symposium. Minds and Machines,17(1), 1–10.

  Article  Google Scholar 

• Anderson, M., & Anderson, S. L. (2008). Ethical healthcare agents. In M. Sordo, S. Vaidya, & L. C. Jain (Eds.), Advanced computational intelligence paradigms in healthcare-3 (pp. 233–257). Berlin: Springer.

  Chapter  Google Scholar 

• Anderson, M., & Anderson, S. L. (2010). Robot be good. Scientific American,303(4), 72–77.

  Article  Google Scholar 

• Anderson, M., & Anderson, S. L. (2014). Geneth: A general ethical dilemma analyzer. In Twenty-eighth AAAI conference on artificial intelligence (pp. 253–261).

• Anderson, M., Anderson, S. L., & Armen, C. (2004). Towards machine ethics. In Proceedings of the AOTP’04—The AAAI-04 workshop on agent organizations: Theory and practice.

• Anderson, M., Anderson, S. L., & Armen, C. (2005). Medethex: Toward a medical ethics advisor. In Proceedings of the AAAI 2005 Fall symposium on caring machines: AI in elder care (pp. 9–16).

• Anderson, M., Anderson, S. L., & Armen, C. (2006a). An approach to computing ethics. IEEE Intelligent Systems,21(4), 56–63.

  Article  Google Scholar 

• Anderson, M., Anderson, S. L., & Armen, C. (2006b). Medethex: A prototype medical ethics advisor. In Proceedings of the national conference on artificial intelligence (Vol. 21, No. 2, pp. 1759–1765). Menlo Park, CA/Cambridge, MA: AAAI Press/MIT Press.

• Andino, C. (2015). Place of ethics between technical knowledge. A philosophical approach. Revista Científica de la UCSA,2(2), 85–94.

  Google Scholar 

• Arkin, R. (2009). Governing lethal behavior in autonomous robots. London: Chapman and Hall/CRC.

  Book  Google Scholar 

• Arkin, R. C. (2010). The case for ethical autonomy in unmanned systems. Journal of Military Ethics,9(4), 332–341.

  Article  Google Scholar 

• Arkin, R. (2018). Lethal autonomous systems and the plight of the noncombatant. In R. Kiggins (Ed.), The political economy of robots (pp. 317–326). Cham: Springer.

  Chapter  Google Scholar 

• Arkoudas, K., Bringsjord, S., & Bello, P. (2005). Toward ethical robots via mechanized deontic logic. In AAAI Fall symposium on machine ethics (pp. 17–23).

• Ashrafian, H. (2015). Artificial intelligence and robot responsibilities: Innovating beyond rights. Science and Engineering Ethics,21(2), 317–326.

  Article  Google Scholar 

• Bandyopadhyay, D., & Sen, J. (2011). Internet of things: Applications and challenges in technology and standardization. Wireless Personal Communications,58(1), 49–69.

  Article  Google Scholar 

• Batty, M., Axhausen, K. W., Giannotti, F., Pozdnoukhov, A., Bazzani, A., Wachowicz, M., et al. (2012). Smart cities of the future. The European Physical Journal Special Topics,214(1), 481–518.

  Article  Google Scholar 

• Beauvisage, T. (2009). Computer usage in daily life. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 575–584). ACM.

• Bedaf, S., Draper, H., Gelderblom, G. J., Sorell, T., & de Witte, L. (2016). Can a service robot which supports independent living of older people disobey a command? The views of older people, informal carers and professional caregivers on the acceptability of robots. International Journal of Social Robotics,8(3), 409–420.

  Article  Google Scholar 

• Belloni, A., Berger, A., Besson, V., Boissier, O., Bonnet, G., Bourgne, G., et al. (2014). Towards a framework to deal with ethical conflicts in autonomous agents and multi-agent systems. In CEPE 2014 well-being, flourishing, and ICTs (pp. 1–10).

• Belloni, A., Berger, A., Boissier, O., Bonnet, G., Bourgne, G., Chardel, P. A., et al. (2015). Dealing with ethical conflicts in autonomous agents and multi-agent systems. In 1st International workshop on artificial intelligence and ethics at the 29th AAAI conference on artificial intelligence.

• Blass, J. A. (2016). Interactive learning and analogical chaining for moral and commonsense reasoning. In Thirtieth AAAI conference on artificial intelligence (pp. 4289–4290).

• Blass, J. A., & Forbus, K. D. (2015). Moral decision-making by analogy: Generalizations versus exemplars. In Twenty-ninth AAAI conference on artificial intelligence (pp. 501–507).

• Bonnemains, V., Saurel, C., & Tessier, C. (2018). Embedded ethics: Some technical and ethical challenges. Ethics and Information Technology,20(1), 41–58.

  Article  Google Scholar 

• Borenstein, J., & Arkin, R. (2019). Robots, ethics, and intimacy: The need for scientific research. In D. Berkich, & M. d'Alfonso (Eds.), On the cognitive, ethical, and scientific dimensions of artificial intelligence (pp. 299–309). Cham: Springer.

  Chapter  Google Scholar 

• Borst, J. P., & Anderson, J. R. (2015). Using the ACT-R cognitive architecture in combination with fMRI data. In B. Forstmann, & E. J. Wagenmakers (Eds.), An introduction to model-based cognitive neuroscience (pp. 339–352). Berlin: Springer.

  Google Scholar 

• Brachman, R. J. (2002). Systems that know what they’re doing. IEEE Intelligent Systems,17(6), 67–71.

  Article  Google Scholar 

• Briggs, G., & Scheutz, M. (2015). Sorry, I can’t do that”: Developing mechanisms to appropriately reject directives in human–robot interactions. In 2015 AAAI Fall symposium series (pp. 1–5).

• Bringsjord, S., Sundar, G. N., Thero, D., & Si, M. (2014). Akratic robots and the computational logic thereof. In Proceedings of the IEEE 2014 international symposium on ethics in engineering, science, and technology (pp. 1–8). IEEE Press.

• Brundage, M. (2014). Limitations and risks of machine ethics. Journal of Experimental & Theoretical Artificial Intelligence,26(3), 355–372.

  Article  Google Scholar 

• Capraro, V., & Rand, D. G. (2018). Do the right thing: Experimental evidence that preferences for moral behavior, rather than equity or efficiency per se, drive human prosociality. Forthcoming in Judgment and Decision Making,13(1), 99–111.

  Google Scholar 

• Cervantes, J. A., Rodríguez, L. F., López, S., Ramos, F., & Robles, F. (2016). Autonomous agents and ethical decision-making. Cognitive Computation,8(2), 278–296.

  Article  Google Scholar 

• Cervantes, J. A., Rosales, J. H., López, S., Ramos, F., & Ramos, M. (2017). Integrating a cognitive computational model of planning and decision-making considering affective information. Cognitive Systems Research,44, 10–39.

  Article  Google Scholar 

• Choi, D., & Langley, P. (2018). Evolution of the icarus cognitive architecture. Cognitive Systems Research,48, 25–38.

  Article  Google Scholar 

• Coeckelbergh, M. (2010). Moral appearances: Emotions, robots, and human morality. Ethics and Information Technology,12(3), 235–241.

  Article  Google Scholar 

• Conway, P., & Gawronski, B. (2013). Deontological and utilitarian inclinations in moral decision making: A process dissociation approach. Journal of Personality and Social Psychology,104(2), 216–235.

  Article  Google Scholar 

• Cook, D. J., & Das, S. K. (2012). Pervasive computing at scale: Transforming the state of the art. Pervasive and Mobile Computing,8(1), 22–35.

  Article  Google Scholar 

• Cristani, M., & Burato, E. (2009). Approximate solutions of moral dilemmas in multiple agent system. Knowledge and Information Systems,18(2), 157–181.

  Article  Google Scholar 

• Czubenko, M., Kowalczuk, Z., & Ordys, A. (2015). Autonomous driver based on an intelligent system of decision-making. Cognitive Computation,7(5), 569–581.

  Article  Google Scholar 

• Dehghani, M., Tomai, E., Forbus, K. D., & Klenk, M. (2008). An integrated reasoning approach to moral decision-making. In Twenty-third AAAI conference on artificial intelligence (pp. 1280–1286).

• Deng, B. (2015). Machine ethics: The robot’s dilemma. Nature,523(7558), 24–26.

  Article  Google Scholar 

• Dennis, L. A., Fisher, M., Lincoln, N. K., Lisitsa, A., & Veres, S. M. (2016a). Practical verification of decision-making in agent-based autonomous systems. Automated Software Engineering,23(3), 305–359.

  Article  Google Scholar 

• Dennis, L., Fisher, M., Slavkovik, M., & Webster, M. (2016b). Formal verification of ethical choices in autonomous systems. Robotics and Autonomous Systems,77, 1–14.

  Article  Google Scholar 

• Epting, S. (2016). A different trolley problem: The limits of environmental justice and the promise of complex moral assessments for transportation infrastructure. Science and Engineering Ethics,22(6), 1781–1795.

  Article  Google Scholar 

• Erdur, M. (2018). Moral realism and the incompletability of morality. The Journal of Value Inquiry,52(2), 227–237.

  Article  Google Scholar 

• Fagin, R., Halpern, J. Y., & Vardi, M. Y. (1990). A nonstandard approach to the logical omniscience problem. In Proceedings of the 3rd conference on theoretical aspects of reasoning about knowledge (pp. 41–55). Morgan Kaufmann Publishers Inc.

• Feil-Seifer, D., & Matarić, M. J. (2011). Socially assistive robotics. IEEE Robotics and Automation Magazine,18(1), 24–31.

  Article  Google Scholar 

• Ferrell, O. C., & Gresham, L. G. (1985). A contingency framework for understanding ethical decision making in marketing. The Journal of Marketing,49(3), 87–96.

  Article  Google Scholar 

• Fleetwood, J., Vaught, W., Feldman, D., Gracely, E., Kassutto, Z., & Novack, D. (2000). Medethex online: A computer-based learning program in medical ethics and communication skills. Teaching and Learning in Medicine,12(2), 96–104.

  Article  Google Scholar 

• Fumagalli, M., & Priori, A. (2012). Functional and clinical neuroanatomy of morality. Brain,135(7), 2006–2021.

  Article  Google Scholar 

• Gerdes, A., & Øhrstrøm, P. (2015). Issues in robot ethics seen through the lens of a moral turing test. Journal of Information, Communication and Ethics in Society,13(2), 98–109.

  Article  Google Scholar 

• Gogoll, J., & Müller, J. F. (2017). Autonomous cars: In favor of a mandatory ethics setting. Science and Engineering Ethics,23(3), 681–700.

  Article  Google Scholar 

• Govindarajulu, N. S., Bringjsord, S., & Ghosh, R. (2018). One formalization of virtue ethics via learning. arXiv preprint arXiv:180507797.

• Greene, J. D., Morelli, S. A., Lowenberg, K., Nystrom, L. E., & Cohen, J. D. (2008). Cognitive load selectively interferes with utilitarian moral judgment. Cognition,107(3), 1144–1154.

  Article  Google Scholar 

• Greene, J., Rossi, F., Tasioulas, J., Venable, K. B., & Williams, B. C. (2016). Embedding ethical principles in collective decision support systems. In Thirtieth AAAI conference on artificial intelligence (pp. 4147–4151).

• Greene, J. D., Sommerville, R. B., Nystrom, L. E., Darley, J. M., & Cohen, J. D. (2001). An fMRI investigation of emotional engagement in moral judgment. Science,293(5537), 2105–2108.

  Article  Google Scholar 

• Guerini, M., Pianesi, F., & Stock, O. (2015). Is it morally acceptable for a system to lie to persuade me? In Workshops at the twenty-ninth AAAI conference on artificial intelligence (pp. 53–60).

• Han, T. A., & Pereira, L. M. (2018). Evolutionary machine ethics. In O. Bendel (Ed.), Handbuch Maschinenethik (pp. 1–25). Wiesbaden: Springer.

  Google Scholar 

• Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017). Neuroscience-inspired artificial intelligence. Neuron,95(2), 245–258.

  Article  Google Scholar 

• Honarvar, A. R., & Ghasem-Aghaee, N. (2009). Casuist BDI-agent: A new extended BDI architecture with the capability of ethical reasoning. In International conference on artificial intelligence and computational intelligence (pp. 86–95). Berlin: Springer.

• Howard, D., & Muntean, I. (2016). A minimalist model of the artificial autonomous moral agent (AAMA). In The 2016 AAAI Spring symposium series (pp. 217–225).

• Hughes, G. J. (2001). Routledge philosophy guidebook to Aristotle on ethics. London: Routledge.

  Google Scholar 

• Kahn, P. H., Jr., Kanda, T., Ishiguro, H., Gill, B. T., Ruckert, J. H., Shen, S., et al. (2012). Do people hold a humanoid robot morally accountable for the harm it causes? In Proceedings of the seventh annual ACM/IEEE international conference on Human–Robot Interaction (pp. 33–40). ACM.

• Kirchin, S. (Ed.). (2012). What is metaethics? In Metaethics (pp. 1–20). London: Palgrave Macmillan.

• Kishi, T., Hashimoto, K., & Takanishi, A. (2017). Human like face and head mechanism. In A. Goswami, & P. Vadakkepat (Eds.), Humanoid robotics: A reference (pp. 1–26). Dordrecht: Springer.

  Google Scholar 

• Kruglanski, A. W., & Gigerenzer, G. (2011). Intuitive and deliberate judgments are based on common principles. Psychological Review,118(1), 97–109.

  Article  Google Scholar 

• Laird, J. E. (2008). Extending the soar cognitive architecture. Frontiers in Artificial Intelligence and Applications,171, 224–235.

  Google Scholar 

• Laird, J. E., Kinkade, K. R., Mohan, S., & Xu, J. Z. (2012). Cognitive robotics using the soar cognitive architecture. In Workshops at the twenty-sixth AAAI conference on artificial intelligence (pp. 46–54).

• Laird, J. E., Lebiere, C., & Rosenbloom, P. S. (2017). A standard model of the mind: Toward a common computational framework across artificial intelligence, cognitive science, neuroscience, and robotics. AI Magazine,38(4), 13–26.

  Article  Google Scholar 

• Lombrozo, T. (2009). The role of moral commitments in moral judgment. Cognitive Science,33(2), 273–286.

  Article  Google Scholar 

• Long, L. N., & Kelley, T. D. (2010). Review of consciousness and the possibility of conscious robots. Journal of Aerospace Computing, Information, and Communication,7(2), 68–84.

  Article  Google Scholar 

• Madl, T., & Franklin, S. (2015). Constrained incrementalist moral decision making for a biologically inspired cognitive architecture. In R. Trappl (Ed.), A construction manual for robots’ ethical systems (pp. 137–153). Cham: Springer.

  Chapter  Google Scholar 

• Malle, B. F. (2016). Integrating robot ethics and machine morality: The study and design of moral competence in robots. Ethics and Information Technology,18(4), 243–256.

  Article  Google Scholar 

• Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015). Sacrifice one for the good of many? People apply different moral norms to human and robot agents. In Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction (pp. 117–124). ACM.

• Mermet, B., & Simon, G. (2016). Formal verification of ethical properties in multiagent systems. In ECAI 2016 workshop on ethics in the design of intelligent agents (EDIA’16). The Netherlands: The Hague.

• Metta, G., Natale, L., Nori, F., Sandini, G., Vernon, D., Fadiga, L., et al. (2010). The iCub humanoid robot: An open-systems platform for research in cognitive development. Neural Networks,23(8), 1125–1134.

  Article  Google Scholar 

• Mikhail, J. (2007). Universal moral grammar: Theory, evidence and the future. Trends in Cognitive Sciences,11(4), 143–152.

  Article  Google Scholar 

• Moor, J. H. (2006). The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems,21(4), 18–21.

  Article  Google Scholar 

• Mordoch, E., Osterreicher, A., Guse, L., Roger, K., & Thompson, G. (2013). Use of social commitment robots in the care of elderly people with dementia: A literature review. Maturitas,74(1), 14–20.

  Article  Google Scholar 

• Mostafa, S. A., Ahmad, M. S., & Mustapha, A. (2019). Adjustable autonomy: A systematic literature review. Artificial Intelligence Review,51(2), 149–186.

  Article  Google Scholar 

• Mostafa, S. A., Mustapha, A., Mohammed, M. A., Ahmad, M. S., & Mahmoud, M. A. (2018). A fuzzy logic control in adjustable autonomy of a multi-agent system for an automated elderly movement monitoring application. International Journal of Medical Informatics,112, 173–184.

  Article  Google Scholar 

• Pellizzoni, S., Siegal, M., & Surian, L. (2010). The contact principle and utilitarian moral judgments in young children. Developmental Science,13(2), 265–270.

  Article  Google Scholar 

• Podschwadek, F. (2017). Do androids dream of normative endorsement? On the fallibility of artificial moral agents. Artificial Intelligence and Law,25(3), 325–339.

  Article  Google Scholar 

• Reig, S., Norman, S., Morales, C. G., Das, S., Steinfeld, A., & Forlizzi, J. (2018). A field study of pedestrians and autonomous vehicles. In Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications (pp. 198–209). ACM.

• Rodríguez, L. F., & Ramos, F. (2014). Development of computational models of emotions for autonomous agents: A review. Cognitive Computation,6(3), 351–375.

  Article  Google Scholar 

• Schaich Borg, J., Hynes, C., Van Horn, J., Grafton, S., & Sinnott-Armstrong, W. (2006). Consequences, action, and intention as factors in moral judgments: An fMRI investigation. Journal of Cognitive Neuroscience,18(5), 803–817.

  Article  Google Scholar 

• Scheutz, M., & Malle, B. F. (2014). Think and do the right thing: A plea for morally competent autonomous robots. In Proceedings of the IEEE 2014 international symposium on ethics in engineering, science, and technology (p. 9). IEEE Press.

• Schroeder, M. (2017). Normative ethics and metaethics. In T. McPherson, & D. Plunkett (Eds.), The Routledge handbook of metaethics (pp. 674–686). London: Routledge.

  Chapter  Google Scholar 

• Sharkey, A., & Sharkey, N. (2012). Granny and the robots: Ethical issues in robot care for the elderly. Ethics and Information Technology,14(1), 27–40.

  Article  Google Scholar 

• Shigemi, S. (2018). ASIMO and humanoid robot research at Honda. In A. Goswami, & P. Vadakkepat (Eds.), Humanoid robotics: A reference (pp. 1–36). Springer.

• Tikhanoff, V., Cangelosi, A., & Metta, G. (2011). Integration of speech and action in humanoid robots: iCub simulation experiments. IEEE Transactions on Autonomous Mental Development,3(1), 17–29.

  Article  Google Scholar 

• Trafton, G., Hiatt, L., Harrison, A., Tamborello, F., Khemlani, S., & Schultz, A. (2013). ACT-R/E: An embodied cognitive architecture for human–robot interaction. Journal of Human–Robot Interaction,2(1), 30–55.

  Article  Google Scholar 

• Van Riemsdijk, M. B., Jonker, C.M., & Lesser, V. (2015). Creating socially adaptive electronic partners: Interaction, reasoning and ethical challenges. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems (pp. 1201–1206). International Foundation for Autonomous Agents and Multiagent Systems.

• Van Staveren, I. (2007). Beyond utilitarianism and deontology: Ethics in economics. Review of Political Economy,19(1), 21–35.

  Article  Google Scholar 

• Van Wynsberghe, A., & Robbins, S. (2018). Critiquing the reasons for making artificial moral agents. Science and Engineering Ethics,25(3), 1–17.

  Google Scholar 

• Vanderelst, D., & Winfield, A. (2018). An architecture for ethical robots inspired by the simulation theory of cognition. Cognitive Systems Research,48, 56–66.

  Article  Google Scholar 

• Vernon, D., Metta, G., & Sandini, G. (2007). A survey of artificial cognitive systems: Implications for the autonomous development of mental capabilities in computational agents. IEEE Transactions on Evolutionary Computation,11(2), 151–180.

  Article  Google Scholar 

• Viroli, M., Pianini, D., Montagna, S., & Stevenson, G. (2012). Pervasive ecosystems: A coordination model based on semantic chemistry. In Proceedings of the 27th annual ACM symposium on applied computing (pp. 295–302). ACM.

• Von der Pfordten, D. (2012). Five elements of normative ethics—A general theory of normative individualism. Ethical Theory and Moral Practice,15(4), 449–471.

  Article  Google Scholar 

• Von Wright, G. H. (1951). Deontic logic. Mind,60(237), 1–15.

  Article  Google Scholar 

• Waldrop, M. M. (2015). Autonomous vehicles: No drivers required. Nature News,518(7537), 20.

  Article  Google Scholar 

• Walker, L. J., & Hennig, K. H. (2004). Differing conceptions of moral exemplarity: Just, brave, and caring. Journal of Personality and Social Psychology,86(4), 629–647.

  Article  Google Scholar 

• Wallach, W. (2008). Implementing moral decision making faculties in computers and robots. AI & Society,22(4), 463–475.

  Article  Google Scholar 

• Wallach, W. (2010). Robot minds and human ethics: The need for a comprehensive model of moral decision making. Ethics and Information Technology,12(3), 243–250.

  Article  Google Scholar 

• Wallach, W., Allen, C., & Smit, I. (2008). Machine morality: Bottom-up and top-down approaches for modelling human moral faculties. AI & Society,22(4), 565–582.

  Article  Google Scholar 

• Wallach, W., Franklin, S., & Allen, C. (2010). A conceptual and computational model of moral decision making in human and artificial agents. Topics in Cognitive Science,2(3), 454–485.

  Article  Google Scholar 

• Wang, S., Wan, J., Zhang, D., Li, D., & Zhang, C. (2016). Towards smart factory for industry 4.0: A self-organized multi-agent system with big data based feedback and coordination. Computer Networks,101, 158–168.

  Article  Google Scholar 

• Wellman, M. P., & Rajan, U. (2017). Ethical issues for autonomous trading agents. Minds and Machines,27(4), 609–624.

  Article  Google Scholar 

• Winfield, A. F., Blum, C., & Liu, W. (2014). Towards an ethical robot: Internal models, consequences and ethical action selection. In Conference towards autonomous robotic systems (pp. 85–96). Cham: Springer.

• Yampolskiy, R. V. (2013). Artificial intelligence safety engineering: Why machine ethics is a wrong approach. In V. Müller (Ed.), Philosophy and theory of artificial intelligence (pp. 389–396). Berlin: Springer.

  Chapter  Google Scholar 

• Young, L., & Durwin, A. (2013). Moral realism as moral motivation: The impact of meta-ethics on everyday decision-making. Journal of Experimental Social Psychology,49(2), 302–306.

  Article  Google Scholar 

• Zambonelli, F., & Viroli, M. (2011). A survey on nature-inspired metaphors for pervasive service ecosystems. International Journal of Pervasive Computing and Communications,7(3), 186–204.

  Article  Google Scholar 

• Zieba, S., Polet, P., Vanderhaegen, F., & Debernard, S. (2010). Principles of adjustable autonomy: A framework for resilient human–machine cooperation. Cognition, Technology & Work,12(3), 193–203.

  Article  Google Scholar 

Download references