Abstract
In this paper, we have highlighted the increasing need for standard metrics to assess and measure the quality of responses produced by conversational agent systems based on different approaches and domains. To demonstrate the approach, we will present the data and results obtained through an evaluation performed on three different conversation systems, namely Eliza (Rogerian Psychotherapist), ALICE (Artificial Linguistic Internet Computer Entity) and AINI (Artificial Intelligent Natural-Language Identity).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
AINI’s research on “Intelligent Agent Technology in E-commerce”, Intelligent Data Engineering and Automated Learning, vol. 2690, pp 10-17, 2003 has been cited by US Pattent examiner for the Patent no US8639638, 2014, http://www.google.com/patents/US8639638.
- 4.
- 5.
In this experiment, Dell Precision PWS380 Server 3 GH with 1 GB of memory was used.
- 6.
The dependency graphs generated using MINIPAR, CMU Link Grammar and Stanford parser parse visualisation tool which can be obtained at http://cgi.stanford.edu.
References
Goh, O.S., Abd Ghani, M.K., Yogan, J.K., Choo, Y.H., Muda, A.K.: Massive Open Online Course (MOOC) with learning objects and intelligent agent technologies. In: IEEE International Conference on IT Convergence and Security, pp. 125–128, Beijing (2014)
Goh, O.S., Depickere, A., Fung, C.C.: Domain knowledge query conversation bots in instant messaging. Knowl.-Based Syst. 21, 681–691 (2008)
Fodor, J.A.: Elm and the Expert: An Introduction to Mentalese and Its Semantics. Cambridge University Press, Cambridge (1994)
McKeown, K.R.: Text Generation. Cambridge University Press, Cambridge (1995)
Moore, J.D., Paris, C.L.: Planning text for advisory dialogues: capturing intentional and rhetorical information. Comput. Linguist. 19, 651–695 (1992)
Radev, D., McKeown, K.: Generating natural language summaries from multiple on-line sources. Comput. Linguist. 24, 469–500 (1998)
Reiter, E., Dale, R.: Building Natural Language Generation Systems. Cambridge University Press, Cambridge (2000)
Winograd, T.: Procedures as a representation for data in a computer program for understanding natural language. Cogn. Psychol. 3(1), 1–191 (1972)
Brooks, R.A., Breazeal, C., Marjanovic, M., Scassellati, B., Williamson, M.M.: The cog project: building a humanoid robot. In: Nehaniv, C.L. (ed.) CMAA 1998. LNCS (LNAI), vol. 1562, pp. 52–87. Springer, Heidelberg (1999)
McCarthy, J.: An example for natural language understanding and the AI problems it raises. Stanford University, 17 November 2015. http://jmc.standford.edu/articles/mrhug.html
Dreyfus, H.L.: What computers still can’t do – a critique of artificial reason. MIT Press, New York (1992)
Newell, A., Shaw, J.C., Simon, H.A.: Report on a general problem–solving program. In: International Conference on Infomnation Processing, pp. 256–264. NESCO House, Paris (1959)
Goh, O.S., Depickere, A., Fung, C.C., Wong, W.: Top-down natural language query approach for embodied conversational agent. In: International Multi Conference of Engineers and Computer Scientists (IMECS 2006), Hong Kong (2006)
Lin, D.: Dependency-based evaluation of MINIPAR. In: Abeillé, A. (ed.) 1st International Conference on Language Resources and Evaluation. Springer, The Netherlands (1998)
Goh, O.S., Fung, C.C., Wong, W., Depickere, A.: An embodied conversational agent for intelligent web interaction on pandemic crisis communication. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IATW 2006), Hong Kong (2006)
Sleator, D., Temperley, D.: Parsing English with a link grammar. Carnegie Mellon University Computer Science Technical report CMU-CS-91–196 (1991)
Klein, D., Manning, C.: Accurate unlexicalized parsing. In: 41st Annual Meeting of the Association for Computational Linguistics, pp. 423–430 (2003)
Baroni, M., Ueyama, M.: BootCaT: bootstrapping corpora and terms from the web. In: Fourth Language Resources and Evaluation Conference (2004)
Zheng, Z.: AnswerBus question answering system. In: Conference on Human Language Technology (2002)
ALICE: ALICE Loebner prize winner transcripts. Loebner Prize Homepage, 17 November 2014. http://loebner.net/Prizef/2005_Contest/Alice/
Weizenbaum, J.: ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 36–45 (1966)
Wallace, R.S.: The anatomy of A.L.I.C.E. Artificial Intelligence Foundation, Inc., 17 November 2014. http://www.alicebot.org/anatomy.html
ALICE: ALICE Silver Edition. ALICE Foundation, 20 August 2014. http://www.alicebot.org/join.html
Acknowledgements
This research is supported and funded through Top Down PJP/2014/FTMK(11B)/S01371 grant, awarded by the Centre for Research and Innovation Management (CRIM), Universiti Teknikal Malaysia Melaka (UTeM). The MeetInventorTM was first trademark (2015002886) and copyright (No LY2015000274) under the Intellectual Property Corporation of Malaysia, 2015. AINI Conversation Agent can be accessed from http://meetinventor.utem.edu.my.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Goh, O.S., Kumar, Y.J., Choon, N.H., Leong, P.H., Safar, M. (2016). An Evaluation of the Conversation Agent System. In: Nguyen, N.T., Trawiński, B., Fujita, H., Hong, TP. (eds) Intelligent Information and Database Systems. ACIIDS 2016. Lecture Notes in Computer Science(), vol 9622. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-49390-8_35
Download citation
DOI: https://doi.org/10.1007/978-3-662-49390-8_35
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-49389-2
Online ISBN: 978-3-662-49390-8
eBook Packages: Computer ScienceComputer Science (R0)