research-article

The oz of wizard: simulating the human for interaction research

Authors:

Aaron Steinfeld,

Odest Chadwicke Jenkins,

Brian ScassellatiAuthors Info & Claims

HRI '09: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction

Pages 101 - 108

https://doi.org/10.1145/1514095.1514115

Published: 09 March 2009 Publication History

Abstract

The Wizard of Oz experiment method has a long tradition of acceptance and use within the field of human-robot interaction. The community has traditionally downplayed the importance of interaction evaluations run with the inverse model: the human simulated to evaluate robot behavior, or Oz of Wizard. We argue that such studies play an important role in the field of human-robot interaction. We differentiate between methodologically rigorous human modeling and placeholder simulations using simplified human models. Guidelines are proposed for when Oz of Wizard results should be considered acceptable. This paper also describes a framework for describing the various permutations of Wizard and Oz states.

References

[1]

Kelley, J. F. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst. 2, 1 (Jan. 1984), 26--41. DOI= http://doi.acm.org/10.1145/357417.357420

Digital Library

[2]

Shiomi, M., Kanda, T., Koizumi, S., Ishiguro, H., and Hagita, N. 2007. Group attention control for communication robots with wizard of OZ approach. In Proceedings of the ACM/IEEE international Conference on Human-Robot interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 121--128. DOI= http://doi.acm.org/10.1145/1228716.1228733

Digital Library

[3]

Kahn, P. H., Freier, N. G., Kanda, T., Ishiguro, H., Ruckert, J. H., Severson, R. L., and Kane, S. K. 2008. Design patterns for sociality in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 97--104. DOI= http://doi.acm.org/10.1145/1349822.1349836

Digital Library

[4]

Shiomi, M., Sakamoto, D., Takayuki, K., Ishi, C. T., Ishiguro, H., and Hagita, N. 2008. A semi-autonomous communication robot: a field trial at a train station. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 303--310. DOI= http://doi.acm.org/10.1145/1349822.1349862

Digital Library

[5]

Trafton, J. G., Schultz, A. C., Perznowski, D., Bugajska, M. D., Adams, W., Cassimatis, N. L., and Brock, D. P. 2006. Children and robots learning to play hide and seek. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 242--249. DOI= http://doi.acm.org/10.1145/1121241.1121283

Digital Library

[6]

Granda, T., Kirkpatrick, M., Julien, T., and Peterson, L. 1990. The evolutionary role of humans in the human-robot system. In Proc. Human Factors Society 34th Annual Meeting. 664--668.

[7]

Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., and Goodrich, M. 2006. Common metrics for human-robot interaction. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 33--40. DOI= http://doi.acm.org/10.1145/1121241.1121249

Digital Library

[8]

Forsyth, D. and Ponce, J. 2003. Computer Vision A Modern Approach, Prentice Hall.

Digital Library

[9]

Jenkins, O., González, G., and Loper, M. 2007. Interactive human pose and action recognition using dynamical motion primitives. International Journal of Humanoid Robotics, 4(2):365--385.

[10]

Grollman, D. H., and Jenkins, O. C. 2008. Sparse incremental learning for interactive robot control policy estimation. In International Conference on Robotics and Automation (ICRA, Pasadena, CA, USA, May 2008).

[11]

Chernova, S. and Veloso, M. 2008. Multi-thresholded approach to demonstration selection for interactive robot learning. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 225--232. DOI= http://doi.acm.org/10.1145/1349822.1349852

Digital Library

[12]

Doshi, F. and Roy, N. 2007. Efficient model learning for dialog management. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 65--72. DOI= http://doi.acm.org/10.1145/1228716.1228726

Digital Library

[13]

Schmidt-Rohr, S. R., Knoop, S., Lösch, M., and Dillmann, R. 2008. Reasoning for a multi-modal service robot considering uncertainty in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 249--254. DOI= http://doi.acm.org/10.1145/1349822.1349855

Digital Library

[14]

Vondrak, M., Sigal, L., and Jenkins, O. 2008. Physical simulation for probabilistic motion tracking. In Computer Vision and Pattern Recognition (CVPR, Anchorage, AK, USA, Jun 2008).

[15]

Srinivasa, S., Ferguson, D., Vande Weghe, M., Diankov, R., Berenson, D., Helfrich, C. and Strasdat, H. 2008. The Robotic Busboy: steps towards developing a mobile robotic home assistant. In International Conference on Intelligent Autonomous Systems.

[16]

Humphrey, C. M. and Adams, J. A. 2008. Compass visualizations for human-robotic interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 49--56. DOI= http://doi.acm.org/10.1145/1349822.1349830

Digital Library

[17]

Yamaoka, F., Kanda, T., Ishiguro, H., and Hagita, N. 2008. How close?: model of proximity control for information-presenting robots. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 137--144. DOI= http://doi.acm.org/10.1145/1349822.1349841

Digital Library

[18]

Hoffman, G. and Breazeal, C. 2007. Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team. In Proc. of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10-12, 2007). ACM, New York, NY, 1-8. DOI= http://doi.acm.org/10.1145/1228716.1228718

Digital Library

[19]

Gemperle, F., DiSalvo, C., Forlizzi, J., and Yonkers, W. 2003. The Hug: a new form for communication. In Proceedings of the 2003 Conference on Designing For User Experiences (San Francisco, California, June 06 - 07, 2003). DUX '03. ACM, New York, NY, 1--4. DOI= http://doi.acm.org/10.1145/997078.997103

Digital Library

[20]

Takayama, L., Ju, W., and Nass, C. 2008. Beyond dirty, dangerous and dull: what everyday people think robots should do. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12-15, 2008). HRI '08. ACM, New York, NY, 25--32. DOI= http://doi.acm.org/10.1145/1349822.1349827

Digital Library

[21]

Scholtz, J., Young, J., Drury, J., and Yanco, H. 2004. Evaluation of human-robot interaction awareness in search and rescue. In Proc. IEEE International Conference on Robotics and Automation.

[22]

Casper, J., and Murphy, R. 2003. Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics B, 33.

Digital Library

[23]

Mutlu, B. and Forlizzi, J. 2008. Robots in organizations: the role of workflow, social, and environmental factors in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 287--294. DOI= http://doi.acm.org/10.1145/1349822.1349860

Digital Library

[24]

Krishnan, H., Gibb, S., Steinfeld, A., and Shladover, S. E. 2001. Rear-end collision-warning system: Design and evaluation via simulation, Transportation Research Record - Journal of the Transportation Research Board 1759, 52--60.

[25]

Bainbridge, W., Hart, J., Kim, E. and Scassellati, B. 2008. The effect of presence on human-robot interaction. In IEEE International Symposium on Robot and Human Interactive Communication (Munich, Germany).

[26]

Tapus, A., Mataric, M. and Scassellati, B. 2007. The grand challenges in socially assistive robotics. IEEE Robotics and Automation Magazine. Vol. 4, No. 1. p. 35--427.

[27]

Jenkins, O. C., González, G., and Loper, M. M. 2007. Tracking human motion and actions for interactive robots. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 365--372. DOI= http://doi.acm.org/10.1145/1228716.1228765

Digital Library

[28]

Koenig, N., Chernova, S., Jones, C., Loper, M., and Jenkins, O. 2008. Hands-free interaction for human-robot teams. In ICRA 2008 Workshop on Social Interaction with Intelligent Indoor Robots (Pasadena, CA, USA, May 2008).

[29]

Koenig, N., Chernova, S., Jones, C., Loper, M., and Jenkins, O. 2008. Hands-free human-robot interaction. In HRI Caught on Film 2, Proc. of the 3rd ACM/IEEE International Conference on Human Robot Interaction. ACM, New York, NY, 383--388.

Digital Library

[30]

Gold, K. and Scassellati, B. 2006. Using context and sensory data to learn first and second person pronouns. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 110--117. DOI= http://doi.acm.org/10.1145/1121241.1121262

Digital Library

[31]

Gold K. and Scassellati, B. 2006. Grounded pronoun learning and pronoun reversal. In 5th International Conference on Development and Learning (ICDL-06, Bloomington, IN).

[32]

Gold, K., Doniec, M., and Scassellati, B. 2007. Learning grounded semantics with word trees: prepositions and pronouns. In Proceedings of the 6th IEEE International Conference on Development and Learning (ICDL, London, England, July 2007).

[33]

Gold, K. and Scassellati, B. 2007. A robot that uses existing vocabulary to infer non-visual word meanings from observation. In Proceedings of the Twenty-Second Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI. Vancouver, BC, Canada, August, 2007).

Digital Library

[34]

MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk, 3rd ed. Mahwah, NJ: Lawrence Erlbaum Associates.

[35]

Gold, K. (2008). Using Sentence Context and Implicit Contrast to Learn Sensor-Grounded Meanings for Relational and Deictic Words: The TWIG System. Yale University, May 2008.

Digital Library

Cited By

Sorbello RCalí C(2024)Perspective Chapter: A Model for Measuring Trust Using BCI in Human-Humanoid InteractionNew Insights in Brain-Computer Interface Systems [Working Title]10.5772/intechopen.115094Online publication date: 1-Oct-2024
https://doi.org/10.5772/intechopen.115094
Kotsios HAsteriou PWintersberger P(2024)Only Trust a Hidden Wizard: Investigating the Effects of Wizard Visibility in Automotive Wizard of Oz StudiesProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675738(74-82)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675738
Ge XXu CMisaki DMarkus HTsai J(2024)How Culture Shapes What People Want From AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642660(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642660
Show More Cited By

Index Terms

The oz of wizard: simulating the human for interaction research
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. HCI theory, concepts and models
  2. Interaction design
    1. Interaction design theory, concepts and paradigms

Recommendations

Wizard of Oz studies in HRI: a systematic review and new reporting guidelines
Inaugural Special Issue: Intersection of Systems Sciences and Human Sciences

Many researchers use Wizard of Oz (WoZ) as an experimental technique, but there are methodological concerns over its use, and no comprehensive criteria on how to best employ it. We systematically review 54 WoZ experiments published in the primary HRI ...
Wizard of Oz experiments and companion dialogues
BCS '10: Proceedings of the 24th BCS Interaction Specialist Group Conference

Novel speech systems such as the conversational agents being developed by the Companions Project (www.companions-project.org) can be simulated using the Wizard of Oz methodology. In this approach technologies that are not yet ready for testing by people ...
Wizard of Oz experiments for companions
BCS-HCI '09: Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology

Wizard of Oz experiments allow designers and developers to see the reactions of people as they interact with to-be-developed technologies. At the Centre for Interaction Design at Edinburgh Napier University we are developing a Wizard of Oz system to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HRI '09: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction

March 2009

348 pages

ISBN:9781605584041

DOI:10.1145/1514095

General Chairs:
Matthias Scheutz
Indiana University, USA
,
François Michaud
Université de Sherbrooke, Canada
,
Program Chairs:
Pamela Hinds
Stanford University, USA
,
Brian Scassellati
Yale University, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

HRI09

Sponsor:

HRI09: International Conference on Human Robot Interaction

March 9 - 13, 2009

California, La Jolla, USA

Acceptance Rates

Overall Acceptance Rate 268 of 1,124 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

125
Total Citations
View Citations
1,644
Total Downloads

Downloads (Last 12 months)122
Downloads (Last 6 weeks)25

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sorbello RCalí C(2024)Perspective Chapter: A Model for Measuring Trust Using BCI in Human-Humanoid InteractionNew Insights in Brain-Computer Interface Systems [Working Title]10.5772/intechopen.115094Online publication date: 1-Oct-2024
https://doi.org/10.5772/intechopen.115094
Kotsios HAsteriou PWintersberger P(2024)Only Trust a Hidden Wizard: Investigating the Effects of Wizard Visibility in Automotive Wizard of Oz StudiesProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675738(74-82)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675738
Ge XXu CMisaki DMarkus HTsai J(2024)How Culture Shapes What People Want From AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642660(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642660
Huang JAronson RShort EGrollman DBroadbent EJu WSoh HWilliams T(2024)Modeling Variation in Human Feedback with User Inputs: An Exploratory MethodologyProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634925(303-312)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634925
Song HRamos GBodik P(2024)Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes2024 IEEE 17th Pacific Visualization Conference (PacificVis)10.1109/PacificVis60374.2024.00043(307-312)Online publication date: 23-Apr-2024
https://doi.org/10.1109/PacificVis60374.2024.00043
Zhao YMantha BKnijnenburg BLi D(2024)Exploring the feasibility of mobile robots for post-occupancy evaluationsBuilding Research & Information10.1080/09613218.2024.2419858(1-18)Online publication date: 30-Oct-2024
https://doi.org/10.1080/09613218.2024.2419858
Zou JGauthier SPellerin HGargot TArchambault DChetouani MCohen DAnzalone S(2024)R2C3, A Rehabilitation Robotic Companion for Children and Caregivers: The Collaborative Design of a Social Robot for Children with Neurodevelopmental DisordersInternational Journal of Social Robotics10.1007/s12369-024-01104-616:3(599-617)Online publication date: 16-Feb-2024
https://doi.org/10.1007/s12369-024-01104-6
Frey AJipp M(2024)Wizard-of-Oz-FahrzeugeHandbuch Assistiertes und Automatisiertes Fahren10.1007/978-3-658-38486-9_11(215-234)Online publication date: 31-Jul-2024
https://doi.org/10.1007/978-3-658-38486-9_11
Lang JJouen FTijus CUzan G(2024)Design of a Virtual Assistant: Collect of User’s Needs for Connected and Automated VehiclesHCI in Mobility, Transport, and Automotive Systems10.1007/978-3-031-60477-5_12(157-170)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60477-5_12
Seraj EXiong JSchrum MGombolay MOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Mixed-initiative multiagent apprenticeship learning for human training of robot teamsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667661(35426-35440)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667661
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents