Design Principles for Building Robust Human-Robot Interaction Machine Learning Models
Authors: 
Josh Bhagat Smith, Vivek Mallampati, Prakash Baskaran, Mark-Robin Giolando, Julie A. AdamsAuthors Info & Claims
Pages 247 - 251
Abstract
Effective collaboration between humans and robots hinges on the robot's ability to comprehend its human teammate. This collaboration demands the development of machine learning models that bridge the gap between human physiological signals and their mental states. However, the challenge lies in developing generalizable machine learning models using data collected in controlled experimental conditions. This manuscript proposes a set of principles for designing human subject evaluations, emphasizing the crucial balance between experimental control and ecological validity while also balancing fundamental machine learning trade-offs.
Supplemental Material
MP4 File
Supplemental video
- Download
- 48.22 MB
References
[1]
P. Baskaran and J. A. Adams. Multi-dimensional task recognition for human-robot teaming: literature review. Frontiers in Robotics and AI, 10, 2023.
[2]
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79:151--175, 2010.
[3]
C. L. Bethel and R. R. Murphy. Review of human studies methods in HRI and recommendations. International Journal of Social Robotics, 2(4):347--359, 2010.
[4]
J. Bhagat Smith, P. Baskaran, and J. A. Adams. Decomposed physical workload estimation for human-robot teams. In International Conference on Human-Machine Systems, pages 1--6. IEEE, 2022.
[5]
J. Bhagat Smith, S. A. Toribio, P. Baskaran, and J. A. Adams. Uncertainty-aware visual workload estimation for human-robot teams. In Conference on Cognitive and Computational Aspects of Situation Management, pages 1--8, 2023.
[6]
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321--357, 2002.
[7]
X.-W. Chen and X. Lin. Big data deep learning: challenges and perspectives. IEEE Access, 2:514--525, 2014.
[8]
S. Dowiasch, P. Wolf, and F. Bremmer. Quantitative comparison of a mobile and a stationary video-based eye-tracker. Behavior research methods, 52:667--680, 2020.
[9]
C. Elkan. The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence, volume 17, pages 973--978. Lawrence Erlbaum Associates Ltd, 2001.
[10]
M. Esterman, B. J. Tamber-Rosenau, Y.-C. Chiu, and S. Yantis. Avoiding non-independence in fMRI data analysis: Leave one subject out. Neuroimage, 50(2):572--576, 2010.
[11]
M. R. Fraune, I. Leite, N. Karatas, A. Amirova, A. Legeleux, A. Sandygulova, A. Neerincx, G. Dilip Tikas, H. Gunes, M. Mohan, N. I. Abbasi, S. Shenoy, B. Scassellati, E. J. de Visser, and T. Komatsu. Lessons learned about designing and conducting studies from HRI experts. Frontiers in Robotics and AI, 8:772141, 2021.
[12]
M. Ginesi, D. Meli, A. Roberti, N. Sansonetto, and P. Fiorini. Autonomous task planning and situation awareness in robotic surgery. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3144--3150. IEEE, 2020.
[13]
G. Hoffman and X. Zhao. A primer for conducting experiments in human--robot interaction. ACM Transactions on Human-Robot Interaction, 10(1), 2020.
[14]
G. A. Holleman, I. T. C. Hooge, C. Kemner, and R. S. Hessels. The "real-world approach' and its problems: A critique of the term ecological validity. Frontiers in Psychology, 11:721, 2020.
[15]
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436--444, 2015.
[16]
A. Luque, A. Carrasco, A. Martín, and A. de Las Heras. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 91:216--231, 2019.
[17]
G. Matthews, J. De Winter, and P. A. Hancock. What do subjective workload scales really measure? operational and representational solutions to divergence of workload measures. Theoretical Issues in Ergonomics Science, 21(4):369--396, 2020.
[18]
L. Mingyue Ma, T. Fong, M. J. Micire, Y. K. Kim, and K. Feigh. Human-robot teaming: Concepts and components for design. In Conference on Field and Service Robotics, pages 649--663. Springer, 2018.
[19]
K. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, Cambridge, MA, 2023.
[20]
S. Rangineni. An analysis of data quality requirements for machine learning development pipelines frameworks. International Journal of Computer Trends and Technology, 71, 2023.
[21]
L. D. Riek. Wizard of oz studies in hri: A systematic review and new reporting guidelines. Journal of Human-Robot Interaction, 1(1):119--136, 2012.
[22]
P. Schmidt, A. Reiss, R. Dürichen, and K. V. Laerhoven. Wearable-based affect recognition-a review. Sensors, 19(19):4079, 2019.
[23]
A. Vabalas, E. Gowen, E. Poliakoff, and A. J. Casson. Machine learning algorithm validation with a limited sample size. PloS one, 14(11):e0224365, 2019.
[24]
J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
[25]
E. Wascher, J. Reiser, G. Rinkenauer, M. Larrá, F. A. Dreger, D. Schneider, M. Karthaus, S. Getzmann, M. Gutberlet, and S. Arnau. Neuroergonomics on the go: An evaluation of the potential of mobile EEG for workplace assessment and design. Human Factors, 65(1):86--106, 2023.
[26]
M. B. Weinger, O. W. Herndon, M. H. Zornow, M. P. Paulus, D. M. Gaba, and L. T. Dallen. An objective methodology for task analysis and workload assessment in anesthesia providers. Anesthesiology, 80(1):77--92, 1994.
[27]
C. D. Wickens, S. E. Gordon, Y. Liu, and J. Lee. An Introduction to Human Factors Engineering, volume 2. Pearson Prentice Hall Upper Saddle River, NJ, 2004.
[28]
Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795--10816, 2023.
Index Terms
- Design Principles for Building Robust Human-Robot Interaction Machine Learning Models
Recommendations
Supervised autonomy for online learning in human-robot interaction
When a robot is learning it needs to explore its environment and how its environment responds on its actions. When the environment is large and there are a large number of possible actions the robot can take, this exploration phase can take ...
Human–robot collaboration and machine learning: A systematic review of recent research
AbstractTechnological progress increasingly envisions the use of robots interacting with people in everyday life. Human–robot collaboration (HRC) is the approach that explores the interaction between a human and a robot, during the completion ...
Highlights- Machine learning is assuming an important role in human–robot collaboration.
- ...
Graphical abstractDisplay Omitted
A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings
Highlights- Comprehensive review of development of adaptive collaborative robots.
- Novel ...
AbstractIncreased global competition has placed a premium on customer satisfaction, and there is a greater demand for manufacturers to be flexible with their products and services. This challenge is usually addressed with the introduction of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
1408 pages
ISBN:9798400703232
DOI:10.1145/3610978
- General Chairs:
- Dan Grollman,
- Elizabeth Broadbent,
- Program Chairs:
- Wendy Ju,
- Harold Soh,
- Tom Williams
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 March 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
HRI '24: ACM/IEEE International Conference on Human-Robot Interaction
March 11 - 15, 2024
CO, Boulder, USA
Acceptance Rates
Overall Acceptance Rate 268 of 1,124 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 142Total Downloads
- Downloads (Last 12 months)142
- Downloads (Last 6 weeks)37
Reflects downloads up to 16 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in