Conversation-based hybrid UI for the repertory grid technique: : A lab experiment into automation of qualitative surveys
Graphical abstract
Display Omitted
References
[1]
Ahn S., Hybrid User Interfaces: Design Guidelines and Implementation Examples, 2006.
[2]
Amaratunga D., Baldry D., Sarshar M., Newton R., Quantitative and qualitative research in the built environment: Application of “mixed” research approach, Work Study 51 (1) (2002) 17–31,.
[3]
Anon D., The future of chatbots, 2022, URL: https://www.tidio.com/blog/chatbot-statistics/.
[4]
Anon D., OpenRepGrid-tools for repertory grids, 2022, URL: https://openrepgrid.org/.
[5]
Ashktorab Z., Jain M., Vera Liao Q., Weisz J.D., Resilient chatbots: Repair strategy preferences for conversational breakdowns, in: Conference on Human Factors in Computing Systems - Proceedings, 2019,.
[6]
Avula S., Chadwick G., Arguello J., Capra R., Searchbots: User engagement with chatbots during collaborative search, in: CHIIR 2018 - Proceedings of the 2018 Conference on Human Information Interaction and Retrieval, Vol. 2018-March, 2018, pp. 52–61,.
[7]
Baez M., Daniel F., Casati F., Benatallah B., Chatbot integration in few patterns, IEEE Internet Comput. 25 (3) (2020) 52–59.
[8]
Bavaresco R., Silveira D., Reis E., Barbosa J., Righi R., Costa C., Antunes R., Gomes M., Gatti C., Vanzin M., Junior S.C., Silva E., Moreira C., Conversational agents in business: A systematic literature review and future research directions, Comp. Sci. Rev. 36 (2020),.
[9]
Bell R.C., Theory-appropriate analysis of repertory grid data, Int. J. Pers. Constr. Psychol. 1 (1) (1988) 101–118,.
[10]
Bender E.M., Friedman B., Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science, Trans. Assoc. Comput. Linguist. 6 (2018) 587–604,.
[11]
Bernard T., Flitman A., Using repertory grid analysis to gather qualitative data for information systems research, 2002.
[12]
Borsci S., Malizia A., Schmettow M., Van Der Velde F., Tariverdiyeva G., Balaji D., Chamberlain A., The Chatbot Usability Scale: The design and pilot of a usability scale for interaction with AI-based conversational agents, Pers. Ubiquitous Comput. 26 (2022) 95–119.
[13]
Borsci S., Schmettow M., Malizia A., Chamberlain A., Van Der Velde F., A confirmatory factorial analysis of the Chatbot Usability Scale: A multilanguage validation, Pers. Ubiquitous Comput. 27 (2) (2023) 317–330.
[14]
Braun V., Clarke V., Boulton E., Davey L., McEvoy C., The online survey as a qualitative research tool, Int. J. Soc. Res. Methodol. 24 (6) (2021) 641–654,. URL: https://www.tandfonline.com/doi/full/10.1080/13645579.2020.1805550.
[15]
Brennan S.E., Conversation as direct manipulation: An iconoclastic view, in: The Art of Human-Computer Interface Design, Addison-Wesley, Reading, Mass, 1990, pp. 393–404.
[16]
Budiu R., Laubheimer P., Intelligent Assistants Have Poor Usability: A User Study of Alexa, Google Assistant, and Siri, Nielsen Norman Group, 2018, URL: https://www.nngroup.com/articles/intelligent-assistant-usability/.
[17]
Celino I., Re Calegari G., Submitting surveys via a conversational interface: An evaluation of user acceptance and approach effectiveness, Int. J. Hum. Comput. Stud. 139 (2020),.
[18]
Edwards H.M., McDonald S., Michelle Young S., The repertory grid technique: Its place in empirical software engineering research, Inf. Softw. Technol. 51 (4) (2009) 785–798,.
[19]
Federici S., de Filippis M.L., Mele M.L., Borsci S., Bracalenti M., Gaudino G., Cocco A., Amendola M., Simonetti E., Inside pandora’s box: A systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs, Disabil. Rehabil.: Assist. Technol. 15 (7) (2020) 832–837.
[20]
Fritzen-Pedicini C., Bleasdale S.C., Brosseau L.M., Moritz D., Sikka M., Stiehl E., Jones R.M., Utilizing the focused conversation method in qualitative public health research: A team-based approach, BMC Health Serv. Res. 19 (1) (2019) 306,. URL: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-019-4107-0.
[21]
Gains N., The repertory grid approach, in: Measurement of Food Preferences, 1994, pp. 51–76,.
[22]
George D., Mallery P., IBM SPSS Statistics 26 Step by Step: a Simple Guide and Reference, Routledge, 2019, pp. 241–244.
[23]
Gozalo-Brizuela R., Garrido-Merchan E.C., ChatGPT is not all you need. a state of the art review of large generative AI models, 2023, arXiv preprint arXiv:2301.04655.
[24]
Grudin J., Jacques R., Chatbots, humbots, and the quest for artificial general intelligence, in: Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2019,.
[25]
Guest G., MacQueen K., Namey E., Applied Thematic Analysis, SAGE Publications, 2012, URL: https://books.google.com/books?id=VuWrexznC7sC.
[26]
Gullickson T., Review of Proposals that Work: A Guide for Planning Dissertations and Grant Proposals, Vol. 39, no. 11, Sage, 1994,.
[27]
Heckmann M., Burk L., Gridsampler – A simulation tool to determine the required sample size for repertory grid studies, J. Open Res. Softw. 5 (1) (2017) 2,.
[28]
Holmes S., Moorhead A., Bond R., Zheng H., Coates V., McTear M., Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?, in: ECCE 2019 - Proceedings of the 31st European Conference on Cognitive Ergonomics: “Design for Cognition”, 2019, pp. 207–214,.
[29]
Jankowicz D., THe Easy Guide to Repertory Grids, John wiley & sons, 2005.
[30]
Kim S., Lee J., Gweon G., Comparing data from chatbot and web surveys, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 2019, pp. 1–12,.
[31]
Kirk D., Blincoe K., Challenges when applying repertory grid technique for software practices, in: ACM International Conference Proceeding Series, Association for Computing Machinery, 2021, pp. 231–240,.
[32]
Kitson-Boyce R., Wheatley R., Blagden N., The application of the repertory grid in forensic practice, J. Constructivist Psychol. 35 (2) (2022) 677–698,. URL: https://www.tandfonline.com/action/journalInformation?journalCode=upcy20.
[33]
Kohavi R., Tang D., Xu Y., Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
[34]
Kumar, R., 2019. Data-Driven Design: Beyond A/B Testing. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. pp. 1–2.
[35]
Lambert R., The Repertory Grid as a Qualitative Interviewing Technique for Use in Survey Development, 1997.
[36]
Lau, T., Cerruti, J., Manzato, G., Bengualid, M., Bigham, J.P., Nichols, J., 2010. A conversational interface to web automation. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. pp. 229–238.
[37]
Li C.H., Chen K., Chang Y.J., When there is no progress with a task-oriented chatbot: A conversation analysis, in: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI 2019, Association for Computing Machinery, Inc, 2019,.
[38]
Li X., Lipton Z.C., Dhingra B., Li L., Gao J., Chen Y.-N., A user simulator for task-completion dialogues, 2016, arXiv preprint arXiv:1612.05688.
[39]
Li J., Zhou M.X., Yang H., Mark G., Confiding in and listening to virtual agents: The effect of personality, in: International Conference on Intelligent User Interfaces, Proceedings IUI, ACM, New York, NY, USA, 2017, pp. 275–286,. URL: https://dl.acm.org/doi/10.1145/3025171.3025206.
[40]
Lister, K., Coughlan, T., Iniesto, F., Freear, N., Devine, P., 2020. Accessible conversational user interfaces: Considerations for design. In: Proceedings of the 17th International Web for All Conference. pp. 1–11.
[41]
Liu Y., Martens J.B., Conversation-based hybrid user interface for structured qualitative survey: A pilot study using repertory grid, in: Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2022,.
[42]
Marr D., Vision: A computational investigation of visual representation in man, Phenomenol. Cogn. Sci. 8 (4) (1982) 397. http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=12242&ref=nf.
[43]
Martens J.B., Comparing experimental conditions using modern statistics, Behav. Res. Methods 53 (3) (2021) 1240–1261,. URL: https://link.springer.com/article/10.3758/s13428-020-01471-8.
[44]
Meersseman E., Vermeir I., Geuens M., The effect of perspectives in food pictures on unhealthy food choices, Food Qual. Pref. 89 (2021),.
[45]
Meredith J., Analysing technological affordances of online interactions using conversation analysis, J. Pragmat. 115 (2017) 42–55,.
[46]
Moore R.J., Arar R., Ren G.J., Szymanski M.H., Conversational UX design, in: Conference on Human Factors in Computing Systems - Proceedings, Vol. Part F1276, 2017, pp. 492–497,.
[47]
Mozafari N., Weiger W.H., Hammerschmidt M., The chatbot disclosure dilemma: Desirable and undesirable effects of disclosing the non-human identity of chatbots, in: International Conference on Information Systems, ICIS 2020 - Making Digital Inclusive: Blending the Local and the Global, 2021, URL: https://aisel.aisnet.org/icis2020/hci%5Fartintel/hci%5Fartintel/6.
[48]
Nielsen J., Landauer T.K., A mathematical model of the finding of usability problems, in: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, Association for Computing Machinery, New York, NY, USA, 1993, pp. 206–213,.
[49]
Nuruzzaman M., Hussain O.K., A survey on chatbot implementation in customer service industry through deep neural networks, in: Proceedings - 2018 IEEE 15th International Conference on E-Business Engineering, ICEBE 2018, IEEE, 2018, pp. 54–61,. URL: https://ieeexplore.ieee.org/document/8592630/.
[50]
Open J.S. Foundation M., About: Node-RED, 2021, URL: https://nodered.org/about/.
[51]
OpenAI M., GPT-4 technical report, 2023, arXiv:2303.08774.
[52]
Pantsar M., Cognitive and Computational Complexity: Considerations from Mathematical Problem Solving, Vol. 86, no. 4, Springer, Netherlands, 2021,.
[53]
Rapp A., Curti L., Boldi A., The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots, Int. J. Hum. Comput. Stud. 151 (2021),.
[54]
Revang M., Mullen A., Magic Quadrant for Enterprise Conversational AI Platforms, 2022, URL: https://www.gartner.com/doc/reprints?id=1-28XJAIXS&ct=220128&st=sb&%5F%5Fhstc=123545108.8bdd386aec815b71400a88628eca825d.16514957745.
[55]
Schrepp M., Thomaschewski J., Hinderks A., Construction of a benchmark for the user experience questionnaire (UEQ), Int. J. Interact. Multimed. Artif. Intell (2017).
[56]
Stein J.P., Appel M., Jost A., Ohler P., Matter over mind? How the acceptance of digital entities depends on their appearance, mental prowess, and the interaction between both, Int. J. Hum. Comput. Stud. 142 (April 2019) (2020),.
[57]
Stevens K.A., The vision of david marr, Perception 41 (9) (2012) 1061–1072.
[58]
Sullivan L.M., Repeated measures, Circulation 117 (9) (2008) 1238–1243.
[59]
SurveyMonkey L.M., SurveyMonkey: Free online survey software & questionnaire tool, 2013, URL: http://www.surveymonkey.net/.
[60]
Taber K.S., The use of Cronbach’s alpha when developing and reporting research instruments in science education, Res. Sci. Educ. 48 (2018) 1273–1296.
[61]
Tamilmani K., Rana N.P., Wamba S.F., Dwivedi R., The extended unified theory of acceptance and use of technology (UTAUT2): A systematic literature review and theory evaluation, Int. J. Inf. Manage. 57 (2021),. URL: https://linkinghub.elsevier.com/retrieve/pii/S0268401220314687.
[62]
Thoppilan R., Freitas D.D., Hall J., Shazeer N., Kulshreshtha A., Cheng H.-T., Jin A., Bos T., Baker L., Du Y., Li Y., Lee H., Zheng H.S., Ghafouri A., Menegali M., Huang Y., Krikun M., Lepikhin D., Qin J., Chen D., Xu Y., Chen Z., Roberts A., Bosma M., Zhao V., Zhou Y., Chang C.-C., Krivokon I., Rusch W., Pickett M., Srinivasan P., Man L., Meier-Hellstern K., Morris M.R., Doshi T., Santos R.D., Duke T., Soraker J., Zevenbergen B., Prabhakaran V., Diaz M., Hutchinson B., Olson K., Molina A., Hoffman-John E., Lee J., Aroyo L., Rajakumar R., Butryna A., Lamm M., Kuzmina V., Fenton J., Cohen A., Bernstein R., Kurzweil R., Aguera-Arcas B., Cui C., Croak M., Chi E., Le Q., LaMDA: Language models for dialog applications, 2022, arXiv:2201.08239.
[63]
Turner C.W., Lewis J.R., Nielsen J., Determining usability test sample size, in: International Encyclopedia of Ergonomics and Human Factors, Vol. 3, no. 2, CRC Press Boca Raton, FL, 2006, pp. 3084–3088.
[64]
van Loggem B., Using the repertory grid technique for mining design patterns, ACM International Conference Proceeding Series, vol. 08-12-July, ACM, New York, NY, USA, 2015,.
[65]
Venkatesh V., Thong J.Y., Xu X., Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology, MIS Q.: Manag. Inf. Syst. 36 (1) (2012) 157–178,.
[66]
Vue V., Introduction – vue.js, 2019, Vue Docs, URL: https://vuejs.org/v2/guide/.
[67]
Weber I., Low-code from frontend to backend: Connecting conversational user interfaces to backend services via a low-code IoT platform, in: ACM International Conference Proceeding Series, 2021,.
[68]
Wen M.H., A conversational user interface for supporting individual and group decision-making in stock investment activities, in: Proceedings of 4th IEEE International Conference on Applied System Innovation 2018, ICASI 2018, IEEE, 2018, pp. 216–219,. URL: https://ieeexplore.ieee.org/document/8394571/.
[69]
Wright K.B., Researching internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services, J. Comput.-Mediated Commun. 10 (3) (2005),. URL: https://academic.oup.com/jcmc/article/4614509.
[70]
Yeh S.F., Wu M.H., Chen T.Y., Lin Y.C., Chang X.J., Chiang Y.H., Chang Y.J., How to guide task-oriented chatbot users, and when: A mixed-methods study of combinations of chatbot guidance types and timings, in: Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2022,.
[71]
Yuan S., Brüggemeier B., Hillmann S., Michael T., User preference and categories for error responses in conversational user interfaces, in: ACM International Conference Proceeding Series, ACM, New York, NY, USA, 2020,.
[72]
Zimmerman J., Oh C., Yildirim N., Kass A., Tung T., Forlizzi J., UX designers pushing AI in the enterprise: A case for adaptive UIs, Interactions 28 (1) (2020) 72–77,. URL: http://library1.nida.ac.th/termpaper6/sd/2554/19755.pdf.
Recommendations
Conversation-Based Hybrid User Interface for Structured Qualitative Survey: A Pilot Study Using Repertory Grid
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing SystemsConversational user interfaces (CUIs) have the potential to substantially influence current practice in diverse application fields, one important example being the way in which online surveys are conducted. Currently, CUIs are mainly applied to help ...
Blending the repertory grid technique with focus groups to reveal rich design relevant insight
DPPI '13: Proceedings of the 6th International Conference on Designing Pleasurable Products and InterfacesIn the context of better understanding and describing the role of modality in data representations, this paper proposes the blending of the Repertory Grid Technique, a proven and tested elicitation method, with focus group methods, thereby offering ...
Challenges when Applying Repertory Grid Technique for Software Practices
EASE '21: Proceedings of the 25th International Conference on Evaluation and Assessment in Software EngineeringThe Repertory Grid Technique (RGT) has been applied within the software engineering domain to investigate a variety of topics. These include topics relating to architectural knowledge, team level tacit knowledge, and project success mechanisms. The ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
The Authors.
Publisher
Academic Press, Inc.
United States
Publication History
Published: 01 April 2024
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025