research-article

Brain Topography Adaptive Network for Satisfaction Modeling in Interactive Information Access System

Authors:

Shaoping MaAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 90 - 100

https://doi.org/10.1145/3503161.3548258

Published: 10 October 2022 Publication History

Abstract

With the growth of information on the Web, most users heavily rely on information access systems (e.g., search engines, recommender systems, etc.) in their daily lives. During this procedure, modeling users' satisfaction status plays an essential part in improving their experiences with the systems. In this paper, we aim to explore the benefits of using Electroencephalography (EEG) signals for satisfaction modeling in interactive information access system design. Different from existing EEG classification tasks, the arisen of satisfaction involves multiple brain functions, such as arousal, prototypicality, and appraisals, which are related to different brain topographical areas. Thus modeling user satisfaction raises great challenges to existing solutions. To address this challenge, we propose BTA, a Brain Topography Adaptive network with a multi-centrality encoding module and a spatial attention mechanism module to capture cognitive connectives in different spatial distances. We explore the effectiveness of BTA for satisfaction modeling in two popular information access scenarios, i.e., search and recommendation. Extensive experiments on two real-world datasets verify the effectiveness of introducing brain topography adaptive strategy in satisfaction modeling. Furthermore, we also conduct search result re-ranking task and video rating prediction task based on the satisfaction inferred from brain signals on search and recommendation scenarios, respectively. Experimental results show that brain signals extracted with BTA help improve the performance of interactive information access systems significantly.

Supplementary Material

MP4 File (MM22-fp2148.mp4)

Presentation video

Download
31.06 MB

References

[1]

Mashael Aldayel, Mourad Ykhlef, and Abeer Al-Nafjan. 2020. Deep learning for EEG-based preference classification in neuromarketing. Applied Sciences, Vol. 10, 4 (2020), 1525.

[2]

Rashid Ali and MM Sufyan Beg. 2011. An overview of Web search evaluation methods. Computers & Electrical Engineering, Vol. 37, 6 (2011), 835--848.

Digital Library

[3]

Xavier Amatriain, Josep M Pujol, and Nuria Oliver. 2009. I like it... i like it not: Evaluating user ratings noise in recommender systems. In International Conference on User Modeling, Adaptation, and Personalization. Springer, 247--258.

Digital Library

[4]

Thomas Baumgartner, Lilian Valko, Michaela Esslen, and Lutz J"ancke. 2006. Neural correlate of spatial presence in an arousing and noninteractive virtual reality: an EEG and psychophysiology study. CyberPsychology & Behavior, Vol. 9, 1 (2006), 30--45.

[5]

Daniel E Berlyne. 1970. Novelty, complexity, and hedonic value. Perception & psychophysics, Vol. 8, 5 (1970), 279--286.

[6]

Anjan Chatterjee. 2003. Prospects for a cognitive neuroscience of visual aesthetics. (2003).

[7]

Xuesong Chen, Ziyi Ye, Xiaohui Xie, Yiqun Liu, Xiaorong Gao, Weihang Su, Shuqi Zhu, Yike Sun, Min Zhang, and Shaoping Ma. 2022. Web Search via an Efficient and Effective Brain-Machine Interface. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1569--1572.

Digital Library

[8]

Ye Chen, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. User satisfaction prediction with mouse movement information in heterogeneous search environment. IEEE Transactions on Knowledge and Data Engineering, Vol. 29, 11 (2017), 2470--2483.

Digital Library

[9]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7--10.

Digital Library

[10]

Keith M Davis III, Michiel Spapé, and Tuukka Ruotsalo. 2021. Collaborative filtering with preferences inferred from brain signals. In Proceedings of the Web Conference 2021. 602--611.

[11]

Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 81--84.

[12]

Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS), Vol. 23, 2 (2005), 147--168.

Digital Library

[13]

Martin Gjoreski, Blagoj Mitrevski, Mitja Luvs trek, and Matjavz Gams. 2018. An inter-domain study for arousal recognition from physiological signals. Informatica, Vol. 42, 1 (2018).

[14]

Shan Guan, Kai Zhao, and Shuning Yang. 2019. Motor imagery EEG classification based on decision tree framework and Riemannian geometry. Computational intelligence and neuroscience, Vol. 2019 (2019).

[15]

Jacek Gwizdka, Rahilsadat Hosseini, Michael Cole, and Shouyi Wang. 2017. Temporal dynamics of eye-tracking and EEG during reading and relevance decisions. Journal of the Association for Information Science and Technology, Vol. 68, 10 (2017), 2299--2312.

Digital Library

[16]

Ahmed Hassan, Ryen W White, Susan T Dumais, and Yi-Min Wang. 2014. Struggling or exploring? Disambiguating long search sessions. In Proceedings of the 7th ACM international conference on Web search and data mining. 53--62.

Digital Library

[17]

Tiffany C Ho, Colm G Connolly, Eva Henje Blom, Kaja Z LeWinn, Irina A Strigo, Martin P Paulus, Guido Frank, Jeffrey E Max, Jing Wu, Melanie Chan, et al. 2015. Emotion-dependent functional connectivity of the default mode network in adolescent depression. Biological psychiatry, Vol. 78, 9 (2015), 635--646.

[18]

Jingzhao Hu, Chen Wang, Qiaomei Jia, Qirong Bu, Richard Sutcliffe, and Jun Feng. 2021. ScalingNet: extracting features from raw EEG data for emotion recognition. Neurocomputing, Vol. 463 (2021), 177--184.

Digital Library

[19]

Kalervo J"arvelin and Jaana Kek"al"ainen. 2017. IR evaluation methods for retrieving highly relevant documents. In ACM SIGIR Forum, Vol. 51. ACM New York, NY, USA, 243--250.

[20]

Ziyu Jia, Youfang Lin, Jing Wang, Zhiyang Feng, Xiangheng Xie, and Caijie Chen. 2021. HetEmotionNet: two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia. 1047--1056.

Digital Library

[21]

Ziyu Jia, Youfang Lin, Jing Wang, Ronghao Zhou, Xiaojun Ning, Yuanlai He, and Yaoshuai Zhao. 2020. GraphSleepNet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification. In IJCAI. 1324--1330.

[22]

Diane Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Now Publishers Inc.

Digital Library

[23]

Hyun Hee Kim and Yong Ho Kim. 2019. ERP/MMR algorithm for classifying topic-relevant and topic-irrelevant visual shots of documentary videos. Journal of the Association for Information Science and Technology, Vol. 70, 9 (2019), 931--941.

Digital Library

[24]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[25]

Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. 2021. BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Frontiers in Human Neuroscience, Vol. 15 (2021).

[26]

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Physical review E, Vol. 69, 6 (2004), 066138.

[27]

Victor Lavrenko and W Bruce Croft. 2017. Relevance-based language models. In ACM SIGIR Forum, Vol. 51. ACM New York, NY, USA, 260--267.

Digital Library

[28]

Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. 2018. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. Journal of neural engineering, Vol. 15, 5 (2018), 056013.

[29]

Rui Li, Yiting Wang, and Bao-Liang Lu. 2021. A Multi-Domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition. In Proceedings of the 29th ACM International Conference on Multimedia. 5565--5573.

Digital Library

[30]

Xiang Li, Dawei Song, Peng Zhang, Guangliang Yu, Yuexian Hou, and Bin Hu. 2016. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In 2016 IEEE international conference on bioinformatics and biomedicine (BIBM). IEEE, 352--359.

[31]

Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. " Satisfaction with Failure" or" Unsatisfied Success" Investigating the Relationship between Search Success and User Satisfaction. In Proceedings of the 2018 world wide web conference. 1533--1542.

[32]

Yiqun Liu, Chao Wang, Ke Zhou, Jianyun Nie, Min Zhang, and Shaoping Ma. 2014. From skimming to reading: A two-stage examination model for web search. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. 849--858.

Digital Library

[33]

Colin Martindale and Kathleen Moore. 1988. Priming, prototypicality, and preference. Journal of Experimental Psychology: Human Perception and Performance, Vol. 14, 4 (1988), 661.

[34]

Juan Abdon Miranda-Correa, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. 2018. Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Transactions on Affective Computing, Vol. 12, 2 (2018), 479--493.

Digital Library

[35]

Yashar Moshfeghi, Peter Triantafillou, and Frank Pollick. 2019. Towards predicting a realisation of an information need based on brain signals. In The World Wide Web Conference. 1300--1309.

Digital Library

[36]

Yashar Moshfeghi, Peter Triantafillou, and Frank E Pollick. 2016. Understanding information need: An fMRI study. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 335--344.

Digital Library

[37]

Marcos Nadal, Enric Munar, Miquel Angel Capo, Jaume Rossello, and Camilo Jose Cela-Conde. 2008. Towards a framework for the study of the neural correlates of aesthetic preference. Spatial vision, Vol. 21, 3 (2008), 379.

[38]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, Vol. 12 (2011), 2825--2830.

[39]

David MW Powers. 2020. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061 (2020).

[40]

B Pyakillya, N Kazachenko, and N Mikhailovsky. 2017. Deep learning for ECG classification. In Journal of physics: conference series, Vol. 913. IOP Publishing, 012004.

[41]

Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International conference on data mining. IEEE, 995--1000.

Digital Library

[42]

Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web. 521--530.

Digital Library

[43]

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc.

[44]

S Rasoul Safavian and David Landgrebe. 1991. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics, Vol. 21, 3 (1991), 660--674.

[45]

Louis A Schmidt and Laurel J Trainor. 2001. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion, Vol. 15, 4 (2001), 487--500.

[46]

Paul J Silvia. 2005. Cognitive appraisals and interest in visual art: Exploring an appraisal theory of aesthetic emotions. Empirical studies of the arts, Vol. 23, 2 (2005), 119--133.

[47]

Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. 2018. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, Vol. 11, 3 (2018), 532--541.

[48]

Akara Supratak and Yike Guo. 2020. TinySleepNet: An efficient deep learning model for sleep stage scoring based on raw single-channel EEG. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 641--644.

[49]

Johan AK Suykens and Joos Vandewalle. 1999. Least squares support vector machine classifiers. Neural processing letters, Vol. 9, 3 (1999), 293--300.

Digital Library

[50]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).

[51]

Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021a. Denoising implicit feedback for recommendation. In Proceedings of the 14th ACM international conference on web search and data mining. 373--381.

Digital Library

[52]

Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021b. Clicks can be cheating: Counterfactual recommendation for mitigating clickbait issue. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1288--1297.

Digital Library

[53]

Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing, Vol. 129 (2014), 94--106.

Digital Library

[54]

Yingying Wu, Yiqun Liu, Yen-Hsi Richard Tsai, and Shing-Tung Yau. 2019. Investigating the role of eye movements and physiological signals in search satisfaction prediction using geometric analysis. Journal of the Association for Information Science and Technology, Vol. 70, 9 (2019), 981--999.

Digital Library

[55]

Dezhong Yao, Yun Qin, Shiang Hu, Li Dong, Maria L Bringas Vega, and Pedro A Valdés Sosa. 2019. Which reference should we use for EEG and ERP practice? Brain topography, Vol. 32, 4 (2019), 530--549.

[56]

Ziyi Ye, Xiaohui Xie, Yiqun Liu, Zhihong Wang, Xuancheng Li, Jiaji Li, Xuesong Chen, Min Zhang, and Shaoping Ma. 2022. Why Don't You Click: Understanding Non-Click Results in Web Search with Brain Signals. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Digital Library

[57]

Zhongliang Yin, Jun Li, Yun Zhang, Aifeng Ren, Karen M Von Meneen, and Liyu Huang. 2017. Functional brain network analysis of schizophrenic patients with positive and negative syndrome based on mutual information of EEG time series. Biomedical Signal Processing and Control, Vol. 31 (2017), 331--338.

[58]

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2114--2124.

Digital Library

[59]

Guanhua Zhang, Minjing Yu, Yong-Jin Liu, Guozhen Zhao, Dan Zhang, and Wenming Zheng. 2021. SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Transactions on Affective Computing (2021).

[60]

Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4653--4664.

Digital Library

[61]

Peixiang Zhong, Di Wang, and Chunyan Miao. 2020. EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing (2020).

[62]

Mu Zhu. 2004. Recall, precision and average precision. Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Vol. 2, 30 (2004), 6.io

Cited By

Ye ZAi QLiu YCao ZJung TXu P(2024)Brain-Computer Interface Meets Information Retrieval: Perspective on Next-generation Information SystemProceedings of the 1st International Workshop on Brain-Computer Interfaces (BCI) for Multimedia Understanding10.1145/3688862.3689114(61-65)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3688862.3689114
Ye ZZhan JAi QLiu Yde Rijke MLioma CRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Query Augmentation with Brain SignalsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681658(7561-7570)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681658
Li JLi HHe ZMa WSun PZhang MMa S(2024)ReChorus2.0: A Modular and Task-Flexible Recommendation LibraryProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688076(454-464)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688076
Show More Cited By

Index Terms

Brain Topography Adaptive Network for Satisfaction Modeling in Interactive Information Access System

Recommendations

Using a trust network to improve top-N recommendation
RecSys '09: Proceedings of the third ACM conference on Recommender systems

Top-N item recommendation is one of the important tasks of recommenders. Collaborative filtering is the most popular approach to building recommender systems which can predict ratings for a given user and item. Collaborative filtering can be extended ...
Information Recommendation Method Research Based on Trust Network and Collaborative Filtering
ICEBE '11: Proceedings of the 2011 IEEE 8th International Conference on e-Business Engineering

Information recommender system is considered to be one of the most effective tools to solve the problem of information overload. Collaborative Filtering (CF), which utilizes similar neighbors to generate recommendations, is believed to be the most ...
Understanding post-adoption behaviors of e-service users in the context of online travel services

This study examines the motivators of continuance intention and WOM of e-service users.This study also examines the relationship between continuance intention and WOM.Satisfaction and perceived usefulness determine continuance intention.Continuance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Tsinghua University Guoqiang Research Institute
the Natural Science Foundation of China

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
194
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)5

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye ZAi QLiu YCao ZJung TXu P(2024)Brain-Computer Interface Meets Information Retrieval: Perspective on Next-generation Information SystemProceedings of the 1st International Workshop on Brain-Computer Interfaces (BCI) for Multimedia Understanding10.1145/3688862.3689114(61-65)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3688862.3689114
Ye ZZhan JAi QLiu Yde Rijke MLioma CRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Query Augmentation with Brain SignalsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681658(7561-7570)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681658
Li JLi HHe ZMa WSun PZhang MMa S(2024)ReChorus2.0: A Modular and Task-Flexible Recommendation LibraryProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688076(454-464)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688076
Ye ZXie XAi QLiu YWang ZSu WZhang M(2024)Relevance Feedback with Brain SignalsACM Transactions on Information Systems10.1145/363787442:4(1-37)Online publication date: 9-Feb-2024
https://dl.acm.org/doi/10.1145/3637874
Huang WChen YJiang XZhang TChen Q(2023)GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal FusionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361750320:2(1-23)Online publication date: 18-Oct-2023
https://dl.acm.org/doi/10.1145/3617503
McGuire NMoshfeghi Y(2023)What Song Am I Thinking Of?Machine Learning, Optimization, and Data Science10.1007/978-3-031-53966-4_31(418-432)Online publication date: 22-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-53966-4_31

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents