Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

PriMonitor: An adaptive tuning privacy-preserving approach for multimodal emotion detection

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

The proliferation of edge computing and the Internet of Vehicles (IoV) has significantly bolstered the popularity of deep learning-based driver assistance applications. This has paved the way for the integration of multimodal emotion detection systems, which effectively enhance driving safety and are increasingly prevalent in our daily lives. However, the utilization of in-vehicle cameras and microphones has raised concerns regarding the extensive collection of driver privacy data. Applying privacy-preserving techniques to a single modality alone proves insufficient in preventing privacy re-identification when correlated with other modalities. In this paper, we introduce PriMonitor, an adaptive tuning privacy-preserving approach for multimodal emotion detection. PriMonitor tackles these challenges by proposing a generalized random response-based differential privacy method that not only enhances the speed and data availability of text privacy protection but also ensures privacy preservation across multiple modalities. To determine suitable weight assignments within a given privacy budget, we introduce pre-aggregator and iterative mechanisms. Our PriMonitor effectively mitigates privacy re-identification due to modal correlation while maintaining a high level of accuracy in multimodal models. Experimental results validate the efficiency and competitiveness of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Algorithm 1
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9

Similar content being viewed by others

Data availability

The dataset CH-SIMS used to support the findings of this study can be download in https://github.com/thuiar/MMSA/.

The dataset CMU-MOSI used to support the findings of this study can be download in https://github.com/A2Zadeh/CMU-MultimodalDataSDK/.

References

  1. Oham, C., Michelin, R.A., Jurdak, R., Kanhere, S.S., Jha, S.: B-FERL: Blockchain based framework for securing smart vehicles. Inf. Process. Manage. 58(1), 102426 (2021)

    Article  Google Scholar 

  2. Li, L., Liu, J., Cheng, L., Qiu, S., Wang, W., Zhang, X., Zhang, Z.: Creditcoin: A privacy-preserving blockchain-based incentive announcement network for communications of smart vehicles. In: IEEE Transactions on Intelligent Transportation Systems, 19(7), 2204–2220 (2018)

  3. Sun, Y., Tian, Z., Li, M., Zhu, C., Guizani, N.: Automated attack and defense framework toward 5G security. IEEE Netw. 34(5), 247–253 (2020)

    Article  Google Scholar 

  4. Sicari, S., Rizzardi, A., Coen-Porisini, A.: 5G in the internet of things era: an overview on security and privacy challenges. Comput. Netw. 179, 107345 (2020)

    Article  Google Scholar 

  5. Qiao, C., Brown, K.N., Zhang, F., Tian, Z.: Adaptive Asynchronous Clustering Algorithms for Wireless Mesh Networks. In: IEEE Transactions on Knowledge and Data Engineering, 35(3), 2610–2627 (2023)

  6. Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., Yang, M.: Multimodal continuous emotion recognition with data augmentation using recurrent neural networks. In: Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop, pp. 57–64 (2018)

  7. Aguilar, G., Rozgic, V., Wang, W., Wang, C.: Multimodal and Multi-view Models for Emotion Recognition. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 991–1002 (2019)

  8. Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. 35(3), 2302–2310 (2021)

  9. Li, A., Duan, Y., Yang, H., Chen, Y., Yang, J.: TIPRDC: task-independent privacy-respecting data crowdsourcing framework for deep learning with anonymized intermediate representations. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 824–832 (2020)

  10. Yin, L., Feng, J., Xun, H., Sun, Z., Cheng, X.: A privacy-preserving federated learning for multiparty data sharing in social IoTs. IEEE Trans. Netw. Sci. Eng. 8(3), 2706–2718 (2021)

    Article  Google Scholar 

  11. Fernandes, N., Dras, M., McIver, A.: Generalised differential privacy for text document processing. In: International Conference on Principles of Security and Trust, pp. 123–148 (2019)

  12. He, Y., Tan, X., Ni, J., Yang, L.T., Deng, X.: Differentially Private Set Intersection for Asymmetrical ID Alignment. In: IEEE Transactions on Information Forensics and Security. 17, 3479–3494 (2022)

  13. Wang, T., Blocki, J., Li, N., Jha, S.: Locally differentially private protocols for frequency estimation. In: 26th USENIX Security Symposium, pp. 729–745 (2017)

  14. Zepf, S., Hernandez, J., Schmitt, A., et al.: Driver emotion recognition for intelligent vehicles: A survey. ACM Comput. Surv. (CSUR) 53(3), 1–30 (2020)

    Article  Google Scholar 

  15. Ma, Z., Mahmoud, M., Robinson, P., Dias, E., Skrypchuk, L.: Automatic detection of a driver’s complex mental states. In: Computational Science and Its Applications - ICCSA 2017: 17th International Conference, Trieste, Italy, July 3–6, 2017, Proceedings, Part III. Series: Lecture notes in computer science (10406). Springer: Cham, pp. 678–691 (2017)

  16. Karimi, S., Sedaaghi, M.H.: Robust emotional speech classification in the presence of babble noise. Int. J. Speech Technol. 16, 215–227 (2013)

    Article  Google Scholar 

  17. Deng, Y., Wu, Z., Chu, C-H., Zhang, Q., Hsu, D. F.: Sensor feature selection and combination for stress identification using combinatorial fusion. Int. J. Adv. Robot. Syst. 306–313 (2013) 

  18. Karaduman, O., Eren, H., Kurum, H., Celenk, M.: An effective variable selection algorithm for aggressive/calm driving detection via CAN bus. In: Proceedings of the 2013 International Conference on Connected Vehicles and Expo (ICCVE’13), Las Vegas, NV, USA, 2013, pp. 586–591 (2013)

  19. Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., Zimmermann, R.: Icon: Interactive conversational memory network for multimodal emotion detection. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2594–2604 (2018)

  20. Mittal, T., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emoticon: Context-aware multimodal emotion recognition using frege's principle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234–14243 (2020)

  21. Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J. Yang, K.: CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718–3727 (2020)

  22. Zheng, W., Yu, J., Xia, R., Wang, S.: A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 15445–15459 (2023)

  23. Illendula, A., Sheth, A.: Multimodal emotion classification. In: companion proceedings of the 2019 world wide web conference, pp. 439–449 (2019)

  24. Ma, J., Tang, H., Zheng, W. L., Lu, B. L.: Emotion recognition using multimodal residual LSTM network. In: Proceedings of the 27th ACM international conference on multimedia, pp. 176–183 (2019)

  25. Lv, F., Chen, X., Huang, Y., Duan, L., Lin, G.: Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2554–2562 (2021)

  26. Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, No. 02, pp. 1359–1367 (2020)

  27. Firdaus, M., Chauhan, H., Ekbal, A., Bhattacharyya, P.: MEISD: a multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 4441–4453 (2020)

  28. Liang, B., Lou, C., Li, X., Yang, M., Gui, L., He, Y., Pei, W., Xu, R.: Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 1767–1777 (2022)

  29. Yang, H., Gao, X., Wu, J., Gan, T., Ding, N., Jiang, F., Nie, L.: Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 6267–6281 (2023)

  30. Tan, Q., Wang, X., Shi, W., Tang, J., Tian, Z.: An Anonymity Vulnerability in Tor. In: IEEE/ACM Transactions on Networking. 30(6), 2574–2587 (2022)

  31. Jaiswal, M., Provost, E. M.: Privacy enhanced multimodal neural representations for emotion recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. 34(05), 7985–7993 (2020)

  32. Gupta, A., Tafasca, S., Odobez, J. M.: A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 5041–5050 (2022)

  33. Zhao, L., Ni, L., Hu, S., Chen, Y., Zhou, P., Xiao, F., Wu, L.: Inprivate digging: Enabling tree-based distributed data mining with differential privacy. In: IEEE INFOCOM 2018-IEEE Conference on Computer Communications, pp. 2087–2095 (2018)

  34. Papernot, N., Thakurta, A., Song, S., Chien, S., Erlingsson, Ú.: Tempered sigmoid activations for deep learning with differential privacy. In: Proceedings of the AAAI Conference on Artificial Intelligence. 35(10), 9312–9321 (2021)

  35. Sun, Y., Tian, Z., Li, M., Su, S., Du, X., Guizani, M.: Honeypot identification in softwarized industrial cyber–physical systems. IEEE Trans. Industr. Inf. 17(8), 5542–5551 (2020)

    Article  Google Scholar 

  36. Chai, Y., Du, L., Qiu, J., Yin, L., Tian, Z.: Dynamic prototype network based on sample adaptation for few-shot malware detection. In: IEEE Trans. Know. Data Eng. 4754–4766 (2020)

  37. Arcolezi, H. H., Gambs, S., Couchot, J. F., Palamidessi, C.: On the Risks of Collecting Multidimensional Data Under Local Differential Privacy. In: Proceedings of the VLDB Endowment (PVLDB). 16 (5), 1126–1139 (2023)

  38. Croft, W.L., Sack, J.R., Shi, W.: Obfuscation of images via differential privacy: From facial images to general images. Peer Peer Netw. Appl. 14(3), 1705–1733 (2021)

    Article  Google Scholar 

  39. Heikkilä, M., Lagerspetz, E., Kaski, S., Shimizu, K., Tarkoma, S., Honkela, A.: Differentially private bayesian learning on distributed data. In: Proceedings of the 31st International Conference on Neural Information Processing Systems December 2017 pp. 3229–3238 (2017)

  40. Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318 (2016)

  41. Xu, C., Ren, J., Zhang, D., Zhang, Y., Qin, Z., Ren, K.: GANobfuscator: Mitigating information leakage under GAN via differential privacy. IEEE Trans. Inf. Forensics Secur. 14(9), 2358–2371 (2019)

    Article  Google Scholar 

  42. Hao, M., Li, H., Xu, G., Liu, S., Yang, H.: Towards efficient and privacy-preserving federated deep learning. In: ICC 2019–2019 IEEE international conference on communications (ICC), pp. 1–6 (2019)

  43. Yang, R., Ma, X., Bai, X., Su, X.: Differential privacy images protection based on generative adversarial network. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 1688–1695 (2020)

  44. Feyisetan, O., Balle, B., Drake, T., Diethe, T.: Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 178–186 (2020)

  45. Lyu, L., Li, Y., He, X., Xiao, T.: Towards differentially private text representations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1813–1816 (2020)

  46. Feyisetan, O., Diethe, T., Drake, T.: Leveraging hierarchical representations for preserving privacy and utility in text. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 210–219 (2019)

  47. Weggenmann, B., Kerschbaum, F.: Syntf: Synthetic and differentially private term frequency vectors for privacy-preserving text mining. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 305–314 (2018)

  48. Devlin, J., Chang, M. W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171– 4186 (2019)

  49. Yue, X., Du, M., Wang, T., Li, Y., Sun, H., Chow, S.S.: Differential Privacy for Text Analytics via Natural Text Sanitization. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP pp. 3853–3866 (2021)

  50. Cormode, G., Jha, S., Kulkarni, T., Li, N., Srivastava, D., Wang, T.: Privacy at scale: Local differential privacy in practice. In: Proceedings of the 2018 International Conference on Management of Data, pp. 1655–1658 (2018)

  51. Liu, Y., Peng, J., Kang, J., Iliyasu, A.M., Niyato, D., Abd El-Latif, A.: A secure federated learning framework for 5G networks. IEEE Wirel. Commun. 27(4), 24–31 (2020)

    Article  Google Scholar 

  52. Li, B., Hou, Y., Che, W.: Data augmentation approaches in natural language processing: A survey. AI Open 3, 71–90 (2022)

    Article  Google Scholar 

  53. Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of the 25st International Conference on Neural Information Processing Systems December 2011 pp. 2546–2554 (2011)

  54. Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L. P.: Tensor fusion network for multimodal sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2017) 

  55. Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., Morency, L. P.: Efficient low-rank multimodal fusion with modality-specific factors. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2247–2256 (2018)

  56. Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., Morency, L. P.: Memory fusion network for multi-view sequential learning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2018)

  57. Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., Morency, L.P.: Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236–2246 (2018)

  58. Ye, J., Maddi, A., Murakonda, S.K., Bindschaedler, V., Shokri, R.: Enhanced Membership Inference Attacks against Machine Learning Models. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS '22), pp. 3093–3106 (2022)

Download references

Acknowledgements

This research is supported by the National Key R&D Program of China (No. 2022YFB3104100), the National Natural Science Foundation of China (No. 62002077, 92167203, 62002127, U20A20177), the Guangzhou Science and Technology Plan Project (No. 2023A03J0119), the Guangzhou University Graduate Student Innovation Ability Cultivation Funding Program (No. 2021GDJC-M37).

Funding

This research is supported by the National Key R&D Program of China (No. 2022YFB3104100), the National Natural Science Foundation of China (No. 62002077, 92167203, 62002127, U20A20177), the Guangzhou Science and Technology Plan Project (No. 2023A03J0119), the Guangzhou University Graduate Student Innovation Ability Cultivation Funding Program (No. 2021GDJC-M37).

Author information

Authors and Affiliations

  1. Cyberspace Institute of Advanced Technology (CIAT), Guangzhou University, Guangzhou, 510006, China

    Lihua Yin, Sixin Lin, Zhe Sun, Simin Wang & Ran Li

  2. Cyberspace Security Research Center, Peng Cheng Laboratory, Shenzhen, 518055, China

    Simin Wang

  3. School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan, 430074, China

    Yuanyuan He

Authors
  1. Lihua Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sixin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhe Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ran Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yuanyuan He
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Lihua Yin and Zhe Sun presented the core concepts and wrote the main manuscript text. Sixin Lin designed algorithm 3.3. Simin Wang completed the preparation of the experiment and analyzed the results. Ran Li prepared figures 1-3 and part of the data processing. Yuanyuan He participated in the scheme design, and revised and edited the manuscript. All authors reviewed the manuscript.

Corresponding author

Correspondence to Zhe Sun.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethical Approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, L., Lin, S., Sun, Z. et al. PriMonitor: An adaptive tuning privacy-preserving approach for multimodal emotion detection. World Wide Web 27, 9 (2024). https://doi.org/10.1007/s11280-024-01246-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11280-024-01246-7

Keywords

Search

Navigation