research-article

EmotionGAN: Unsupervised Domain Adaptation for Learning Discrete Probability Distributions of Image Emotions

Authors:

Kurt KeutzerAuthors Info & Claims

MM '18: Proceedings of the 26th ACM international conference on Multimedia

Pages 1319 - 1327

https://doi.org/10.1145/3240508.3240591

Published: 15 October 2018 Publication History

Abstract

Deep neural networks have performed well on various benchmark vision tasks with large-scale labeled training data; however, such training data is expensive and time-consuming to obtain. Due to domain shift or dataset bias, directly transferring models trained on a large-scale labeled source domain to another sparsely labeled or unlabeled target domain often results in poor performance. In this paper, we consider the domain adaptation problem in image emotion recognition. Specifically, we study how to adapt the discrete probability distributions of image emotions from a source domain to a target domain in an unsupervised manner. We develop a novel adversarial model for emotion distribution learning, termed EmotionGAN, which alternately optimizes the Generative Adversarial Network (GAN) loss, semantic consistency loss, and regression loss. The EmotionGAN model can adapt source domain images such that they appear as if they were drawn from the target domain, while preserving the annotation information. Extensive experiments are conducted on the FlickrLDL and TwitterLDL datasets, and the results demonstrate the superiority of the proposed method as compared to state-of-the-art approaches.

References

[1]

Xavier Alameda-Pineda, Elisa Ricci, Yan Yan, and Nicu Sebe. 2016. Recognizing emotions from abstract paintings using non-linear matrix completion. In CVPR . 5240--5248.

[2]

Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM MM . 223--232.

Digital Library

[3]

Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. 2017. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR . 3722--3731.

[4]

Tao Chen, Felix X Yu, Jiawei Chen, Yin Cui, Yan-Ying Chen, and Shih-Fu Chang. 2014. Object-based visual sentiment concept analysis and application. In ACM MM . 367--376.

Digital Library

[5]

Paul Ekman. 1992. An argument for basic emotions. Cognition & Emotion, Vol. 6, 3--4 (1992), 169--200.

[6]

Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. 2013. Unsupervised visual domain adaptation using subspace alignment. In ICCV . 2960--2967.

Digital Library

[7]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. JMLR, Vol. 17, 1 (2016), 2096--2030.

Digital Library

[8]

Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015. Domain generalization for object recognition with multi-task autoencoders. In ICCV . 2551--2559.

Digital Library

[9]

Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. 2016. Deep reconstruction-classification networks for unsupervised domain adaptation. In ECCV . 597--613.

[10]

Boqing Gong, Kristen Grauman, and Fei Sha. 2013. Connecting the dots with landmarks: Discriminatively learning domain-invariant features for unsupervised domain adaptation. In ICML. 222--230.

Digital Library

[11]

Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. 2012. Geodesic flow kernel for unsupervised domain adaptation. In CVPR . 2066--2073.

Digital Library

[12]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680.

Digital Library

[13]

Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. 2011. Domain adaptation for object recognition: An unsupervised approach. In ICCV . 999--1006.

Digital Library

[14]

Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. 2014. Unsupervised adaptation across domain shifts by generating intermediate data representations. IEEE TPAMI, Vol. 36, 11 (2014), 2288--2302.

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.

[16]

Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. 2017. CyCADA: Cycle-Consistent Adversarial Domain Adaptation. arXiv preprint arXiv:1711.03213 (2017).

[17]

Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, and Alex J Smola. 2007. Correcting sample selection bias by unlabeled data. In NIPS. 601--608.

Digital Library

[18]

Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang. 2018. Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation: the Benefit of Target Expectation Maximization. arXiv preprint arXiv:1801.10068 (2018).

[19]

Joonwhoan Lee and EunJong Park. 2011. Fuzzy similarity-based emotional classification of color images. IEEE TMM, Vol. 13, 5 (2011), 1031--1039.

Digital Library

[20]

Ming-Yu Liu and Oncel Tuzel. 2016. Coupled generative adversarial networks. In NIPS. 469--477.

Digital Library

[21]

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015. Learning transferable features with deep adaptation networks. In ICML. 97--105.

Digital Library

[22]

Xin Lu, Poonam Suryanarayan, Reginald B Adams Jr, Jia Li, Michelle G Newman, and James Z Wang. 2012. On shape and the computability of emotions. In ACM MM. 229--238.

Digital Library

[23]

Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In ACM MM. 83--92.

Digital Library

[24]

Joseph A Mikels, Barbara L Fredrickson, Gregory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz. 2005. Emotional category data on images from the International Affective Picture System. BRM, Vol. 37, 4 (2005), 626--630.

[25]

Jie Ni, Qiang Qiu, and Rama Chellappa. 2013. Subspace interpolation via dictionary learning for unsupervised domain adaptation. In CVPR . 692--699.

Digital Library

[26]

Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. 2015. Visual domain adaptation: A survey of recent advances. IEEE SPM, Vol. 32, 3 (2015), 53--69.

[27]

Kuan-Chuan Peng, Amir Sadovnik, Andrew Gallagher, and Tsuhan Chen. 2015. A Mixed Bag of Emotions: Model, Predict, and Transfer Emotion Distributions. In CVPR . 860--868.

[28]

Tianrong Rao, Min Xu, and Dong Xu. 2016. Learning multi-level deep representations for image emotion classification. arXiv preprint arXiv:1611.07145 (2016).

[29]

Artem Rozantsev, Mathieu Salzmann, and Pascal Fua. 2016. Beyond sharing weights for deep domain adaptation. arXiv preprint arXiv:1603.06432 (2016).

[30]

Harold Schlosberg. 1954. Three dimensions of emotion. Psychological Review, Vol. 61, 2 (1954), 81.

[31]

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. 2017. Learning from simulated and unsupervised images through adversarial training. In CVPR . 2242--2251.

[32]

Baochen Sun, Jiashi Feng, and Kate Saenko. 2017. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications. 153--171.

[33]

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR. 2818--2826.

[34]

Antonio Torralba and Alexei A Efros. 2011. Unbiased look at dataset bias. In CVPR. 1521--1528.

Digital Library

[35]

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In CVPR. 2962--2971.

[36]

Jufeng Yang, Dongyu She, and Ming Sun. 2017a. Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network. In IJCAI. 3266--3272.

Digital Library

[37]

Jufeng Yang, Ming Sun, and Xiaoxiao Sun. 2017b. Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network. In AAAI. 224--230.

[38]

Yun Yang, Peng Cui, Wenwu Zhu, and Shiqiang Yang. 2013. User interest and social influence based emotion prediction for individuals. In ACM MM . 785--788.

Digital Library

[39]

Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2016. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In AAAI. 308--314.

Digital Library

[40]

Jianbo Yuan, Sean Mcdonough, Quanzeng You, and Jiebo Luo. 2013. Sentribute: image sentiment analysis from a mid-level perspective. In ACM WISDOM . 10.

Digital Library

[41]

Sicheng Zhao, Guiguang Ding, Yue Gao, and Jungong Han. 2017a. Approximating Discrete Probability Distribution of Image Emotions by Multi-Modal Features Fusion. In IJCAI. 4669--4675.

Digital Library

[42]

Sicheng Zhao, Guiguang Ding, Yue Gao, and Jungong Han. 2017b. Learning Visual Emotion Distributions via Multi-Modal Features Fusion. In ACM MM . 369--377.

Digital Library

[43]

Sicheng Zhao, Guiguang Ding, Yue Gao, Xin Zhao, Youbao Tang, Jungong Han, Hongxun Yao, and Qingming Huang. 2018. Discrete Probability Distribution Prediction of Image Emotions With Shared Sparse Learning. IEEE TAFFC (2018).

[44]

Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. 2014a. Exploring principles-of-art features for image emotion recognition. In ACM MM . 47--56.

Digital Library

[45]

Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, and Guiguang Ding. 2017c. Continuous Probability Distribution Prediction of Image Emotions via Multi-Task Shared Sparse Regression. IEEE TMM, Vol. 19, 3 (2017), 632--645.

Digital Library

[46]

Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, and Tat-Seng Chua. 2016. Predicting personalized emotion perceptions of social images. In ACM MM . 1385--1394.

Digital Library

[47]

Sicheng Zhao, Hongxun Yao, You Yang, and Yanhao Zhang. 2014b. Affective image retrieval via multi-graph learning. In ACM MM . 1025--1028.

Digital Library

[48]

Xinge Zhu, Liang Li, Weigang Zhang, Tianrong Rao, Min Xu, Qingming Huang, and Dong Xu. 2017. Dependency exploitation: a unified CNN-RNN approach for visual emotion recognition. In IJCAI . 3595--3601.

Digital Library

[49]

Junbao Zhuo, Shuhui Wang, Weigang Zhang, and Qingming Huang. 2017. Deep Unsupervised Convolutional Domain Adaptation. In ACM MM. 261--269.

Digital Library

Cited By

Wu DYang DZhou YMa CCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text PairsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680875(602-611)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680875
Korinth B(2024)Using paintings to teach about the impact of environmental hazardsEnvironmental Hazards10.1080/17477891.2024.2358043(1-13)Online publication date: 27-May-2024
https://doi.org/10.1080/17477891.2024.2358043
Manaka TZyl TKar DWade A(2024)Multi-step Transfer Learning in Natural Language Processing for the Health DomainNeural Processing Letters10.1007/s11063-024-11526-y56:3Online publication date: 20-May-2024
https://doi.org/10.1007/s11063-024-11526-y
Show More Cited By

Index Terms

EmotionGAN: Unsupervised Domain Adaptation for Learning Discrete Probability Distributions of Image Emotions
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Social media
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Sentiment analysis

Recommendations

DC-FUDA: Improving deep clustering via fully unsupervised domain adaptation
Abstract
By transferring knowledge from a source domain, the performance of deep clustering on an unlabeled target domain can be greatly improved. When achieving this, traditional approaches assume that an adequate amount of labeled data are available in ...
Unsupervised Domain Adaptation on Sentence Matching Through Self-Supervision
Abstract
Although neural approaches have yielded state-of-the-art results in the sentence matching task, their performance inevitably drops dramatically when applied to unseen domains. To tackle this cross-domain challenge, we address unsupervised domain ...
Soft Labels Transfer with Discriminative Representations Learning for Unsupervised Domain Adaptation
Machine Learning and Knowledge Discovery in Databases
Abstract
Domain adaptation aims to address the challenge of transferring the knowledge obtained from the source domain with rich label information to the target domain with less or even no label information. Recent methods start to tackle this problem by ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '18: Proceedings of the 26th ACM international conference on Multimedia

October 2018

2167 pages

ISBN:9781450356657

DOI:10.1145/3240508

General Chairs:
Susanne Boll
University of Oldenburg, Germany
,
Kyoung Mu Lee
Seoul National University, Korea
,
Jiebo Luo
University of Rochester, USA
,
Wenwu Zhu
Tsinghua University, China
,
Program Chairs:
Hyeran Byun
Yonsei University, Korea
,
Chang Wen Chen
State Univ. Of New York at Buffalo, USA
,
Rainer Lienhart
University of Augsburg, Germany
,
Tao Mei
JD AI, China

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Berkeley Deep Drive
Project Funded by China Postdoctoral Science Foundation
National Natural Science Foundation of China
National Key R&D Program of China

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22 - 26, 2018

Seoul, Republic of Korea

Acceptance Rates

MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
1,075
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu DYang DZhou YMa CCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text PairsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680875(602-611)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680875
Korinth B(2024)Using paintings to teach about the impact of environmental hazardsEnvironmental Hazards10.1080/17477891.2024.2358043(1-13)Online publication date: 27-May-2024
https://doi.org/10.1080/17477891.2024.2358043
Manaka TZyl TKar DWade A(2024)Multi-step Transfer Learning in Natural Language Processing for the Health DomainNeural Processing Letters10.1007/s11063-024-11526-y56:3Online publication date: 20-May-2024
https://doi.org/10.1007/s11063-024-11526-y
Kong XJiang WJia JShi YXu RLiu SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)DUSA: Decoupled Unsupervised Sim2Real Adaptation for Vehicle-to-Everything Collaborative PerceptionProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611948(1943-1954)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611948
Zhao SHong XYang JZhao YDing G(2023)Toward Label-Efficient Emotion and Sentiment AnalysisProceedings of the IEEE10.1109/JPROC.2023.3309299111:10(1159-1197)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3309299
Wu HHuang YNan G(2023)Doubled coupling for image emotion distribution learningKnowledge-Based Systems10.1016/j.knosys.2022.110107260(110107)Online publication date: Jan-2023
https://doi.org/10.1016/j.knosys.2022.110107
Zhu SQing CXu X(2023)Text-Guided Generative Adversarial Network for Image Emotion TransferAdvanced Intelligent Computing Technology and Applications10.1007/978-981-99-4742-3_42(506-522)Online publication date: 30-Jul-2023
https://doi.org/10.1007/978-981-99-4742-3_42
Dai QGao YDai QGao Y(2023)Hypergraph Computation for Social Media AnalysisHypergraph Computation10.1007/978-981-99-0185-2_9(159-189)Online publication date: 17-Jan-2023
https://doi.org/10.1007/978-981-99-0185-2_9
Wang YZhu LHuang SHui TLi XWang FLiu SMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Cross-Modality Domain Adaptation for Freespace Detection: A Simple yet Effective BaselineProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3547752(4031-4042)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3547752
Zhao SYue XZhang SLi BZhao HWu BKrishna RGonzalez JSangiovanni-Vincentelli ASeshia SKeutzer K(2022)A Review of Single-Source Deep Unsupervised Visual Domain AdaptationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2020.302850333:2(473-493)Online publication date: Feb-2022
https://doi.org/10.1109/TNNLS.2020.3028503
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents