research-article

Towards a Unified Middle Modality Learning for Visible-Infrared Person Re-Identification

Authors:

Hanzi WangAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 788 - 796

https://doi.org/10.1145/3474085.3475250

Published: 17 October 2021 Publication History

Abstract

Visible-infrared person re-identification (VI-ReID) aims to search identities of pedestrians across different spectra. In this task, one of the major challenges is the modality discrepancy between the visible (VIS) and infrared (IR) images. Some state-of-the-art methods try to design complex networks or generative methods to mitigate the modality discrepancy while ignoring the highly non-linear relationship between the two modalities of VIS and IR. In this paper, we propose a non-linear middle modality generator (MMG), which helps to reduce the modality discrepancy. Our MMG can effectively project VIS and IR images into a unified middle modality image (UMMI) space to generate middle-modality (M-modality) images. The generated M-modality images and the original images are fed into the backbone network to reduce the modality discrepancy.Furthermore, in order to pull together the two types of M-modality images generated from the VIS and IR images in the UMMI space, we propose a distribution consistency loss (DCL) to make the modality distribution of the generated M-modalities images as consistent as possible. Finally, we propose a middle modality network (MMN) to further enhance the discrimination and richness of features in an explicit manner. Extensive experiments have been conducted to validate the superiority of MMN for VI-ReID over some state-of-the-art methods on two challenging datasets. The gain of MMN is more than 11.1% and 8.4% in terms of Rank-1 and mAP, respectively, even compared with the latest state-of-the-art methods on the SYSU-MM01 dataset.

References

[1]

Yehansen Chen, Lin Wan, Zhihang Li, Qianyan Jing, and Zongyuan Sun. 2021. Neural Feature Search for RGB-Infrared Person Re-Identification. In Proceedings of the IEEE CVPR. 587--597.

[2]

Seokeon Choi, Sumin Lee, Youngeun Kim, Taekyung Kim, and Changick Kim. 2020. Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification. In Proceedings of the IEEE CVPR. 10257--10266.

[3]

Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, and Yuyu Huang. 2018. Cross-Modality Person Re-Identification with Generative Adversarial Training. In Proceedings of the IJCAI. 677--683.

Digital Library

[4]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im-ageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE CVPR. 248--255.

[5]

Zhongying Deng, Xiaojiang Peng, and Yu Qiao. 2019. Residual Compensation Networks for Heterogeneous Face Recognition. In Proceedings of the AAAI, Vol. 33. 8239--8246.

[6]

Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. 2019. Bilinear Attention Networks for Person Retrieval. In Proceedings of the IEEE ICCV. 8030--8039.

[7]

Zhan-Xiang Feng, Jianhuang Lai, and Xiaohua Xie. 2020. Learning Modality-Specific Representations for Visible-Infrared Person Re-Identification. In IEEE TIP, Vol. 29. 579--590.

[8]

Lishuai Gao, Hua Zhang, Zan Gao, Weili Guan, Zhiyong Cheng, and Meng Wang. 2020. Texture Semantically Aligned with Visibility-aware for Partial Person Re-identification. In Proceedings of the ACM MM. 3771--3779.

Digital Library

[9]

Shang Gao, Jingya Wang, Huchuan Lu, and Zimo Liu. 2020. Pose-Guided Visible Part Matching for Occluded Person ReID. In Proceedings of the IEEE CVPR. 11744--11752.

[10]

Yixiao Ge, Zhuowan Li, Haiyu Zhao, Guojun Yin, Shuai Yi, Xiaogang Wang, and hongsheng Li. 2018. FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification. In Proceedings of the NeurIPS, Vol. 31. 1230--1241.

Digital Library

[11]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proceedings of the NeurIPS, Vol. 27. 2672--2680.

Digital Library

[12]

Yi Hao, Nannan Wang, Jie Li, and Xinbo Gao. 2019. HSME: Hypersphere Manifold Embedding for Visible Thermal Person Re-identification. In Proceedings of the AAAI, Vol. 33. 8385--8392.

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning For Image Recognition. In Proceedings of the IEEE CVPR. 770--778.

[14]

Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In Defense of The Triplet Loss for Person Re-identification. ArXiv (2017).

[15]

Ruibing Hou, Bingpeng Ma, Hong Chang, Xinqian Gu, Shiguang Shan, and Xilin Chen. 2019. Interaction-and-Aggregation Network for Person Re-identification. In Proceedings of the IEEE CVPR. 9317--9326.

[16]

Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang. 2020. Style Normalization and Restitution for Generalizable Person Re-identification. In Proceedings of the IEEE CVPR. 3143--3152.

[17]

Mahdi M Kalayeh, Emrah Basaran, Muhittin Gökmen, Mustafa E Kamasak, and Mubarak Shah. 2018. Human Semantic Parsing for Person Re-Identification. In Proceedings of the IEEE CVPR. 1062--1071.

[18]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classifi-cation with Deep Convolutional Neural Networks. In Proceedings of the NeurIPS, Vol. 25. 1106--1114.

Digital Library

[19]

Van Der Maaten Laurens and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. In JMLR, Vol. 9. 2579--2605.

[20]

Diangang Li, Xing Wei, Xiaopeng Hong, and Yihong Gong. 2020. Infrared-Visible Cross-Modal Person Re-Identification with an X Modality. In Proceedings of the AAAI. 4610--4617.

[21]

Yu-Jhe Li, Ci-Siang Lin, Yan-Bo Lin, and Yu-Chiang Frank Wang. 2019. Cross-Dataset Person Re-Identification via Unsupervised Pose Disentanglement and Adaptation. In Proceedings of the IEEE ICCV. 7919--7929.

[22]

Yongguo Ling, Zhun Zhong, Zhiming Luo, Paolo Rota, Shaozi Li, and Nicu Sebe. 2020. Class-Aware Modality Mix and Center-Guided Metric Learning for Visible-Thermal Person Re-Identification. In Proceedings of the ACM MM. 889--897.

Digital Library

[23]

Chong Liu, Xiaojun Chang, and Yi-Dong Shen. 2020. Unity Style Transfer for Person Re-Identification. In Proceedings of the IEEE CVPR. 6887--6896.

[24]

Yan Lu, Yue Wu, Bin Liu, Tianzhu Zhang, Baopu Li, Qi Chu, and Nenghai Yu. 2020. Cross-Modality Person Re-Identification With Shared-Specific Feature Transfer. In Proceedings of the IEEE CVPR. 13379--13389.

[25]

Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. 2019. Bag of Tricks and a Strong Baseline for Deep Person Re-Identification. In Proceedings of the IEEE CVPR Workshops. 0--0.

[26]

Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. 2019. Pose-Guided Feature Alignment for Occluded Person Re-Identification. In Proceedings of the IEEE ICCV. 542--551.

[27]

Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, and Kang Ryoung Park. 2017. Person Recognition System Based on a Combination of Body Images From Visible Light and Thermal Cameras. In Sensors, Vol. 17. 605.

[28]

Peixi Peng, Yonghong Tian, Yangru Huang, Xiangqian Wang, and Huilong An. 2020. Discriminative Spatial Feature Learning for Person Re-Identification. In Proceedings of the ACM MM. 274--283.

Digital Library

[29]

Nan Pu, Wei Chen, Yu Liu, Erwin M Bakker, and Michael S Lew. 2020. Dual Gaussian-based Variational Subspace Disentanglement for Visible-Infrared Per-son Re-Identification. In Proceedings of the ACM MM. 2149--2158.

Digital Library

[30]

Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. 2018. Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline). In Proceedings of the ECCV. 480--496.

[31]

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE CVPR. 2818--2826.

[32]

Chiat-Pin Tay, Sharmili Roy, and Kim-Hui Yap. 2019. AANet: Attribute Attention Network for Person Re-Identifications. In Proceedings of the IEEE CVPR. 7134--7143.

[33]

Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. 2018. Learning Discriminative Features with Multiple Granularities for Person Re-identification. In Proceedings of the ACM MM. 274--282.

Digital Library

[34]

Guan'an Wang, Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, and Zengguang Hou. 2019. RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment. In Proceedings of the ICCV. 3623--3632.

[35]

Guan-An Wang, Tianzhu Zhang, Yang Yang, Jian Cheng, Jianlong Chang, Xu Liang, and Zeng-Guang Hou. 2020. Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification. In Proceedings of the AAAI, Vol. 34. 12144--12151.

[36]

Zhixiang Wang, Zheng Wang, Yinqiang Zheng, Yung-Yu Chuang, and Shin'ichi Satoh. 2019. Learning to Reduce Dual-level Discrepancy for Infrared-visible Person Re-identification. In Proceedings of the IEEE CVPR. 618--626.

[37]

Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. 2018. Person Transfer GAN to Bridge Domain Gap for Person Re-identification. In Proceedings of the IEEE CVPR. 79--88.

[38]

Xing Wei, Diangang Li, Xiaopeng Hong, Wei Ke, and Yihong Gong. 2020. Co-attentive Lifting for Infrared-visible Person Re-identification. In Proceedings of the ACM MM. 1028--1037.

Digital Library

[39]

Ancong Wu, Wei-Shi Zheng, Shaogang Gong, and Jianhuang Lai. 2020. RGB-IR Person Re-identification by Cross-Modality Similarity Preservation. In IJCV, Vol. 128. 1765--1785.

[40]

Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, and Jianhuang Lai. 2017. Rgb-Infrared Cross-Modality Person Re-identification. In Proceedings of the ICCV. 5380--5389.

[41]

Fan Yang, Zheng Wang, Jing Xiao, and Shin'ichi Satoh. 2020. Mining on Hetero-geneous Manifolds for Zero-Shot Cross-Modal Image Retrieval. In Proceedings of the AAAI. 12589--12596.

[42]

Hanrong Ye, Hong Liu, Fanyang Meng, and Xia Li. 2020. Bi-directional Exponen-tial Angular Triplet Loss for Rgb-infrared Person Re-identification. In IEEE TIP, Vol. 30. 1583--1595.

[43]

Mang Ye, Xiangyuan Lan, Qingming Leng, and Jianbing Shen. 2020. Cross-Modality Person Re-Identification via Modality-aware Collaborative Ensemble Learning. In IEEE TIP, Vol. 29. 9387--9399.

[44]

Mang Ye, Xiangyuan Lan, Jiawei Li, and Pong Yuen. 2018. Hierarchical Discrimi-native Learning for Visible Thermal Person Re-identification. In Proceedings of the AAAI, Vol. 32.

[45]

Mang Ye, Jianbing Shen, David J. Crandall, Ling Shao, and Jiebo Luo. 2020. Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification. In Proceedings of the ECCV. 229--247.

[46]

Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C Yuen. 2018. Visible Thermal Person Re-Identification via Dual-Constrained Top-Ranking. In Proceedings of the IJCAI. 1092--1099.

Digital Library

[47]

Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Xin Jin, and Zhibo Chen. 2020. Relation-Aware Global Attention for Person Re-Identification. In Proceedings of the IEEE CVPR. 3186--3195.

[48]

Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji. 2019. Pyramidal Person Re-IDentification via Multi-Loss Dynamic Training. In Proceedings of the IEEE CVPR. 8514--8522.

[49]

Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz. 2019. Joint Discriminative and Generative Learning for Person Re-identification. In Proceedings of the IEEE CVPR. 2138--2147.

[50]

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. 2020. Random Erasing Data Augmentation. In Proceedings of the AAAI, Vol. 34. 13001--13008.

[51]

Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang. 2020. Identity-Guided Human Semantic Parsing for Person Re-Identification. In Proceedings of the ECCV. 346--363.

[52]

Yang Zou, Xiaodong Yang, Zhiding Yu, BVK Vijaya Kumar, and Jan Kautz. 2020. Joint Disentangling and Adaptation for Cross-domain Person Re-identification. In Proceedings of the ECCV. 87--104.

Cited By

Cheng XDeng SYu HZhao G(2025)DMANet: Dual-modality alignment network for visible–infrared person re-identificationPattern Recognition10.1016/j.patcog.2024.110859157(110859)Online publication date: Jan-2025
https://doi.org/10.1016/j.patcog.2024.110859
Lu LWang RChen ZChen J(2024)Cross-modal person re-identification using fused local effective features and multi-scale featuresTransactions of the Institute of Measurement and Control10.1177/01423312241266275Online publication date: 8-Aug-2024
https://doi.org/10.1177/01423312241266275
Hao CLi YSun JChen H(2024)MFEN: Multi-scale Feature Expansion Network for Visible-Infrared Person Re-IdentificationProceedings of the International Conference on Computer Vision and Deep Learning10.1145/3653781.3653824(1-6)Online publication date: Jun-2024
https://doi.org/10.1145/3653781.3653824
Show More Cited By

Index Terms

Towards a Unified Middle Modality Learning for Visible-Infrared Person Re-Identification
1. Information systems
  1. Information retrieval

Recommendations

Video-based Visible-Infrared Person Re-Identification via Style Disturbance Defense and Dual Interaction
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Video-based visible-infrared person re-identification (VVI-ReID) aims to retrieve video sequences of the same pedestrian from different modalities. The key of VVI-ReID is to learn discriminative sequence-level representations that are invariant to both ...
Cross-Modality Transformer for Visible-Infrared Person Re-Identification
Computer Vision – ECCV 2022
Abstract
Visible-infrared person re-identification (VI-ReID) is a challenging task due to the large cross-modality discrepancies and intra-class variations. Existing works mainly focus on learning modality-shared representations by embedding different ...
Credible Dual-X Modality Learning for Visible and Infrared Person Re-Identification
PRICAI 2023: Trends in Artificial Intelligence
Abstract
Visible-Infrared person Re-Identification (VI-ReID) is essential for public security. However, it poses a significant challenge due to the distinct reflection frequencies of visible and infrared modalities, leading to a substantial semantic gap ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
881
Total Downloads

Downloads (Last 12 months)297
Downloads (Last 6 weeks)20

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng XDeng SYu HZhao G(2025)DMANet: Dual-modality alignment network for visible–infrared person re-identificationPattern Recognition10.1016/j.patcog.2024.110859157(110859)Online publication date: Jan-2025
https://doi.org/10.1016/j.patcog.2024.110859
Lu LWang RChen ZChen J(2024)Cross-modal person re-identification using fused local effective features and multi-scale featuresTransactions of the Institute of Measurement and Control10.1177/01423312241266275Online publication date: 8-Aug-2024
https://doi.org/10.1177/01423312241266275
Hao CLi YSun JChen H(2024)MFEN: Multi-scale Feature Expansion Network for Visible-Infrared Person Re-IdentificationProceedings of the International Conference on Computer Vision and Deep Learning10.1145/3653781.3653824(1-6)Online publication date: Jun-2024
https://doi.org/10.1145/3653781.3653824
Dai LLuo ZLi SZheng ZWang YQian XZhong ZWang ZZheng L(2024)Exploring Part Features for Unsupervised Visible-Infrared Person Re-IdentificationProceedings of the 1st ICMR Workshop on Multimedia Object Re-Identification10.1145/3643490.3661809(1-5)Online publication date: 10-Jun-2024
https://doi.org/10.1145/3643490.3661809
Kim SGwon SSeo K(2024)Enhancing Diverse Intra-identity Representation for Visible-Infrared Person Re-Identification2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00250(2501-2510)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00250
Yang XDong WLi MWei ZWang NGao X(2024)Cooperative Separation of Modality Shared-Specific Features for Visible-Infrared Person Re-IdentificationIEEE Transactions on Multimedia10.1109/TMM.2024.337713926(8172-8183)Online publication date: 13-Mar-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3377139
Li ZLiu HPeng XJiang W(2024)Inter-Intra Modality Knowledge Learning and Clustering Noise Alleviation for Unsupervised Visible-Infrared Person Re-IdentificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.336730436:8(3934-3947)Online publication date: Aug-2024
https://doi.org/10.1109/TKDE.2024.3367304
Niu XLi ELiu JWang YOsadchy MFang Y(2024)Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNetIEEE Transactions on Image Processing10.1109/TIP.2023.334865633(655-670)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2023.3348656
Liu MSun YWang XBian YZhang ZWang Y(2024)Pose-Guided Modality-Invariant Feature Alignment for Visible–Infrared Object Re-IdentificationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.338455873(1-10)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3384558
Chan SMeng WBai CHu JChen S(2024)Diverse-Feature Collaborative Progressive Learning for Visible-Infrared Person Re-IdentificationIEEE Transactions on Industrial Informatics10.1109/TII.2024.335943220:5(7754-7763)Online publication date: May-2024
https://doi.org/10.1109/TII.2024.3359432
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents