Empowering MultiModal Models’ In-Context Learning Ability through Large Language Models
Authors: 
Wenjuan Han, Haozhe Zhao, Zefan CaiAuthors Info & Claims
Pages 9 - 10
Abstract
Pretrained visual-language models (VLMs) have made progress in developing multimodal models to improve various tasks. However, they lack reasoning and in-context learning ability. Building on the success of large language models (LLMs) in general-purple NLP tasks, researchers anticipate that the VLM should also have the same strong reasoning and ICL ability through specific techniques, for example benefiting from LLMs. To boost VLMs to solve vision-language problems via few-shot exemplars, we suggest a vision-language model, called MIC1.
References
[1]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
[2]
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023. Vlp: A survey on vision-language pre-training. Machine Intelligence Research 20, 1 (2023), 38–56.
[3]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arxiv:2305.06500 [cs.CV]
[4]
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems 35 (2022), 30583–30598.
[5]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023).
[6]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 12888–12900.
[7]
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. 2023. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931 (2023).
[8]
Chao Lou, Wenjuan Han, Yuhuan Lin, and Zilong Zheng. 2022. Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs With Language Structures via Dependency Relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 15607–15616.
[9]
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 (2023).
[10]
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, 2023. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205 (2023).
Index Terms
- Empowering MultiModal Models’ In-Context Learning Ability through Large Language Models
Recommendations
Prompting large language model with context and pre-answer for knowledge-based VQA
AbstractExisting studies apply Large Language Model (LLM) to knowledge-based Visual Question Answering (VQA) with encouraging results. Due to the insufficient input information, the previous methods still have shortcomings in constructing the prompt for ...
Highlights- We propose a novel framework that prompts LLM for knowledge-based VQA.
- We add dynamic routing to vanilla VQA model to further inspire the LLM.
- we add a pre-answer selection layer to generate more suitable pre-answers.
- We build ...
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
MM '24: Proceedings of the 32nd ACM International Conference on MultimediaThis tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and video. Participants will gain an understanding of the foundational concepts of ...
Can Large Language Models Be Good Companions?: An LLM-Based Eyewear System with Conversational Common Ground
Developing chatbots as personal companions has long been a goal of artificial intelligence researchers. Recent advances in Large Language Models (LLMs) have delivered a practical solution for endowing chatbots with anthropomorphic language capabilities. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2023
173 pages
ISBN:9798400702334
DOI:10.1145/3603165
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 September 2023
Check for updates
Author Tags
Qualifiers
- Poster
- Research
- Refereed limited
Funding Sources
- the Talent Fund of Beijing Jiaotong University
Conference
ACM TURC '23
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 135Total Downloads
- Downloads (Last 12 months)107
- Downloads (Last 6 weeks)5
Reflects downloads up to 18 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format