Nothing Special   »   [go: up one dir, main page]

SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs

Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu


Abstract
Large language models hold significant potential for integrating various data types, such as text documents and database records, for advanced analytics. However, blending text and numerical data presents substantial challenges. LLMs need to process and cross-reference entities and numbers, handle data inconsistencies and redundancies, and develop planning capabilities such as building a working memory for managing complex data queries. In this paper, we introduce four novel tasks centered around sports data analytics to evaluate the numerical reasoning and information fusion capabilities of LLMs. These tasks involve providing LLMs with detailed, play-by-play sports game descriptions, then challenging them with adversarial scenarios such as new game rules, longer durations, scrambled narratives, and analyzing key statistics in game summaries. We conduct extensive experiments on NBA and NFL games to assess the performance of LLMs on these tasks. Our benchmark, SportsMetrics, introduces a new mechanism for assessing LLMs’ numerical reasoning and fusion skills.
Anthology ID:
2024.acl-long.17
Volume:
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
267–278
Language:
URL:
https://aclanthology.org/2024.acl-long.17
DOI:
10.18653/v1/2024.acl-long.17
Bibkey:
Cite (ACL):
Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, and Fei Liu. 2024. SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 267–278, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs (Hu et al., ACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.acl-long.17.pdf