Jul 9, 2024 · We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error ...
Jul 9, 2024 · StorySumm consists of 96 short stories and LLM-generated summaries with localized faithfulness errors and explanations. Each unfaithful summary is labeled as ...
The StorySumm dataset is in the file storysumm.json. Description of data fields: label - final label for the summary (0 is unfaithful, 1 is faithful)
Jul 9, 2024 · We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error ...
STORYSUMM: Evaluating Faithfulness in Story Summarization
www.aimodels.fyi › papers › arxiv › stor...
Jul 9, 2024 · This paper introduces StorySumm, a new dataset for evaluating the faithfulness of story summarization models.
Jul 10, 2024 · Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source ...
Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative ...
People also ask
What is the theme of faithfulness?
What is faithfulness of explanations?
STORYSUMM: Evaluating Faithfulness in Story Summarization · Melanie Subbiah ... A new dataset, STORYSUMM, comprising LLM summaries of short stories with localized ...
We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This ...
By focusing on faithfulness in narrative summarization and using real-world data from LLMs and Reddit, STORYSUMM poses a realistic but hard benchmark to push ...