We propose a regression-based automatic evaluation method that evaluates the utterances generated by chat-oriented dialogue systems based on the similarities to ...

Automatic Evaluation of Chat-Oriented Dialogue Systems Using Large-Scale Multi-references: 8th International Workshop on Spoken Dialog Systems. January 2019.

Sugiyama H. et al. Automatic Evaluation of Chat-Oriented Dialogue Systems Using Large-Scale Multi-references // Lecture Notes in Electrical Engineering. 2018.

Jun 10, 2020 · The evaluation of chat-oriented dialogue systems has been typically accomplished through the use of automated metrics and human evaluation (Sec-.

270 References · Automatic Evaluation of Chat-Oriented Dialogue Systems Using Large-Scale Multi-references · Real User Evaluation of Spoken Dialogue Systems Using ...

This book presents lectures given at the 8th International Workshop on Spoken Dialog Systems. As agents evolve in terms of their ability to carry on a dialog ...

The automatic evaluation of chat-oriented dialogue systems remains an open problem. Most studies have evaluated them by hand, but this approach requires huge ...

The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi- reference evaluation.

A series of experiments show that the use of multiple references results in improved correlation between several automatic metrics and human judgement for ...

Dec 1, 2020 · We propose a new BERT-based evaluation metric called DEB, which is pretrained on 727M Reddit conversations and then finetuned on our dataset.