On the Relation Between Products’ Relevance and Customers’ Satisfaction in Voice Shopping

In this talk we consider the domain of voice shopping in Alexa, Amazon’s voice assistant. In this domain, search scenarios are integral part of the shopping sessions, where users seek for a product to buy, or for some information about a product. The fact that in voice search, both the input and output are spoken, involves many challenges in automatic speech recognition, natural language understanding, question answering, and new user experiences. We will elaborate on customers’ behavior in voice shopping, where we have observed an interesting and surprising phenomenon that many customers purchase or engage with irrelevant search results. The term “irrelevance” may mislead, since a relevant item is typically interpreted as “anything that satisfies the user needs”. Thus, the title of this work may look as an oxymoron – the purchase of a product is a strong signal of relevance to the customer. In the context of this work we take a simplified approach. We mark product items as relevant or irrelevant to the user query based on the relevance judgments of several human annotators. However, even in the context of objective relevance judgments, it is still surprising that so many customers engage with irrelevant results. We will analyze this phenomenon and demonstrate its significance. We will offer several hypotheses as to the reasons behind customers’ purchase and engagement with irrelevant results, including customers’ personal preferences, trendiness of the products and their relatedness to the query, the query intent and the product price.