A confidence-aware top-k query processing toolkit on crowdsourcing

Ranking techniques have been widely used in ubiquitous applications like recommendation, information retrieval, etc. For ranking computation hostile but human friendly items, crowdsourcing is considered as an emerging technique to process the ranking by human power. However, there is a lack of an easy-to-use toolkit for answering crowdsourced top-k query with minimal effort.

In this work, we demonstrate an interactive programming toolkit that is a unified solution for answering the crowd-sourced top-k queries. The toolkit employs a new confidence-aware crowdsourced top-k algorithm, SPR. The whole progress of the algorithm is monitored and visualized to end users in a timely manner. Besides the visualized result and the statistics, the system also reports the estimation of the monetary cost and the breakdown of each phase. Based on the estimation, end users can strike a balance between the budget and the quality through the interface of this toolkit.