Evorus

Crowd-powered Conversational Assistant Built to Automate Itself Over Time

Kenneth Huang, Joseph Chee Chang, Jeff Bigham. ACM SIGCHI 2018 (r=26% N=2595)

BEST PAPER NOMINATION
Crowd-powered chatbots are robust than current pure AI approach, but can be slower and more expensive at runtime. We attempted to combine the two approaches for high quality, low latency, and low cost. We introduce Evorus, a crowd-powered chatbot that automate itself over time by learning to integrate AI chatbots, reusing responses, and assess response quality. A 5-month-long public deployment study shows promising results. You can try talking to Evorus today.

Abstract

Crowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, and low cost solutions. In this paper, we introduce Evorus, a crowd-powered conversational assistant built to automate itself over time by i) allowing new chatbots to be easily integrated to automate more scenarios, ii) reusing prior crowd answers, and iii) learning to automatically approve response candidates. Our 5-month-long deployment with 80 participants and 281 conversations shows that Evorus can automate itself without compromising conversation quality. Crowd-AI architectures have long been proposed as a way to reduce cost and latency for crowd-powered systems; Evorus demonstrates how automation can be introduced successfully in a deployed system. Its architecture allows future researchers to make further innovation on the underlying automated components in the context of a deployed open domain dialog system.

Try Evorus on Google Talk

TalkingToTheCrowd.org

Overview Video

Media Coverage

Downloads

PDF Download arXiv

Citation

1
2
3
4

Kenneth Huang, Joseph Chee Chang, Jeffrey P. Bigham. 2018.
Evorus: A Crowd-powered Conversational AssistantBuilt to Automate Itself Over Time.
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18).
ACM, Montreal, QC, Canada.

Bibtex

1
2
3
4
5
6
7
8
9

@inproceedings{Huang:2018:Evorus,
 author = {Huang, Kenneth and Chang, Joseph Chee and Bigham, Jeffrey P.},
 title = {Evorus: A Crowd-powered Conversational AssistantBuilt to Automate Itself Over Time},
 booktitle = {Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '18},
 year = {2018},
 publisher = {ACM},
 address = {Montreal, QC, Canada},
}