Evorus: A crowd-powered conversational assistant that automates itself over time

Kenneth Huang, Joseph Chee Chang, Saiganesh Swaminathan, Jeffrey P. Bigham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Crowd-powered conversational assistants have found to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. One promising direction is to combined the two approaches for high quality and low cost solutions. However, traditional offline approaches of building automated systems with the crowd requires first collecting training data from the crowd, and then training a model before an online system can be launched. In this paper, we introduce Evorus, a crowd-powered conversational assistant with online-learning capability that automate itself over time. Evorus expands a previous crowd-powered conversation system by reducing its reliance on the crowd over time while maintaining the robustness and reliability of human intelligence, by (i) allowing new chatbots to be added to help contribute possible answers, (ii) learning to reuse past responses to similar queries over time, and (iii) learning to reduce the amount of crowd oversight necessary to retain quality. Our deployment study with 28 users show that automated responses were chosen 12.84% of the time, and voting cost was reduced by 6%. Evorus introduced a new framework for constructing crowd-powered conversation systems that can gradually automate themselves using machine learning, a concept that we believe can be generalize to other types of crowd-powered systems for future research.

Original languageEnglish (US)
Title of host publicationUIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
Pages155-157
Number of pages3
ISBN (Electronic)9781450354196
DOIs
StatePublished - Oct 20 2017
Event30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017 - Quebec City, Canada
Duration: Oct 22 2017Oct 25 2017

Publication series

NameUIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology

Conference

Conference30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017
CountryCanada
CityQuebec City
Period10/22/1710/25/17

Fingerprint

Costs
Online systems
Learning systems

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Cite this

Huang, K., Chang, J. C., Swaminathan, S., & Bigham, J. P. (2017). Evorus: A crowd-powered conversational assistant that automates itself over time. In UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 155-157). (UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology). Association for Computing Machinery, Inc. https://doi.org/10.1145/3131785.3131823
Huang, Kenneth ; Chang, Joseph Chee ; Swaminathan, Saiganesh ; Bigham, Jeffrey P. / Evorus : A crowd-powered conversational assistant that automates itself over time. UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2017. pp. 155-157 (UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology).
@inproceedings{24542f40492d463bbaecf9ccdb5ba15a,
title = "Evorus: A crowd-powered conversational assistant that automates itself over time",
abstract = "Crowd-powered conversational assistants have found to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. One promising direction is to combined the two approaches for high quality and low cost solutions. However, traditional offline approaches of building automated systems with the crowd requires first collecting training data from the crowd, and then training a model before an online system can be launched. In this paper, we introduce Evorus, a crowd-powered conversational assistant with online-learning capability that automate itself over time. Evorus expands a previous crowd-powered conversation system by reducing its reliance on the crowd over time while maintaining the robustness and reliability of human intelligence, by (i) allowing new chatbots to be added to help contribute possible answers, (ii) learning to reuse past responses to similar queries over time, and (iii) learning to reduce the amount of crowd oversight necessary to retain quality. Our deployment study with 28 users show that automated responses were chosen 12.84{\%} of the time, and voting cost was reduced by 6{\%}. Evorus introduced a new framework for constructing crowd-powered conversation systems that can gradually automate themselves using machine learning, a concept that we believe can be generalize to other types of crowd-powered systems for future research.",
author = "Kenneth Huang and Chang, {Joseph Chee} and Saiganesh Swaminathan and Bigham, {Jeffrey P.}",
year = "2017",
month = "10",
day = "20",
doi = "10.1145/3131785.3131823",
language = "English (US)",
series = "UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology",
publisher = "Association for Computing Machinery, Inc",
pages = "155--157",
booktitle = "UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology",

}

Huang, K, Chang, JC, Swaminathan, S & Bigham, JP 2017, Evorus: A crowd-powered conversational assistant that automates itself over time. in UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology, Association for Computing Machinery, Inc, pp. 155-157, 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, Quebec City, Canada, 10/22/17. https://doi.org/10.1145/3131785.3131823

Evorus : A crowd-powered conversational assistant that automates itself over time. / Huang, Kenneth; Chang, Joseph Chee; Swaminathan, Saiganesh; Bigham, Jeffrey P.

UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2017. p. 155-157 (UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Evorus

T2 - A crowd-powered conversational assistant that automates itself over time

AU - Huang, Kenneth

AU - Chang, Joseph Chee

AU - Swaminathan, Saiganesh

AU - Bigham, Jeffrey P.

PY - 2017/10/20

Y1 - 2017/10/20

N2 - Crowd-powered conversational assistants have found to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. One promising direction is to combined the two approaches for high quality and low cost solutions. However, traditional offline approaches of building automated systems with the crowd requires first collecting training data from the crowd, and then training a model before an online system can be launched. In this paper, we introduce Evorus, a crowd-powered conversational assistant with online-learning capability that automate itself over time. Evorus expands a previous crowd-powered conversation system by reducing its reliance on the crowd over time while maintaining the robustness and reliability of human intelligence, by (i) allowing new chatbots to be added to help contribute possible answers, (ii) learning to reuse past responses to similar queries over time, and (iii) learning to reduce the amount of crowd oversight necessary to retain quality. Our deployment study with 28 users show that automated responses were chosen 12.84% of the time, and voting cost was reduced by 6%. Evorus introduced a new framework for constructing crowd-powered conversation systems that can gradually automate themselves using machine learning, a concept that we believe can be generalize to other types of crowd-powered systems for future research.

AB - Crowd-powered conversational assistants have found to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. One promising direction is to combined the two approaches for high quality and low cost solutions. However, traditional offline approaches of building automated systems with the crowd requires first collecting training data from the crowd, and then training a model before an online system can be launched. In this paper, we introduce Evorus, a crowd-powered conversational assistant with online-learning capability that automate itself over time. Evorus expands a previous crowd-powered conversation system by reducing its reliance on the crowd over time while maintaining the robustness and reliability of human intelligence, by (i) allowing new chatbots to be added to help contribute possible answers, (ii) learning to reuse past responses to similar queries over time, and (iii) learning to reduce the amount of crowd oversight necessary to retain quality. Our deployment study with 28 users show that automated responses were chosen 12.84% of the time, and voting cost was reduced by 6%. Evorus introduced a new framework for constructing crowd-powered conversation systems that can gradually automate themselves using machine learning, a concept that we believe can be generalize to other types of crowd-powered systems for future research.

UR - http://www.scopus.com/inward/record.url?scp=85041422598&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041422598&partnerID=8YFLogxK

U2 - 10.1145/3131785.3131823

DO - 10.1145/3131785.3131823

M3 - Conference contribution

AN - SCOPUS:85041422598

T3 - UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology

SP - 155

EP - 157

BT - UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology

PB - Association for Computing Machinery, Inc

ER -

Huang K, Chang JC, Swaminathan S, Bigham JP. Evorus: A crowd-powered conversational assistant that automates itself over time. In UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc. 2017. p. 155-157. (UIST 2017 Adjunct - Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology). https://doi.org/10.1145/3131785.3131823