BEST PAPER NOMINATION
HCOMP 2016 INVITED ENCORE TALK
Many crowd clustering approaches have difficulties providing global context to workers in order to generate meaningful categories. Alloy uses a sample-and-search technique to provide a better understanding of the global context. It also combines the in-depth semantic knowledge from human computation and the scalability of machine learning models to create rich structures from unorganized documents with high quality and efficiency.

Abstract

Crowdsourced clustering approaches present a promising way to harness deep semantic knowledge for clustering complex information. However, existing approaches have difficulties supporting the global context needed for workers to generate meaningful categories, and are costly because all items require human judgments. We introduce Alloy, a hybrid approach that combines the richness of human judgments with the power of machine algorithms. Alloy supports greater global context through a new sample and search crowd pattern which changes the crowd’s task from classifying a fixed subset of items to actively sampling and querying the entire dataset. It also improves efficiency through a two phase process in which crowds provide examples to help a machine cluster the head of the distribution, then classify low-confidence examples in the tail. To accomplish this, Alloy introduces a modular cast and gather approach which leverages a machine learning backbone to stitch together different types of judgment tasks.

Downloads

PDF Download ACM Digital Library

Media Coverage

• Pittsburgh Post-Gazette: Crowdsourcing work? Get rid of the human supervisor
• Campus Technology: Research Project Mixes Humans and Machines for Better Crowdsourcing
• Neuraoscience News: Crowd Augmented Cognition: Combining Human and Machine Intelligence to Accelerate Learning
• DZone: Research Suggests AI Managers Effective for Crowdsourcing
• PhysOrg: Crowd-augmented cognition: Team develops tools that combine human and machine intelligence to accelerate learning
• TechExplore: Big thinking in small pieces: Computer guides humans in crowdsourced research
• Spend Matters: Crowdsourcing and Cognitive Computing: Are You Ready for the Future of Work?
• Science Daily: Crowd-augmented cognition - combine human, machine intelligence to accelerate learning
• EurekAlert: Big thinking in small pieces: Computer guides humans in crowdsourced research
• EurekAlert: Crowd-augmented cognition
• NSF News: Big thinking in small pieces: Computer guides humans in crowdsourced research
• NSF News: Crowd-augmented cognition
• CMU SCS News: Big thinking in small pieces: Computer guides humans in crowdsourced research
• CMU HCII News: Computer Guides Humans in Crowdsourced Research

Citation

1
2
3
4

Joseph Chee Chang, Aniket Kittur, and Nathan Hahn. 2016.
Alloy: Clustering with Crowds and Computation.
In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16).
ACM, New York, NY, USA, 3180-3191. DOI: http://dx.doi.org/10.1145/2858036.2858411

Bibtex

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

@inproceedings{Chang:2016:ACC:2858036.2858411,
 author = {Chang, Joseph Chee and Kittur, Aniket and Hahn, Nathan},
 title = {Alloy: Clustering with Crowds and Computation},
 booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '16},
 year = {2016},
 isbn = {978-1-4503-3362-7},
 location = {Santa Clara, California, USA},
 pages = {3180--3191},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/2858036.2858411},
 doi = {10.1145/2858036.2858411},
 acmid = {2858411},
 publisher = {ACM},
 address = {New York, NY, USA},
}