Efficient Crowdsourcing via Bayes Quality Control

Hao Su; Shan Liao

doi:10.4028/www.scientific.net/AMR.998-999.1576

Paper Titles

Application of Fast Mapping Techniques in Sudden Crisis
p.1553

Cloudy-Based Optimization of the Human Resource Management System
p.1557

Development of Logistics Park Based on Symbiosis Theory
p.1563

Dynamic Analysis of Human Behavior Based on User Log of ACM
p.1570

Efficient Crowdsourcing via Bayes Quality Control
p.1576

Evaluation Research of Pig Farmers’ Behaviors of Veterinary Drug Usage Based on Factor Analysis
p.1581

Facility-Optimizing Plan for Intersection System Based on Intelligence Traffic Management Technique
p.1586

Performance Evaluation of Logistics and Distribution
p.1591

Research of Bid Evaluation Risk Analysis Model Based on Decision Tree
p.1595

HomeAdvanced Materials ResearchAdvanced Materials Research Vols. 998-999Efficient Crowdsourcing via Bayes Quality Control

Efficient Crowdsourcing via Bayes Quality Control

Abstract:

Crowdsourcing has become an important tool to aggregate the wisdom of the crowd in this Internet age. A central problem in building an online crowdsourcing system is to determine the appropriate number of workers to assign tasks to. We study this problem by formulating a Bayes Net framework and introduce a quantity derived from posterior distribution to measure the convergence of crowd opinions. Using this quantity, our algorithm could stop soliciting opinions from more workers if the distribution of opinions is unlikely to change in future predictions. We empirically demonstrate the effectiveness of the designed strategy by building a challenging fine-grained image annotation task on Amazon Mechanical Turk. Experiment results show that our approach not only saves annotation cost but also guarantees high annotation quality.

You might also be interested in these eBooks

Advances in Applied Sciences, Engineering and Technology II

View Preview

Info:

Periodical:

Advanced Materials Research (Volumes 998-999)

Pages:

1576-1580

DOI:

https://doi.org/10.4028/www.scientific.net/AMR.998-999.1576

Citation:

Cite this paper

Online since:

July 2014

Authors:

Hao Su*, Shan Liao

Keywords:

Bayes Network, Crowd-Sourcing, Quality Control

Export:

RIS, BibTeX

Price:

Permissions CCC:

Request Permissions

Permissions PLS:

Request Permissions

Сopyright:

Citation:

* - Corresponding Author

References

[1] Doan, A; Ramarkrishnan, R; Halevy, A (2011), Crowdsourcing Systems on the World Wide Web, Communications of the ACM 54 (4): 86–96.

DOI: 10.1145/1924421.1924442

Google Scholar

[2] Rulison, Larry (February 14, 2012). A Winning App? They Hope So,. Times Union (Albany).

Google Scholar

[3] Deng, J.; Dong, W.; Socher, R.; Li, L. -J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical ImageDatabase. In CVPR09.

DOI: 10.1109/cvpr.2009.5206848

Google Scholar

[4] Sorokin, A., and Forsyth, D. 2008. Utility data annotationwith amazon mechanical turk. 2008 IEEE Computer SocietyConference on Computer Vision and Pattern RecognitionWorkshops 51(c): 1–8.

DOI: 10.1109/cvprw.2008.4562953

Google Scholar

[5] P. Welinder, P. P. 2010. Online crowdsourcing: rating annotatorsand obtaining cost-effective labels. In CVPR.

Google Scholar

[6] Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The multidimensional wisdom of crowds. In NIPS, 2424–2432.

Google Scholar

[7] Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan,J. R. 2009. Whose vote should count more: Optimal integrationof labels from labelers of unknown expertise. In NIPS, 2035–(2043).

Google Scholar

[8] Branson, S.; Perona, P.; and Belongie, S. 2011. Strong supervisionfrom weak annotation: Interactive training of deformablepart models. In ICCV, 1832–1839.

DOI: 10.1109/iccv.2011.6126450

Google Scholar

[9] Vijayanarasimhan, S., and Grauman, K. 2011. Large-scalelive active learning: Training object detectors with crawleddata and crowds. In CVPR, 1449–1456.

DOI: 10.1109/cvpr.2011.5995430

Google Scholar