Quality Control Method in Crowdsourcing Platform for Professional Dictionary Compilation Process (PDCCP) - Process-Aware Systems - page 92

7 Conclusion

In this paper, we introduced a large-scale non-computing automatic testing
crowdsourcing task for professional dictionary compilation (PDCCP) and mainly focused
on the quality control method in it. We proposed a quality testing method that uses
another crowdsourcing task for the quality testing (audition) part of translation task. We
also explored some task distribution strategy, static or competency, to stimulate task
attendants to finish their work earlier. From the experiments, we concluded that quality
testing method has great influences to efficiency while less impact on quality due to the
ineffective task distribution strategy that should take advantage of quality feedbacks or
the ensured high quality of task attendants. Competency strategy compared to static
strategy has more remarkable effects on efficiency but it does not have notable influences
on quality that might due to the overall high quality of task attendants. In the future, (1)
we should increase the experiment size and complexity and further measure the
influences of quality testing method and task distribution strategy to crowdsourcing task
efficiency and quality. (2) We should improve task distribution strategy based on the
feedback from quality testing method. (3) Explore other potential of crowdsourcing task,
such as try to determine the difficulty level of subtasks and better the task distribution
method to enhance efficiency.

Acknowledgement. This work is partially supported by China National Science
Foundation (Granted Number 61272438, 61472253), Research Funds of Science and
Technology Commission of Shanghai Municipality (Granted Number 14511107702,
12511502704).

References

[1] Howe, J.: The rise of crowdsourcing. Wired Magazine 14(6), 1-4 (2006)
[2] Brabham, D.C.: Crowdsourcing as a model for problem solving an introduction and
cases. Convergence: The International Journal of Research into New Media
Technologies 14(1), 75-90 (2008)
[3] Walter, S.D.: Gold Standard Test Encyclopedia of Biostatistics
[4] Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data
via the EM algorithm. Journal of the Royal Statistical Society 39(1), 1-38 (1977)
[5] Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk.
In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64-67.
ACM (2010)
[6] Zhang, Z., Pang, J., Xie, X.: Research on crowdsourcing strategy of quality control and
evaluation algorithm. Chinese Journal of Computers 36(8), 1636-1649 (2013)
[7] Howe, J.: The rise of crowdsourcing. Wired Magazine 14(6), 1-4 (2006)
[8] Brabham, D.C.: Crowdsourcing as a model for problem solving an introduction and
cases. Convergence: the International Journal of Research into New Media
Technologies 14(1), 75-90 (2008)
[9] Doan, A., Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing systems on the world-wide
web. Communications of the ACM 54(4), 86-96 (2011)
[10] Franklin, M.J., Kossmann, D., Kraska, T., et al.: CrowdDB: answering queries with
crowdsourcing. In: Proceedings of the 2011 ACM SIGMOD International Conference on
Management of Data, pp. 61-72. ACM (2011)