Crowdsourcing platforms like Amazon MTurk are being used to collect manually generated data and annotations at scale. Such platforms can be accessed programmatically and are commonly used to build hybrid human-machine systems that leverage machines to scale over large amounts of data and keep humans in the loop to increase data processing quality.

After an introduction on crowdsourcing and hybrid human-machine systems, in this talk we will present a longitudinal data science study on the evolution of the most popular micro-task crowdsourcing platform. We will then focus on some of the open research challenges around crowdsourcing effectiveness and efficiency. In this context, we will present a study on the effect of limiting the time available to complete a task on the accuracy of the work done by the crowd. We will then present work on understanding malicious worker behaviours and work on understanding of the effects of crowdsourcing work environments.

Dr. Gianluca Demartini is a Senior Lecturer in Data Science at the University of Queensland, School of Information Technology and Electrical Engineering. His main research interests are Human Computation, Information Retrieval, and Semantic Web. His research has been supported by the UK Engineering and Physical Sciences Research Council (EPSRC) and by the EU H2020 framework program. He received the Best Paper Award at the European Conference on Information Retrieval (ECIR) in 2016 and the Best Demo Award at the International Semantic Web Conference (ISWC) in 2011. He has published more than 70 peer-reviewed scientific publications including papers at major venues such as WWW, ACM SIGIR, VLDBJ, ISWC, and ACM CHI. He has given several invited talks, tutorials, and keynotes at a number of academic conferences, companies, and Dagstuhl seminars. He is an ACM Distinguished Speaker since 2015.

Before joining the University of Queensland, he was Lecturer at the University of Sheffield in UK, post-doctoral researcher at the eXascale Infolab at the University of Fribourg in Switzerland, visiting researcher at UC Berkeley, junior researcher at the L3S Research Center in Germany, and intern at Yahoo! Research in Spain. In 2011, he obtained a Ph.D. in Computer Science at the Leibniz University of Hanover focusing on Semantic Search.