If you do not see this message displayed properly, please click here


The Faculty of Informatics is pleased to announce a seminar given by Stefano Mizzaro 



Crowdsourcing Relevance Magnitudes
Speaker: Stefano Mizzaro
University of Udine, Italy
Date: Friday, February 5, 2016
Place: USI Lugano Campus, room SI-006, Informatics building (Via G. Buffi 13)
Time: 14:30



Magnitude estimation is a psychophysical scaling technique for the measurement of sensations, where observers assign numbers to stimuli in response to their perceived intensity. I will report some results of a study on the use of magnitude estimation for judging the relevance of documents in the context of information retrieval evaluation. This large scale study involved collecting more than 50,000 magnitude estimation relevance judgments by means of crowdsourcing.
The analysis shows that, perhaps surprisingly, crowdsourcing did work, allowing us to gather good quality data. On average, magnitude estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of magnitude estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used: I will also talk about applying magnitude estimation for calibrating gains in gain-based effectiveness metrics, like nDCG and ERR, directly from user-reported perceptions of relevance. Finally I will also anticipate some results on analyzing free text comments provided by the workers.



Stefano Mizzaro is Associate Professor at the University of Udine, Italy, since 2006. His main research interests for the last 20 years have been (and still are) information retrieval evaluation, mobile and context-aware retrieval, and scholarly publishing and peer review. After publishing is 100th paper, he stopped counting them. He is co-founder of MoBe Ltd., a spinoff company dealing with software and services for mobile devices. In 2014 he spent his sabbatical at RMIT, Melbourne; now he is back home enjoying the good old world


Host: Prof. Fabio Crestani


Faculty of Informatics

Faculty of Informatics
Università della Svizzera italiana
Via Giuseppe Buffi 13
CH-6904 Lugano
Tel.: +41 (0)58 666 46 90
Fax: +41 (0)58 666 45 36
Email: decanato.inf@usi.ch
Web: www.inf.usi.ch
Twitter: @USI_INF


Segui USI@EXPO2015 su Twitter Segui USI@EXPO2015 su Facebook Segui USI@EXPO2015 su Linkedin Segui USI@EXPO2015 su YouTube