You are using an outdated browser. Please upgrade your browser
Our user interface for collecting annotations shows the user an image and asks them, for a particular pair of pixels (indicated with crosshairs and labeled Points 1 and 2), which of the two points has a darker surface color. The user can then select one of three options: Point 1, Point 2, and About the same. We ask users to specify their confidence in their assessment as Guessing, Probably, or Definitely, as was done by [Branson et al. 2010].
We aggregate judgements from 5 users for each pair of points and use the CUBAM machine learning model [Welinder et al. 2010] to model two forms of bias.
See our publication for more details.
Our user interface
