module: mturk.cubam

Utilities for using the CUBAM model (python package cubam), as described in the paper

Welinder P., Branson S., Belongie S., Perona, P. “The Multidimensional Wisdom of Crowds.” Conference on Neural Information Processing Systems (NIPS) 2010.
mturk.cubam.compute_cubam_1d_signal_model_scores(labels)

Takes an array of labels with multiple votes per object, and computes a score of how likely each object is to be 1. It runs the 1d Binary Signal Model described in:

Welinder P., Branson S., Belongie S., Perona, P. “The Multidimensional Wisdom of Crowds.” Conference on Neural Information Processing Systems (NIPS) 2010.
Parameters:labels – array of tuples (object_id, user_id, label)
Returns:Dictionary mapping {object_id: score}. The more positive the score, the more likely it is to be a 1. To minimize the error rate, threshold at 0. To avoid false positives, threshold at a higher value.
Return type:dict
mturk.cubam.update_votes_cubam(object_model, labels, object_attr, label_attr, object_label_attr, user_attr='user_id', quality_method_attr='quality_method', score_threshold=0, min_votes=4, show_progress=False, return_changed_objects=False, experiment_filter=None)

Updates all binary labels for an object model (object_model) using the CUBAM model from:

Welinder P., Branson S., Belongie S., Perona, P. “The Multidimensional Wisdom of Crowds.” Conference on Neural Information Processing Systems (NIPS) 2010.
Parameters:
  • object_model (django.db.models.Model) – django model for objects being labeled
  • labels (models.Model or django.db.models.query.QuerySet or list) – list of labels. Each item in the list is expected to be an object x such that getattr(x, label_attr) is the Boolean label to be used.
  • object_attr – name of the object id on each label. So, for each label x, getattr(x, object_attr) should give the object that the label is describing.
  • label_attr – name of the label attribute (e.g. 'correct' or 'flat')
  • object_label_attr – name of the aggregated attribute on the object (e.g. 'scene_quality_correct'). Note: object_label_attr + '_score' is also set to be the CUBAM score.
  • user_attr – (default: 'user_id') name of the user attribute on the label. So, for each label x, getattr(x, user_attr) should give the user that provided that label.
  • quality_method_attr – (default 'quality_method') name of the attribute on the label that records which method set the label. If not None, this attribute is set to 'C'.
  • score_threshold – threshold for CUBAM scores where a label is considered True.
  • min_votes – objects with fewer than this many votes are not updated.
  • show_progress – if True, render progress bars during the operation.
  • return_changed_objects – if True, return a list of objects that were updated.
  • experiment_filter – if specified, the filter to obtain the experiments that this label was obtained from. Typically the filter is {'slug': '<your_slug>'}. This is used to check whether anything new was submitted since the last time this was run. If nothing changed, this funciton returns without updating.
Returns:

the changed objects if return_changed_objects=True, else None
Return type:

list or None