Calibration Workflow

Before you can send off loads of data to the labelers, you're required to calibrate your project by running a Calibration Batch. This will allow taskers to test your project, and they'll give you feedback on the instructions and attempt to label a small subset of your data.

Calibration Batches serve the purpose of making sure

You have a properly functioning Project
You have completed tasks that can be turned into Training tasks for the labelers
You have completed tasks that can be turned in Evaluation tasks to track labeler performance

Note that this is an iterative process. You may have to create more than one Calibration Batch to properly fine-tune your instructions and create the necessary number of Quality Tasks.

Creating a Calibration batch

You can create a Calibration Batch from the Batches page of your project.

Calibration batches are required before you can unlock a regular production batch

You'll select a subset of your data for the Calibration Batch.

Select data for Calibration Batch. Ideally you'll select around 20 items.

Once you've created your Calibration Batch, it should be completed within a few hours.

Auditing a Calibration batch

The second part of calibrating your project is to view the results. It is important to take the time to review instruction feedback as well as the actual labelling attempt.

Instruction feedback
Calibration Batches provide the inputs you'll need to refine your instructions. The higher the clarity of your instructions, the better each Tasker will be able to perform on your project. You can view this feedback in the Calibration Batch view under Tasks.

Any available feedback can also be seen in the Instruction Editor, so you can view the feedback as you refine your project's instructions.

Auditing the batch

At the top of the Calibration Batch view you should be able to open the Audit Tool to go through the tasks and grade the attempt.

Click Open Audit Tool to launch the audit

Go through each item of the Calibration to fully complete the Calibration Batch

Once you've finished your audit, you'll have a returned Calibration Score that is represents how accurately the taskers labeled.

This score is only available after you complete the audit. It takes into account a confidence score reported by the Taskers as well as how accuracy their labels actually were.

You might need to run more than one Calibration Batches until the calibration scores are high. Calibration Batches help to resolve initial confusions about the instructions and help you iterate.

Note, for tasks that require a higher level of english understanding or other language requirements, you can use advanced controls to specify

Advanced controls