Calibration Workflow
Before you can send off loads of data to the labelers, you're required to calibrate your project by running a Calibration Batch. This will allow taskers to test your project, and they'll give you feedback on the instructions and attempt to label a small subset of your data.
Calibration Batches serve the purpose of making sure
You have a properly functioning Project
You have completed tasks that can be turned into Training tasks for the labelers
You have completed tasks that can be turned in Evaluation tasks to track labeler performance
Note that this is an iterative process. You may have to create more than one Calibration Batch to properly fine-tune your instructions and create the necessary number of Quality Tasks.
Creating a Calibration batch
You can create a Calibration Batch from the Batches page of your project.
Calibration batches are required before you can unlock a regular production batch
You'll select a subset of your data for the Calibration Batch.
Select data for Calibration Batch. Ideally you'll select around 20 items.
Once you've created your Calibration Batch, it should be completed within a few hours.
Auditing a Calibration batch
The second part of calibrating your project is to view the results. It is important to take the time to review instruction feedback as well as the actual labelling attempt.
Instruction feedback
Calibration Batches provide the inputs you'll need to refine your instructions. The higher the clarity of your instructions, the better each Tasker will be able to perform on your project. You can view this feedback in the Calibration Batch view under Tasks.
Any available feedback can also be seen in the Instruction Editor, so you can view the feedback as you refine your project's instructions.
Auditing the batch
At the top of the Calibration Batch view you should be able to open the Audit Tool to go through the tasks and grade the attempt.
Click Open Audit Tool to launch the audit
Go through each item of the Calibration to fully complete the Calibration Batch
Once you've finished your audit, you'll have a returned Calibration Score that is represents how accurately the taskers labeled.
This score is only available after you complete the audit. It takes into account a confidence score reported by the Taskers as well as how accuracy their labels actually were.
You might need to run more than one Calibration Batches until the calibration scores are high. Calibration Batches help to resolve initial confusions about the instructions and help you iterate.
Note, for tasks that require a higher level of english understanding or other language requirements, you can use advanced controls to specify
Advanced controls