Improving Calibration Score

Calibration Batches are a critical step in the iteration of your project. Getting a good Calibration Score indicates a healthy project where taskers are able to understand your instructions and label your data.

Understanding Calibration Score

Calibration Score is a function of audit results and Tasker confidence
It is recommended that your Calibration Score is above 80% before moving on to a Production Batch

Inputs that go into calculating the Calibration Score include:

• The number of rejections you made while auditing the returned completed tasks

• How confident the Tasker self-reported after reading the instructions & completing your task

Here a score of 42% indicates improvements should be made before progressing to a Production Batch

Implement Tasker feedback

Iterate on instructions
Write Instructions that cater better to the moments of confusion taskers may be having

For each Calibration Batch you will receive feedback on each task, based on how easy or confusing the Tasker found the process of reading the instructions and executing on the task.

Feedback is on the right side of each Task in the Calibration Batch view

Each task can be expanded to see complete feedback & all actions that can be taken

Per feedback, you may want to view the Audit Response (to see tasker response and any of your audit versions) and see the instructions attached to that task. To edit the Instructions, you'll need to go to Task Setup > Current Instructions, where you can view the feedback and Calibration Score next to your instruction workspace.

Feedback from the most recent Calibration Batch is situated next to the workspace

Commonly, the feedback may point to adding more context, explanations, and examples of properly and improperly labelled tasks.

Show examples allows you to see all examples and add newly labelled examples

You can include examples as Well labeled or Poorly labeled

In general, you should aim to write instructions in a way that shows how one should do the task as opposed to just describing it.

Rapid Tip: Show how to do the task instead of just describing it

🐶 Example Image Annotation Project

Dogs & their coat color

❌ Don't do: Describing the task

Label dogs and their coat colors [solid, merle, brindle, harlequin, ticked, spotted, roan, tricolor]

✔️ Do: Show how to do the task

Describe overall goal of labelling dogs and their coat colors.

"The goal of this project is to properly distinguish between dog coat colors across a variety of breeds. We would like to have each dog in the dataset labeled and also have their coat coloring marked."

Give an overview of how to label dogs:

"Draw a bounding polygon around the dogs, making sure to include all legs, ears, and tails"

→ Include examples of well drawn and poorly drawn polygons.

Give an overview of coat colors:

"Here are multiple examples of all of the coat colors across different breeds..."

→ Include various examples for coat colors.