Writing Instructions: Best Practices
Below are some best practices to consider as you write instructions:
Dive into your data: Before writing instructions, it’s important to review a representative sample of your data to understand what global and edge case rules need to be written. As you’re reviewing the data, put yourself in the labeler's shoes; document areas you’d need guidance on, as well as any tricky cases that need to be addressed in the instructions.
Writing instructions:
Structure: Following a clear structure in your instructions will help labelers better understand the rules, leading to higher quality annotations. The below structure is a good starting point:
Summary of Task: Provide a few pithy bullets describing the basics of your task.
Workflow: Provide the steps labelers must take to annotate your task from start to finish.
Annotation Rules: Start with the global rules (i.e. rules that apply to all tasks, then zoom in to more specific rules). Areas you’ll likely want to touch on:
Minimum pixel size
What to label and what NOT to label
How to manage occlusion, truncation, and low visibility cases
Geometry sizing if relevant
How to manage cases you’re not sure about (i.e. “err on the side of selecting XXX”)
Etc.
Label and Attribute Definitions and Examples
Common Errors
Edge Cases
Communication:
Communicate succinctly and clearly (ie pithy bullets, use spaces to decrease cognitive load for the labeler)
Include at least 1 image for each point you’re looking to make. Make sure each image has a description of WHY it is correctly or incorrectly annotated.
Pressure test your instructions: After writing the instructions, review your data again and make sure your instructions cover the majority (90+% of cases you see). Continue to iterate on instructions until you have hit the 90+% number.
Final tips:
It can be helpful to include a short video walking through the instructions
If there are certain tools / features that you expect labelers to use when annotating your data, we’d suggest mentioning them in the instructions and providing a description of how to use the tool (e.g. interpolation)