Writing Instructions: Best Practices

Below are some best practices to consider as you write instructions:

Dive into your data: Before writing instructions, it’s important to review a representative sample of your data to understand what global and edge case rules need to be written. As you’re reviewing the data, put yourself in the labeler's shoes; document areas you’d need guidance on, as well as any tricky cases that need to be addressed in the instructions.
Writing instructions:
- Structure: Following a clear structure in your instructions will help labelers better understand the rules, leading to higher quality annotations. The below structure is a good starting point:
- Summary of Task: Provide a few pithy bullets describing the basics of your task.
- Workflow: Provide the steps labelers must take to annotate your task from start to finish.
- Annotation Rules: Start with the global rules (i.e. rules that apply to all tasks, then zoom in to more specific rules). Areas you’ll likely want to touch on:
  - Minimum pixel size
  - What to label and what NOT to label
  - How to manage occlusion, truncation, and low visibility cases
  - Geometry sizing if relevant
  - How to manage cases you’re not sure about (i.e. “err on the side of selecting XXX”)
  - Etc.
- Label and Attribute Definitions and Examples
- Common Errors
- Edge Cases
Communication:
- Communicate succinctly and clearly (ie pithy bullets, use spaces to decrease cognitive load for the labeler)
- Include at least 1 image for each point you’re looking to make. Make sure each image has a description of WHY it is correctly or incorrectly annotated.
Pressure test your instructions: After writing the instructions, review your data again and make sure your instructions cover the majority (90+% of cases you see). Continue to iterate on instructions until you have hit the 90+% number.
Final tips:
- It can be helpful to include a short video walking through the instructions
- If there are certain tools / features that you expect labelers to use when annotating your data, we’d suggest mentioning them in the instructions and providing a description of how to use the tool (e.g. interpolation)