Scale Studio allows teams to add review levels to projects. Reviews help teams better ensure data quality, especially when the initial attempters are new to a project and may not be fully aware of what is expected.
Similar to auditors, reviewers can "Accept," "Reject," or "Fix/Make Changes" to tasks that an attempter has completed.
Every annotator can have one of three types of roles:
- Attempter | These annotators can only work at the "attempt" level, receiving fresh new tasks or tasks that reviewers have rejected.
- Reviewer | These annotators can only work at the "review" level, receiving tasks that an attempter has already worked on.
- Attempter + Reviewer | These annotators can work at both levels. However, if tasks need review, these annotators will be served reviews before new tasks.
Review levels are project dependent, so this setting will need to be configured for each new project. For projects with a review level, teams must ensure that there is at least one reviewer.
Review vs. Audits
Reviews are meant as a method for quality assurance and are required for task completion. If a review is enabled for a project, all tasks will pass through the review level before completion. For example, if the number of review levels is set to "1", then each task will have at least two touches (i.e., one attempt and one review) - more if there are rejections. Generally, it is recommended only to have one review level.
On the other hand, audits are meant to be used to sample tasks by managers and admins to check quality. Generally, it is recommended that teams audit 10 to 20% of their tasks.
Setting Up Review Levels (Admin and Managers Only)
To add review levels:
- Go to the Batches tab in the project dashboard.
- Select the "Settings" button in the center of the page.
- Under Pipeline configuration, adjust the "Number of reviews" to your desired levels.
- Press "Save."
Setting Up Reviewers (Admin and Managers Only)
To set up a reviewer:
- Under Label Team in the account settings, ensure that the annotator is assigned to the project.
- Go to the Labeler Management tab in the project dashboard.
- Under "Labeling Team for [Project Name]," select the annotator(s) whose role will be changed.
- Select the "Change role" button.
- Select the role desired for the annotator(s).
- Press "Update Roles."