Improving General Quality

When properly set up, Evaluation Tasks are the primary way to monitor the overall quality of a project, and will also be a big indicator of how successful your Production batches will be.

It is generally helpful to make as many evaluation tasks as possible that cover the range of examples that might be seen in your project. For all task types besides document transcription and lidar, Rapid will try to suggest tasks that will round out your evaluation task set to ensure that there is quality monitoring across your whole project taxonomy. Tasks are suggested upon production batch completion and can be accepted, fixed, or rejected from the issues queue.

Understanding why quality is poor

Inspect common answers
Look to see what are the trends in the task completions through the Common Answers feature.

Inspect common answers
Look to see what are the trends in the task completions through the Common Answers feature.

This specific selection will show results that are 0-50% accurate

If taskers seem to make a lot of common mistakes...

Create more comprehensive training
Through looking at Common Answers you can identify which tasks have a lot of common mistakes. These tasks are good candidates to be used as Training Tasks.

For any task, you can easily Convert into a training task right at the Task view.

Clicking Convert allows you to convert to training task

Add common mistakes to your Instructions
Common mistakes can also be included in your Instructions to provide examples of what not to do to the taskers.

In Definition you can access Advanced controls and automatically convert rejected audits into common errors in your Project Instructions.

Common errors updates adjust your Instructions to include rejected audits

Tag all of your evaluation tasks with difficulty and concepts
When editing or creating evaluation tasks, you can classify their difficulty and tag them with "concepts" that you define. Ensuring that all evaluation tasks have concepts and difficulty allows Rapid to more comprehensively evaluate labelers and ensure that only those who are best suited for your project are working on it. If labelers are making common mistakes, it might make sense to create a concept for the particular mistake you're guarding against

You can edit difficulty and concepts at the top of the evaluation task window, highlighted in the red box

If reported quality seems mismatched with your expectations...

Tune grading
Adjusting the ranking of responses will update the grading and create a quality grading system more in tune with your needs

Access Tune Grading through Quality Lab

Adjust a random sample of 5 tasks to be ranked from best to worst - this ranking will tune the grading if a valid grading configuration exists for that order