How To Use Quality Lab

Quality Lab is your mission control for all the training & evaluation tasks you have created. Here, you can see all the tasks you've turned into training tasks or evaluation tasks - as well as their performance track record.

See average accuracy scores and number of completions by clicking into any of the quality tasks.

From this view, you can also see the responses from individuals who have completed each benchmark. You can go through and see how different labelers performed, and the accuracy of each attempt. Simply click into the benchmark you're interested in seeing more details about, hit "View sample responses", and scroll through the responses.

Retiring Training/Evaluation Tasks
If you feel like one (or more) of your quality tasks were poorly made, you have the option to retire it. You may choose to do this if a task is too easy, or too hard - or it's just been done by all your annotators already and there's no point in continuing to have it around. Simply click into the task that you'd like to retire, and click the "Retire" button at the bottom right.

You can always change your mind later re-enable the task from the "Disabled Tasks" section.

Advanced Settings

There are many advanced options you can take advantage of to further boost your quality game.

Rank Responses
Adjusting the ranking of responses will update the grading and create a quality grading system more in tune with your needs. We will infer, based on how you've ranked your annotated results, what you prioritize in your tasks - and we will update our grading schema in the backend to better reflect that.

Mass Retire
You can specify criteria to help auto-retire your training & evaluation tasks. You can specify the pass rate, as well as the # of completions that are required in order to require a task. For example, you can say: all tasks with over 80% accuracy, and have been completed over 20 times will be retired.

Evaluation Task Frequency
You can specify how often you want an evaluation task to come up. The higher you set the frequency to, the more often your annotators will be served an evaluation task. If you set your frequency to 10% that means that 1 in every 10 tasks your annotators get will be an evaluation task.

Evaluation Task Recycling
Decide whether you want people to be able to do the same evaluation task, if you've run out of others. If turned on, people may see evaluation tasks they've already attempted in the past.