​

Auditing on Rapid

As tasks are completed, you can view and audit them by clicking on a batch in the batches table page. From here you should see a grid of tasks, each with its preview and status shown in a card. On this page you can filter tasks by task status and completion date, as well as randomize the ordering for a non-biased sample to audit from. To view task instructions, view task metadata, and download an individual task response, use the three dot menu in the top right corner of the card. To see which tasks are ready for your attention, filter by “Ready to Audit”. Click on a task card with the “Ready to Audit” status to start auditing that task. You will enter the task audit view, where you’ll find the Task Details (label date, pin note, instruction feedback, task metadata, audit history) sidebar on the right and a bottom navigation bar to quickly switch to other tasks. You can collapse both of these panels to enlarge the audit interface. When you audit, you can choose to either “Accept” the task because the response looks good, “Reject” because the response is completely wrong, or “Fix” to correct the response. Fixing a task response will require you to add/edit/delete annotations in our labeling tool and clicking “Save”. After auditing a task, a modal will pop up to ask if you want to perform any of the following actions:

• Create Training Task: This task will be used to train labelers before they can start working on your project.
• Create Custom Courses Task: Use this task in a custom training course for your labelers to take before they start working on your project.
• Create Evaluation Task: Labelers will randomly receive this task while working on your project and will be kicked off the project if their accuracy is low.
• Insert Example into Instructions: Make this task available for reference to your labelers in your instructions

Audit Modal Creating a training task, custom task, or evaluation task prompts you to add concepts (labeling principles you assess quality for) and rate the difficulty of the task. When creating an instruction example, you will have to choose where to place the example in your instructions. Example tasks can be added into either a Rule or Label section. The example can be inserted into an existing block of examples, or into a new example block. If you have an existing example block that this example would be relevant to, you can simply insert it there. Otherwise, you should create a new block with a relevant description. If you do not care to create quality tasks/instruction examples after each audit, check the “Quick Audit Mode” checkbox at the top of the task audit page (only appears after the minimum number of benchmarks has been met). Once in quick audit mode, pinning tasks will also not prompt the auditor to leave a pin note. If you need to cancel an active batch for any reason, you can find that option in the top right corner of the batch grid page. We will cancel any tasks that have not been completed by our labelers. To receive your task responses, either click “Download Tasks” or “Email Tasks” in the top right corner of the batch grid page. Downloading directly from the dashboard is limited to 1000 task responses (based on your filter parameters). To receive more than 1000 task responses, please use the email button. To bulk create quality tasks or instruction examples from audited tasks, select the “Bulk Create Quality Tasks” button on the batch grid page. You can then select multiple tasks (indicated by the checkbox on the task card) and confirm by clicking “Create Quality Tasks”. This will bring up the quality task creation modal which you’re already familiar with. A recommended workflow is to pin tasks that you would like to use as quality tasks/examples as you audit, filter by pinned tasks on the batch grid page, then use the ‘Bulk Create Quality Tasks’ feature.