Documentation Index
Fetch the complete documentation index at: https://api-reference.scale.com/llms.txt
Use this file to discover all available pages before exploring further.
Fixless Audits
Fixless Audits improve the quality and accuracy of task auditing by allowing auditors to provide feedback without making any fixes. Fixless Audits are quicker and easier to create when compared to standard audits where customers fix tasks. These audits can be created through the API or in Scale’s LidarLite auditing tool. Fixless Audits submitted via the API can be reviewed and adjusted in LidarLite, allowing auditors to add, delete, or edit Feedback Items as needed. This guide walks through Fixless Audit creation and review using the API.Create a Fixless Audit
Fixless Audits first require some basic information: the relevant task ID, audit result, and the audited task response URL. When you create a fixless audit via the API, you’ll need to create an audit payload in the format outlined in this documentation.Retrieve a Fixless Audit
Fixless audits can be retrieved via our endpoint, and are structured in the same format as when you create an audit. An example response format has been included below the Creation Payload below.Feedback Items
Fixless Audits primarily consist of Feedback Items, which indicate errors, comments, confirmations, or flags on the task. Fixless Audits can be submitted without any Feedback Items. This is a valid audit and indicates that the task has no errors.Feedback Item Type (required)
We usetype to indicate the purpose of the Feedback Item. The most common type is error which indicates a mistake on the task. Other types: flag, confirmation, comment, are less common and are not fixed by Scale or considered in quality score calculations.
Error: The annotation is incorrectFlag: The annotation needs to be reviewed by a customer auditor, does not get fixed by Scale, and does not affect quality scores.Confirmation: The annotation is correct, does not affect quality scoresComment: Generic feedback, does not affect quality scores
Feedback Item Category (required)
category indicates the kind of issue found on the task.
missing- indicates that an annotation is missing. This is also referred to as a false negative.extraneous- indicates that the annotation should not be present. This is also referred to as a false positive.geometry- indicates that the annotation has the wrong dimensions or shapeposition- indicates that the annotation is in the wrong placeattribute- indicates that the annotation has an incorrect attribute propertylabel- indicates that the annotation label (also referred to as class) is incorrect
missing are specific to an annotation_id. missingerrors require a point/ polygon and a missing_annotation_typeto indicate what type of annotation is missing and where it should be.
Feedback Item Scope (required)
scope indicates the location and duration of the issue. We have 4 different schemas for Feedback Item Scopes, which apply to different error categories. Details on typing and required properties can be found in the **Create Fixless Audit **section, below.
FeedbackItemScopeAnnotation- indicates the annotation ID, frame interval, and attribute where the issue is present. This schema is used when the category is:extraneous,geometry,position,attribute, orlabelFeedbackItemScopePolygon- indicates the 3D or 2D position and frame interval where the issue is present. This schema is used when the category is:missingFeedbackItemScopePoint- indicates the 3D or 2D position and frame interval where the issue is present. This schema is used when the category is:missingFeedbackItemScopeScene- indicates that the issue is related to a scene attribute (not an individual annotation). This schema is used when the category is:attribute
Feedback Item Severity (optional - defaults to standard)
severity indicates the seriousness of the issue. Severity can be mild, standard or severe.
-
Mild: Used when the auditor wants to flag an error, but the error doesn’t exceed our SLA error threshold (e.g. cuboid position is off by <30cm). ** **These errors will not affect the quality score -
Standard:Used when the auditor wants to flag an error that exceeds our SLA error threshold (e.g. cuboid position is off by >30cm). These errors will affect the quality score -
Severe: Used when the auditor wants to flag an error that exceeds our SLA error threshold (e.g. cuboid position is off by >30cm) and the error is critical and requires special attention. These errors will affect the quality score
Feedback Item Description and Metadata (optional)
description is an open text field shown in the tasking and auditing UI. This field can be used to describe the error or fix.
metadata is an object that can be used to store data for internal tracking. For instance, in metadata you might store metadata.is_verified_by_human: trueand metadata.confidence_level: 0.83 if you were using an automated system to generate Feedback Items.
Feedback Item State
state does **not **need to be included in the Fixless Audit creation payload. Newly created Feedback Items are defaulted to state: open. Feedback Item states may change as Scale reviews and fixes tasks.
open- default state. Indicates the issue has not been fixed yetresolved- indicates the issue has been fixeddisputed- indicates that Scale disagrees with the issueescalated- indicates the customer disagrees with Scale’s dispute of the issuerejected- terminal state indicating that the issue was incorrectly reported after review of the escalation
Calculating Quality Scores
Scale only uses Feedback Items oftype: error and severity: standard | severe to compute quality scores.
Only the latest Fixless Audit is considered when computing task and batch quality scores.
Invalid Feedback Items are not considered. Invalid feedback items are marked in the grader_output property on the Feedback Item. Grader outputs are computed after the Fixless Audit is submitted and may not be available for a few minutes:
grader_output.conflict = truegrader_output.explanation = "description of the issue..."
- Extraneous, attribute, geometry, position, or label error where a valid annotation ID is not provided
- Attribute error where the attribute name does not match an attribute on the annotation
- Missing error where a point or polygon indicating the location of the error is not specified