Validators
You can add validators to your project to help improve quality. Before a labeler submits a task, validators on the project will run and verify the response. A validator can be either blocking or non-blocking. Blocking validators force labelers to fix their responses to pass the validator before submitting; whereas, non-blocking validators appear as dismissible warnings which can remind labelers to double-check their work before submitting.
NLP Validators
These validators can be setup for text labels which are available for the following use cases: Text Generation, Text Transcription, Content Collection
| Validator Name | Description | Parameters |
|---|---|---|
| Word Count | This validator prevents responses that are too short or too long. | Min words (optional) Max words (optional) Warning message (optional) |
| Language & Gibberish Detection | This validator detects the language of the response and whether or not the response is gibberish. | Language |
| Regex Blocklist | You can use this validator to build regex rules to prevent the most common errors or enforce certain response formats. | Regex rules Matching policy This parameter defines the logical relationship between rules: - Every rule should match - Some rules should match - None of the rules should match” Warning message |
| String Blocklist | Use this validator to prevent a blocklist of strings. These strings can be words, phrases, etc. | Strings to block Warning message |
| Response Similarity | Use this validator to prevent labelers from submitting text that overlaps with other text fields within the same response. | Threshold The threshold is the number from 0 to 1 which means the relative Levenshtein distance between two pieces of text. We recommend using 0.3 and tuning this number by running the validator in Task Preview. Warning message |
| Instructions Similarity | This validator prevents labelers from submitting spam that overlaps with the task instructions. | Threshold The threshold is the number from 0 to 1 which means the relative Levenshtein distance between two pieces of text. We recommend using 0.3 and tuning this number by running the validator in Task Preview. Warning message |
| Grammar & Spellcheck | This validator identifies grammar and spelling errors and suggests fixes. | |
| URL Formatting | This validator enforces certain text labels to have URL format. | |
| Text Formatting | Use this validator to ensure clean text responses | Enforce sentence case capitalization Prevent double whitespaces and trailing whitespaces Enforce punctuation at the end of the response |
| Diversity | This validator prevents labelers from submitting text that follow the same patterns as previous attempts | Regex Min Attempts to Apply This is the minimum number of attempts for a labeler before this validator is applied to their future attempts. Max Attempts to Look Back This is the maximum number of previous attempts from the labeler to analyze. Threshold (optional) This is a number from 0 to 1 that represents the proportion of attempts that are analyzed and match the regex pattern before this validator triggers. Number of Matches to Trigger (optional) This is the number of previous attempts that match the regex pattern for this validator to trigger. Warning message |
CV Validators
These validators can be setup for geometry labels (box, ellipse, point, etc.) or segmentation clusters which are available for the following use cases: Object Detection, Object & Event Detection, Semantic Segmentation
| Size | This validator restricts sizes of annotations to a specified minimum or maximum. | Width (px) Height (px) Width (%) The % is based on the corresponding image/video width Height (%) The % is based on the corresponding image/video height Diagonal (px) The diagonal is calculated as the hypotenuse of the bounding box surrounding the object/segmentation. Min/max constraint Specifies whether or not ALL of the above constraints are to be used as minimum or maximum constraints. Logical operator If multiple above constraints are utilizes, specifies whether to combine them under "AND" logic or "OR" logic. |
| Label Usage | Use this validator to define a minimum required or maximum allowed number of annotations per label. | Min uses Max uses Warning message |
| Contiguous Objects | This validator checks that each annotation has contiguous frames. | |
| Collinear Points | This validator checks that polygon and line annotations contain no more than 2 collinear points, which would imply that the same polygon could be drawn with fewer points. | |
| Duplicate Vertex | This validator checks that polygon and line annotations do not contain multiple vertices with the exact same coordinate location. | |
| Overlap | This validator checks that all annotations of each provided label overlap with all other such annotations, or that each is completely disjoint. | Overlap validation If set to "should not overlap," checks for disjointness. If set to "should overlap," checks for overlaps. |
| Containment | This validator checks that each annotation in a provided list of "inner labels" is contained by at least one annotation in a provided list of "outer labels," or contained by exactly zero such annotations. | Inner labels The list of inner labels to check containment of. Outer labels The list of outer labels to check containment with. Containment validation If set to "should be contained," checks for containment. If set to "should not be contained," checks for the opposite. |
| Pixel Percentage | This validator checks that the total percentage of pixels in a given label does not fall outside a specified minimum and maximum percentage. | Min % Max % |
Updated 8 months ago