Skip to main content

Documentation Index

Fetch the complete documentation index at: https://api-reference.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Annotation Attributes Overview

In many cases, it is useful to have more human-judged metadata on top of each annotation for a given task, for example, measuring the occlusion-level of all vehicles in an image. To achieve this, we support annotation_attributes, an object representing additional attributes that you’d like to capture per image annotation. You may use annotation_attributes to define categorical attributes, numerical attributes, angle attributes, text attributes and more for each annotation. You define the type of the attribute using the type property of the attribute, if no type is specified, it will default to a category type attribute. See the Annotation Attribute Types below for more details. The format for annotation_attributes is an object whose key-value pairs all specify attributes of each annotation that you want to capture. The schema differs slightly based on the type of attribute.

​
Annotation Attribute Types

Attribute TypeDescription
CategoricalMultiple choice attribute, with an optional ability to enable selecting multiple options simultaneously.
NumericalInteger input
AngleInput to select a value between 0 and 360 with a visual interface supporting angles.
TextInput for free text
X/Y OffsetInput to select width or height within a box annotation

​
Request Format

To create a task with attributes, simply add the annotation_attributes parameter to your task creation request using the format described above. 
{
  "callback_url": "http://www.example.com/callback",
  "instruction": "Draw boxes around the vehicles in the image.",
  "attachment_type": "image",
  "attachment": "http://i.imgur.com/v4cBreD.jpg",
  "geometries": {
    "box": {
      "objects_to_annotate": ["car","pedestrian"],
    }
  }
  "annotation_attributes": {
    "parked": {
      "description": "Is the car currently parked?",
      "choices": [
        "Yes",
        "No"
      ]
    },
    "heading": {
      "description": "Which direction is the car heading",
      "choices": [
        "left",
        "right",
        "back",
        "front"
      ],
      "conditions": {
        "label_condition": {
          "label": "car"
        },
        "attribute_conditions": [
          {
            "parked": "No"
          }
        ]
      }
    }
  }
}

​
Response Format

In addition to the standard attributes for the annotation response, if you specified annotation_attributes in the request, each annotation object will contain an attributes JSON. It will contain keys for each of the attributes you defined in the annotation_attributes schema, and the values will be the categorical value or numerical value chosen by a labeler. 📘 See the Callback Section for more details about callbacks. 
{
  "response": {
    "annotations": [
      {
        "left": 123,
        "top": 10,
        "width": 121,
        "height": 39,
        "label": "car",
        "attributes": {
          "parked": "Yes",
          "heading": "left"
        }
      },
      {
        "left": 82,
        "top": 56,
        "width": 64,
        "height": 30,
        "label": "car",
        "attributes": {
          "parked": "No",
          "heading": "front"
        }
      },
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // populated task for convenience
    ...
  }
}

​
Numerical Attributes

  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • type: a string describing the type of attribute. Must be number for numerical attributes.
    • min (optional): The minimum number that may be chosen as the attribute.
    • max (optional): The maximum number that may be chosen as the attribute.
    • step (optional, default 1): The legal number intervals for the attribute.
    • conditions (optional): a JSON that describes the conditions under which this attribute should be collected.
{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "example_number": {
      "type": "number",
      "description": "What is the answer to this example question?",
      "min": 0,
      "max": 100
    }
  }
}

​
Categorical Attributes

Each key-value defining a categorical attribute should have the following structure:
  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • choices: a list of strings or LabelDescription objects corresponding to the categorical choices.
    • type (default category): a string describing the type of attribute. Must be category for categorical attributes.
    • conditions (optional): a JSON that describes the conditions under which this attribute should be collected.
    • allow_multiple (optional): a boolean value, if true, allows multiple values to be selected. The response will be an array of values.

​
Request Example

{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "example_attribute": {
      "type": "category",
      "description": "What is the answer to this example question?",
      "choices": [
        "Answer1",
        "Answer2"
      ],
      "allow_multiple": true
    }
  }
 ...
}

​
Response Examples

{
  "example_attribute": "Answer1"
}

​
Angle Attributes

An angle attribute allows you to receive some angular information of the attribute, such as the heading yaw angle or heading pitch angle. This will return a number from 0 to 360, which is the value of the angle in degrees. Each key-value defining an angle attribute should have the following structure:
  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • type: a string describing the type of attribute. Must be angle for angle attributes.
    • conditions (optional): a JSON that describes the conditions under which this attribute should be collected.
{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "example_angle": {
      "type": "angle",
      "description": "What is the angle of this object?",
    }
  }
}

​
Text Attributes

A text attribute allows you to receive freeform text, such as a transcription of a road sign or a car’s license plate. This will return a string. Each key-value defining a text attribute should have the following structure:
  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • type: a string describing the type of attribute. Must be text for text attributes.
    • conditions (optional): a JSON that describes the conditions under which this attribute should be collected.
{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "example_text": {
      "type": "text",
      "description": "What does this sign say?",
    }
  }
}

​
X/Y Offset Attributes

X/Y offset attributes are supported for box and cuboid geometries and represent some x- or y-offset within the annotation. They may be used to mark key positions such as the center of the annotated object. In visual representations of task responses, they will be rendered as a horizontal/vertical lines across the annotation. Each key-value defining an X/Y offset attribute should have the following structure:
  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • type: "x_line" or "y_line"
{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "center": {
      "type": "x_line",
      "description": "Where is the center of the vehicle?",
    }
  }
}

{
  "response": {
    "annotations": [
      {
        "left": 123,
        "top": 10,
        "width": 121,
        "height": 39,
        "label": "car",
        "attributes": {
          "center": 60
        }
      }
    ]
  }
}

​
Linked Attributes

Linked attributes are used to link one annotation in a task to another annotation - for example, specifying the car that a tire belongs to. Each key-value defining a linked attribute should have the following structure:
  • key: a short string that serves as the identifier for the attribute
  • value: an object that should have the following keys:
    • description: a human-readable string describing the attribute to a labeler.
    • allowed_labels: a list of strings corresponding to the label choices this object can be linked to. If using Label Nesting, the strings in allowed_labels can only be the final “leaf” nodes.
    • type: linked
    • required: For imageannotation, segmentannotation, videoannotation, and videoplaybackannotation tasks, a boolean value representing whether this attribute is required to be annotated. Defaults to false. Note that for 3d Sensor Fusion Tasks, this option is not supported. For Sensor Fusion tasks, specifying a linked attribute will require an object to be linked before a task can be returned to you.
    • conditions (optional): a JSON that describes the conditions under which this attribute should be collected. See Conditionality for more details.
If the linked attribute has a value, the value of the attribute will be the annotation UUID of the object it was linked to. If the linked attribute is not used, the linked attribute will not be included in the response at all. 
{
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "example_linked_attribute": {
      "type": "linked",
      "description": "What is this object connected to?",
      "allowed_labels": [
        "Truck",
        "Car"
      ],
      "required": true
    }
  }
 ...
}

// This example shows two polygons representing friends. 
// The first has no linked friends. The second example is linked to the first. 

{
  "annotations": [
    {
      "label": "Friend",
      "uuid": "a6092706-266d-4de1-8a14-fa6c682a57ea",
      "vertices": [
        {
          "x": 360.73736572265625,
          "y": 200.52745056152344
        },
        {
          "x": 365.7757263183594,
          "y": 348.65576171875
        },
        {
          "x": 510.88104248046875,
          "y": 295.24896240234375
        },
        {
          "x": 468.55865478515625,
          "y": 176.34324645996094
        }
      ],
      "geometry": "polygon"
    },
    {
      "label": "Friend",
      "attributes": {
        "friend": "a6092706-266d-4de1-8a14-fa6c682a57ea"
      },
      "uuid": "c6e00dc4-a3c0-4d48-9203-8f02d9c76328",
      "vertices": [
        {
          "x": 619.7100219726562,
          "y": 166.2664794921875
        },
        {
          "x": 443.36676025390625,
          "y": 255.94961547851562
        },
        {
          "x": 507.8580017089844,
          "y": 367.8016052246094
        },
        {
          "x": 693.2703247070312,
          "y": 313.3871154785156
        }
      ],
      "geometry": "polygon"
    }
  ]
}

​
Global Attributes

Using the "is_global": true flag in the conditions JSON when defining attributes marks the attribute as global. For imageannotation and segmentannotation tasks, this means the attribute applies to the scene as a whole instead of any particular annotation. Therefore, the label_condition condition will never apply. For videoannotation and videoplaybackannotation tasks, this means the attribute applies to events which occur on a frame or during a group of frames. The label_condition can be used to restrict the attribute to appear on specific event labels. For imageannotation tasks, the global attribute image_rotation will sync with the image rotation in the task. 
{
  "annotation_attributes": {
    "is_night": {
      "type": "category",
      "description": "Does this scene take place at night?",
      "choices": ["Yes", "No"],
      "conditions": {
        "is_global": true
      }
    }
  }
}

​
Conditionality

The conditions JSON describes the set of conditions under which the attribute should be collected. Each of these conditions objects described in the JSON are OR’d together, and each property within these objects is AND’d together.

​
Definition: Conditions

The conditions object describes which labels and attributes to ask for based on already provided responses. conditions can have up to two key-values: label_condition and attribute_conditions.

​
Definition: Condition

A Condition is an object defined by a single key and a value. The key describes the semantic string that you want to check, whether it is 'label' for the label_condition or the attribute identifier for attribute_conditions. The value is either a string or an array of strings. In the case of a string, the label or attribute will be checked for strict equality with the string. In the case of an array of strings, the label or attribute will be checked for membership within the array.

​
Condition Parameters

The label_condition key-value will describe the label or labels that you want to collect this attribute for. The value is a Condition object, which must have a key equal to 'label' and a value corresponding to the label or furthest “leaf” node / subchoice of a LabelDescription object. For example, if in the above example, you wanted an “occlusion” attribute for some mix of the “leaf” labels, you could specify the label condition for the attribute as e.g. \{ "label": \["Car", "Truck", "Motorcycle", "Adult"\] \}. If you wanted the attribute to be conditional on all the subchoices of a given label (e.g., all “Vehicle”s), you would simply specify all of those subchoices in the array,
e.g. \{ "label": \["Car", "Truck", "Train", "Motorcycle"\] \}. The attribute_conditions key-value will describe under what conditions with the other attributes you’d like to collect this attribute for. The value should be a Condition object or a list of Condition objects. attribute_conditions are automatically verified for circular dependencies, and tasks will be rejected if any circular dependencies are found. 
"conditions": {
  "label_condition": {
  	"label": ["Car", "Truck", "Motorcycle"]
  },
  "attribute_conditions": [{ "is_moving": "No" }, { "is_parked": "Yes" }]
}

// condition for the label being equal to vehicle
{ "label": "vehicle" }

// condition for the label to be equal to either car or truck
{ "label": ["car", "truck"] } 

// condition for the is_parked attribute to be equal to be Yes
{ "is_parked": "Yes" }

{
  ...,
  "geometries": {
     ...
  },
  "annotation_attributes": {
    "occlusion": {
      "description": "What percent of the object is occluded?",
      "choices": ["0%", "25%", "50%", "75%"],
      "conditions": {
        "label_condition": {
        	"label": ["Car", "Truck", "Motorcycle"]
        }
      }
    }
  },
  ...,
}

​
Global Track Attributes (Multiframe Tasks)

For videoannotation, videoplaybackannotation, and lidarannotation tasks, using the "global": true flag when defining attributes marks an attribute as a global track attribute. This means the attribute’s value will be constant across the entire object track, instead of being allowed to vary from frame to frame.

​
Multiple Types of “Global” Attributes

Global Attributes (here) are different from Global Track Attributes for Multiframe Tasks. Global Attributes specify a global, scene-level attribute, or an attribute of events if using events in videoannotation or videoplaybackannotation tasks. It is not used to customize the attributes of a single annotation. Global Track Attributes (Multiframe Tasks) specify one attribute of annotations when using multiframe tasks, like videoannotation or videoplaybackannotation. Once set, all frames of the annotation must have the same, consistent attribute value. Global Track Attributes should only be used to specify annotation attributes of videoannotation , videoplaybackannotation or lidarannotation tasks. 
{
  "annotation_attributes": {
    "car_color": {
      "type": "category",
      "description": "What is the color of the car?",
      "choices": ["red", "green", "blue"],
      "global": true,
    }
  }
}