Image Annotation Overview

This is the recommended task type for annotating images with vector geometric shapes. The available geometries are box, polygon, line, point, cuboid, and ellipse.
This endpoint creates an imageannotation task. Given an image, Scale will annotate the image with the geometries you specify.
The required parameters for this task are attachment and geometries.

Body Params

projectstring

The name of the project to associate this task with.

batchstring

The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch's project. For Scale Rapid projects specifying a batch is required. See Batches section for more details.

instructionstring

A markdown-enabled string or iframe embedded Google Doc explaining how to do the task. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.

callback_urlstring

The full url (including the scheme http:// or https://) or email address of the callback that will be used when the task is completed.

attachmentstringrequired

A URL to the image you'd like to be annotated.

context_attachmentsarray of objects

An array of objects in the form of {"attachment": "<link to actual attachment>"} to show to taskers as a reference. Context images themselves can not be labeled. Context images will appear like this in the UI. You cannot use the task's attachment url as a context attachment's url.

geometriesobjectrequired

This object is used to define which objects need to be annotated and which annotation geometries (box, polygon, line, point, cuboid, or ellipse) should be used for each annotation. Further description of each geometry can be found in each respective section below

annotation_attributesobject

This field is used to add additional attributes that you would like to capture per annotation. See Annotation Attributes for more details about annotation attributes.

linksobject

Use this field to define links between annotations. See Links for more details about links.

hypothesisobject

Editable annotations that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.

layerobject

Read-only annotations to be pre-drawn on the task. See the Layers section for more details.

base_annotationsboolean

Editable annotations, with the option to be "locked", that a task should be initialized with. This is useful when you've run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.

can_add_base_annotationsboolean

Whether or not new annotations can be added to the task if base_annotations are used. If set to true, new annotations can be added to the task in addition to base_annotations. If set to false, new annotations will not be able to be added to the task.

can_edit_base_annotationsboolean

Whether or not base_annotations can be edited in the task. If set to true, base_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base_annotations will be locked.

can_edit_base_annotation_labelsboolean

Whether or not base_annotations labels can be edited in the task. If set to true, the label of base_annotations can be edited by the tasker. If set to false, the label will be locked.

can_delete_base_annotationsboolean

Whether or not base_annotations can be removed from the task. If set to true, base_annotations can be deleted from the task. If set to false, base_annotations cannot be deleted from the task.

image_metadataobject

This field accepts specified image metadata, supported fields include:
- date_time - displays the date and time the image is taken
- resolution - configures the units of the ruler tools, resolution_ratio holds the number of resolution_units corresponding to one pixel; e.g. {resolution_ratio: 3, resolution_unit: 'm'}, one pixel in the image corresponds to three meters in the real world.
- location - the real-world location where this image was captured, in the standard geographic coordinate system; e.g. {lat: 37.77, long: -122.43}

metadataobject

A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB. See the Metadata section for more detail.

paddinginteger

paddingXinteger

The amount of padding in pixels added to the left and right of the image. Overrides padding if set.

paddingYinteger

The amount of padding in pixels added to the top and bottom of the image. Overrides padding if set.

priorityinteger

A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.

unique_idstring

A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See Avoiding Duplicate Tasks for more details.

clear_unique_id_on_errorboolean

If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically

tagsarray of strings

Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.

Request

POST/v1/task/imageannotation

import requests

url = "https://api.scale.com/v1/task/imageannotation"

payload = {
    "instruction": "**Instructions:** Please label all the things",
    "attachment": "https://i.imgur.com/iDZcXfS.png",
    "geometries": {
        "box": {
            "min_height": None,
            "min_width": None,
            "can_rotate": None,
            "integer_pixels": None
        },
        "polygon": {
            "min_vertices": None,
            "max_vertices": None
        },
        "line": {
            "min_vertices": None,
            "max_vertices": None
        },
        "cuboid": {
            "min_height": None,
            "min_width": None,
            "camera_intrinsics": {
                "fx": None,
                "fy": None,
                "cx": None,
                "cy": None,
                "skew": None,
                "scalefactor": None
            },
            "camera_rotation_quaternion": {
                "w": None,
                "x": None,
                "y": None,
                "z": None
            },
            "camera_height": None
        }
    },
    "padding": None,
    "paddingX": None,
    "paddingY": None,
    "priority": None
}
headers = {
    "accept": "application/json",
    "content-type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

Response

{
  "task_id": "string",
  "created_at": "string",
  "type": "imageannotation",
  "status": "pending",
  "instruction": "string",
  "is_test": false,
  "urgency": "standard",
  "metadata": {},
  "project": "string",
  "callback_url": "string",
  "updated_at": "string",
  "work_started": false,
  "params": {
    "attachment_type": "image",
    "attachment": "http://i.imgur.com/3Cpje3l.jpg",
    "geometries": {
      "box": {
        "objects_to_annotate": [
          null
        ],
        "min_height": 5,
        "min_width": 5
      },
      "polygon": {
        "objects_to_annotate": [
          null
        ]
      },
      "point": {
        "objects_to_annotate": [
          null
        ]
      }
    },
    "annotation_attributes": {
      "additionalProp": {
        "type": "category",
        "description": "string",
        "choice": "string"
      }
    }
  }
}

Boxes

Given a box entry in params.geometries, Scale will annotate your image or video with boxes and return the position and dimensions of the boxes.

Request Parameters

objects_to_annotatearray of strings

A list of string or LabelDescription objects.

min_heightinteger

The minimum height in pixels of the bounding boxes you'd like to be made.

min_widthinteger

The minimum width in pixels of the bounding boxes you'd like to be made.

can_rotateboolean

Allows a tasker to rotate the bounding box.

integer_pixelsboolean

Response fields denoting box location and size (top, left, width, height) will be returned as integers instead of floats. This does not work with rotated boxes.

Response Fields

Key	Type	Description
uuid	string	A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames.
type	string	String indicating geometry type: `box`
label	string	The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames.
attributes	object	See the Annotation Attributes section for more details about the `attributes` response field.
left	float	The distance, in pixels, between the left border of the bounding box and the left border of the image.
top	float	The distance, in pixels, between the top border of the bounding box and the top border of the image.
width	float	The width, in pixels, of the bounding box.
height	float	The height, in pixels, of the bounding box.

If can_rotate was set to true, the following fields will supersede the above fields:

Key	Type	Description
rotation	float	The clockwise rotation in radians
vertices	An array of objects with a schema {x: 0, y: 0}	The vertices of the rotated bounding box
left	float	The distance, in pixels, between the left border of the unrotated bounding box and the left border of the image.
top	float	The distance, in pixels, between the top border of the unrotated bounding box and the top border of the image.

Example Box Request

{
  "geometries": {
    "box": {
      "objects_to_annotate": [
        "traffic_sign",
        {
          "choice": "vehicle",
          "subchoices": [
          	"Car",
            {
              "choice": "truck_suv",
              "display": "truck or SUV"
            }
          ]
        },
        "pedestrian"
      ],
      "min_height": 5,
      "min_width": 5,
      "can_rotate": false
    },
    ...
  },
  ...
}

Example Box Response

{
  "response": {
    "annotations": [
      {
        "type": "box",
        "label": "pedestrian",
        "attributes": {
            "moving": "yes"
        },
        "left": 2,
        "top": 4,
        "width": 3,
        "height": 5,
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
      },
      {
        "type": "box",
        "label": "car",
        "attributes": {
            "moving": "yes"
        },
        "left": 7,
        "top": 5,
        "width": 14,
        "height": 5,
        "uuid": "0a6cd019-a014-4c67-bd49-c269ba08028a"
      },
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // populated task for convenience
    ...
  }
}

Example Rotated Box Response

{
  "response": {
    "annotations" : [ 
      {
        "label" : "car",
        "attributes" : {},
        "uuid" : "122a4270-f9b2-4f66-a9ca-2e06f0de66e5",
        "width" : 121.878523862864,
        "height" : 71.6961921895555,
        "rotation" : 1.2440145049532,
        "left" : 613.440037825633,
        "top" : 199.208745812549,
        "type" : "box",
        "vertices" : [ 
          {
            "x" : 688.769014855216,
            "y" : 165.835344251165
          }, 
          {
            "x" : 727.891633787782,
            "y" : 281.264089660824
          }, 
          {
             "x" : 659.989584658913,
            "y" : 304.27833956349
          }, 
          {
            "x" : 620.866965726348,
            "y" : 188.84959415383
          }
        ]
      }
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // populated task for convenience
    ...
  }
}

Polygons

Given a polygon entry in params.geometries, Scale will annotate your image or video with polygons and return the vertices of the polygons.

Request Parameters

objects_to_annotatearray of objects

A list of string or LabelDescription objects.

min_verticesinteger

The minimum number of vertices in a valid line annotation for your request.

max_verticesinteger

The maximum number of vertices in a valid line annotation for your request. Must be at least min_vertices.

Response Fields

Key	Type	Description
uuid	string	A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames.
type	string	String to indicate geometry type: `polygon`
label	string	The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames.
attributes	object	See the Annotation Attributes section for more details about the `attributes` response field.
vertices	array	An array of vertex objects describing the vertices of the polygon, listed in the order they were annotated. In other words, the point order will be either clockwise or counter-clockwise for each annotation.

Definition: Vertex

Key	Type	Description
x	number	The distance, in pixels, between the vertex and the left border of the image.
y	number	The distance, in pixels, between the vertex and the top border of the image.

Example Polygon Request

{
  "geometries": {
    "polygon": {
      "objects_to_annotate": [
        "traffic_sign",
        {
          "choice": "vehicle",
          "subchoices": [
          	"Car",
            {
              "choice": "truck_suv",
              "display": "truck or SUV"
            }
          ]
        },
        "pedestrian"
      ],
      "min_vertices": 4,
      "max_vertices": 15
    },
    ...
  },
  ...
}

Example Polygon Response

{
  "response": {
    "annotations": [
      {
        "type": "polygon",
        "label": "car",
        "vertices": [
            {
                "x": 123,
                "y": 10
            },
            {
                "x": 140,
                "y": 49
            },
            {
                "x": 67,
                "y": 34
            }
        ],
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
      },
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // task inlined for convenience
    ...
  }
}

Lines

Given a line entry in params.geometries, Scale will annotate your image or video with polylines (segmented lines) and return the vertices of the lines.

Request Parameters

objects_to_annotatearray of objects

A list of string or LabelDescription objects.

min_verticesinteger

The minimum number of vertices in a valid line annotation for your request.

max_verticesinteger

The maximum number of vertices in a valid line annotation for your request. Must be at least min_vertices.

Response Fields

Key	Type	Description
uuid	string	A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames.
type	string	String to indicate geometry type: `line`
label	string	The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames.
attributes	object	See the Annotation Attributes section for more details about the `attributes` response field.
vertices	array	An array of vertex objects describing the vertices of the polygon, listed in the order they were annotated. In other words, the point order will be either clockwise or counter-clockwise for each annotation.

Definition: `Vertex`

Key	Type	Description
x	number	The distance, in pixels, between the vertex and the left border of the image.
y	number	The distance, in pixels, between the vertex and the top border of the image.

Example Line Request

{
  "geometries": {
    "line": {
      "objects_to_annotate": [
        "unmarked_lane",
        {
          "choice": "marked lanes",
          "subchoices": [
            "solid",
            {
              "choice": "dashed",
              "display": "dashed or dotted"
            }
          ]
        },
        "shoulder"
      ],
      "min_vertices": 2,
      "max_vertices": 15
    },
    ...
  },
  ...
}

Example Line Response

{
  "response": {
    "annotations": [
      {
        "type": "line",
        "label": "solid line",
        "vertices": [
            {
                "x": 123,
                "y": 10
            },
            {
                "x": 140,
                "y": 49
            },
            {
                "x": 67,
                "y": 34
            }
        ],
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
      },
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // populated task for convenience
    ...
  }
}

Ellipses

Given an ellipse entry in params.geometries, Scale will annotate your image or video with ellipses and return the extremal points of the ellipses. The ellipses may be rotated relative to the X and Y axes.

Request Parameters

objects_to_annotatearray of objects

A list of string or LabelDescription objects.

Response Fields

Key	Type	Description
uuid	string	A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames.
type	string	String to indicate geometry type: `ellipse`
label	string	The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames.
attributes	object	See the Annotation Attributes section for more details about the `attributes` response field.
vertices	array	A list of Vertex objects of length 4 describing the extremal vertices of the ellipse

Example Ellipse Request

{
    ...
    "geometries": {
        "ellipse": {
            "objects_to_annotate": ["wheel"]
        }
    },
    "annotation_attributes": {
        "position": {
            "type": "category",
            "description": "What is the position of this wheel?",
            "choices": [
              "front_left",
              "front_right",
              "back_left",
              "back_right",
            ]
        }
    },
    ...
}

Example Ellipses Response

{
  "response": {
    "annotations": [
      {
        "type": "ellipse",
        "label": "wheel",
        "attributes": {
            "position": "front_left"
        },
        "vertices": [
            {
                "x": 123,
                "y": 92
            },
            {
                "x": 173,
                "y": 113
            },
            {
                "x": 123,
                "y": 134
            },
            {
                "x": 73,
                "y": 113
            }
        ],
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
      },
      { ... },
      { ... }
    ]
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // task inlined for convenience
    ...
  }
}

Cuboids

Given a cuboid entry in params.geometries, Scale will annotate your image or video with perspective cuboids and return the vertices of the cuboids. If camera intrinsics and extrinsics are provided as well, Scale will return scale-invariant 3D coordinates with respect to the camera, i.e. assuming the camera is at the origin. See https://scale.com/blog/3d-cuboids-annotations for a detailed explanation of how we can augment 2D cuboid responses.

Request Parameters

objects_to_annotatearray of objects

A list of string or LabelDescription objects.

min_heightinteger

The minimum height in pixels of the cuboids you'd like to be made.

min_widthinteger

The minimum width in pixels of the cuboids you'd like to be made.

camera_intrinsicsobject

An object that defines camera intrinsics, in format {fx: number, fy: number, cx: number, cy: number, scalefactor: number, skew: number} (skew defaults to 0, scalefactor defaults to 1). scalefactor is used if the image sent is of different dimensions from the original photo (if the attachment is half the original, set scalefactor to 2) to correct the focal lengths and offsets. Use in conjunction with camera_rotation_quaternion and camera_height to get perspective-corrected cuboids and 3d points.

camera_rotation_quaternionobject

Object that defines the rotation of the camera in relation to the world. Expressed as a quaternion, in format {w: number, x: number, y: number, z: number}. Use in conjunction with camera_intrinsics to get perspective-corrected cuboids and 3d points. Note that the z-axis of the camera frame represents the camera's optical axis. Use in conjunction with camera_intrinsics and camera_height to get perspective-corrected cuboids and 3d points.

camera_heightinteger

The height of camera above the ground, in meters. Use in conjunction with camera_rotation_quaternion and camera_intrinsics to get perspective-corrected cuboids and 3d points.

Response Fields

Key	Type	Description
uuid	string	A computer-generated unique identifier for this annotation. In video annotation tasks, this can be used to track the same object across frames.
type	string	String to indicate geometry type: `cuboid`
label	string	The label of this annotation, chosen from the `objects_to_annotate` array for its geometry. In video annotation tasks, any annotation objects with the same `uuid` will have the same `label` across all frames.
attributes	object	See the Annotation Attributes section for more details about the `attributes` response field.
vertices	array of `Vertex` objects	A list of `Vertex` objects defining all visible vertices of the cuboid. See the Vertex section for more details.
edges	array of `Edge` objects	A list of `Edge` objects defining the edges of the cuboid.. See the Edge section for more details.
points_2d	array of `{x, y}` coordinate objects	If `camera_rotation_quaternion`, `camera_intrinsics`, and `camera_height` were provided, contains projected 2D coordinates of all 8 vertices of the cuboid after perspective correction. See diagram below for the order that the points are returned in.
points_3d	array of `{x, y, z}` coordinate objects	If `camera_rotation_quaternion`, `camera_intrinsics`, and `camera_height` were provided, contains 3D coordinates (arbitrarily scaled, relative to the camera location) of all 8 vertices of the cuboid after perspective correction. See diagram below for the order that the points are returned in.

Definition: `Vertex`

Key	Type	Description
x	number	The distance, in pixels, between the vertex and the left border of the image.
y	number	The distance, in pixels, between the vertex and the top border of the image.
type	string	Always `vertex`.
description	string	An enum describing the position of the vertex, which is one of: `face-topleft` `face-bottomleft` `face-topright` `face-bottomright` `side-topcorner` `side-bottomcorner`

Definition: `Edge`

Key	Type	Description
x1	number	The distance, in pixels, between the first vertex of the edge and the left border of the image.
y1	number	The distance, in pixels, between the first vertex of the edge and the top border of the image.
x2	number	The distance, in pixels, between the second vertex of the edge and the left border of the image.
y2	number	The distance, in pixels, between the second vertex of the edge and the top border of the image.
type	string	Always `edge`.
description	string	An enum describing the position of the edge, which is one of:: `face-top` `face-bottom` `face-left` `face-right` `side-top` `side-bottom`

Example Cuboid Request

{
  ...
  "geometries": {
    "cuboid": {
      "objects_to_annotate": [
        "car"
      ],
      "min_height": 10,
      "min_width": 10,
      "camera_intrinsics": {
        "fx": 986.778503418,
        "fy": 984.4254150391,
        "cx": 961.078918457,
        "cy": 586.9694824219,
        "skew": 0,
        "scale_factor": 1
      },
      "camera_rotation_quaternion": {
        "w": 0.0197866653,
        "x": 0.0181939654,
        "y": 0.6981190587,
        "z": -0.715476937
      },
      "camera_height": -0.2993970777
    }
  },
  ...
}

Text

Points on the cuboid are returned in this order for both points_2d and points_3d:

       3-------2
      /|      /|
     / |     / |
    0-------1  |
    |  7----|--6
    | /     | /
    4-------5

Example Cuboid Response

{
  ...,
  "response": {
    "annotations": [
      {
        "label": "car",
        "vertices": [
          {
            "description": "face-topleft",
            "y": 270,
            "x": 293,
            "type": "vertex"
          },
          {
            "description": "face-bottomleft",
            "y": 437,
            "x": 293,
            "type": "vertex"
          },
          {
            "description": "face-topright",
            "y": 270,
            "x": 471,
            "type": "vertex"
          },
          {
            "description": "face-bottomright",
            "y": 437,
            "x": 471,
            "type": "vertex"
          },
          {
            "description": "side-topcorner",
            "y": 286,
            "x": 607,
            "type": "vertex"
          },
          {
            "description": "side-bottomcorner",
            "y": 373,
            "x": 607,
            "type": "vertex"
          }
        ],
        "edges": [
          {
            "description": "face-top",
            "x1": 293,
            "y1": 270,
            "x2": 471,
            "y2": 270,
            "type": "edge"
          },
          {
            "description": "face-right",
            "x1": 471,
            "y1": 270,
            "x2": 471,
            "y2": 437,
            "type": "edge"
          },
          {
            "description": "face-bottom",
            "x1": 471,
            "y1": 437,
            "x2": 293,
            "y2": 437,
            "type": "edge"
          },
          {
            "description": "face-left",
            "x1": 293,
            "y1": 437,
            "x2": 293,
            "y2": 270,
            "type": "edge"
          },
          {
            "description": "side-top",
            "x1": 471,
            "y1": 270,
            "x2": 607,
            "y2": 286,
            "type": "edge"
          },
          {
            "description": "side-bottom",
            "x1": 471,
            "y1": 437,
            "x2": 607,
            "y2": 373,
            "type": "edge"
          }
        ],
        "points_2d": [
          {
            "y": 270,
            "x": 293
          },
          {
            "y": 437,
            "x": 293
          },
          {
            "y": 270,
            "x": 471
          },
          {
            "y": 437,
            "x": 471
          },
          {
            "y": 286,
            "x": 607
          },
          {
            "y": 373,
            "x": 607
          },
          {
            "y": 373,
            "x": 607
          },
          {
            "y": 373,
            "x": 607
          }
        ],
        "points_3d": [
          {
            "z": 0,
            "y": 270,
            "x": 293
          },
          {
            "z": 0,
            "y": 437,
            "x": 293
          },
          {
            "z": 0,
            "y": 270,
            "x": 471
          },
          {
            "z": 0,
            "y": 437,
            "x": 471
          },
          {
            "z": 0,
            "y": 286,
            "x": 607
          },
          {
            "z": 0,
            "y": 373,
            "x": 607
          },
          {
            "z": 0,
            "y": 373,
            "x": 607
          },
          {
            "z": 0,
            "y": 373,
            "x": 607
          }
        ],
      }
    ]
  },
  ...
}

Image Response Format

The response field, which is part of the callback POST request and permanently stored as part of the task object, will contain an annotations field (and a global_attributes field, if Global Attributes were specified in the task creation request).

The annotations field will contain an array of Annotation objects. The schema of each Annotation object depends on the Geometry of the Annotation. See the Boxes, Polygons, Lines, Points, Cuboids, and Ellipses sections for descriptions of the schemas.

Example response

{
  "response": {
    "annotations": [
      {
        "type": "box",
        "label": "small vehicle",
        "attributes": {
          "moving": "yes"
        },
        "left": 2,
        "top": 4,
        "width": 3,
        "height": 5,
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef42"
      },
      {
        "type": "box",
        "label": "large vehicle",
        "attributes": {
          "moving": "yes"
        },
        "left": 7,
        "top": 5,
        "width": 14,
        "height": 5,
        "uuid": "0a6cd019-a014-4c67-bd49-c269ba08028a"
      },
      {
        "type": "polygon",
        "label": "car",
        "vertices": [
          {
            "x": 123,
            "y": 10
          },
          {
            "x": 140,
            "y": 49
          },
          {
            "x": 67,
            "y": 34
          }
        ],
        "uuid": "65ec1f52-5902-4b39-bea9-ab6b4d58ef43"
      },
      { ... },
      { ... }
    ],
    "global_attributes": {
      "driving": "Yes",
      "night": "No"
    }
  },
  "task_id": "5774cc78b01249ab09f089dd",
  "task": {
    // populated task for convenience
    ...
  }
}

Image Annotation Hypothesis

When creating a imageannotation task, you can provide prelabels in the hypothesis field, so that workers don't have to start from scratch to annotate the image.

In order to add pre-labels in a task using hypothesis, you’ll need to provide these in the hypothesis field of the payload when creating the task. The schema of the hypothesis object must match the schema of the task response.

Verify the task response field schema for the desired task type.
Review your project taxonomy (label names, attribute conditions, annotation types, etc).
Generate pre-labels that are formatted to match the aforementioned schema and taxonomy.
Create a task, including a hypothesis field that contains the pre-labels at the same top-level as other task fields such as project and instructions.

The hypothesis format will largely mirror Scale’s task response format. In this particular task type, annotations field array is mandatory inside the hypothesis object for simple annotations.

Note: UUIDs are not mandatory, if you want to use a particular UUID to track an annotation you can add it to the hypothesis, if not, Scale will generate one for you.

For Image Annotation, you can also add Global Attributes in the hypothesis object at the same level of annotations in the global_attributes field.

task_payload_with_hypothesis

{
 ...
 "attachment": "https://example.com/attachment.png",
 "hypothesis": {
   "annotations":  [
     {
       "label": "car",
       "left": 90,
       "top": 66,
       "height": 94,
       "width": 96,
       "type": "box"
     }
   ]
 },
 ...
}

task_taxonomy

{
 "geometries": {
   "box": {
     "objects_to_annotate": [
       "car"
     ],
     "min_height": 10,
     "min_width": 10
   }
 },
 "annotation_attributes": {}
}

scale_task_response

{
 "links": [],
 "annotations": [
   {
     "label": "car",
     "uuid": "xfb506ca-d742-4e75-bb52-0725f099b238",
     "left": 115,
     "top": 68,
     "height": 97,
     "width": 69,
     "type": "box"
   },
 ],
 "global_attributes": {}
}

Video Annotation Overview

Note: Scale VideoAnnotation has been deprecated in favor of Video V2 (/task/videoplayback).

Note: Scale Video is only available for our Enterprise customers. If you want to learn more, please contact our sales team.

This endpoint creates a videoannotation task. Given a series of images sampled from a video (which we will refer to as "frames"), Scale will annotate each frame with the Geometries (box, polygon, line, point, cuboid, and ellipse) you specify.

The required parameter for this task is geometries.

You can optionally provide additional markdown-enabled or Google Doc-based instructions via the instruction parameter.

You may also optionally specify events_to_annotate, a list of strings describing events section to annotate in the video.

If the request is successful, Scale will return the generated task object, at which point you should store the task_id to have a permanent reference to the task.

Label Nesting and Options

There are often annotation tasks that have too many label choices for a tasker to efficiently sort through them all at once, or times when you want to show one version of a label name to a tasker, but would like another version in the response.

In those cases, you can utilize LabelDescription objects to support nested labels, where labels may have subcategories within them, as well as setting display values for the label.

When declaring objects_to_annotate in your task parameters, we accept a mixed array of strings and the more complex LabelDescription objects.

Definition: `LabelDescription`

A simple example is illustrated in the example JSON below, where objects_to_annotate can simply be a string, a nested label with choices and subchoices, or a nested label where the subchoices themselves are LabelDescription objects with a display value.

While there may be a large number of total labels, using subchoices a tasker can first categorize an object as a road, pedestrian, or vehicle, and based on that choice, further select the specific type of pedestrian or vehicle.

Nested labels may be specified both for the object labels (the objects_to_annotate array parameter), as well as in the choices array of a categorical annotation attribute. In both cases, you would specify a nested label by using a LabelDescription object instead of a string.

For example, for an objects_to_annotate array of ["Vehicle", "Pedestrian"], you could instead add a nested label by passing an array, like ["Vehicle", {"choice": "Pedestrian", "subchoices": ["Animal", "Adult", "Child"]}]. Then, if a tasker selected "Pedestrian" for an annotation, they would be further prompted to choose one of the corresponding subchoices for that annotation.

The LabelDescription object has the following structure:

Parameter	Type	Description
choice*	string	The name of the label. This should be singular and descriptive (ex: `car`, `background`, `pole`). When both a `choice` and `subchoices` are defined, the `choice` will not be selectable, it will only be used for UX navigation. Only the "leaf" nodes will be returned in Scale's response.
subchoices	Array<LabelDescription \| string>	Optional: Descriptions of the sub-labels to be shown under this parent label. Array can be a mix of LabelDescription objects or strings.
instance_label	boolean default `false`	Optional: For Segmentation-based Tasks - Whether this label should be segmented on a per-instance basis. For example, if you set `instance_label` to `true`, each individual car would get a separate mask in the image, allowing you to distinguish between them.
display	string default `choice`	Optional: The value to be shown to a Tasker for a given label. Visually overrides the `choice` field in the user experience, but does not affect the task response or conditionality.

LabelDescription Example

objects_to_annotate = [
  "Road",
  {
    "choice": "Vehicle",
    "subchoices": ["Car", "Truck", "Train", "Motorcycle"]
  },
  {
    "choice": "Pedestrian",
    "subchoices": [
      "Animal", 
      {"choice": "Ped_HeightOverMeter", "display": "Adult" }, 
      {"choice": "Ped_HeightUnderMeter", "display": "Child" }, 
    ]
  }
]

Image Annotation Overview

Body Params

Boxes

Request Parameters

Response Fields

Polygons

Request Parameters

Response Fields

Lines

Request Parameters

Response Fields

Definition: Vertex

Ellipses

Request Parameters

Response Fields

Cuboids

Request Parameters

Response Fields

Definition: Vertex

Definition: Edge

Image Response Format

Image Annotation Hypothesis

Video Annotation Overview

Note: Scale VideoAnnotation has been deprecated in favor of Video V2 (/task/videoplayback).

Label Nesting and Options

Definition: LabelDescription

Definition: `Vertex`

Definition: `Vertex`

Definition: `Edge`

Definition: `LabelDescription`