Skip to main content

Documentation Index

Fetch the complete documentation index at: https://api-reference.scale.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Create Semantic Segmentation Annotation Task

This endpoint creates a **segmentannotation** task. In this task, one of our labelers will view the given image and classify pixels in the image according to the labels provided. You will receive a semantic, pixel-wise, dense segmentation of the image. We also support instance-aware semantic segmentations, also called panoptic segmentation, via LabelDescription objects. The required parameters for this task are **attachment** and **labels**. The **attachment** is a URL to an image you’d like to be segmented. **labels** is an array of strings or LabelDescription objects describing the different types of objects you’d like to segment the image with. You can optionally provide additional markdown-enabled or Google Doc-based instructions via the **instruction** parameter. You can also optionally set **allow_unlabeled** to true, which will allow the existence of unlabeled pixels in the task response - otherwise, all pixels in the image will be classified (in which case it’s important that there are labels for everything in the image, to avoid misclassification). The response you will receive will be a series of images where each pixel’s value corresponds to the label, either via a numerical index or a color mapping. You will also get separate masks for each label for convenience.If the request successful, Scale will return the generated task object, at which point you should store the **task_id** to have a permanent reference to the task.
​
Body Params
object
​
project
string
The name of the project to associate this task with.
​
batch
string
A markdown-enabled string or iframe embed google doc explaining how to do the segmentation. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
​
callback_url
string
The full url (including the scheme **http://** or **https://**) or email address of the callback that will be used when the task is completed.
​
attachment
string
required
A URL to the image you’d like to be segmented.
​
attachment_type
string
Describes what type of file the attachment is. We currently only support image for the segmentannotation.
​
labels
array of strings
required
An array of strings or LabelDescription objects describing the different types of objects you’d like to be used to segment the image. You may include at most 50 labels.
​
annotation_attributes
object
This field is used to add additional attributes that you would like to capture per annotation. This only applies to instance annotations. See Annotation Attributes for more details about annotation attributes.
​
allow_unlabeled
boolean
Whether or not this image can be completed without every pixel being labeled.
​
hypothesis
object
Editable annotations that a task should be initialized with. This is useful when you’ve run a model to prelabel the task and want annotators to refine those prelabels. Review the Segmentation Hypothesis Format for more details.
​
metadata
object
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB. See the Metadata section for more detail.
​
context_attachments
array of objects
An array of objects in the form of {“attachment”: “<link to actual attachment>”} to show to taskers as a reference. Context images themselves can not be labeled. Context images will appear like this in the UI. You cannot use the task’s attachment url as a context attachment’s url.
​
unique_id
string
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See Avoiding Duplicate Tasks for more details.
​
clear_unique_id_on_error
boolean
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
​
tags
array of strings
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
import requests

# Replace with your actual API key
API_KEY = 'your_api_key_here'

# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/segmentannotation"

# Define the payload for the segment annotation task
payload = {
    "instruction": "**Instructions:** Please label all the things",
    "attachment": "https://i.imgur.com/iDZcXfS.png",
    "attachment_type": "image",
    "annotation_attributes": {
        "newKey": {
            "type": "type",
            "description": "description",
            "choices": "choices",
            "conditions": {
                "label_condition": ["car", "car2"],
                "attribute_conditions": {
                    "newKey": "New Value",
                    "newKey-1": "New Value"
                }
            }
        }
    },
    "allow_unlabeled": False,
    "metadata": {
        "newKey": "New Value",
        "newKey-1": "New Value"
    },
    "project": "Project Name",
    "batch": "Batch Name",
    "callback_url": "http://www.example.com/callback",
    "labels": [["vehicle"], "vehicle 2", "vehicle 3"],
    "context_attachments": [{"attachment": "attachment"}, {"attachment": "attachment2"}],
    "unique_id": "unique_id",
    "clear_unique_id_on_error": True,
    "tags": ["tag", "tag2"]
}

# Set up the headers for the request
headers = {
    "accept": "application/json",       # Specify that we want the response in JSON format
    "content-type": "application/json"  # Specify the content type of the request
}

# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))

# Print the response text to see the result
print(response.text)


{
  "task_id": "string",
  "created_at": "string",
  "type": "segmentannotation",
  "status": "pending",
  "instruction": "string",
  "is_test": false,
  "urgency": "standard",
  "metadata": {},
  "project": "string",
  "callback_url": "string",
  "updated_at": "string",
  "work_started": false,
  "params": {
    "allow_unlabeled": false,
    "labels": [
      null
    ],
    "instance_labels": [
      null
    ],
    "attachment_type": "image",
    "attachment": "https://i.imgur.com/SudOKhq.jpg"
  }
}

​
Create General Video Annotation Task

This endpoint creates a **videoannotation** task. Given a series of images sampled from a video (which we will refer to as “frames”), Scale will annotate each frame with the Geometries (box, polygon, line, point, cuboid, and ellipse) you specify. The required parameter for this task is **geometries**. You can optionally provide additional markdown-enabled or Google Doc-based instructions via the **instruction** parameter. You may also optionally specify **events_to_annotate**, a list of strings describing events section to annotate in the video. If the request is successful, Scale will return the generated task object, at which point you should store the **task_id** to have a permanent reference to the task.
​
Body Params
object
​
project
string
The name of the project to associate this task with.
​
batch
string
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch’s project. For Scale Rapid projects specifying a batch is required. See Batches section for more details.
​
instruction
string
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
​
callback_url
string
The full url (including the scheme **http://** or **https://**) or email address of the callback that will be used when the task is completed.
​
attachments
array of strings
An array of URLs for the frames you’d like to be annotated. These image frames are stitched together to create a video. This is required if attachment_type is image and must be omitted if attachment_type is video.
​
attachment
string
A URL pointing to the video file attachment. Only the mp4, webm, and ogg formats are supported.
​
attachment_type
string
Describes what type of file the attachment(s) are. The only options are image and video.
​
geometries
object
required
An object mapping **box**, **polygon**, **line**, **point**, **cuboid**, or **ellipse** to Geometry objects
​
annotation_attributes
object
See the Annotation Attributes section for more details about annotation attributes.
​
events_to_annotate
array of strings
The list of events to annotate.
​
links
object
Use this field to define links between annotations. See Links for more details about links.
​
frame_rate
int32
The number of frames per second to annotate.
​
padding
int32
The amount of padding in pixels added to the top, bottom, left, and right of each video frame. This allows labelers to extend annotations outside of the frames.
​
paddingX
int32
The amount of padding in pixels added to the left and right of each video frame. Overrides **padding** if set.
​
paddingY
int32
The amount of padding in pixels added to the top and bottom of each video frame. Overrides padding if set.
​
hypothesis
object
Editable annotations that a task should be initialized with. This is useful when you’ve run a model to prelabel the task and want annotators to refine those prelabels. Review the Segmentation Hypothesis Format for more details.
​
base_annotations
object
Editable annotations, with the option to be “locked”, that a task should be initialized with. This is useful when you’ve run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.
​
can_add_base_annotations
boolean
Whether or not new annotations can be added to the task if base_annotations are used. If set to true, new annotations can be added to the task in addition to base_annotations. If set to false, new annotations will not be able to be added to the task.
​
can_edit_base_annotations
boolean
Whether or not base_annotations can be edited in the task. If set to true, base_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base_annotations will be locked.
​
can_edit_base_annotation_labels
boolean
Whether or not base_annotations labels can be edited in the task. If set to true, the label of base_annotations can be edited by the tasker. If set to false, the label will be locked.
​
can_delete_base_annotations
boolean
Whether or not base_annotations can be removed from the task. If set to true, base_annotations can be deleted from the task. If set to false, base_annotations cannot be deleted from the task.
​
metadata
object
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
​
priority
int32
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
​
unique_id
string
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See Avoiding Duplicate Tasks for more details.
​
clear_unique_id_on_error
boolean
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
​
tags
array of strings
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
import requests

# Replace with your actual API key
API_KEY = 'your_api_key_here'

# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/videoannotation"

# Define the payload for the video annotation task
payload = {
    "instruction": "**Instructions:** Please label all the things",
    "attachments": [
        "https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000005.png",
        "https://static.scale.com/scaleapi-lidar-images/2011_09_29_drive_0071_sync/image_02/data/0000000008.png"
    ],
    "attachment_type": "image",
    "geometries": {
        "box": {
            "min_height": 10,
            "min_width": 10,
            "can_rotate": True,
            "integer_pixels": False
        },
        "polygon": {
            "min_vertices": 10,
            "max_vertices": 20,
            "objects_to_annotate": ["large vehicle"]
        },
        "line": {
            "min_vertices": 10,
            "max_vertices": 20,
            "objects_to_annotate": ["large vehicle"]
        },
        "point": {
            "objects_to_annotate": ["large vehicle", "large vehicle"]
        },
        "cuboid": {
            "min_height": 10,
            "min_width": 10,
            "camera_intrinsics": {
                "fx": 10,
                "fy": 10,
                "cx": 10,
                "cy": 10,
                "skew": 10,
                "scalefactor": 10
            },
            "camera_rotation_quaternion": {
                "w": 10,
                "x": 10,
                "y": 10,
                "z": 10
            },
            "camera_height": 10
        },
        "ellipse": {
            "objects_to_annotate": ["large vehicle"]
        }
    },
    "events_to_annotate": ["event_1_name", "event_2_name"],
    "frame_rate": 1,
    "start_time": 10,
    "padding": 10,
    "paddingX": 10,
    "metadata": {
        "newKey": "New Value",
        "newKey-1": "New Value"
    },
    "priority": 30,
    "project": "Project Name",
    "batch": "Batch Name",
    "callback_url": "http://www.example.com/callback",
    "attachment": "attachment_url",
    "duration_time": 10,
    "paddingY": 10,
    "unique_id": "unique_id",
    "clear_unique_id_on_error": True,
    "tags": ["tag1", "tag2"]
}

# Set up the headers for the request
headers = {
    "accept": "application/json",       # Specify that we want the response in JSON format
    "content-type": "application/json"  # Specify the content type of the request
}

# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))

# Print the response text to see the result
print(response.text)


{
  "task_id": "string",
  "created_at": "string",
  "type": "videoannotation",
  "status": "pending",
  "instruction": "string",
  "is_test": false,
  "urgency": "standard",
  "metadata": {},
  "project": "string",
  "callback_url": "string",
  "updated_at": "string",
  "work_started": false,
  "params": {
    "attachment_type": "website",
    "attachment": [
      null
    ],
    "geometries": {
      "box": {
        "objects_to_annotate": [
          null
        ],
        "min_height": 10,
        "min_width": 10
      },
      "polygon": {
        "objects_to_annotate": [
          null
        ]
      },
      "point": {
        "objects_to_annotate": [
          null
        ]
      }
    },
    "annotation_attributes": {
      "additionalProp": {
        "description": "string",
        "choice": "string"
      }
    },
    "events_to_annotate": [
      null
    ],
    "with_labels": true
  }
}

​
Create Video Playback Annotation Task

This endpoint creates a **videoplaybackannotation** task. In this task, we will view the given video file and draw annotations around the specified objects. You are required to provide a URL to the video file as the **attachment**. It can be in **mp4**, **webm**, or **ogg** format. You can optionally provide additional markdown-enabled or Google Doc-based instructions via the **instruction** parameter. You may optionally specify a **frame_rate**, which will determine how many frames per second will be used to annotate the given video. The default value is **1**. You may also optionally specify **events_to_annotate**, a list of strings describing events section to annotate in the video. If the request is successful, Scale will return the generated task object, at which point you should store the **task_id** to have a permanent reference to the task.
​
Body Params
object
​
project
string
The name of the project to associate this task with.
​
batch
string
The name of the batch to associate this task with. Note that if a batch is specified, you need not specify the project, as the task will automatically be associated with the batch’s project. For Scale Rapid projects specifying a batch is required. See Batches section for more details.
​
instruction
string
A markdown-enabled string or iframe embed google doc explaining how to do the task. You can use markdown to show example images, give structure to your instructions, and more. See our instruction best practices for more details. For Scale Rapid projects, DO NOT set this field unless you specifically want to override the project level instructions.
​
callback_url
string
The full url (including the scheme **http://** or **https://**) or email address of the callback that will be used when the task is completed.
​
attachments
array of strings
An array of URLs for the frames you’d like to be annotated. These image frames are stitched together to create a video. This is required if attachment_type is image and must be omitted if attachment_type is video.
​
attachment
string
A URL pointing to the video file attachment. Only the mp4, webm, and ogg formats are supported.
​
attachment_type
string
Describes what type of file the attachment(s) are. The only options are image and video.
​
geometries
object
required
An object mapping box, polygon, line, point, cuboid, or ellipse to Geometry objects
​
annotation_attributes
object
See the Annotation Attributes section for more details about annotation attributes.
​
events_to_annotate
int32
The list of events to annotate.
​
duration_time
array of strings
The duration of the video in seconds. This is ignored if attachment_type is image. Default is full video length.
​
frame_rate
object
The number of frames to capture in one second. This is ignored if attachment_type is image.
​
start_time
int32
The start time in seconds. This is ignored if attachment_type is image.
​
padding
int32
The amount of padding in pixels added to the top, bottom, left, and right of each video frame. This allows labelers to extend annotations outside of the frames.
​
paddingX
int32
The amount of padding in pixels added to the left and right of each video frame. Overrides padding if set.
​
paddingY
int32
The amount of padding in pixels added to the top and bottom of each video frame. Overrides padding if set.
​
base_annotations
object
Editable annotations, with the option to be “locked”, that a task should be initialized with. This is useful when you’ve run a model to prelabel the task and want annotators to refine those prelabels. Must contain the annotations field, which has the same format as the annotations field in the response.
​
can_add_base_annotations
boolean
Whether or not new annotations can be added to the task if base_annotations are used. If set to true, new annotations can be added to the task in addition to base_annotations. If set to false, new annotations will not be able to be added to the task.
​
can_edit_base_annotations
boolean
Whether or not base_annotations can be edited in the task. If set to true, base_annotations can be edited by the tasker (position of annotation, attributes, etc). If set to false, all aspects of base_annotations will be locked.
​
can_edit_base_annotation_labels
boolean
Whether or not base_annotations labels can be edited in the task. If set to true, the label of base_annotations can be edited by the tasker. If set to false, the label will be locked.
​
can_delete_base_annotations
boolean
Whether or not base_annotations can be removed from the task. If set to true, base_annotations can be deleted from the task. If set to false, base_annotations cannot be deleted from the task.
​
metadata
object
A set of key/value pairs that you can attach to a task object. It can be useful for storing additional information about the task in a structured format. Max 10KB.
​
priority
int32
A value of 10, 20, or 30 that defines the priority of a task within a project. The higher the number, the higher the priority.
​
unique_id
string
A arbitrary ID that you can assign to a task and then query for later. This ID must be unique across all projects under your account, otherwise the task submission will be rejected. See Avoiding Duplicate Tasks for more details.
​
clear_unique_id_on_error
boolean
If set to be true, if a task errors out after being submitted, the unique id on the task will be unset. This param allows workflows where you can re-submit the same unique id to recover from errors automatically
​
tags
array of strings
Arbitrary labels that you can assign to a task. At most 5 tags are allowed per task. You can query tasks with specific tags through the task retrieval API.
import requests

# Replace with your actual API key
API_KEY = 'your_api_key_here'

# Define the URL for the API endpoint
url = "https://api.scale.com/v1/task/videoplaybackannotation"

# Define the payload for the video playback annotation task
payload = {
    "instruction": "**Instructions:** Please label all the things",
    "attachments": [
        "https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000000.png",
        "https://static.scale.com/scaleapi-lidar-images/2011_09_26_drive_0051_sync/image_02/data/0000000001.png"
    ],
    "attachment": "https://scale-static-assets.s3-us-west-2.amazonaws.com/demos/multimodal-video.mp4",
    "attachment_type": "image",
    "geometries": {
        "box": {
            "min_height": 10,
            "min_width": 10
        },
        "polygon": {
            "min_vertices": 1,
            "max_vertices": " "
        },
        "line": {
            "min_vertices": 1,
            "max_vertices": " "
        },
        "point": {
            "x": " ",
            "y": " "
        },
        "cuboid": {
            "min_height": 0,
            "min_width": 0,
            "camera_intrinsics": {
                "fx": " ",
                "fy": " ",
                "cx": " ",
                "cy": " ",
                "skew": 0,
                "scalefactor": 1
            },
            "camera_rotation_quaternion": {
                "w": " ",
                "x": " ",
                "y": " ",
                "z": " "
            },
            "camera_height": " "
        }
    },
    "frame_rate": 1,
    "padding": 0,
    "paddingX": 0,
    "paddingY": 0,
    "priority": 30
}

# Set up the headers for the request
headers = {
    "accept": "application/json",       # Specify that we want the response in JSON format
    "content-type": "application/json"  # Specify the content type of the request
}

# Adding authentication to the POST request
# The auth parameter requires a tuple with the API key and an empty string
response = requests.post(url, json=payload, headers=headers, auth=(API_KEY, ''))

# Print the response text to see the result
print(response.text)


{
  "task_id": "string",
  "created_at": "string",
  "type": "imageannotation",
  "status": "pending",
  "instruction": "string",
  "is_test": false,
  "urgency": "standard",
  "metadata": {},
  "project": "string",
  "callback_url": "string",
  "updated_at": "string",
  "work_started": false,
  "params": {
    "attachment_type": "image",
    "attachment": "http://i.imgur.com/3Cpje3l.jpg",
    "geometries": {
      "box": {
        "objects_to_annotate": [
          null
        ],
        "min_height": 5,
        "min_width": 5
      },
      "polygon": {
        "objects_to_annotate": [
          null
        ]
      },
      "point": {
        "objects_to_annotate": [
          null
        ]
      }
    },
    "annotation_attributes": {
      "additionalProp": {
        "type": "category",
        "description": "string",
        "choice": "string"
      }
    }
  }
}