Overview

In this guide, we'll walk through the steps to upload category ground truth annotations and model predictions to Nucleus.

Adding Taxonomies

Adding taxonomies to your dataset allows you to upload category annotations and predictions to your dataset items. Nucleus currently only supports category taxonomies. (Support for a default taxonomy that will allow you to upload category annotations and predictions without first explicitly defining a taxonomy is coming soon!)

Uploading Ground Truth Annotations and Model Predictions

Once you've added a taxonomy to your dataset, you can upload category annotations and predictions from that taxonomy to the items in your dataset in much the same way that you would other annotation types.

from nucleus import (
  NucleusClient,
  CategoryAnnotation,
  CategoryPrediction,
)

# setup
client = nucleus.NucleusClient("YOUR_SCALE_API_KEY")
dataset = client.get_dataset("ds_bw6de8s84pe0vbn6p5zg")

# create taxonomy
dataset.add_taxonomy(
  taxonomy_name="clothing_type", 
  taxonomy_type="category", 
  labels=["shirt", "trousers", "dress"]
)

# create + upload ground truth annotation
category_gt = CategoryAnnotation(
  label="dress", 
  taxonomy_name="clothing_type", 
  reference_id="image_1", 
  metadata={"dress_color": "navy"}
)

response = dataset.annotate
  annotations=[category_gt],
  update=True,
  asynchronous=False # async is recommended, but sync jobs are easier to debug
)

# create model
model = client.add_model(
  name="My Model",
  reference_id="My-CNN",
  metadata={"timestamp": "121012401"}
)

# create + upload model prediction
category_pred = CategoryPrediction(
  label="dress",
  taxonomy_name="clothing_type",
  reference_id="image_1",
  confidence=0.5,
  class_pdf={"shirt": 0.4, "trousers": 0.1, "dress": 0.5}
)

job = dataset.upload_predictions(
  	model=model,
  	predictions=[category_pred],
  	update=True,
    asynchronous=True # async is recommended, but sync jobs are easier to debug
)

# poll current status
job.status()

# block until upload completes
job.sleep_until_complete()

Unlike with other annotation types, you don't need to call the calculate model metrics endpoint to update matches against ground truth annotations and calculate various metrics such as IOU for categorization. Sorting by metrics, as well as the evaluation plots and metrics present in the Insights page will be automatically refreshed when you upload new category annotations and/or predictions. This is because category ground truth annotations and predictions are matched explicitly by taxonomy.