Uploading 3D Data to Nucleus

Overview

In this guide, we'll walk through the steps to upload your 3D and sensor fusion data into Nucleus.

Nucleus currently supports LidarScenes, which are sequences of Frames for each timestep of capture. Each Frame is a collection of sensor data, namely a single lidar pointcloud with any number of associated camera images.

Preparing Your Data

Nucleus expects a remote JSON for pointclouds and a remote image file. For camera images, Nucleus also expects a metadata field camera_params, which gives details about the extrinsics and intrinsics of your camera sensor. This information is used to create integrated 2D visualizations such as cuboid projections and FOV highlighting.

Please refer to our API documentation for more specifics.

Lidar Pointclouds

Lidar pointclouds must follow a specific JSON schema for uploads into Nucleus. This schema is very similar to that of Frame payloads for Scale annotation projects. However, Nucleus payloads only pertain to the pointcloud and should not have camera information.

Pointcloud JSONs can have five properties:

Property
Type
Description

points

List[LidarPoint]

List of points (and their attributes) in the pointcloud. Each LidarPoint must have an (x,y,z) position and optionally, attributes like intensity. See API reference.

device_position

Location of the ego vehicle. Should use the same coordinate system as points.
Defaults to (0,0,0) if not provided.

device_heading

Orientation of the ego vehicle expressed as a quaternion. See Heading Examples.
Defaults to (0,0,0,0) if not provided.

device_gps_pose

Location and direction of the ego vehicle on the globe.
Optional.

timestamp

int

Starting timestamp of the sensor rotation in nanoseconds.
Optional.

The most important property to have is points. Meanwhile, device_position and device_heading are helpful for calibrating a static offset of the sensor relative to the captured points. Providing device_gps_pose and timestamp is optional, but may prove helpful when querying your data in Nucleus.

{
    "points": [
        {
            "x": 100.0,
            "y": 200.0,
            "z": 300.0,
            "i": 0.7
        },
        {
            "x": 400.0,
            "y": 500.0,
            "z": 600.0,
            "i": 0.5
        },
        {
            "x": 700.0,
            "y": 800.0,
            "z": 900.0,
            "i": 0.3
        }
    ],
    "device_position": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0
    },
    "device_heading": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0,
        "w": 0.0
    },
    "device_gps_pose": {
      	"lat": 90.0,
      	"lon": 180.0,
        "bearing": 360.0
    },
    "timestamp": 9001
}

Once you've constructed a JSON for each of your pointclouds, you can upload them to your favorite remote file system or cloud storage provider. Nucleus expects each DatasetItem to be initialized with the URL of a pointcloud JSON -- we'll show how to construct these in a later section.

Privacy Mode

As an enterprise customer, you can use Nucleus without ever transferring data onto Scale servers! Read more on Privacy Mode here.

Camera Images

Nucleus processes each camera image as a single DatasetItem, even if there are multiple camera images associated with the same Frame (e.g. multiple angles captured at the same timestep).

Intrinsics and extrinsics for each camera, such as focal length or camera position and heading, should be organized into a Python dictionary as in the below example. These dictionaries should use the same schema as Scale annotation CameraImages, excluding the image_url property (schema defined here).

camera_params = {
    "position": {
        "x": -0.6,
        "y": -0.1,
        "z": 1.7
    },
    "heading": { # Hamiltonian quaternion
        "x": 0.5,
        "y": 0.5,
        "z": -0.5,
        "w": -0.5
    },
    "cx": 123.0, # principal point x value
    "cy": 456.0, # principal point y value
    "fx": 900.0, # focal length in x direction (pixels)
    "fy": 901.0, # focal length in x direction (pixels)
    # other optional camera intrinsics keys:
    # - timestamp
    # - scale_factor
    # - priority
    # - camera_index
    # - camera_model
    # - skew
    # - k1, k2, k3, k4 (radial distortion coeffs)
    # - p1, p2 (tangential distortion coeffs)
    # - xi (reference frame offset)
}

All camera images should also be uploaded to a remote file system from which Scale can access URLs to each image.

As for camera instrinsics like the dictionary above, we recommend serializing each camera's intrinsics to JSON or a similar key-value format. These files should be organized such that it's easy to link them back to their associated camera images. More on how we'll use these in the next section.

Constructing Lidar Scenes

Now that your data is formatted properly, we can start using the Nucleus Python client to construct LidarScene objects for upload via API.

There are three main steps in this process:

  1. Create pointcloud and camera image DatasetItems
  2. Create Frames from 1.
  3. CreateLidarScenes from 2.

DatasetItem

Pointclouds

Since we've already packaged much of the necessary pointcloud information into the JSON blobs, creating pointcloud DatasetItems is very straightforward.

from nucleus import DatasetItem

pointcloud = nucleus.DatasetItem(
    pointcloud_location="s3://your-bucket-name/001/lidar/00.json",
    reference_id="scene-1-pointcloud-0",
    metadata={"foo": "bar"}
)

Camera Images

Camera intrinsics such as focal length or camera position and heading must be packaged as a dictionary and passed into image metadata using the camera_params key. As mentioned, this dictionary must follow the schema defined here, excluding image_url.

import json
import boto3

# remote camera_params JSON
s3 = boto3.client("s3")
res = s3.get_object(Bucket="your-bucket-name", Key="001/front-camera/intrinsics.json")
front_camera_params = json.loads(res["Body"].read())

frontcam_image = DatasetItem(
    image_location="s3://your-bucket-name/001/front-camera/00.jpeg",
    reference_id="frontcam-scene-1-image-0",
    metadata={
        # arbitrary metadata
        "foo": "bar",
        "hello": "world",
      
      	# camera paramaters/intrinsics
      	# MUST USE KEY `camera_params`
        "camera_params": front_camera_params
    }
)

# local camera_params JSON
with open("local/path/to/001/rear-camera/intrinsics.json", "r") as f:
  rear_camera_params = json.loads(f.read())

rearcam_image = DatasetItem(
    image_location="s3://your-bucket-name/001/rear-camera/00.jpeg",
    reference_id="rearcam-scene-1-image-0",
    metadata={
        "foo": "bar",
        "hello": "world",
        "camera_params": rear_camera_params
    }
)

Frame

Next, you'll want to group your newly created pointcloud and camera image DatasetItems into Frames.

A Frame corresponds to a single timestep of sensor capture. It must contain exactly one pointcloud, and any number of camera images. Sensor names may be passed in as arbitrary keyword arguments, mapped to their respective DatasetItems.

from nucleus import Frame

frame = Frame(
  	lidar=pointcloud,
  	front_camera=frontcam_image,
  	rear_camera=rearcam_image
)

LidarScene

After creating several Frames for each timestep of capture, you can string them together into a LidarScene.

from nucleus import (DatasetItem, Frame, LidarScene)

scene = LidarScene(
  	reference_id="scene_by_list",
  	frames=[frame0, frame1, frame2]
)

You can also make changes to an existing LidarScene. The add_frame method allows you to add or update frames in the sequence by index.

# scene.frames: [frame0, frame1, frame2]

# add a new Frame
scene.add_frame(
  	frame=frame3,
  	index=3 # add to end of sequence
)

# overwrite existing Frame
scene.add_frame(
  	frame=frame0_new
  	index=0,
  	update=True # default is False, which will ignore collisions
)

# scene.frames: [frame0_new, frame1, frame2, frame3]

Similarly, you can add pointclouds and camera images to specific frames of a scene by index.

# scene.frames[0].items: {
#			'lidar': pointcloud0,
#			'front_camera': frontcam_image0,
#			'rear_camera': rearcam_image0
# }

# add a new camera image to frame 0
scene.add_item(
  	sensor_name='left_camera',
  	item=leftcam_image0,
  	index=0
)

# overwrite existing pointcloud in frame 0
scene.add_item(
  	sensor_name='lidar',
  	item=pointcloud0_new,
  	index=0
)

# scene.frames[0].items: {
#			'lidar': pointcloud0_new,
#			'front_camera': frontcam_image0,
#			'rear_camera': rearcam_image0,
#			'left_camera': leftcam_image0
# }

Uploading to Nucleus

Creating/Retrieving a Dataset

If you don't yet have a Dataset to populate with scenes, it's fairly straightforward to create one -- see the example below.

If you do have an existing Dataset, you can retrieve it by datasetid, which are always prefixed with `ds. You can list all of your dataset_ids usingNucleusClient.list_datasets()`, or extract it from the Nucleus dashboard's URL upon clicking into the Dataset.

from nucleus import (Dataset, NucleusClient)

client = NucleusClient(YOUR_API_KEY)

dataset = client.create_dataset(YOUR_DATASET_NAME)
from nucleus import (Dataset, NucleusClient)

client = NucleusClient(YOUR_API_KEY)

dataset = client.get_dataset(YOUR_DATASET_ID) # id is prefixed with `ds_`

With your scenes and dataset ready, you can now upload to Nucleus!

# after creating or retrieving a Dataset
job = dataset.append(
  	items=[scene0, scene1, scene2, ...],
  	update=True,
  	asynchronous=True # highly recommended for 3D uploads!
)

# async jobs will run in the background, poll using:
job.status()

By setting the update flag to True, your upload will overwrite any existing scene- or item-level metadata for any collisions on reference_id.

A note on `update=True`

Nucleus expects scenes to maintain the same structure in terms of frame sequence, and items per frame.

Intended scene/frame updates with a different structure will be ignored.

It is also highly recommended to set asynchronous=True for 3D uploads! This will dramatically increase upload speeds, particularly because it takes considerable time to process pointclouds.

Review

  1. Schematize and upload remote pointcloud JSONs
  2. Upload remote camera images
    a. Schematize camera intrinsics
  3. Create pointcloud and camera image DatasetItems
  4. Collect the above into Frames
  5. Collect the above into LidarScenes
  6. Create/retrieve Dataset
  7. Upload LidarScenes to Dataset
Updated 18 days ago