Nuro mines for rare classes in unlabeled data with Nucleus Object Autotag


Nucleus Enables Rare Class Mining—Essential for Safe Autonomy.

Headquartered in Mountain View, CA, Nuro exists to better everyday life through robotics. The company’s custom electric autonomous vehicles are designed to bring the things people need—from produce to prescriptions—right to your home. Its R2 robot was the first completely autonomous, zero-occupant on-road vehicle approved for commercial delivery by the U.S. Department of Transportation. Last April, the company partnered with Domino’s to autonomously deliver pizza to customers in Houston and last June, Nuro announced a new partnership with FedEx to add to a growing roster of some of the largest brands in the US. Nuro has piloted autonomous, eco-friendly local delivery for communities in Texas, Arizona, and California, aiming to make streets safer and cities more livable.

Nuro’s approach to building a custom-goods only vehicle allows it to turn safety innovations that have traditionally been focused internally on passengers (seatbelts, airbags, etc) inside out to prioritize vulnerable road users such as bikers and pedestrians. The R2 robot features 360° and thermal cameras, as well as lidar and radar for a rich representation of the road. Nuro builds its own HD maps, and its AV stack combines engineered and machine-learned components for maximum robustness. A redundant collision prevention system provides an additional layer of safety. Nuro uses Scale to label both 2D and 3D data, generate and label HD maps, curate its training data, and identify new edge cases to solve with new labels and better models.

The Problem

Nuro Needed Better Tools to Identify Infrequent but Meaningful Scenarios in Training Data.

Nuro faced the daunting task of identifying thousands of images from their training set that include small animals as obstacles, yet these labels were not in the ground truth of their training data. Specifically, the team needed to identify unlabeled images from their training set that include:

  • Pedestrians in unusual postures (~60 found with Nuro’s internal tool vs. 1000+ with Nucleus)
  • Animal cases—horses, large birds, small non-pet animals (~50 found with Nuro’s internal tool vs. 400+ with Nucleus)
  • Occluded and backlit pedestrians (10-20 found with Nuro’s internal tool vs. 500+ with Nucleus)
  • Infrequently encountered vehicles such as excavators (10-20 found with Nuro’s internal tool vs. 500+ with Nucleus)

Yet these labels were not present in the ground truth of their training data.

“ML models only deliver the highest accuracy when they can handle edge cases that might be challenging, uncommon, or even dangerous. The Autotag functionality in Nucleus helps us immensely by identifying examples of infrequent scenarios in our dataset, all with a simple query. As Nuro works to ensure efficient deliveries as safely as possible, we depend on tools like Nucleus to curate edge cases which we can use to train ever more accurate and capable models.”

Jack Guo

Jack Guo

Head of Autonomy Platform Nuro

The Solution

Nucleus Helps Nuro’s Perception Team Collaborate to Unearth Safety-Critical Edge Cases.

Nuro aims to ingest a substantial portion of its training dataset into Nucleus to identify rare edge cases from a massive collected dataset with minimal effort or intervention from its ML team. Thus, Nuro can build robust, accurate models that can even handle rare but significant (or even dangerous) edge cases, like a sidewalk collision with a toddler or domestic animals.

Together, Nucleus’ Object Autotag and data curation workflows enable ML engineers to use a consistent platform for these common modeling tasks. Autotag lets users select unlabeled images of a certain category, and then using an internal model and its feature vectors, provide a set of similar images, suitable to be sent out for labeling with new class labels. That way, team members can all collaborate on the same unified platform. Nuro’s ML team can now incorporate newly identified edge cases (rare scenarios) and re-train a new model in just a few days rather than over a week.

“Nucleus makes it easy for us to identify edge cases, create an Autotag, and then prioritize what labels we need from Scale. Our collaboration with Scale began with more and more targeted labels for 2D and 3D data, progressed to HD map labeling, and today extends to dataset management and curation. Identifying and labeling edge cases helps us train more robust and generalizable models for our delivery robots in the real world. Our goal is always higher than 95% accuracy for rare cases, often comprising 1% of the dataset. Nucleus sets us up to train models that achieve those metrics.”

Peter Wang

Peter Wang

Project Manager Nuro

The Result

With Nucleus, Nuro Tackles the Ever Growing Long Tail of Rare Scenarios.

Today, Nuro’s Perception Team maintains an impressive dataset at a scale of over 500 million images. For training data, when the team holds the amount of labeled data fixed for supervised learning, they’ve been able to handle more and more edge cases like avoiding collisions with large birds. Thus, Nucleus helps Nuro achieve targeted levels of performance on specific subdomains.

Looking forward, Nuro intends to continue to scale up the size of its dataset that it curates with Nucleus so that the perception team will be ready to target any new edge cases, should they appear in the suburban wild. Future training data will likely include changing environmental conditions such as snow; noisy obstacles like tree, lawn, and leaf clippings; and identifying actions, such as the hand-wave of a pedestrian.

After re-training their models with data and labels that address these new scenarios, Nuro can improve the safety and efficiency of their automated deliveries with fewer human interventions required.

“Before Nucleus, we used several scripts to find similar examples in our dataset, but the experience wasn’t reproducible or particularly collaborative. Nucleus helps us quickly establish new slices of our dataset suitable for labeling. Thus, with Nucleus, we are able to uncover rare classes in our dataset, finding larger sample sizes than before.”

Mehul Smriti Raje

Mehul Smriti Raje

ML Ground Truth Engineer Nuro

Get Started Today