experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

General

Engineering

Clement Delangue HuggingFace CEO, shares what Transformers can do. He explores how we might ensure they are used ethically and transparently.

Clement Delangue HuggingFace CEO on Transformers - TransformX 2021 Conference - Scale AI

HuggingFace CEO, Clement Delangue: Are Transformers Becoming the Most Impactful Tech of the Decade?

Today we&rsquo;re thrilled to share that Nucleus now supports 3D Sensor Fusion data, including LiDAR point clouds.
<a href="https://scale.com/nucleus" target="_blank" rel="noopener noreferrer">Scale Nucleus</a> is the mission control for your data, providing advanced tooling for curating, visualizing and exploring machine learning datasets and debugging ML models. We recently launched an all-new Nucleus for dataset management, complete with privacy features and updated Autotag capabilities, all to help teams build better ML models by building better datasets. Now, Nucleus can help teams manage 3D datasets to develop highly accurate perception models for a range of use cases, including autonomous vehicles, robotics, and augmented/virtual reality (AR and VR).
Scale Nucleus helps you visualize and debug 3D Sensor Fusion data by making it easy to:
<ul>
<li>Construct Scenes that contain a sequence of LiDAR point clouds and their corresponding camera images over time</li>
<li>Visualize, query and share sensor fusion data with colleagues</li>
<li>Integrate Sensor Fusion data with your 3D model predictions to uncover and identify:</li>
<li>Discrepancies between 3D object detection model outputs and ground truth</li>
<li>False positive, common class confusions, and cuboid predictions that have low Intersection over Union (IoU) with ground truth</li>
<li class="ql-indent-1">Model performance and improvements over time</li>
</ul>
The Nucleus API makes it easy to import your data, construct scenes, and add model predictions. You can also import data from an existing Scale project directly into Nucleus.
<h2>A Primer: Nucleus Scenes</h2>
Scenes are the building blocks of 3D datasets in Nucleus and are designed with point clouds as the primary unit of data. More formally, a Scene is a sequence of Frames over time, where each Frame contains exactly one point cloud and an optional set of camera images. Here we can see a LiDAR point cloud above two paired images (the full dataset includes more than just front and back cameras, but a full suite of sensor data), which helps explain why we designed Nucleus Scenes to encapsulate data across both time and sensor modalities:
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/00157f16-6ed9-4ffc-45cc-60b676806e00/public" alt="Structure Diagram of Nucleus 3D Scenes">One useful way to interpret a LiDAR Scene is as a table of sensor readings. A row in the table corresponds to a sequence of readings from a particular sensor, while a column in the table represents a collection of sensor data from the same timestamp (we call this a &ldquo;frame&rdquo; of the scene).
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6ea98ea3-b4d4-40eb-f36f-3562514cab00/public" alt="Complex Nucleus 3D Scene on SF's Embarcadero, Quarter Angle View">
On their own, point clouds can be difficult to interpret, due to lack of motion cues or associated camera data. Nucleus Scenes make sets of 3D and 2D data easier to interpret for the following reasons: first, any number of 2D camera frames can be attached to each 3D LiDAR frame; and second, Scenes link frames from the same timespan so that you can examine all frames in a linear fashion.
Let&rsquo;s now view the same Scene in Nucleus and see if we can identify the mystery object from above:
<iframe class="ql-video" src="https://scaleai.wistia.com/embed/medias/8ikm69knof" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
After uploading predicted cuboids from an ML model for your Scene, Nucleus automatically computes performance metrics under the hood. You can sort predicted cuboids by IoU or filter by false positives to quickly narrow in on your model&rsquo;s failure modes. (These features have been available for 2D image analysis for some time, but today we are announcing their ability for 3D data, too.)
To summarize, by creating and uploading 3D Scenes to Nucleus, you can easily visualize LiDAR point clouds and images side-by-side, quality-check your ground truth annotations, and search your data based on metadata or even ML-produced attributes. Lastly, you can use Nucleus to measure key prediction performance metrics, such as IoU and false positives.
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6908e9df-249d-47df-4db5-a39827556c00/public" alt="Birds' Eye View of SF Embarcadero as Seen through LiDAR and Multiple Image Sensors in Nucleus Scene">
<h2>Getting Started with 3D Data in Nucleus</h2>
Visualizing, querying, and measuring the quality of sensor fusion datasets is a challenge facing even the most sophisticated ML teams. With the newly launched support of multimodal scenes, Nucleus makes it easy to navigate through massive datasets, and quickly narrow down on the most difficult examples. Unleash the full potential of your datasets by leveraging Nucleus for debugging, search, and quality related workflows.
The complete code for this tutorial is available in <a href="https://colab.research.google.com/github/brwillia/nuc-3d/blob/main/nucleus_3d_demo.ipynb#scrollTo=rr88VqEHaL2_" target="_blank" rel="noopener noreferrer">this Colab notebook</a>. If you&rsquo;d like to follow along step by step and create a personal dataset on Nucleus, you will first need to make a <a href="http://scale.com/signup" target="_blank" rel="noopener noreferrer">free Scale account</a> and obtain a test API key (which you can <a href="https://dashboard.scale.com/settings/profile" target="_blank" rel="noopener noreferrer">find here</a> if you navigate to your user icon, and then click on API keys). You can then copy the notebook and add your personal key to upload your data and try out Nucleus for yourself. At the end, you&rsquo;ll also find a section that shows you how to format and upload your own LiDAR data for curation and management in Nucleus if you&rsquo;re ready to manage your own 3D data in Nucleus.

Learn how to ingest and curate 3D LiDAR point cloud data in Nucleus.

Nucleus now supports 3D Scenes.

Managing 3D Sensor Fusion Data with Scenes in Scale Nucleus

Kevin Scott, CTO of Microsoft shares how we can democratize AI for the benefit of everyone in society.

Highlights From Kevin Scott Microsoft - TransformX 2021 Conference - Scale AI

Microsoft CTO Kevin Scott's Seven Insights on the Future of AI

Editor’s note: This post originally appeared on <a href="https://lambdalabs.com/" rel="noopener noreferrer" target="_blank">lambdalabs.com</a>. You can find a link to it <a href="https://lambdalabs.com/blog/lambda-scale-nucleus/" rel="noopener noreferrer" target="_blank">here</a>. Training accurate models typically requires a deep understanding of your training data. Through visualization, it’s important to both identify potential flaws in your ground truth labels, and to look for insights that will guide model development. Thus, visualizing your training data is the foundation of every successful deep learning project. Nucleus is a tool from Scale that allows you to visualize and debug your training data. It provides an intuitive web interface to visualize and explore a wealth of different types of data and their labels. It also gives you the tools to explore and understand the relationship between your data and the model predictions that you generate. In this post, we'll show you how you can get started with Nucleus by training a pedestrian detection model. To do this, we will use Nucleus to visualize an object detection dataset containing images of common street scenes, and compare the ground truth to the predictions from a trained neural network. This loop of inspecting your training data and looking for failure modes in a network's predictions is a crucial process in refining both your training dataset and your neural network.<h2>Getting Started</h2> The example model in this post was trained on a <a href="https://lambdalabs.com/gpu-workstations/vector" rel="noopener noreferrer" target="_blank">Lambda Vector workstation</a> running <a href="https://lambdalabs.com/lambda-stack-deep-learning-software" rel="noopener noreferrer" target="_blank">Lambda Stack</a>. If you want to follow along, please see the <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ" rel="noopener noreferrer" target="_blank">example notebook here</a>. To start, we’ll download the publicly available <a href="https://www.cis.upenn.edu/~jshi/ped_html/" rel="noopener noreferrer" target="_blank">Penn-Fudan Dataset</a>, which consists of photographs of pedestrian data with labeled bounding boxes. To load these into Nucleus, we need to do two things: first, sign up for a free Scale account and corresponding API key, and second, install the Nucleus client on our Lambda workstation from a command prompt or shell:<pre class="ql-syntax" spellcheck="false">pip install scale-nucleus
</pre> <h2>Training Data</h2> Once we've installed the Nucleus client and downloaded the dataset, we can upload the images and ground truth labels to a new Dataset we created in Nucleus. (For the full details of how to do this, see <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ#scrollTo=jC-728nqvD0L" rel="noopener noreferrer" target="_blank">this section</a> of the notebook.) Now that we’ve uploaded the data, we can visualize the ground truth labels in the Nucleus web dashboard without writing any code.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/edf2a44b-d1e4-4dcc-d6f4-70b3587ac100/public" alt="PennFudan pedestrian dataset with bounding boxes in Nucleus"> <h2>Model Predictions</h2> Now that we've "<a href="https://karpathy.github.io/2019/04/25/recipe/#1-become-one-with-the-data" rel="noopener noreferrer" target="_blank">become one with our data</a>," we can start thinking about training a neural network for object detection. For this example we fine-tuned a <a href="https://arxiv.org/abs/1506.01497" rel="noopener noreferrer" target="_blank">Faster R-CNN</a> model pre-trained on the <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">COCO dataset</a>. Training it for 10 epochs only takes a few minutes on an <a href="https://lambdalabs.com/blog/nvidia-rtx-a6000-benchmarks/" rel="noopener noreferrer" target="_blank">NVIDIA RTX A6000</a> in a Lambda Vector Workstation. After training is finished we can see that our model has reached an Average Precision (AP@IoU=0.50:0.95) of 0.83. This sounds OK. It's not too far from 1, but what does it mean? Do we have a good pedestrian detector yet? The AP metric of object detection is notoriously hard to get your head around, and there are endless <a href="https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173" rel="noopener noreferrer" target="_blank">blog posts</a> that aim to provide an intuition for it, but you’ll ultimately need to look at our results to see how we're doing. Nucleus allows us to upload our model predictions and compare them to the ground truth. Here we can see that blue boxes predicted by our model match pretty well with the green ground truths from the dataset: <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/1dc58fa6-c049-4610-324b-aab8d114a600/public" alt="Image Grid, Ground Truth versus Fast-RCNN Bounding Boxes"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/005b8ff2-c8e7-44a0-42bc-615be98ba000/public" alt="Nucleus Annotation, Classification, and IoU Detail"> <h2>What to Try Next?</h2> Nucleus also gives some nice tools to help us figure out what else we should try to improve our accuracy results. Here we visualize the ground truth labels with the model predictions so that we can get a feel for where our model is going off the rails. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/08ad13c7-2552-45ce-0650-cc8e4115a200/public" alt="Correcting the Ground Truth Bounding Box in Nucleus"> In many cases, it looks like our model is already doing a pretty good job, and it's the ground truth labels that have actually missed some pedestrians! For anyone who has worked on a practical machine learning project, this is probably a familiar story—finding and fixing errors and problems in your dataset is just as much part of deep learning as designing and training models. In this case, we’ll need to go back to our data labels and fix the ground truth errors in order to get a more accurate evaluation of our model. This is also something we can do directly in Nucleus with its convenient annotation tools. <h2>Conclusion</h2> Nucleus is a convenient way to visualize and organize your training data for a machine learning project, as we showed in our object detection example. Most importantly, it makes it easy to inspect and track the predictions from your trained models, and helps you establish the virtuous cycle of training networks, evaluating predictions, identifying failure cases, and improving both data and models: all of these steps prove to be crucial in developing an effective deep learning project. To learn more about Lambda workstations, <a href="https://lambdalabs.com/" rel="noopener noreferrer" target="_blank">check out Lambda’s website</a>, or to get started managing and improving your dataset, <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">sign up for Scale Nucleus here</a>. If you want to try this workflow out on your own <a href="https://lambdalabs.com/gpu-workstations/vector" rel="noopener noreferrer" target="_blank">Lambda workstation</a> or Lambda <a href="https://lambdalabs.com/service/gpu-cloud" rel="noopener noreferrer" target="_blank">Cloud instance</a>, you can explore the <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ" rel="noopener noreferrer" target="_blank">example notebook here</a>.

Scale partners with Lambda to help businesses train better models on better data.

Data Curation with Nucleus and Better Training on Lambda 

Lambda and Scale Nucleus: Empowering Your Model Training with Better Data

TransformX Conference brought together leaders, engineers, & researchers to explore the shift from research to reality within AI & Machine Learning.

Highlights From TransformX 2021 Conference - Scale AI

Highlights From TransformX 2021 Conference

Scale AI is proud to announce the availability of our ISO 27001 certificate 

Scale AI | ISO 27001 Compliance

Our Commitment to Securing Customer Data: Scale + ISO 27001

Research conducted by Scale AI's ML team has found that human annotations remain indispensable for deep learning models

Scale AI Research Highlight

Human Annotations Remain Indispensable for Developing Deep Learning Models

Scale AI | Enhancing Taxonomy Categorization with Human & ML Consensus

Enhancing Taxonomy Categorization for eCommerce, Marketplace, and Retail AI

ML & Human Consensus: The Best of Both Worlds

It’s been an exciting journey since we shipped Scale Nucleus for the first time <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">just over a year ago</a>. We’ve helped over 100 different organizations explore, curate and improve their datasets and debug their machine learning (ML) models, across semantic segmentation, object detection, and even 3D point clouds. Today, we’re excited to share an all-new Nucleus, with powerful new features and a redesigned user interface that make it easier than ever to create better models with better datasets. To improve ML models requires first understanding where they fail. Nucleus helps you go beyond aggregate error metrics, and instead quickly discover specific patterns of failure across your model predictions and dataset labels. Once you’ve identified problems, Nucleus makes them easy to fix. Address bad labels through efficient, model-assisted data quality assurance (QA) and one-click labeling integration, or improve model predictions through targeted data curation of what to label next.<a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> is one of the most intuitive and efficient ways to select the right data for labeling: random sampling shouldn’t be the way you select data to label. Training better-performing models requires a focus on edge cases, false positives, and the portion of minority classes or edge cases that are relevant to the modeling problem you’re trying to solve. And your curation process shouldn’t be overly time-intensive or manual. As we launched Nucleus, we <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">explained</a> that better ML starts with understanding your data in depth. To improve production ML, you need to understand your models’ qualitative failure modes, fix them by gathering the right data, and curate diverse scenarios.<h2>From Similarity Search to Autotag:</h2> When Scale customers look to improve ML accuracy, they find that their models often struggle with minority classes and edge cases, what we refer to as “the long tail.” It might be a QA team that highlights challenges with a self-driving vehicle, like the relatively rare combination of entering a tunnel at nighttime behind a truck. Although this cross-section accounts for only a small fraction of driving scenarios, their computer vision models must be capable of handling them just as well. In the below example, we’ve identified uncertain samples, and using a similarity search based on internal models and feature vectors, surfaced similar portions of the dataset for labeling or QA. This process culminates in our Autotag feature:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/907a4a44-3760-4607-8981-4fd69c669700/public" alt="Similarity search and Autotag"> Autotag is an incredibly efficient way to tag similar objects or scenes with a new class. Simply select a few similar images of the class you are trying to classify or create a tag for, click ‘Autotag,’ and then further refine your set of images with positive and negative examples. Scale Nucleus will then create an Autotag, an internal classification attached to the subset of your dataset that Scale identified as similar. You can then retrieve objects in your dataset based on this classification, even if labels for it don’t exist in your ground truth. <h2> </h2><h2> The same useful core, with refined usability:</h2><h2> </h2> Nucleus focuses on several main use cases that we’ve developed in conjunction with our customers: Understand the strengths and weaknesses in your dataset as you identify ways to improve quality: <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3b36ea98-b2c3-42e9-c063-a4a2a400b200/public" alt="Understand the strengths and weaknesses in your dataset as you identify ways to improve quality."> Using the query bar at top or powerful search options at left , you can discover edge cases, debug model failures, and efficiently QA any part or your dataset. Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU): <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/94c22661-e595-47af-3446-858a7ae4e700/public" alt="Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU)."> Object view helps you assess all classes, labels, and bounding boxes in your dataset and make quick tweaks to a label, approve/reject the object as well-labeled, or send the object to be labeled. Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fd250417-e7f1-42bc-8df6-f1e141092d00/public" alt="Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data."> The Insights tab provides you with interactive class distributions, correlations, and confusion matrices, from which you can access the underlying data for each category in one click. Scale Nucleus was built with the explicit goal of helping ML teams improve their datasets to extract better and better performance out of models they’re training on their data. Whether you’re starting out with an industry-standard dataset, and then execute transfer learning with the addition of proprietary data, or training a model entirely on your own data, Nucleus helps you train your model on the data that matters. <h2> </h2><h2> Privacy Mode: using Nucleus without sharing sensitive data</h2><h2> </h2> Several prospective customers asked if they could use Scale Nucleus without uploading their training datasets to our cloud. Accordingly, we created Privacy Mode in conjunction with our existing API, letting you use Nucleus to curate your dataset without sensitive raw data ever leaving your servers With Privacy Mode, you can submit URLs to Nucleus that link to raw data assets like images or point clouds, instead of transferring that data to Scale. These URLs may optionally be protected behind your corporate VPN or an IP whitelist. When you load a Nucleus web page, your browser will directly fetch the raw data from your servers without it ever being accessible to Scale. Privacy Mode even works well with similarity search and Autotag, as users can <a href="https://dashboard.scale.com/nucleus/docs/api#create-a-custom-index" rel="noopener noreferrer" target="_blank">create custom model embedding indexes</a> for their datasets. <h2> </h2><h2> What customers are saying about Nucleus:</h2><h2> </h2><blockquote> </blockquote><blockquote> “KeepTruckin encounters all manners of surprising edge cases in real world</blockquote><blockquote> data collection, so when it comes to knowing we’re labeling the most</blockquote><blockquote> valuable subset of our collected data, we turn to Scale Nucleus. Its</blockquote><blockquote> intuitive visualizations, query engine and Autotag help our teams improve</blockquote><blockquote> both data quality and models, all in the same motion.”</blockquote><blockquote> </blockquote>—Ali Rehan, Engineering Manager AI/Vision Products, KeepTruckin Nucleus makes it easy to query data samples based on their metadata, or simply bucket images based on “similarity,” specifically along feature vectors of a Scale-trained model. For example, if you know your model should be detecting police vehicles and it’s not doing so already, you can quickly query for a handful of examples, find similar images, and through simple positive and negative feedback to Nucleus, identify a subset of your dataset to send out for labeling. One year in, Scale Nucleus has already come a long way towards building a new generation of ML tooling, but we’re just getting started. If you’d like to join us in this journey and try Scale Nucleus, sign up <a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">on our website</a> or <a href="https://calendly.com/russell-kaplan/30min" rel="noopener noreferrer" target="_blank">schedule a demo</a>.

Scale Nucleus helps you curate and automatically tag your training data.

Nucleus relaunched: making data and model quality even easier.

Improving Datasets and Debugging Machine Learning Models with Scale Nucleus, Rebuilt from the Ground Up 

At Scale AI, customers come first. This is a core credo that drives many of our daily decisions. We take the trust of our customers and the security of their data seriously, and today, we are proud to announce the availability of our System and Organizational Controls (SOC) 2 Type II report, and the achievement of HIPAA compliance. As a cloud-based data platform for AI, our customers need to be confident that we securely handle their data. SOC reports are prepared by independent Certified Public Accountant (CPA) auditors based on the internationally recognized <a href="https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf" rel="noopener noreferrer" target="_blank">Trust Services Criteria</a> framework developed by the American Institute of Certified Public Accountants (<a href="https://www.aicpa.org/" rel="noopener noreferrer" target="_blank">AICPA</a>). Achieving compliance with SOC 2 Type II requires Scale to demonstrate its compliance with the service commitments and system requirements in the Security Trust Services Category. The Health Insurance Portability and Accountability Act (<a href="https://www.hhs.gov/hipaa/index.html" rel="noopener noreferrer" target="_blank">HIPAA</a>) created national standards to protect sensitive patient health information (PHI). Achieving compliance with the regulations set forth by HIPAA demonstrates Scale’s commitment to safeguarding PHI. Our HIPAA compliant service allows customers in the highly-regulated and security-conscious healthcare industry to utilize Scale as their secure, compliant data platform for AI. Our commitment to enterprise-grade security is one reason leading ML teams such as NVIDIA, iRobot, Toyota Research Institute, Open AI, and more rely on Scale AI to accelerate the development of their AI applications. If you’d like further details on our SOC 2 Type II report or HIPAA compliance, please contact us at security@scale.com or sales@scale.com

Scale AI is proud to announce the availability of our SOC2 Type II report and achievement of HIPAA compliance 

Scale AI | SOC 2 Type II and HIPAA Compliance

Our Commitment to Securing Customer Data: Scale + SOC 2 Type II and HIPAA

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog