BlogCategoryTag

Header

Footer

PreFooter

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

PaginationButton

AnnouncementContextProvider

UnnamedGlobalGroupOfVariants2

Screen

Segment

Experiment

Experiment2

scale.com (Production)

plasmic-nav

plasmic-embed-css

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

react-slick

react-quill

Product

Company

General

Engineering

TransformX Conference brought together leaders, engineers, & researchers to explore the shift from research to reality within AI & Machine Learning.

Highlights From TransformX 2021 Conference - Scale AI

Highlights From TransformX 2021 Conference

Scale AI is proud to announce the availability of our ISO 27001 certificate 

Scale AI | ISO 27001 Compliance

image.jpg

Our Commitment to Securing Customer Data: Scale + ISO 27001

Today, we are thrilled to announce the General Availability of <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a>, the fastest way to production-level quality labels, with no data minimums. With Scale Rapid, ML teams can get an initial set of high-quality training data in a matter of hours. We are also pleased to announce a new <a href="http://scale.com/university-research" rel="noopener noreferrer" target="_blank">University Research Program</a>, offering students and researchers reduced rates on Scale Rapid. Often, quality labeled data will be the blocker in experimenting with different model setups. Slow iteration speed with labeling can significantly slow down AI development. Using Scale Rapid, ML teams can stand up production-quality data pipelines in three simple steps: set up a project, calibrate on instructions and quality, then scale to production.With Scale Rapid, ML teams can: <ol><li>Experiment quickly by setting up labeling projects in</li><li> minutes and receiving initial datasets within hours.</li><li> </li><li>Iterate over potential edge cases and instructions by</li><li> getting fast and real-time feedback on labeling instructions and ensure</li><li> high-quality annotations.</li><li> </li><li>Scale to production-level pipelines in days with precision</li><li> quality control. No need for lengthy contract negotiations, purchase order,</li><li> or committed spend.</li><li> </li><li> </li></ol><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8876b33d-d9c6-41e8-f868-4f8c485c6200/public" alt="Scale Rapid - Grata Quote"> Scale Rapid has already annotated more than 750k image and text tasks during the Early Access period with usage doubling month over month. We look forward to improving more ML teams’ iteration speed by providing extremely fast turnarounds for high-quality data. Stop waiting for your labels and <a href="https://dashboard.scale.com/signup?product=rapid" rel="noopener noreferrer" target="_blank">get started with Scale Rapid</a> today. For university rates, apply <a href="http://scale.com/university-research" rel="noopener noreferrer" target="_blank">here</a>.

Stop Waiting for Your Labels. Scale Rapid is the fastest path to production-quality labels with no data minimums. 

Stop Waiting for Your Labels with Scale Rapid

Scale Rapid: Now Generally Available

Research conducted by Scale AI's ML team has found that human annotations remain indispensable for deep learning models

Scale AI Research Highlight

Human Annotations Remain Indispensable for Developing Deep Learning Models

Scale AI | Enhancing Taxonomy Categorization with Human & ML Consensus

Enhancing Taxonomy Categorization for eCommerce, Marketplace, and Retail AI

ML & Human Consensus: The Best of Both Worlds

It’s been an exciting journey since we shipped Scale Nucleus for the first time <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">just over a year ago</a>. We’ve helped over 100 different organizations explore, curate and improve their datasets and debug their machine learning (ML) models, across semantic segmentation, object detection, and even 3D point clouds. Today, we’re excited to share an all-new Nucleus, with powerful new features and a redesigned user interface that make it easier than ever to create better models with better datasets. To improve ML models requires first understanding where they fail. Nucleus helps you go beyond aggregate error metrics, and instead quickly discover specific patterns of failure across your model predictions and dataset labels. Once you’ve identified problems, Nucleus makes them easy to fix. Address bad labels through efficient, model-assisted data quality assurance (QA) and one-click labeling integration, or improve model predictions through targeted data curation of what to label next.<a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> is one of the most intuitive and efficient ways to select the right data for labeling: random sampling shouldn’t be the way you select data to label. Training better-performing models requires a focus on edge cases, false positives, and the portion of minority classes or edge cases that are relevant to the modeling problem you’re trying to solve. And your curation process shouldn’t be overly time-intensive or manual. As we launched Nucleus, we <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">explained</a> that better ML starts with understanding your data in depth. To improve production ML, you need to understand your models’ qualitative failure modes, fix them by gathering the right data, and curate diverse scenarios.<h2>From Similarity Search to Autotag:</h2> When Scale customers look to improve ML accuracy, they find that their models often struggle with minority classes and edge cases, what we refer to as “the long tail.” It might be a QA team that highlights challenges with a self-driving vehicle, like the relatively rare combination of entering a tunnel at nighttime behind a truck. Although this cross-section accounts for only a small fraction of driving scenarios, their computer vision models must be capable of handling them just as well. In the below example, we’ve identified uncertain samples, and using a similarity search based on internal models and feature vectors, surfaced similar portions of the dataset for labeling or QA. This process culminates in our Autotag feature:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/907a4a44-3760-4607-8981-4fd69c669700/public" alt="Similarity search and Autotag"> Autotag is an incredibly efficient way to tag similar objects or scenes with a new class. Simply select a few similar images of the class you are trying to classify or create a tag for, click ‘Autotag,’ and then further refine your set of images with positive and negative examples. Scale Nucleus will then create an Autotag, an internal classification attached to the subset of your dataset that Scale identified as similar. You can then retrieve objects in your dataset based on this classification, even if labels for it don’t exist in your ground truth. <h2> </h2><h2> The same useful core, with refined usability:</h2><h2> </h2> Nucleus focuses on several main use cases that we’ve developed in conjunction with our customers: Understand the strengths and weaknesses in your dataset as you identify ways to improve quality: <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3b36ea98-b2c3-42e9-c063-a4a2a400b200/public" alt="Understand the strengths and weaknesses in your dataset as you identify ways to improve quality."> Using the query bar at top or powerful search options at left , you can discover edge cases, debug model failures, and efficiently QA any part or your dataset. Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU): <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/94c22661-e595-47af-3446-858a7ae4e700/public" alt="Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU)."> Object view helps you assess all classes, labels, and bounding boxes in your dataset and make quick tweaks to a label, approve/reject the object as well-labeled, or send the object to be labeled. Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fd250417-e7f1-42bc-8df6-f1e141092d00/public" alt="Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data."> The Insights tab provides you with interactive class distributions, correlations, and confusion matrices, from which you can access the underlying data for each category in one click. Scale Nucleus was built with the explicit goal of helping ML teams improve their datasets to extract better and better performance out of models they’re training on their data. Whether you’re starting out with an industry-standard dataset, and then execute transfer learning with the addition of proprietary data, or training a model entirely on your own data, Nucleus helps you train your model on the data that matters. <h2> </h2><h2> Privacy Mode: using Nucleus without sharing sensitive data</h2><h2> </h2> Several prospective customers asked if they could use Scale Nucleus without uploading their training datasets to our cloud. Accordingly, we created Privacy Mode in conjunction with our existing API, letting you use Nucleus to curate your dataset without sensitive raw data ever leaving your servers With Privacy Mode, you can submit URLs to Nucleus that link to raw data assets like images or point clouds, instead of transferring that data to Scale. These URLs may optionally be protected behind your corporate VPN or an IP whitelist. When you load a Nucleus web page, your browser will directly fetch the raw data from your servers without it ever being accessible to Scale. Privacy Mode even works well with similarity search and Autotag, as users can <a href="https://dashboard.scale.com/nucleus/docs/api#create-a-custom-index" rel="noopener noreferrer" target="_blank">create custom model embedding indexes</a> for their datasets. <h2> </h2><h2> What customers are saying about Nucleus:</h2><h2> </h2><blockquote> </blockquote><blockquote> “KeepTruckin encounters all manners of surprising edge cases in real world</blockquote><blockquote> data collection, so when it comes to knowing we’re labeling the most</blockquote><blockquote> valuable subset of our collected data, we turn to Scale Nucleus. Its</blockquote><blockquote> intuitive visualizations, query engine and Autotag help our teams improve</blockquote><blockquote> both data quality and models, all in the same motion.”</blockquote><blockquote> </blockquote>—Ali Rehan, Engineering Manager AI/Vision Products, KeepTruckin Nucleus makes it easy to query data samples based on their metadata, or simply bucket images based on “similarity,” specifically along feature vectors of a Scale-trained model. For example, if you know your model should be detecting police vehicles and it’s not doing so already, you can quickly query for a handful of examples, find similar images, and through simple positive and negative feedback to Nucleus, identify a subset of your dataset to send out for labeling. One year in, Scale Nucleus has already come a long way towards building a new generation of ML tooling, but we’re just getting started. If you’d like to join us in this journey and try Scale Nucleus, sign up <a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">on our website</a> or <a href="https://calendly.com/russell-kaplan/30min" rel="noopener noreferrer" target="_blank">schedule a demo</a>.

Scale Nucleus helps you curate and automatically tag your training data.

Nucleus relaunched: making data and model quality even easier.

Improving Datasets and Debugging Machine Learning Models with Scale Nucleus, Rebuilt from the Ground Up 

At Scale AI, customers come first. This is a core credo that drives many of our daily decisions. We take the trust of our customers and the security of their data seriously, and today, we are proud to announce the availability of our System and Organizational Controls (SOC) 2 Type II report, and the achievement of HIPAA compliance. As a cloud-based data platform for AI, our customers need to be confident that we securely handle their data. SOC reports are prepared by independent Certified Public Accountant (CPA) auditors based on the internationally recognized <a href="https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf" rel="noopener noreferrer" target="_blank">Trust Services Criteria</a> framework developed by the American Institute of Certified Public Accountants (<a href="https://www.aicpa.org/" rel="noopener noreferrer" target="_blank">AICPA</a>). Achieving compliance with SOC 2 Type II requires Scale to demonstrate its compliance with the service commitments and system requirements in the Security Trust Services Category. The Health Insurance Portability and Accountability Act (<a href="https://www.hhs.gov/hipaa/index.html" rel="noopener noreferrer" target="_blank">HIPAA</a>) created national standards to protect sensitive patient health information (PHI). Achieving compliance with the regulations set forth by HIPAA demonstrates Scale’s commitment to safeguarding PHI. Our HIPAA compliant service allows customers in the highly-regulated and security-conscious healthcare industry to utilize Scale as their secure, compliant data platform for AI. Our commitment to enterprise-grade security is one reason leading ML teams such as NVIDIA, iRobot, Toyota Research Institute, Open AI, and more rely on Scale AI to accelerate the development of their AI applications. If you’d like further details on our SOC 2 Type II report or HIPAA compliance, please contact us at security@scale.com or sales@scale.com

Scale AI is proud to announce the availability of our SOC2 Type II report and achievement of HIPAA compliance 

Scale AI | SOC 2 Type II and HIPAA Compliance

Our Commitment to Securing Customer Data: Scale + SOC 2 Type II and HIPAA

For many ML teams, acquiring high quality labeled data is a barrier to iterating quickly on different models. That’s why we’re thrilled to offer early access to <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a>, a new product that enables machine learning engineers and researchers to receive high-quality labels and instruction feedback in as little as one hour.With Scale Rapid, you can: <ul><li>Create your own labeling projects</li><li>Upload your data labeling via UI or API</li><li>Design and submit your own labeling instructions</li><li>Direct quality improvements with new or updated evaluation tasks</li><li> </li></ul> Scale was founded to solve the challenge of scaling data labeling pipelines to production-level volumes. Today, as more diverse and agile teams experiment with ML, those teams are all too often blocked by the inability to update and receive prompt feedback on labeling instructions for rapid iteration. Scale Rapid solves for this by providing:<ol><li>Production-quality labels at lower volumes</li><li> </li><li> More insight and control over labelling workflow through quality and</li><li> throughput metrics, including edge case detection</li><li> </li><li> </li><li> No minimum commitments, annual contracts, or platform fees. You can pay as</li><li> you go, per label, not hourly rates—with a credit card, not a contract</li><li> </li><li> </li></ol> Our product architects have worked closely with some of the most seasoned deep learning and ML teams in the world. We quickly realized that getting quality labeled data, and getting it fast, is often the obstacle to getting good results from experimental models. In order to stay agile, research teams and startups alike need access to the best training data possible in the shortest amount of time. Scale Rapid solves these problems by using Scale’s core operations infrastructure and enabling an iterative feedback loop with the customer. Currently in Early Access, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> gives customers the fastest way to production-quality labels with no data minimums, while offering full control.<h2> </h2><h2> What customers are saying about Rapid:</h2><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/73fe19dd-5b59-4bd0-d61f-eea2b5a28400/public" alt="Collectability Quote"> Other research and engineering groups at Adobe, Bossanova, Grata, Square, and X2 AI are also labeling rightsized batches of data with Scale Rapid. If you are looking for a cost-effective data annotation solution for rapid experimentation, join the waitlist or contact us today for more information on our early access program. To learn more about Scale Rapid, watch our <a href="https://scale.com/events/scale-rapid" rel="noopener noreferrer" target="_blank">Tech Talk</a> that was held on August 18th, we walked through a live demo and took questions. <h2>Other recent product advances:</h2> Scale Rapid isn’t our only recent product development. To expand on the image training needs for autonomous vehicle customers, we launched <a href="https://scale.com/mapping" rel="noopener noreferrer" target="_blank">Scale Mapping</a> to address both computer vision and path planning needs. Also, <a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> helps you curate your dataset and select edge cases, false positives, and under-represented classes for labeling rather than simply randomly sampling your dataset for labeling. Join <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">the waitlist</a> or <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">contact us</a> today for more information on our early access program.

Get early access to Rapid, Scale’s new onramp for rapid experimentation.

Quickly Iterate on Datasets with Higher-Quality Annotations

Scale Rapid: Fast Experimentation with Production-Quality Labels

If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home" rel="noopener noreferrer" target="_blank">Join the Scale Exchange Community</a> to access these recordings and previous events. Scale’s mission is to accelerate the development of AI. For eCommerce and consumer marketplaces, AI powers high-quality catalog curation, user-to-product personalization, intelligent customer support, and real-time demand forecasting. Through the adoption of AI, leaders in the space have the opportunity to create tens of millions of dollars of value via better recommendations, increased conversion, and deep user satisfaction. Last week we hosted Scale Converge, to showcase the latest advances in AI and ML for eCommerce and online marketplaces. As we shared in our opening remarks, “it is imperative for every industry to leverage AI, or otherwise risk being left behind.” We brought together technical leaders to hear from the people and companies leading the eCommerce and retail industry forward. It was an event full of thoughtful conversations that examined the advancements made, the challenges that still lie ahead, and what we can expect in the years to come. <h2>Session Recap: Using AI to Launch New Products</h2> Toby Espinosa, VP at DoorDash joined Alex Wang, Founder &amp; CEO at Scale to discuss how DoorDash launches new products and services and how AI can help drive scale. Toby discussed how AI comes into play for DoorDash’s white label business and their framework for launching new products stating “We do a very simple process of, what is the problem for one of our core customers? How big is this? Is it a large enough business opportunity for us to dive head in first? And then do we have a competitive advantage? And if we do, if all those things match, it's like you're in Vegas and you see the slot machines and they go ding, ding, ding, they all match.” Toby also shared his insights into hiring the best talent profiles for those who can do the “and” in our current world. Given the operational nature of DoorDash’s business, their approach to innovation and problem solving often involves putting a small team on a problem or opportunity to solve at a micro level, and then using AI and technology to scale. Thus, when searching for product talent DoorDash looks for people that can do the AND. Those who can start solving the problem in the weeds, but have a vision for scale. <h2>Session Recap: Combining AI and Human Insights to Accelerate AI Adoption in eCommerce</h2> Sebastian Barrios, VP of Technology at Mercado Libre, Pranam Kolari, Sr. Director of Engineering at Walmart Technology, and Jason Sleight, Group Tech Lead at Yelp joined Aatish Nayak, Head of Content &amp; Language for an expert panel to discuss how the field is leveraging advancements in AI/ML to accelerate the world’s move to online retail. The group discussed a variety of topics including: <ul><li> </li><li>Applications of AI/ML across eCommerce marketplaces </li><li> </li><li> </li><li>Building MLOps platforms to support eCommerce use-cases</li><li> </li><li> </li><li>Challenges in personalization &amp; discoverability for ever-changing global customer interests</li><li> </li><li> </li><li>Utilizing human-in-the-loop for high search and catalog quality</li><li> </li></ul><h2>Challenges Ahead</h2> Because of the challenges posed by our ever-changing world, scaling ML from proofs of concept to production is nearly impossible without the most up-to-date data, continuous re-training of models, and human-in-the-loop to handle long-tail edge cases. But it’s not enough just to have a lot of data. AI must be developed with the most accurate, high-quality data. Without it, AI systems won’t live up to its full potential and bad data can lead to less than ideal outcomes: <ul><li> </li><li>Off-topic recommendations to users causing churn</li><li> </li><li> </li><li>Incorrect product catalogs</li><li> </li><li> </li><li>Slow customer support cycles</li><li> </li><li> </li><li>Fragility to changing global circumstances</li><li> </li></ul>Yet it is imperative for every industry to leverage AI, or otherwise risk being left behind. High-quality ground truth data must be the foundation of every AI strategy and it is important that companies have a trusted partner helping them build their AI models correctly from day one. Scale serves as that trusted partner, providing the critical data infrastructure for AI. We work on the most ambitious AI projects in the world to accelerate the development of this technology - from eCommerce and finance to transportation, robotics, and government - for companies like Instacart, Etsy, PayPal, OpenAI, Brex, and Flexport. <h2>What’s Next</h2> The continued advancements in AI have created incredible applications in today’s world but there is an exciting future that awaits. The AI revolution is inevitable – it’s imperative we build it right. At Scale, we’re excited to have a hand in creating trusted, ubiquitous use of AI. We are also grateful for all our incredible speakers representing organizations including DoorDash, Mercado Libre, Walmart Technology, and Yelp. If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home" rel="noopener noreferrer" target="_blank">Join the Scale Exchange Community</a> to access these recordings and previous events.

Scale AI Hosts Scale Converge, The Future of eCommerce & Retail with AI

Scale Converge 2021 | The Future of AI

Scale Converge: The Future of eCommerce & Retail with AI

Many AI systems rely on supervised learning methods in which neural networks train on labeled data. The challenge with supervised methods is getting models to perform well on examples not adequately represented in the training dataset. Typically, as the frequency of a particular category decreases, so does average model performance on this category (see Figure 1). It is often difficult and costly to achieve strong performance on the rare edge cases that make up the long tail of a data distribution. In this blog post, we’ll take a deeper look at how sophisticated data curation tools can help machine learning teams target their experiments toward taming the long tail.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fa8816c4-0552-489a-3131-4577f439ab00/public" alt="Figure 1: Long Tail Distribution"> Figure 1: This type of distribution, in which there are a few common categories followed by many rare categories, is called a long tail distribution. In the majority of deep learning applications, datasets collected in the real world tend to have this long-tail shape. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/703dc6dd-ba3f-4caf-380b-d77dbffb8800/public" alt="Figure 2: Ubiquity of Long Tail Distributions"> Figure 2: Long tail distributions occur frequently in the real world. For example, the frequency of words in English writing follows a long-tail distribution. The ugly truth is that almost all AI problems worth solving are made difficult by the challenge of a long-tail. Imagine you’re an engineer at an autonomous vehicle company, working on an object detection model trained on image data captured from vehicles on the road. This real-world data provides a textbook long-tail distribution: There are many thousands of images of cars on the highway, but very few depicting bicycles at night (see Figure 3). In a preliminary experiment, the subsequent model performs badly when localizing cyclists at night (see Figure 4). Neglecting this rare scenario is likely to result in high-severity errors during testing, but it’s critical for the algorithm to detect cyclists properly in order to prevent collisions. What’s the best method to address this model failure in a targeted way? There are many approaches to improving performance, including experimenting with new architectures or additional hyperparameter tuning. But if the goal is to improve model performance on this specific edge case, the most targeted approach is to add more examples of cyclists to the training dataset. The problem is that unlabeled data is inherently difficult to search; sampling randomly, one is unlikely to find more cyclists at night, and it’s too expensive to simply label all collected data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e7bb4d6-f3cb-4206-5f23-a30613bac500/public" alt="Figure 3: Berkeley Deep Drive Class Distribution"> Figure 3: In the Berkeley Deep Drive dataset, the category “rider” occurs infrequently compared to the category “car.” It’s no surprise that when we view individual examples of riders, our model trained on Berkeley Deep Drive does not do a good job of localizing these objects. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/01a950ec-c9ea-4901-65d7-befbc8d51900/public" alt="Figure 4: Examples of poor localization for the class “rider”"> Figure 4: Object detection predictions from an EfficientDet D7 architecture trained on Berkeley Deep Drive. Examples of poor localization for the class “rider”, a long tail class in the training dataset. To improve performance on the long tail of edge cases, machine learning practitioners can get caught up in an endless cycle of collecting more training data, subsampling, and retraining the model. While this type of experimentation is essential to machine learning development, it is incredibly costly with respect to time, compute, and data labeling. The cost of improving performance on the long tail often threatens the economics of AI products. <a href="https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/" rel="noopener noreferrer" target="_blank">This blog post</a> from Andreesen Horowitz details how AI businesses seldom have the attractive economic properties of traditional software businesses. Where software products benefit from economies of scale, AI businesses experience the opposite; as model performance improves over time, the marginal cost of improvement increases exponentially. It may require ten times the initial training data to yield any significant model improvement. Given that expensive model retraining is unavoidable, machine learning teams should focus on making iterative experimentation as efficient as possible. One way to focus experiments on improving the long tail is to use model failures to identify gaps in the training dataset and then source additional data to fill those gaps. Think of this approach to machine learning experimentation as “mining the long tail.” With each experiment, identify a failure case, find more examples of this rare scenario, add new examples to the training dataset, and repeat. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b271a31-9926-4679-4e58-29e90178b500/public" alt="Figure 5: ML Development Lifecycle"> Figure 5: The ML development lifecycle consists of collecting, generating, annotating, managing or curating, training, and evaluating. The data curation or management step has the biggest impact on an experiment’s success, yet is most overlooked in ML discourse. It’s difficult to target experiments in the most under-represented scenarios without sophisticated tools for dataset curation. There are three operationally critical capabilities for most AI teams: <ol><li>Identify long-tail scenarios during model evaluation</li><li>Reliably and repeatedly source similar examples from unlabeled data</li><li>Add these examples to a labeled training dataset.</li></ol> Some of the most effective teams we work with at Scale face the challenge of the long tail head-on by building their tools, operations, and workflows proactively around an iterative process of data curation. Hussein Mehanna, VP &amp; Head of AI/ML at Cruise, highlighted this at Scale’s Transform conference earlier this year: “You have to have a robust, fast, continuous learning cycle, so when you find something that you haven't seen before, you add it quickly to your data, and you learn from it, and you put it back into the model.” This idea of targeted dataset curation is simple; in practice, however, it’s difficult to achieve because of the substantial engineering work required to index millions of raw images and surface difficult scenarios automatically during model evaluation. Let’s put ourselves back in the shoes of an ML engineer. Having diagnosed poor performance on cyclists at night, we now want to target the next experiment at improving this scenario. With a complete set of data curation tools, this workflow would be simple. From one difficult example, source similar images from the large set of unlabeled data and send it through the labeling pipeline for annotation. Once this new batch of data is annotated, include it in the training dataset for subsequent experiments. We call this data-centric approach to experimentation model-in-the-loop dataset curation. Not only is this a highly targeted approach for addressing long-tail performance but, unlike random sampling, it results in better balance across classes and avoids labeling redundant examples that yield little marginal improvement of model performance. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/0o9gb9d3ss"></iframe> Performance on the long tail is often a make-or-break situation when it comes to AI in production. Especially in high-impact applications like healthcare — where data is difficult to collect and equity is critical — being proactive on edge cases is of utmost importance. Given the ubiquity of long-tail challenges in real-world applications, it’s surprising that there isn’t more discussion around how to tailor machine learning experiments toward these cases. ML teams should not run the risk of leaving precious rare scenarios undiscovered in a sea of unlabeled data. When it comes to the long tail, targeted dataset curation is an important tool for achieving performant AI models robust to the complexities of the real world.

The success of real-world machine learning applications often depends on model performance on edge cases.

How to Tame the Long Tail in Machine Learning

open-dataset-nucleus

slugs must be all lowercase. ie: ‘my-slug’

events-banner

This entry represents both interview quote highlight and interview chat highlight because of Builder limitations. Choose the fields that matches either of them.

interview-highlight

If it's selected, the author and quote fields will be taken, otherwise the chat fields. The default is for chat.

resources-download-model

transform-session

CTA to open the session. Leave blank if you don't want a cta for that session

Link for the CTA. Usually the link to the exchange session

blog-filter-type

event-transform-x-ticket

solution

homepage-quotes

logos

homepage-content

builder-debug

transfrom-x-in-person

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

guides-cta

summit-2023-partners

sales

z[deprecated] page

quotes

z[deprecated] compare

product

blog-category-model

announcement-banner

accelerate

Vercel Powered and Edge Computed OG Image. Preferred over og img file upload. If this field is non empty, it will take precedent over the uploaded file

summit-2023-speakers

homepage-resources

customers

Display order on the page /customers. Higher values show first

page-cross-product-webinar

interview

event-transform

Event model for new marketing events page

event-2-0

homepage-event

custom-fonts

transform-x-speaker

Be careful to not add any trailing white spaces to the slug. Slugs must be all lowercase. ie: ‘my-slug’

Order of the featured speakers displayed on the home page for transformX

z[deprecated] document-ai

summit-2023-agenda

event-converge

hosts

author

transform-x

Page: /In-Person. Section: Hero. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Card Body

Page: /In-Person. Section: Hero. Text: Tagline

open-dataset

z[deprecated] customers

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

legal

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

z[deprecated] guides-old

z[deprecated] guides-lp

partners

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Government

Solutions

Resources

Customers

Pricing →

Blog

The future of your industry starts here.