experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

Scale AI | Enhancing Taxonomy Categorization with Human & ML Consensus

Enhancing Taxonomy Categorization for eCommerce, Marketplace, and Retail AI

ML & Human Consensus: The Best of Both Worlds

It’s been an exciting journey since we shipped Scale Nucleus for the first time <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">just over a year ago</a>. We’ve helped over 100 different organizations explore, curate and improve their datasets and debug their machine learning (ML) models, across semantic segmentation, object detection, and even 3D point clouds. Today, we’re excited to share an all-new Nucleus, with powerful new features and a redesigned user interface that make it easier than ever to create better models with better datasets. To improve ML models requires first understanding where they fail. Nucleus helps you go beyond aggregate error metrics, and instead quickly discover specific patterns of failure across your model predictions and dataset labels. Once you’ve identified problems, Nucleus makes them easy to fix. Address bad labels through efficient, model-assisted data quality assurance (QA) and one-click labeling integration, or improve model predictions through targeted data curation of what to label next.<a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> is one of the most intuitive and efficient ways to select the right data for labeling: random sampling shouldn’t be the way you select data to label. Training better-performing models requires a focus on edge cases, false positives, and the portion of minority classes or edge cases that are relevant to the modeling problem you’re trying to solve. And your curation process shouldn’t be overly time-intensive or manual. As we launched Nucleus, we <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">explained</a> that better ML starts with understanding your data in depth. To improve production ML, you need to understand your models’ qualitative failure modes, fix them by gathering the right data, and curate diverse scenarios.<h2>From Similarity Search to Autotag:</h2> When Scale customers look to improve ML accuracy, they find that their models often struggle with minority classes and edge cases, what we refer to as “the long tail.” It might be a QA team that highlights challenges with a self-driving vehicle, like the relatively rare combination of entering a tunnel at nighttime behind a truck. Although this cross-section accounts for only a small fraction of driving scenarios, their computer vision models must be capable of handling them just as well. In the below example, we’ve identified uncertain samples, and using a similarity search based on internal models and feature vectors, surfaced similar portions of the dataset for labeling or QA. This process culminates in our Autotag feature:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/907a4a44-3760-4607-8981-4fd69c669700/public" alt="Similarity search and Autotag"> Autotag is an incredibly efficient way to tag similar objects or scenes with a new class. Simply select a few similar images of the class you are trying to classify or create a tag for, click ‘Autotag,’ and then further refine your set of images with positive and negative examples. Scale Nucleus will then create an Autotag, an internal classification attached to the subset of your dataset that Scale identified as similar. You can then retrieve objects in your dataset based on this classification, even if labels for it don’t exist in your ground truth. <h2> </h2><h2> The same useful core, with refined usability:</h2><h2> </h2> Nucleus focuses on several main use cases that we’ve developed in conjunction with our customers: Understand the strengths and weaknesses in your dataset as you identify ways to improve quality: <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3b36ea98-b2c3-42e9-c063-a4a2a400b200/public" alt="Understand the strengths and weaknesses in your dataset as you identify ways to improve quality."> Using the query bar at top or powerful search options at left , you can discover edge cases, debug model failures, and efficiently QA any part or your dataset. Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU): <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/94c22661-e595-47af-3446-858a7ae4e700/public" alt="Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU)."> Object view helps you assess all classes, labels, and bounding boxes in your dataset and make quick tweaks to a label, approve/reject the object as well-labeled, or send the object to be labeled. Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fd250417-e7f1-42bc-8df6-f1e141092d00/public" alt="Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data."> The Insights tab provides you with interactive class distributions, correlations, and confusion matrices, from which you can access the underlying data for each category in one click. Scale Nucleus was built with the explicit goal of helping ML teams improve their datasets to extract better and better performance out of models they’re training on their data. Whether you’re starting out with an industry-standard dataset, and then execute transfer learning with the addition of proprietary data, or training a model entirely on your own data, Nucleus helps you train your model on the data that matters. <h2> </h2><h2> Privacy Mode: using Nucleus without sharing sensitive data</h2><h2> </h2> Several prospective customers asked if they could use Scale Nucleus without uploading their training datasets to our cloud. Accordingly, we created Privacy Mode in conjunction with our existing API, letting you use Nucleus to curate your dataset without sensitive raw data ever leaving your servers With Privacy Mode, you can submit URLs to Nucleus that link to raw data assets like images or point clouds, instead of transferring that data to Scale. These URLs may optionally be protected behind your corporate VPN or an IP whitelist. When you load a Nucleus web page, your browser will directly fetch the raw data from your servers without it ever being accessible to Scale. Privacy Mode even works well with similarity search and Autotag, as users can <a href="https://dashboard.scale.com/nucleus/docs/api#create-a-custom-index" rel="noopener noreferrer" target="_blank">create custom model embedding indexes</a> for their datasets. <h2> </h2><h2> What customers are saying about Nucleus:</h2><h2> </h2><blockquote> </blockquote><blockquote> “KeepTruckin encounters all manners of surprising edge cases in real world</blockquote><blockquote> data collection, so when it comes to knowing we’re labeling the most</blockquote><blockquote> valuable subset of our collected data, we turn to Scale Nucleus. Its</blockquote><blockquote> intuitive visualizations, query engine and Autotag help our teams improve</blockquote><blockquote> both data quality and models, all in the same motion.”</blockquote><blockquote> </blockquote>—Ali Rehan, Engineering Manager AI/Vision Products, KeepTruckin Nucleus makes it easy to query data samples based on their metadata, or simply bucket images based on “similarity,” specifically along feature vectors of a Scale-trained model. For example, if you know your model should be detecting police vehicles and it’s not doing so already, you can quickly query for a handful of examples, find similar images, and through simple positive and negative feedback to Nucleus, identify a subset of your dataset to send out for labeling. One year in, Scale Nucleus has already come a long way towards building a new generation of ML tooling, but we’re just getting started. If you’d like to join us in this journey and try Scale Nucleus, sign up <a href="http://scale.com/nucleus" rel="noopener noreferrer" target="_blank">on our website</a> or <a href="https://calendly.com/russell-kaplan/30min" rel="noopener noreferrer" target="_blank">schedule a demo</a>.

Scale Nucleus helps you curate and automatically tag your training data.

Nucleus relaunched: making data and model quality even easier.

Improving Datasets and Debugging Machine Learning Models with Scale Nucleus, Rebuilt from the Ground Up 

At Scale AI, customers come first. This is a core credo that drives many of our daily decisions. We take the trust of our customers and the security of their data seriously, and today, we are proud to announce the availability of our System and Organizational Controls (SOC) 2 Type II report, and the achievement of HIPAA compliance. As a cloud-based data platform for AI, our customers need to be confident that we securely handle their data. SOC reports are prepared by independent Certified Public Accountant (CPA) auditors based on the internationally recognized <a href="https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf" rel="noopener noreferrer" target="_blank">Trust Services Criteria</a> framework developed by the American Institute of Certified Public Accountants (<a href="https://www.aicpa.org/" rel="noopener noreferrer" target="_blank">AICPA</a>). Achieving compliance with SOC 2 Type II requires Scale to demonstrate its compliance with the service commitments and system requirements in the Security Trust Services Category. The Health Insurance Portability and Accountability Act (<a href="https://www.hhs.gov/hipaa/index.html" rel="noopener noreferrer" target="_blank">HIPAA</a>) created national standards to protect sensitive patient health information (PHI). Achieving compliance with the regulations set forth by HIPAA demonstrates Scale’s commitment to safeguarding PHI. Our HIPAA compliant service allows customers in the highly-regulated and security-conscious healthcare industry to utilize Scale as their secure, compliant data platform for AI. Our commitment to enterprise-grade security is one reason leading ML teams such as NVIDIA, iRobot, Toyota Research Institute, Open AI, and more rely on Scale AI to accelerate the development of their AI applications. If you’d like further details on our SOC 2 Type II report or HIPAA compliance, please contact us at security@scale.com or sales@scale.com

Scale AI is proud to announce the availability of our SOC2 Type II report and achievement of HIPAA compliance 

Scale AI | SOC 2 Type II and HIPAA Compliance

Our Commitment to Securing Customer Data: Scale + SOC 2 Type II and HIPAA

If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home" rel="noopener noreferrer" target="_blank">Join the Scale Exchange Community</a> to access these recordings and previous events. Scale’s mission is to accelerate the development of AI. For eCommerce and consumer marketplaces, AI powers high-quality catalog curation, user-to-product personalization, intelligent customer support, and real-time demand forecasting. Through the adoption of AI, leaders in the space have the opportunity to create tens of millions of dollars of value via better recommendations, increased conversion, and deep user satisfaction. Last week we hosted Scale Converge, to showcase the latest advances in AI and ML for eCommerce and online marketplaces. As we shared in our opening remarks, “it is imperative for every industry to leverage AI, or otherwise risk being left behind.” We brought together technical leaders to hear from the people and companies leading the eCommerce and retail industry forward. It was an event full of thoughtful conversations that examined the advancements made, the challenges that still lie ahead, and what we can expect in the years to come. <h2>Session Recap: Using AI to Launch New Products</h2> Toby Espinosa, VP at DoorDash joined Alex Wang, Founder &amp; CEO at Scale to discuss how DoorDash launches new products and services and how AI can help drive scale. Toby discussed how AI comes into play for DoorDash’s white label business and their framework for launching new products stating “We do a very simple process of, what is the problem for one of our core customers? How big is this? Is it a large enough business opportunity for us to dive head in first? And then do we have a competitive advantage? And if we do, if all those things match, it's like you're in Vegas and you see the slot machines and they go ding, ding, ding, they all match.” Toby also shared his insights into hiring the best talent profiles for those who can do the “and” in our current world. Given the operational nature of DoorDash’s business, their approach to innovation and problem solving often involves putting a small team on a problem or opportunity to solve at a micro level, and then using AI and technology to scale. Thus, when searching for product talent DoorDash looks for people that can do the AND. Those who can start solving the problem in the weeds, but have a vision for scale. <h2>Session Recap: Combining AI and Human Insights to Accelerate AI Adoption in eCommerce</h2> Sebastian Barrios, VP of Technology at Mercado Libre, Pranam Kolari, Sr. Director of Engineering at Walmart Technology, and Jason Sleight, Group Tech Lead at Yelp joined Aatish Nayak, Head of Content &amp; Language for an expert panel to discuss how the field is leveraging advancements in AI/ML to accelerate the world’s move to online retail. The group discussed a variety of topics including: <ul><li> </li><li>Applications of AI/ML across eCommerce marketplaces </li><li> </li><li> </li><li>Building MLOps platforms to support eCommerce use-cases</li><li> </li><li> </li><li>Challenges in personalization &amp; discoverability for ever-changing global customer interests</li><li> </li><li> </li><li>Utilizing human-in-the-loop for high search and catalog quality</li><li> </li></ul><h2>Challenges Ahead</h2> Because of the challenges posed by our ever-changing world, scaling ML from proofs of concept to production is nearly impossible without the most up-to-date data, continuous re-training of models, and human-in-the-loop to handle long-tail edge cases. But it’s not enough just to have a lot of data. AI must be developed with the most accurate, high-quality data. Without it, AI systems won’t live up to its full potential and bad data can lead to less than ideal outcomes: <ul><li> </li><li>Off-topic recommendations to users causing churn</li><li> </li><li> </li><li>Incorrect product catalogs</li><li> </li><li> </li><li>Slow customer support cycles</li><li> </li><li> </li><li>Fragility to changing global circumstances</li><li> </li></ul>Yet it is imperative for every industry to leverage AI, or otherwise risk being left behind. High-quality ground truth data must be the foundation of every AI strategy and it is important that companies have a trusted partner helping them build their AI models correctly from day one. Scale serves as that trusted partner, providing the critical data infrastructure for AI. We work on the most ambitious AI projects in the world to accelerate the development of this technology - from eCommerce and finance to transportation, robotics, and government - for companies like Instacart, Etsy, PayPal, OpenAI, Brex, and Flexport. <h2>What’s Next</h2> The continued advancements in AI have created incredible applications in today’s world but there is an exciting future that awaits. The AI revolution is inevitable – it’s imperative we build it right. At Scale, we’re excited to have a hand in creating trusted, ubiquitous use of AI. We are also grateful for all our incredible speakers representing organizations including DoorDash, Mercado Libre, Walmart Technology, and Yelp. If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home" rel="noopener noreferrer" target="_blank">Join the Scale Exchange Community</a> to access these recordings and previous events.

Scale AI Hosts Scale Converge, The Future of eCommerce & Retail with AI

Scale Converge 2021 | The Future of AI

Scale Converge: The Future of eCommerce & Retail with AI

Many AI systems rely on supervised learning methods in which neural networks train on labeled data. The challenge with supervised methods is getting models to perform well on examples not adequately represented in the training dataset. Typically, as the frequency of a particular category decreases, so does average model performance on this category (see Figure 1). It is often difficult and costly to achieve strong performance on the rare edge cases that make up the long tail of a data distribution. In this blog post, we’ll take a deeper look at how sophisticated data curation tools can help machine learning teams target their experiments toward taming the long tail.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fa8816c4-0552-489a-3131-4577f439ab00/public" alt="Figure 1: Long Tail Distribution"> Figure 1: This type of distribution, in which there are a few common categories followed by many rare categories, is called a long tail distribution. In the majority of deep learning applications, datasets collected in the real world tend to have this long-tail shape. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/703dc6dd-ba3f-4caf-380b-d77dbffb8800/public" alt="Figure 2: Ubiquity of Long Tail Distributions"> Figure 2: Long tail distributions occur frequently in the real world. For example, the frequency of words in English writing follows a long-tail distribution. The ugly truth is that almost all AI problems worth solving are made difficult by the challenge of a long-tail. Imagine you’re an engineer at an autonomous vehicle company, working on an object detection model trained on image data captured from vehicles on the road. This real-world data provides a textbook long-tail distribution: There are many thousands of images of cars on the highway, but very few depicting bicycles at night (see Figure 3). In a preliminary experiment, the subsequent model performs badly when localizing cyclists at night (see Figure 4). Neglecting this rare scenario is likely to result in high-severity errors during testing, but it’s critical for the algorithm to detect cyclists properly in order to prevent collisions. What’s the best method to address this model failure in a targeted way? There are many approaches to improving performance, including experimenting with new architectures or additional hyperparameter tuning. But if the goal is to improve model performance on this specific edge case, the most targeted approach is to add more examples of cyclists to the training dataset. The problem is that unlabeled data is inherently difficult to search; sampling randomly, one is unlikely to find more cyclists at night, and it’s too expensive to simply label all collected data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e7bb4d6-f3cb-4206-5f23-a30613bac500/public" alt="Figure 3: Berkeley Deep Drive Class Distribution"> Figure 3: In the Berkeley Deep Drive dataset, the category “rider” occurs infrequently compared to the category “car.” It’s no surprise that when we view individual examples of riders, our model trained on Berkeley Deep Drive does not do a good job of localizing these objects. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/01a950ec-c9ea-4901-65d7-befbc8d51900/public" alt="Figure 4: Examples of poor localization for the class “rider”"> Figure 4: Object detection predictions from an EfficientDet D7 architecture trained on Berkeley Deep Drive. Examples of poor localization for the class “rider”, a long tail class in the training dataset. To improve performance on the long tail of edge cases, machine learning practitioners can get caught up in an endless cycle of collecting more training data, subsampling, and retraining the model. While this type of experimentation is essential to machine learning development, it is incredibly costly with respect to time, compute, and data labeling. The cost of improving performance on the long tail often threatens the economics of AI products. <a href="https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/" rel="noopener noreferrer" target="_blank">This blog post</a> from Andreesen Horowitz details how AI businesses seldom have the attractive economic properties of traditional software businesses. Where software products benefit from economies of scale, AI businesses experience the opposite; as model performance improves over time, the marginal cost of improvement increases exponentially. It may require ten times the initial training data to yield any significant model improvement. Given that expensive model retraining is unavoidable, machine learning teams should focus on making iterative experimentation as efficient as possible. One way to focus experiments on improving the long tail is to use model failures to identify gaps in the training dataset and then source additional data to fill those gaps. Think of this approach to machine learning experimentation as “mining the long tail.” With each experiment, identify a failure case, find more examples of this rare scenario, add new examples to the training dataset, and repeat. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b271a31-9926-4679-4e58-29e90178b500/public" alt="Figure 5: ML Development Lifecycle"> Figure 5: The ML development lifecycle consists of collecting, generating, annotating, managing or curating, training, and evaluating. The data curation or management step has the biggest impact on an experiment’s success, yet is most overlooked in ML discourse. It’s difficult to target experiments in the most under-represented scenarios without sophisticated tools for dataset curation. There are three operationally critical capabilities for most AI teams: <ol><li>Identify long-tail scenarios during model evaluation</li><li>Reliably and repeatedly source similar examples from unlabeled data</li><li>Add these examples to a labeled training dataset.</li></ol> Some of the most effective teams we work with at Scale face the challenge of the long tail head-on by building their tools, operations, and workflows proactively around an iterative process of data curation. Hussein Mehanna, VP &amp; Head of AI/ML at Cruise, highlighted this at Scale’s Transform conference earlier this year: “You have to have a robust, fast, continuous learning cycle, so when you find something that you haven't seen before, you add it quickly to your data, and you learn from it, and you put it back into the model.” This idea of targeted dataset curation is simple; in practice, however, it’s difficult to achieve because of the substantial engineering work required to index millions of raw images and surface difficult scenarios automatically during model evaluation. Let’s put ourselves back in the shoes of an ML engineer. Having diagnosed poor performance on cyclists at night, we now want to target the next experiment at improving this scenario. With a complete set of data curation tools, this workflow would be simple. From one difficult example, source similar images from the large set of unlabeled data and send it through the labeling pipeline for annotation. Once this new batch of data is annotated, include it in the training dataset for subsequent experiments. We call this data-centric approach to experimentation model-in-the-loop dataset curation. Not only is this a highly targeted approach for addressing long-tail performance but, unlike random sampling, it results in better balance across classes and avoids labeling redundant examples that yield little marginal improvement of model performance. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/0o9gb9d3ss"></iframe> Performance on the long tail is often a make-or-break situation when it comes to AI in production. Especially in high-impact applications like healthcare — where data is difficult to collect and equity is critical — being proactive on edge cases is of utmost importance. Given the ubiquity of long-tail challenges in real-world applications, it’s surprising that there isn’t more discussion around how to tailor machine learning experiments toward these cases. ML teams should not run the risk of leaving precious rare scenarios undiscovered in a sea of unlabeled data. When it comes to the long tail, targeted dataset curation is an important tool for achieving performant AI models robust to the complexities of the real world.

The success of real-world machine learning applications often depends on model performance on edge cases.

How to Tame the Long Tail in Machine Learning

Artificial Intelligence (AI) and machine learning (ML) are increasingly being applied to high-stakes domains. One emergent domain involves applying ML to healthcare to assist doctors in providing better care for patients. In the United States, however, people of color face disparities in access to healthcare, the quality of care received, and overall health outcomes. In dermatology, for example, darker skin tones are underrepresented in dermatology residency programs, textbooks, research, and diagnoses. We aren’t the first researchers to investigate this problem. Academics such as <a href="https://www.mdedge.com/dermatology/article/216697/pigmentation-disorders/racial-limitations-fitzpatrick-skin-type" rel="noopener noreferrer" target="_blank">Dr. Susan Taylor</a>, <a href="https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212" rel="noopener noreferrer" target="_blank">Joy Boulamwini</a>, and <a href="https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf" rel="noopener noreferrer" target="_blank">Timnit Gebru</a> have made great strides in identifying racial limitations around critical issues such as facial recognition and healthcare. In reviewing these various papers and studies, we recognized a need for a better understanding of the underlying data fueling this research. Specifically, we sought to understand how this underrepresentation impacts an ML model’s ability to accurately diagnose various skin conditions. In collaboration with our research partner, MIT Media Lab, we analyzed and evaluated deep neural networks trained on clinical images in dermatology.<h1>Methodology:</h1> The team at Scale annotated 16,577 clinical images sourced from two dermatology atlases — DermaAmin and Atlas Dermatologico — with Fitzpatrick skin type labels. The Fitzpatrick labeling system, while not perfect, is a six-point scale originally developed for classifying sun reactivity of skin phenotype. The Fitzpatrick scale served as the basis for skin color in emojis and, more recently, the Fitzpatrick scale has been used in computer vision applications to evaluate algorithmic fairness and model accuracy. The annotated images represent 114 skin conditions with at least 53 images and a maximum of 653 images per skin condition. The annotations, which we are calling the Fitzpatrick 17k Dataset, have been open-sourced <a href="https://github.com/mattgroh/fitzpatrick17k" rel="noopener noreferrer" target="_blank">here</a> if you would like to explore the dataset in greater detail.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8176b7a8-94e7-494b-746f-17563912e200/public" alt="The Fitzpatrick Skin Type Scale"> After the dataset had been labeled, researchers at MIT trained a transfer learning model based on a VGG-16 deep neural network architecture pre-trained on the seminal ImageNet dataset to classify various skin conditions. For more details on how the model was trained, we encourage you to read the full paper <a href="https://arxiv.org/abs/2104.09957" rel="noopener noreferrer" target="_blank">here</a>. <h2>Research Findings:</h2> The research found that the data used to train a model does matter. By tagging the images with Fitzpatrick labels, researchers found that the dataset contains 3.6 times more images of the two lightest Fitzpatrick skin types than the two darkest Fitzpatrick skin types. The underrepresentation of dark skin images in the dataset — and in dermatology atlases more broadly — led to larger disparities in the model’s ability to correctly diagnose skin conditions involving darker skin tones. While more research is required to identify where accuracy disparities are greatest across skin types, this research empirically shows that the data a model is trained on matters. More importantly, it matters how data is collected and what type of data is collected. Before developing and deploying large-scale ML models in the healthcare sector, we encourage researchers and practitioners to consider and examine biases in training datasets to ensure healthcare disparities are not unintentionally amplified by these models.

Improving clinical dermatology diagnostics with better data

Scale AI & MIT Media Lab: Improving Clinical Dermatology Diagnostics

Scale AI and Research Partner MIT Media Lab Analyze Bias in Dermatological Datasets to Improve Diagnoses

Have you ever missed the on-ramp for a freeway because you were in the wrong
lane? Or had to circle a block in search of a package drop box or specific
entrance? These moments may just be minor, albeit annoying, inconveniences for
us as individuals. But for autonomous vehicles, ride sharing, or logistics
companies that make thousands of daily trips and deliveries, the costs of such
navigation errors add up quickly.
&nbsp;
Companies that depend on high-quality maps for precise navigation have two
off-the-shelf solutions:
<ol>
<li>Regular map applications like those used by individual</li>
<li>consumers do not provide enough detail and users can&rsquo;t layer their own data</li>
<li>on top of these maps in a scalable way.</li>
<li>HD map companies offer more turnkey solutions but customers</li>
<li>still can&rsquo;t edit or update the data themselves because the maps are owned by</li>
<li>the mapping companies. Furthermore, it&rsquo;s a time-consuming process to get</li>
<li>maps updated and customers must wait for vendors to cover new regions.</li>
</ol>
&nbsp;
Lacking a viable off-the-shelf option, many companies opt to build their own
maps. Scaling the process to generate large volumes of extremely precise
annotated map data poses the same challenges as scaling data pipelines for any
other AI application.
<h2>Introducing: Scale Mapping</h2>
&nbsp;
Scale Mapping gives customers the most
flexible, scalable, and transparent mapping solution. With Scale Mapping,
customers can generate high-precision maps down to the centimeter level for
simulations and real-world testing, enhance prediction and motion planning by
effectively predicting another agent&rsquo;s intent, train perception models to
live-detect map features, and improve navigation and route planning to
maximize efficiency.
<h3>How It Works:</h3>
&nbsp;
&nbsp;
High-precision annotations of large-scale maps are extremely challenging. To
minimize cognitive load for human annotators (Taskers) and to maximize
labeling efficiency and accuracy, Scale Mapping first breaks map sections down
into tiles. From there Taskers start with vector annotations, or annotations
of physical map features such as lane boundaries and crosswalks. Once the
vector annotations are complete, semantic annotations &mdash; or the annotation of
concepts &mdash; such as the location of package drop boxes or a bike lane becoming
a turn lane is layered on. Lastly, relationships between map features and map
semantics are linked, such as a traffic light to a specific traffic lane.
After all three layers of annotations are complete, the individual map tiles
are stitched back together to create a comprehensive map.
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b4e2e6c5-8eb3-4305-9a65-afb211dbc100/public" alt="How Scale Mapping Works Diagram">
&nbsp;
<h2>Real-World Impact</h2>
&nbsp;
Because it can support data inputs from LiDAR sensors, cameras, and even
existing maps, this workflow allows customers to develop, update, and own
their own proprietary HD maps at speed. Scale Mapping is already trusted by
leaders in the autonomous driving industry.
&nbsp;
&nbsp;
Nuro, one of the leaders in the self-driving delivery space, and Domino&rsquo;s,
the largest pizza company in the world based on global retail sales,
launched autonomous pizza delivery in Houston. Nuro&rsquo;s R2 robot is the first
completely autonomous, occupantless on-road delivery vehicle with regulatory
approval from the U.S. Department of Transportation.
&nbsp;
&nbsp;
<iframe class="ql-video" src="https://nuro.sfo3.digitaloceanspaces.com/Nuro-Delivery-B-Roll-2021.mp4?mtime=20210411191557&amp;focal=none" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
&nbsp;
&nbsp;
Scale partnered with Nuro to improve perception and mapping models for the
R2, starting with providing high-quality, rapidly labeled data. With Scale
Mapping, Scale is moving up the autonomous driving stack by enhancing
localization and mapping capabilities. Put simply, the R2 needs to know
where it is, what&rsquo;s around it, where it&rsquo;s going, and how to get there. The
data Scale provides lets the R2 do exactly that while ensuring the safe,
accurate, and efficient delivery of pizzas.
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d156b7f7-dc7e-4f28-0442-9c4317206b00/public" alt="Nuro Quote">
&nbsp;
&nbsp;
In terms of the impact across our Nuro projects, we doubled the amount of
data annotated from 2020 to 2021.
&nbsp;
&nbsp;
Talk to our team today for a demo and more information about our
industry-leading quality with transparency, flexibility, and rapid
turnaround.
&nbsp;

The flexible solution to develop and scale your own custom maps.

Introducing: Scale Mapping

Own Your Own Maps. Introducing: Scale Mapping

Many industries that impact our day-to-day lives, including logistics,
manufacturing, financial services, insurance, and government, still run on
paper. These documents contain critical information that powers important
workflows like clearing shipments past customs, processing insurance claims,
underwriting loans, tracking machinery parts, issuing tax refunds, and parsing
clinical lab reports. Unfortunately, document processes still involve people
manually keying in information into digital systems. These labor-intensive
workflows significantly increase the time it takes to extract information,
leading to added costs, poor customer experience, and lack of scalability for
high volumes of documents.
&nbsp;
To tackle this challenge, companies previously used technologies like Optical
Character Recognition (OCR). However, OCR solutions only transcribe text
without extracting the most relevant fields. Most &ldquo;intelligent&rdquo; extraction
solutions rely on customers to manually define templates or hard-coded sets of
rules to handle data extraction. The rules dictate which part of the document
corresponds to a certain field, which means the system can only process the
document layouts that fit the original template. But, if you change vendors,
document types, or update the document in any way, then you need to manually
define another set of rules. Solutions like these take a lot of upkeep and
result in poor quality when dealing with complicated, variable document types.
&nbsp;
Imagine being able to feed any type of document into a system that can quickly
and accurately extract fields and link entities, even from new layouts or
formats &mdash; all without any setup or maintenance effort on your end.
&nbsp;
Tech-forward companies are recognizing the importance of deploying a robust
solution for automating entity extraction and linking because high quality,
structured data can:
<ul>
<li>Improve existing services by making them faster, cheaper, and more accurate</li>
<li>Enable new products based on the data unlocked.</li>
</ul>
&nbsp;
The challenge is building a system that can adapt to real-world variations,
especially across semi-structured and unstructured documents without
sacrificing accuracy. Here is how we do it.
<h2>Scale Document AI</h2>
&nbsp;
Scale Document AI relies on our
latest technology, Adaptive AI, to deploy refined machine learning models for
customers who demand high quality and low latency when it comes to document
processing. We leverage base models trained on millions of data points, and
further refine those models for each customer use case. This enables us to
extract and link entities from highly variable documents in seconds without
putting the burden of setup on the customer.
&nbsp;
What&rsquo;s unique about Adaptive AI is that we deliver a solution tailored to each
use case to extract the data our customers need at high quality &mdash; regardless
of any changes in document layouts. Unlike existing solutions, our machine
learning models thrive on challenging and varied documents by parsing the
structural layout of pages, contextualizing the meaning of words, and
understanding the relationships between different fields. We developed
Adaptive AI to actually understand the structure and the form fields&rsquo; meaning,
rather than simply learning where on a document to find a field (e.g.
understanding the vendor name instead of instructing that it is usually on top
left of document).
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7356c354-9a87-455d-d0e7-6618e6d73100/public" alt="Scale Document Workflow">
&nbsp;
With Scale Document AI, you get:
<ul>
<li>Fine-tuned document processing models trained to meet</li>
<li>your exact requirements. Achieve higher accuracy than off-the-shelf</li>
<li>solutions across a variety of document layouts.</li>
<li>&nbsp;</li>
<li>Guaranteed quality up to 99%+ and turnaround SLA times as fast as</li>
<li>seconds</li>
<li>with a fully automated or hybrid document processing workflow using our</li>
<li>API. Scale provides integrated models and optional human-in-the-loop</li>
<li>quality assurance workforce.</li>
<li>&nbsp;</li>
<li>Painless and quick setup so your teams can focus on what&rsquo;s</li>
<li>important.</li>
<li>No more configuring templates or fixing errors. Upload sample documents</li>
<li>and provide a list of fields that you need extracted. There is no further</li>
<li>setup or maintenance required to start processing documents.</li>
<li>&nbsp;</li>
<li>Dedicated machine learning and engineering support. Work</li>
<li>with a world class technical team of machine learning researchers and</li>
<li>engineers dedicated to your document processing needs. We can help you</li>
<li>adapt as you scale your business.</li>
<li>&nbsp;</li>
</ul>
&nbsp;
<h2>Real-World Impact</h2>
&nbsp;
Scale Document AI serves many industries across financial services,
insurance, real estate, logistics, manufacturing, energy, healthcare, and
government, across a variety of document types and use cases such as loan
origination, customer onboarding, invoice/receipt/bill processing, fraud
prevention, claims processing, title closing, shipment processing, and
clinical lab report extraction.
&nbsp;
&nbsp;
In logistics, we partnered with
<a href="https://www.flexport.com/" target="_blank" rel="noopener noreferrer">Flexport</a>, which is building the
platform for global trade, to significantly increase efficiency and reduce
costs in what used to be a manual effort for processing critical shipping
paperwork. We quickly deployed our Adaptive AI for important logistics
documents like Bill of Lading, Arrival Notices, and others to reduce data
extraction errors and improve compliance across Flexport&rsquo;s global trade
network. James Chen, Flexport CTO, explained the impact of our partnership
at
Scale Transform, our conference earlier this year:
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/a186b4ed-c72c-4575-e374-aeb715927c00/public" alt="James_Chen_Quote">
&nbsp;
&nbsp;
Another applicable area for Scale Document AI is financial services,
specifically bill pay services, loan origination, and claims processing.
<a href="https://www.brex.com/" target="_blank" rel="noopener noreferrer">Brex</a>, a $7.4B startup modernizing
banking for businesses, uses Scale Document to power its new premium bill
pay product. When customers submit invoices in the Brex app or forward an
email with an invoice to the designated Brex email, our Adaptive AI extracts
all relevant information from these invoices instantly and accurately to pay
the bill, at higher quality and faster than off the shelf solutions.
&nbsp;
&nbsp;
We are excited to reimagine document processing to enable faster and better
workflows for challenging document types where conventional approaches don&rsquo;t
hit the mark. If you want to achieve modern operational efficiencies through
AI and see tangible downstream improvements in business metrics, contact us
at documentai@scale.com or sign up on our
<a href="https://scale.com/sales" target="_blank" rel="noopener noreferrer">website</a>.
&nbsp;

Scale Document's Adaptive AI ensures fast, accurate document processing, removing costly errors through machine learning

OCR is not Enough. Try Scale Document AI.

OCR Is Not Enough. Try Scale Document AI.

<h3>How did you join Scale?</h3> I knew of Scale because one of my friends from college worked here. When I was choosing my first job, I was considering either joining a large tech company or a small startup. All my internships had been at large tech companies, so at the time I was nervous about whether I wanted to join a small startup. But I knew I wanted to do something that had a higher ceiling, where I could be really excited about the possibilities. <h3>What about the company’s mission is the most compelling to you?</h3> I strongly believe in our mission to democratize and accelerate the development of AI. What excites me is that we're solving problems that are new, that other companies haven't solved before. We're tackling issues that are going to be more and more important as AI becomes more and more prevalent. So, I think the most compelling part of the mission for me is the impact of AI and the role that we get to play in it. <h3> </h3><h3> You worked at big companies prior to joining Scale. What was the</h3><h3> transition like? What advice do you share with others contemplating a</h3><h3> similar move?</h3><h3> </h3> I liked that at Scale, I was given a lot of ownership from day one. At larger companies, sometimes it can take years before you're really trusted to make large decisions, and at Scale I was given the opportunity to grow really quickly, to give input, and to contribute to work that I felt was very impactful very quickly. In terms of advice, I would say don't feel like you're confined to just what your job title might be. There might often be many different things you need to do to achieve an outcome. Sometimes that's engineering, sometimes that's operations, sometimes that's product management. And I think one of the things that has made Scale successful is we have a lot of people who are willing to do things outside of their job description to achieve important outcomes, and so I think that is a valuable piece of advice. <h3>What is the most exciting project you’ve worked on?</h3> What I'm most proud of is my work as the product manager for the 2D team. While I was working on 2D, we saw large growth to the business, in terms of number of customers and revenue. At the same time, we did a lot to improve our margins and reduce our costs. I'm very proud of that team and how we were able to achieve that amount of growth. It was the first time in my career that I was able to work on something at that level. <h3>What are some of your favorite things about the culture?</h3> I really like that people are encouraged to disagree with each other and to have conversations about those disagreements. I think that helps make Scale a really great place to learn. I also really like the people at Scale - they’re a lot of fun to work with and have really great senses of humor. <h3>What is the most valuable thing you’ve learned here?</h3> Problem-solving skills and the ability to own a business outcome. And there's not one specific skill that's necessary for that, but I think gaining confidence in my ability to do that, and gaining experience in doing that has been very valuable. <a href="https://scale.ai/jobs" rel="noopener noreferrer" target="_blank">Explore Open Positions</a>

Scale Product Manager Sara Xiang shares insights on why she joined, projects she's worked on and life at Scale.

Life at Scale AI | Sara Xiang

Life at Scale

Scale AI was founded on the belief that better data → better AI. There has recently been a renewed focus on “good” data and its impact on the quality of models in the industry. Andrew Ng, who spoke at our first Transform conference earlier this year, emphasized this point: “We spend a lot of time sourcing and preparing high-quality data before we train the model, just like a chef would spend a lot of time to source and prepare high-quality ingredients before they cook a meal. I feel like a lot of the emphasis of the work in AI should shift to systematic data preparation.” While we have intuitively understood the importance of data quality, the Google Research Team paper <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank">"Everyone wants to do the model work, not the data work’: Data Cascades in</a><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank"> High-Stakes AI"</a> published earlier this year was the first we’ve come across that attempts to quantify and provide empirical evidence of “bad” data’s impacts.<h3>Overview of the Paper</h3> AI models increasingly are being applied in a broad range of domains from healthcare to climate change. Based on interviews conducted with 53 global AI practitioners working on a variety of applications, the paper’s researchers define, identify causes of, and present empirical evidence on the problem of data cascades. Data cascades are compounding events caused by data issues that lead to negative, downstream effects, resulting in the accumulation of tech debt over time. For more details on the paper’s findings, we encourage you to read the full research <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank">here</a>.<h3>Key Takeaways</h3> The paper notes that 92% of AI practitioners interviewed experienced at least one data cascade or adverse outcomes caused by problems in a data processing pipeline. The paper also cites several causes of cascades, but the number one cause — cited by 54.7% of the practitioners interviewed — was “interacting with physical world bitterness.” In other words, when AI systems move from their well-defined, static development environments and are deployed in the physical world, the systems are challenged by the dynamic changes inherent in the real world.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5e7113c0-9478-4cf4-54d3-839083523d00/public" alt="Data Cascades Causes"> Table 1: Prevalence and distribution of data cascades, triggers, and impacts. IN is short for India, EA &amp; WA for East African and West African countries respectively, and US is for the United States. Many ML researchers and engineers today are taught to develop models using pristine datasets in academic settings. In real-world applications, however, the data ML engineers must work with is messy or even non-existent. While the paper’s focus is on high-stakes domains with limited resources, even in high-resource settings, ML engineers or data scientists will hack together a data pipeline leveraging whatever tools and services they can find. The lack of unified infrastructure to support comprehensive data operations creates structural failures that ultimately cause data cascades and their commensurate tech debt. Exacerbating the issue, as noted by <a href="https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf" rel="noopener noreferrer" target="_blank">Sculley et al., 2015</a>, complex models erode abstraction boundaries for ML systems. Developing and maintaining traditional code already results in tech debt. But that is significantly compounded by ML-specific issues — including data dependencies, analysis debt, and configuration debt — that are much harder to detect, diagnose, and fix. The longer tech debt accumulates, the more pervasive it becomes. The more pervasive the tech debt, the more time and effort ML teams ultimately must spend to resolve issues, resulting in productivity loss and slower deployments. <h1> </h1><h1> Where Scale Comes In: Minimizing Data Cascades by Focusing on Data Work</h1><h1> </h1> Scale was founded on the belief that for AI and ML, data is the new code. By providing a data-centric, end-to-end solution to manage the entire ML lifecycle, we bridge that skill gap, so ML engineers don’t have to change how they work. By focusing on the data, we enable our customers to focus on the model work, not the data work, minimizing the risk of data cascades and making AI systems safer and more robust. <h2>Data Annotation</h2> We started this journey by focusing on data annotation. When we first began providing data annotation solutions to our autonomous vehicle customers, our quality assurance (QA) systems were highly manual. Human QA Agents or Queue Managers would audit and re-check sample tasks and manually assign scores to individual Taskers. Over the years, as we expanded into new use cases and new industries, we have built out a variety of internal tools to automate many of these QA processes including: <h4>Automated Benchmarks</h4> With benchmarks, we programmatically take an annotation task we know is correct, create an error in that task (usually a common mistake we’ve seen Taskers make), and serve them as regular tasks to our Taskers. By leveraging benchmark tasks, we can automatically give Taskers scores based on whether they fixed the error before submitting. This then allows us to understand how well each Tasker is performing without manually scoring each individual on a variety of tasks. <h4>Linters</h4> Linters are tools designed to catch errors and flag anomalies. At Scale, we leverage a variety of types of linters. For example, a response-based linter looks at a submitted task and checks for possible errors such as annotations that should not exist, like duplicate bounding boxes for the same object. Over time, we have layered response-based linters into the UI to flag issues for our Taskers as they annotate -- for example, warnings when the size of a bounding box appears too large. Moving beyond our core expertise in data annotation, our team has been hard at work building products to support the entire ML lifecycle. These include: <h2>Data Collection</h2> Collecting diverse and representative data, which is the first step in developing a machine learning model, can be a challenge, especially for content and language use cases where there is much more subjectivity. Leveraging our existing global Tasker base, our text and audio collection workflows are API supported and seamlessly integrate with our data labeling pipeline for seamless end-to-end dataset creation. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/uq2zsk4cy0"></iframe> <h2>Data Management</h2> While more training data is generally better than less data, once data is collected, ML teams need to select what data to annotate. Today, most ML teams do this either by randomly selecting data to label, or manually parsing through their data to identify valuable examples. Scale Nucleus uses active learning and advanced querying to help ML teams visualize their data, understand over- or under-represented labels and attributes to help mitigate bias, and identify edge cases to address the long-tail challenges of machine learning development. Nucleus also enables model-assisted QA, using model predictions (model errors in particular) to identify errors in labeled data and to improve data quality. In the video below, we show how active learning powers the Autotag feature of Nucleus to allow users to automatically search for and tag similar images — in this case flowers. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/yocc96733p"></iframe> <h2>Data Generation</h2> The ability to understand how ML models will react to rare or dangerous real-world scenarios is crucial for safety-critical computer vision applications. However, it’s not always feasible to collect enough examples of edge cases in the real world. Currently in alpha, Scale customers can augment ground-truth training data or generate scenarios from scratch to simulate a variety of cameras, sensors, or environmental conditions such as fog or night-time lighting. With data generation, ML teams can synthesize specific, troublesome scenarios (often pre-labeled) to expose their models to more edge cases and under-represented scenarios than they could otherwise collect. <h1>Real-World Impact</h1> As we noted in our <a href="https://scale.com/blog/series-e" rel="noopener noreferrer" target="_blank">Series E Announcement</a>, we’ve already seen the impact that the right data infrastructure can bring to our customers. From 10X’ing throughput for Toyota to improving States Title’s document processing model to more than 95%, we have freed up many of our customers’ ML teams to focus on the model work instead of the data work. We’re proud to be trusted by the world’s most ambitious AI teams to build the data foundation that will set up their organizations for success for decades to come. In the coming weeks, we’ll also dive deeper into some of the topics — like how to solve for the long-tail distributions of data with better data management — on our blog. In the meantime, if you or your team are looking for an end-to-end solution for your datasets, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">reach out to us</a>. We’d love to hear from you.

Better Data = Better AI. Understanding the downstream impact of "bad" data and how to mitigate these issues.

Scale AI: Doing the Data Work so Customers Can Focus on the Model Work

Doing the Data Work to Mitigate Data Cascades

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog