experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

Engineering

Scale AI is pleased to announce the launch of Scale Document - our endpoint for the secure processing of documents. Advancements in natural language processing (NLP) technologies have spurred initiatives across a wide range of industries to digitize document-based content. Led by the logistics, banking, financial services, and insurance industries, the intelligent document processing (IDP) market is expected to reach $1.1 billion by the end of 2020 according to <a href="https://www.everestgrp.com/2019-06-intelligent-document-processing-market-is-exploding-expected-to-grow-70-80-over-next-2-years-press-release-50323.html" rel="noopener noreferrer" target="_blank">research conducted by the Everest Group</a>. Scale Document builds on the previously released Scale Text product to better address customers with document processing and understanding use cases with tailored features. <img src="https://cdn.builder.io/o/assets%2Fe0438815ba51486bbb6a202747122d4b%2F90e89f53cb924b7fbf7860831a532383%2Fcompressed?apiKey=e0438815ba51486bbb6a202747122d4b&amp;token=90e89f53cb924b7fbf7860831a532383&amp;alt=media&amp;optimized=true">Multi-line transcription: one of many ML-powered tooling features to accelerate human review. Capable of supporting a range of inputs from PDFs, Microsoft Word Documents, JPEGs and more, Scale Document has a few key features: <ul><li>Machine learning to automatically classify and pre-label documents while also accelerating human review with active tooling. In addition to increasing the quality of documents delivered, this also enables the turnaround time for tasks to be as fast as a few hours to meet critical business requirements. (The workflow for how a document is processed is below.)</li></ul> <ul><li>Anonymization of personally identifiable information (PII) either with name and face blurring or placement of dummy content ensures compliance with data privacy principles. </li></ul> <ul><li>Flexible and varying levels of workforce security. Customers can select a global or U.S.-based workforce with options to add background checks or annotation in secure facilities depending on requirements. Our global work-from-home workforce also ensures continued delivery of data.</li></ul> <a href="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/000a7403-f16e-4886-03b2-7d9163505700/public" rel="noopener noreferrer" target="_blank" style="color: inherit;"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/98b87f7a-9829-4af5-6ff6-71d7bf000b00/public">The ML + human-in-the-loop workflow that processes documents at high quality and with fast turnaround.</a> Rigorously tested by a handful of customers during a private beta, Classification, Multilingual Transcription and Data Extraction annotation provide fine-grained understanding of document type, and allows for the ability to extract and understand document content and generate responses based off of document content, such as approving invoices or flagging applications for human review. Take a look at the <a href="https://scale.com/document" rel="noopener noreferrer" target="_blank">Scale Document product page</a> or <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">contact us</a> to talk about your business needs.

Scale Document is the secure endpoint for document processing.

Introducing Scale AI's latest product: Scale Document

Now Available: Scale Document

<h2>Notice any changes to our website?</h2> The Scale AI team is excited to introduce a new look and new structure for our product line-up. Scale AI was founded almost four years ago because we believed that high quality training data is the key bottleneck to the development of AI applications. Thanks to feedback from incredible customers, we have significantly expanded our data annotation products over the years to support an increasing range of data inputs and annotation types for computer vision and natural language (NLP) applications. As we expanded our data annotation capabilities, it became clear that we needed a better way to group and name our products to reflect both the current state and future vision of our product offering.<h2>What exactly are the changes?</h2> Sensor Fusion Cuboids and Sensor Fusion Segmentation is now one product called <a href="https://scale.com/3d-sensor-fusion" rel="noopener noreferrer" target="_blank">Scale 3D Sensor Fusion</a>.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8ef57c88-a258-41a3-a359-5c9fed59b400/public"> 3D Cuboids, Bounding Boxes, Image Categorization, Lines &amp; Splines, Polygons and Semantic Segmentation are also now one product called <a href="https://scale.com/image" rel="noopener noreferrer" target="_blank">Scale Image</a> with an addition of Point Annotation.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/132eb95b-d69d-47a9-f657-bb3ec16e2e00/public"> Video Annotation is now <a href="https://scale.com/video" rel="noopener noreferrer" target="_blank">Scale Video</a>, and has been expanded to include classification, cuboid and polygon annotation in addition to the existing bounding box annotation.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/28098202-ca0e-436e-606d-f39ff82eea00/public"> Our Natural Language products have been updated to <a href="https://scale.com/text" rel="noopener noreferrer" target="_blank">Scale Text</a> with Scale Audio currently in private beta.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/41c67cce-2a87-4057-2353-667527172500/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8b1ab848-ccda-4691-3f23-77daa83f0100/public"> You may also wonder, “how does this product restructure affect me?” If you are one of our amazing customers, from a practical standpoint, impact to you is minimal. While the product names are different, the products themselves are still the same, and we will continue to deliver data at the quality you expect. If you’re learning about Scale AI for the first time, we hope this rebrand allows you to better understand our full product suite and its capabilities.<h2>What’s Next?</h2> The Scale AI team remains committed to delivering products customers love and rely on. We are grateful for all of our incredible and innovative customers for their support. In the coming weeks and months, we’ll be announcing and rolling out new products and additional annotation types to existing products. In the meantime, if you have any questions about todays's announcement, reach out to us at support@scale.com. Otherwise, we invite you to explore our updated product line up.<img src="https://images.ctfassets.net/1qgw036lpkfx/4WVGAESruBGqfVBitOG0FX/3be113efdad428155b3cce7b86e373f4/06.jpg">

Explore the new product line-up: Scale 3D Sensor Fusion, Scale Image, Scale Video and Scale Text

Introducing a new look and new structure for our core product line-up

Introducing a New Look for Scale AI Products

Machine learning is a field of study with tremendous strides being made through active academic research. The machine learning team at Scale AI regularly hosts reading groups to discuss papers they find interesting to give us context in a rapidly evolving, research-heavy field. In this blog post, we go over the papers the team has read throughout the quarter and provide insights on how a paper influenced our own work here at Scale AI when relevant.<h3><a href="https://arxiv.org/abs/1803.03635" rel="noopener noreferrer" target="_blank">The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks</a></h3> Authors: Jonathan Frankle, Michael Carbin Presenter:<a href="https://www.linkedin.com/in/ppyht2/" rel="noopener noreferrer" target="_blank">Hongbo Tian</a> Pruned neural network architectures created by removing unnecessary weights from trained dense neural networks can attain various benefits without sacrificing accuracy. For example, smaller architectures have lower storage requirements, and certain hardware architectures can take advantage of sparsity, leading to massive computational performance gains at inference time. However, training the pruned network from scratch (as opposed to training the full network and pruning) has historically achieved poor accuracy. This paper presents the hypothesis that dense neural networks will contain some subnetwork that can reach the same performance of the original network when trained in isolation, and the authors cover some techniques to uncover the “winning subnetwork.” Larger network architectures have typically been considered superior, but this paper demonstrates that such a belief is not necessarily true. The lottery ticket conjecture matters when developing machine learning models here at Scale AI, since we know that we can build networks that tackle ambitious challenges without being blocked on unreasonable monetary or compute requirements.<h3><a href="https://arxiv.org/abs/1911.08265" rel="noopener noreferrer" target="_blank">Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model</a></h3> Authors: Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver Presenter:<a href="https://www.linkedin.com/in/ppyht2/" rel="noopener noreferrer" target="_blank">Hongbo Tian</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d923bed9-5bf7-471d-b24b-5171bb888a00/public"> MuZero’s performance, compared to the state of the art. After sufficient training, MuZero’s Elo (denoted by the blue lines) surpasses that of AlphaZero for boardgames and R2D2 in Atari. Games like chess and Go tend to have deterministic and well-defined rules. Such games aren’t representative of real world problems because most realistic problems occur in an environment with complex or unknown rules. This paper presents MuZero, a successor to AlphaZero that matches AlphaZero’s performance in Go but can learn to play Atari games, all without any prior knowledge to the game rules. MuZero builds off of AlphaZero’s search algorithms, but it also uses a learned environment model when training to capture aspects of the future that are relevant for planning. Although MuZero don’t know the game rule or environmental dynamics of the challenges it faces, it matches AlphaZero in logically complex games and beats the state-of-the-art model-free reinforcement learning algorithms in visually complex games (e.g. Atari games). Such a feat makes MuZero a pioneer in developing AI to solve realistic problems where there’s no perfect simulator and demonstrates the feasibility of applying learning methods to more ambiguous problems.<h3><a href="https://arxiv.org/abs/2001.07685" rel="noopener noreferrer" target="_blank">FixMatch: Simplifying Semi-Supervised Learning with Consistency and</a></h3><h3><a href="https://arxiv.org/abs/2001.07685" rel="noopener noreferrer" target="_blank"> Confidence</a></h3>Authors: Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel Presenter:<a href="https://www.linkedin.com/in/felix-lau-0386641a/" rel="noopener noreferrer" target="_blank">Felix Lau</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/dfa605ca-6f5c-4108-6219-ff05ea4ba300/public"> The general architecture of FixMatch. The model is trained to make predictions on a strongly-augmented version of an unlabeled image, based off the prediction results of a weakly-augmented version of the same image. The authors combine two common semi-supervised learning methods, consistency regularization and pseudo-labeling, to create a new algorithm that achieves state-of-the-art performance on standard, semi-supervised learning benchmarks. For unlabeled data, the algorithm applies both strong augmentation and weak augmentation. If the confidence for unlabeled data is higher than a certain threshold, the algorithms apply a cross-entropy loss between the prediction from weakly and strongly augmented images.<h3><a href="https://arxiv.org/abs/2001.04451" rel="noopener noreferrer" target="_blank">Reformer: The Efficient Transformer</a></h3> Authors: Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya Presenter: <a href="https://www.linkedin.com/in/ppyht2/" rel="noopener noreferrer" target="_blank">Hongbo Tian</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/1b7fdb75-94c4-4d06-f940-35f3660be800/public"> A visual depiction of attention with locality sensitive hashing. By hashing and then bucketing keys, attention doesn’t need to handle all pair-wise keys. Training good transformers can be prohibitively expensive: the largest version of GPT-2 contains 1.5 billion parameters, while NVIDIA’s Megatron-LM has up to 8.3 billion parameters. The authors of Reformer present a model that performs on par with preceding transformers while being more memory-efficient and faster to train. Due to the decreased resource requirements, Reformer makes Transformers more widely available to use in practice while also being able to handle longer sequences in general. Rather than using the standard dot-product attention, Reformer uses a less expressive attention mechanism combined with locality sensitive hashing, which reduces the complexity of attention. The classical self-attention mechanism attends to all pair-wise elements in the input sequence, giving it quadratic algorithmic complexity. By reducing the expressiveness of self-attention, Reformer can leverage locality-sensitive hashing via bucketing similar elements together, thus simplifying self-attention to linearithmic complexity. This mechanism ultimately enables Reformer to process the entirety of Crime and Punishment in a single training example.<h3><a href="https://arxiv.org/abs/1712.01208" rel="noopener noreferrer" target="_blank">The Case for Learned Index Structures</a></h3> Authors: Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis Presenter:<a href="https://www.linkedin.com/in/ppyht2/" rel="noopener noreferrer" target="_blank">Hongbo Tian</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/64d24dc4-d8a4-43ba-59f5-d437cf189200/public"> A comparison of a traditional and learned hash map. Learned data structures replace the hashing function with a model that achieves better performance. Many classic data structures (such as B-Trees, Hash maps, and Bloom filters) rely on some index structure to map a key to a position of a record. However, index structures can be generalized as a model that takes in a key and infers information about a data record. The authors explore replacing traditional index structures with deep-learning models, and their results demonstrate that machine learning can greatly increase speed of classic data structures that have already been optimized over decades. This paper is novel in the sense that it exposes opportunities for large performance improvements in data structures that have already been optimized for decades. It also serves as a reminder to our team at Scale AI, that excellence can exceed that of our initial expectations and that machine learning can have very non-obvious use cases.<h3><a href="https://arxiv.org/abs/1912.08193" rel="noopener noreferrer" target="_blank">PointRend: Image Segmentation as Rendering</a></h3> Authors: Alexander Kirillov, Yuxin Wu, Kaiming He, Ross Girshick Presenter:<a href="https://www.linkedin.com/in/felix-lau-0386641a/" rel="noopener noreferrer" target="_blank">Felix Lau</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3d238643-0e8b-4136-427f-65b72c477f00/public"> Inference results on a scene with and without PointRend. PointRend is able to handle the edges, such that it properly segments the side mirrors of a car. Image segmentation can suffer in accuracy around the borders of various classes and difficult regions. As a result, the authors of this paper create a neural network module call PointRend that adaptively runs point-based segmentation in higher resolutions at selected locations, leading to crisp boundaries compared to previous segmentation techniques. PointRend can be applied on top of existing instance and semantic segmentation tasks. At Scale AI, we’ve found that getting the edges between classes right is the most critical when it comes to producing high quality labels for 2D segmentation.<h3><a href="http://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/milioto2019iros.pdf" rel="noopener noreferrer" target="_blank">RangeNet++: Fast and Accurate LiDAR Semantic Segmentation</a></h3> Authors: Andres Milioto, Ignacio Vizzo, Jens Behley, Cyrill Stachniss Presenter:<a href="https://www.linkedin.com/in/danielhavir/" rel="noopener noreferrer" target="_blank">Daniel Havir</a> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/098bb5c7-5edc-42af-8003-dbc16ba10b00/public"> Diagram of the various states of data representation during LiDAR segmentation. Each letter corresponds to a step in the approach that RangeNet++ takes. High-level semantic perception tasks in autonomous vehicles are primarily solved through high resolution cameras, even though there are a wider variety of sensor modalities available for autonomous vehicles. At Scale AI, we are familiar with the problems defined in this paper. The paper details an approach to improving LiDAR-only semantic segmentation. The authors use a 4-step approach:<ol><li> </li><li> Reduce the LiDAR point cloud to two-dimensional range images that encode the</li><li> Euclidean distance along with the original coordinates and LiDAR remission.</li><li> </li><li> </li><li> Run the range image representation through a 2D semantic segmentation model.</li><li> </li><li>Project the model output back to the original point cloud format.</li><li> </li><li> Post-process the output using a novel KNN approach to account for occlusion</li><li> errors introduced with the 2D projection.</li><li> </li><li> </li></ol><h3><a href="https://arxiv.org/abs/2002.08871" rel="noopener noreferrer" target="_blank">Fast Differentiable Sorting and Ranking</a></h3> Authors: Mathieu Blondel, Olivier Teboul, Quentin Berthet, Josip Djolonga Presenter:<a href="https://www.linkedin.com/in/chiao-lun-cheng-87a4a34/" rel="noopener noreferrer" target="_blank">Chiao-Lun Cheng</a><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/75bc368c-8496-47be-c579-dde544e2c300/public"> Illustration of the soft sorting and ranking operators. Sorting and ranking are basic and common operations in computer science and machine learning. However, due to their natural properties, they are not differentiable. The authors propose the first differentiable sorting and ranking operations, while maintaining their time and space complexity. To reduce sorting and ranking to a differentiable form, the authors construct the operators as projections onto a permutohedron, take the convex hull of the ordering permutations, and convert the results to an isotonic optimization problem. If you’re interested in joining our growing engineering team, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for open positions.

The machine learning research papers the Scale AI team read and discussed in Q1 2020.

Research Papers the Scale AI Team has Read in Q1 2020

Scale AI Machine Learning Digest - Q1 2020

As the COVID-19 pandemic continues to spread, it affects all of our communities in different ways. The health and safety of Scale AI’s employees, as well as our global community of labelers, is a critical matter and a responsibility we do not take lightly. That’s why we’ve taken action over the past several weeks to protect both our full-time and contract employees, and ensure we can continue to offer the high-quality services our customers expect. We also want Scale AI to make a contribution to combating this disease if we can. Research to track and contain the virus is in its early days, but we know that better data could make a difference. To help these efforts, Scale AI will provide free data annotation services for any researcher working on diagnosing, containing, tracking or finding a vaccine or long-term cure for the virus. If you’re working on a project, you can reach out to us <a href="https://events.scale.com/accelerating-covid-19-research" rel="noopener noreferrer" target="_blank">here</a>. <h2>Scale AI's Employees, Labeling Community and Customers</h2> The safety of our team, both full-time and contractors, is a top priority. As a technology company, we are fortunate to be able to leverage existing communications tools to maintain collaboration. Scale AI’s ML-enabled platform also gives a global, work-from-home workforce the ability to deliver data annotation at the level of quality and volume our customers expect. Anyone working with a basic laptop and internet connection can utilize our platform. We are proud to provide a safe option for workers to continue to earn wages to support themselves and their families throughout this period of uncertainty. We have also reached out to our customers to let them know how we're working to deliver products and services. We do not expect any disruption to our work at the moment, however, because the situation remains fluid, we will continue to closely monitor our operations. In the event of a disruption, we will make sure to coordinate with your teams to minimize any impact. <h2>Accelerating COVID-19 Research</h2> Adjusting to the reality of a new disease has been difficult and unsettling, but it has been heartening to see a range of companies and organizations come together to help combat this pandemic. We want to make our own contribution to these efforts where we can. Given the scale of the pandemic, researchers should have every tool at their disposal as they try to track and counter this virus. Researchers have <a href="https://deepmind.com/research/open-source/computational-predictions-of-protein-structures-associated-with-COVID-19" rel="noopener noreferrer" target="_blank">already shown</a> how new machine learning techniques can help shed new light on this virus. But as with all new diseases, this work is much harder when there is <a href="https://www.wsj.com/articles/coronavirus-reveals-limits-of-ai-health-tools-11582981201" rel="noopener noreferrer" target="_blank">so little existing data to go on</a>. In those situations, the role of well-annotated data to train models or diagnostic tools is even more critical. Our mission at Scale AI is to accelerate the development of AI applications. To help in those efforts, Scale AI will provide free data annotation services for any researcher working on diagnosing, containing, tracking or finding a vaccine or long-term cure for the virus. Advancing and accelerating health research and science is critical to reducing the spread of viral infections. If you have data labeling needs related to this research, please let us know <a href="https://events.scale.com/accelerating-covid-19-research" rel="noopener noreferrer" target="_blank">here</a>.

Scale AI will offer free labeling to researchers working to contain the spread of the virus.

Scale AI is committed to containing the spread and being a resource to researchers working on COVID-19.

Supporting researchers and our community during COVID-19

Self-driving cars promise to improve mobility, build a more efficient transportation system and even save lives. But while the industry has been making great progress, to date the field has struggled to build cars and artificial intelligence models that can handle all weather conditions. Snow is hard to drive in—as many drivers are well aware. But wintry conditions are especially hard for self-driving cars because of the way snow affects the critical hardware and AI algorithms that power them. Take a car trained to drive on the wide, sunny boulevards of Arizona or California and place it on a snowy street like this, and things will go wrong:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8403db34-aa39-4091-ee14-85abee753600/public"> Snow covering the road markings would make it hard for the car to stay in its lane. Accumulated snow significantly reduces the color contrast, making it harder for the car’s system to recognize objects, like trees or parked cars covered in snow—or even pedestrians. Most problematically, snowfall makes it hard for a vehicle’s cameras and LiDAR sensors to see, which significantly reduces safety:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b8835be9-1a7e-4ece-2e1f-eba02a3ce900/public"> A skilled human driver can handle the same road in all weathers—but today’s AV models can’t generalise their experience in the same way. To do so, they need much more data. Until now, there has not yet been an open dataset of annotated LiDAR data and images of driving in snowy weather that researchers can use to develop more robust systems.<h2> </h2><h2> CADC: A new dataset for wintry conditions</h2> In partnership with the University of Waterloo and the University of Toronto, we’ve been applying our technology to this challenge. Today, we’re open sourcing the world’s first AV dataset in wintry conditions—the Canadian Adverse Driving Conditions dataset (CADC, or cad-see.) Drawn from 20 km of driving in harsh, snowy conditions in southwestern Ontario over the past two winters, CADC contains 7,000 frames of LiDAR and camera data spread over 70 driving sequences—featuring a wide range of perception challenges for AV models that can handle wintry conditions. Scale provided the annotations that makes the data usable by researchers. Researchers at the Universities of Waterloo and Toronto have also released a detailed academic paper outlining the technical features of the data collected and object detection labels provided, which you can view on arXiv <a href="https://arxiv.org/abs/2001.10117" rel="noopener noreferrer" target="_blank">here</a>. Ultimately, this is about safety. Without robust data, self-driving models can’t learn how to handle snowy conditions—preventing us from being able to make AV systems a reality for the many millions of people who live in places that receive wintry weather. As Professor Krzyzstof Czarnecki at the University of Waterloo says, “We want to engage the research community to generate new ideas and enable innovation. This is how you can solve really hard problems, the problems that are just too big for anyone to solve on their own.” We hope this dataset will make it easier for researchers to achieve the next-generation breakthroughs in AV systems that will take us one step closer to self-driving vehicles that work safely in a wide range of different environments. As Professor Steven Waslander at the University of Toronto says, “We’re hoping that both industry and academia go nuts with it. We want the world to be working on driving everywhere.“Bad weather is a condition that is going to happen.” You can find the dataset on the <a href="http://cadcd.uwaterloo.ca/" rel="noopener noreferrer" target="_blank">Waterloo website</a>, with dataset support tools available on <a href="https://github.com/mpitropov/cadcd_devkit" rel="noopener noreferrer" target="_blank">GitHub</a>.

Open source autonomous vehicle dataset for wintry conditions.

Overcoming a critical challenge to the widespread use of autonomous vehicles

Open Sourcing the World's First AV Dataset for Wintry Environments

OpenAI recently published <a href="https://openai.com/blog/fine-tuning-gpt-2/" rel="noopener noreferrer" target="_blank">a paper on fine-tuning GPT-2</a>, where they used Scale AI to collect the preferences of human labelers to improve their language models. Although we were already labeling millions of text and computer vision tasks per day at the time, the unique latency requirements and subjective nature of OpenAI’s tasks posed a new challenge for us. In particular: how do you scalably maintain the quality of labels, without having labelers check each other’s work? Today we’re sharing a deep dive into our approach to the problem, the automatic benchmark mining system we built to solve it, and the things we learned along the way. In doing so, we hope to illustrate some of the many challenges that make scalable data labeling such an interesting area of work.<h2>The Problem</h2>First, let’s take a look at one of these tasks. Given a snippet of text and some possible continuations for it, the researchers at OpenAI wanted humans to pick the continuation that best fit a given prompt, such as “which continuation has the most positive sentiment” or “which continuation is the most descriptive”.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fd227eaf7e823414c8446e5095ab2bbe2">But instead of labeling this data in a big batch like we usually do (offline), OpenAI wanted to put their model in a closed loop with our labelers, so the data was labeled online: the model would generate some text samples, have the humans rate it using our API, train based on human preferences, and then repeat this process over a few days.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F7eeb3638a0c745c3bc984d9f713e545b"><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fd64dabc8b64848299a8faecdcd826910">This meant that in order for an experiment to have enough epochs of training, and still finish quickly, the tasks needed to be done with very low latency (&lt;30 mins) at high throughput (~5,000 labels/hr). As we will explain later, these constraints were unique and required us to build additional functionality into our data labeling pipeline.Furthermore, the answers to these tasks were also extremely subjective. As an example, here is a real task where one has to pick which continuation is the most descriptive:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F5becb9abfe314f64a61d36cc38c86699">Can you pick an answer with certainty?This ambiguity makes it hard to tell the people that are making a genuine effort apart from the people (or bots!) that are guessing randomly. However, we must make that determination correctly in order to scale this up to thousands of labelers, and still deliver high quality data to our customer.<h2>Measuring Quality at Scale</h2>When we first built our labeling pipeline, our customers wanted us to prioritize the highest quality data over other factors such as the fastest turnaround SLAs. Now we had to figure out a way to keep our high quality bar while boosting our turnaround from days to minutes.One reason for higher turn-around time was redundancy. On most projects, we send each task through a pipeline of multiple people, either in series or in parallel. Having this redundancy greatly reduces the probability of errors. It also allows us to estimate the accuracies of each person by grading their responses against the completed version of the task.The other reason for higher turn-around time was lagging quality indicators. We usually hold each batch of tasks in a staging state before it is sent back to a customer, so that our confidence indicators on labelers can catch up based on new information. We can then relabel tasks we have low confidence in.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F55389ccfedf84869aa27c1a33210a20f">However, due to OpenAI’s latency and throughput constraints, we would not have been able to use either of these features of our architecture — tasks would have to be immediately sent back to the customer after a single person had attempted them.Fortunately, having people review other people’s work is just one of the ways we get signal on labeler quality. For example, on some computer vision projects, one feature that determines labeler quality is based on localization loss from a deep learning model.<h2>Benchmark Tasks</h2>Another way we measure quality is using a mechanism called benchmark tasks. The idea is to collect high-confidence responses to a subset of tasks first (called benchmark tasks — or a golden dataset in <a href="https://www.aclweb.org/anthology/D08-1027.pdf" rel="noopener noreferrer" target="_blank">literature</a>), and then use them to estimate the quality of active labelers. The benchmark tasks are sprinkled among new tasks served to people, and disguised to be indistinguishable from them. We can then guarantee that we never return tasks to a customer from labelers who do not meet the quality threshold on benchmark tasks.We had been successfully using benchmark tasks as another way of measuring quality, but these were typically created manually by our in-house quality team. This manual creation would not have worked for this project for a few reasons:<ul><li>we wanted to support OpenAI in creating new projects and iterating quickly on experiments without being blocked on the Scale team creating benchmarks.</li><li>due to the ambiguous nature of tasks, it was harder to manually find tasks for which we could have high certainty in the answer.</li></ul>To solve these problems, we augmented our pipeline with a system that automatically mines and maintains a set of benchmark tasks.Once a project came in, our system would take a small subset of tasks from it, and have each one done by multiple trusted labelers. If those people almost all reached consensus that one of the responses on a task was correct, we would make this task a benchmark task, with that response. The broader remote workforce would only be allowed on the project once we had enough benchmarks.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F1a8013001a90431ca2b5186b48729688">For added safety, only people we had the most confidence in were selected for benchmark generation. This is in line with a general principle we use in designing our operational systems:* leverage people we have highest trust in*. Here, if the most accurate labelers are working on creating benchmarks, their work can then be used to validate the work of a much larger set of people.Since we only pick tasks that reach consensus, unambiguous tasks are more likely to become benchmarks than ambiguous ones with no clear answer. To an extent, this is helpful because we can be confident our evaluation of quality has low noise. The caveat is that this set of benchmark tasks is no longer representative of our original set of tasks.In this particular case, this was fine, since we were hoping to eliminate malicious and under-qualified labelers, while still allowing for expression of human preference and subjectivity. The need to understand this relationship between test question difficulty and false positives/negatives is one that recurs frequently for us, and has also been studied extensively in fields such as Item Response Theory.<h2>Making This Work</h2>With a complex system that includes free-willed actors (human labelers) with unpredictable behavior, every new feature needs multiple iterations — cycles of deploying changes, observing the system’s reaction to them, and then responding to new developments — in order to make it actually work. The benchmark generation system was no exception. Here we go over two of the many improvements that we made.<h3>Dynamic Serving</h3>As with any job, even the highest quality labelers can be inconsistent on their quality from one day to the next. Initially, we distributed benchmark tasks at a fixed uniform sample rate, which meant that it took a long time before our system could confidently adjust its quality read for a given labeler. We evolved our logic for serving benchmark tasks to instead vary its rate dynamically, and serve more benchmarks as soon as a dip in quality was suspected.<h3>Preventing Heuristics And Bias</h3>Things ran smoothly until we got to the descriptiveness project, in which labelers had to pick the most descriptive continuation among 4 options. Our quality audit showed a dip, confirmed by the fact that the average time people were spending on these tasks had also started to fall.What was special about this project?It turns out that the correct answer (the most descriptive option) had an extremely high likelihood of also being the longest option. This was true often enough that a strategy of “pick the longest option” would be within our tolerance of quality, and prove far more lucrative than acting honestly.This demonstrates the need to care about the distribution of benchmark tasks. By explicitly including and weighting anti-bias examples (in this case, examples where the longest option is not correct) in our benchmark set, we could correct for this issue.<h2>A Note on Testing and Monitoring</h2>Every distributed system needs to be tested and robustly monitored, and a human-based distributed system like ours is no different.How do we test changes if our system depends on unpredictable human behavior? At Scale, we usually test things like that by launching “synthetic projects” — a sandboxed copy of a real project we’ve already done before, with a similar population of labelers as the real project. This allows us to experiment and iterate quickly without fear of disrupting the actual labeling pipeline for a customer. We leveraged synthetic projects to tune our parameters, test hypotheses and also at one point to keep our workforce “warmed up” by giving them tasks to do as we waited for the customer to send in more tasks.In addition to this, we need to monitor the system and output quality to ensure that there are no regressions. This is how we quickly caught and responded to some of the issues described above. We set up <a href="https://scale.com/blog/alert-from-reports" rel="noopener noreferrer" target="_blank">alerts</a> for when inter-labeler agreement rate, throughput or average time per task fell outside the expected range of values.In addition to automated alerts, we need constant qualitative monitoring on all of our projects, which is done by our in-house quality team. This project was no different. To check this system, we set up a process to audit labelers close to our filtering boundary based on benchmarks. This allowed them to alert us if the system was erroneously filtering out labelers who were good (false positives) or not filtering out labelers below quality (false negatives).Finally, since we had access to a small sample of data labeled by the researchers at OpenAI themselves, we set up a daily cron on our async job service that would have the same sample re-labeled by workers, and alert if the quality on this was lower than desired.<h2>What's Next</h2>Since building the prototype of this system and learning from it, we have incorporated it into all natural language and categorization use cases as well as parts of our computer vision pipeline. We have also improved the accuracy of our classifier that determines whether labelers are good quality using a Bayesian model. We will publish blog posts describing those efforts soon.This post emphasizes how we're solving quality measurement, but there's a whole world of other problems we're tackling that are also vital to the data labeling process, such as: using ML to make data labeling orders of magnitude faster, scalably teaching complex instructions, building state of the art 3D point cloud rendering tools, and setting up the data infrastructure to support all these systems. If you’re interested in joining us in solving these problems, and powering the next generation of ML applications, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for the latest open positions. If you have projects that require high quality data labeling, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">let our team know</a>!Find out more about OpenAI <a href="https://openai.com/about/" rel="noopener noreferrer" target="_blank">here</a>.

assets-e0438815ba51486bbb6a202747122d4b-4ba57966ec8e4f30a76576f000b8f572.jpeg

Labeling 1M Data Points/Week for OpenAI's GPT-2

How to Label 1M Data Points/Week

Building on the news of our <a href="https://scale.com/blog/series-c" rel="noopener noreferrer" target="_blank">Series C</a> round of funding valuing Scale AI over $1B, and our selection as one of <a href="https://scale.com/blog/linkedin-top-startup-2019" rel="noopener noreferrer" target="_blank">LinkedIn’s Top Startups in the US</a>, we are pleased to be selected as one of <a href="https://www.forbes.com/sites/jilliandonfro/2019/09/17/ai-50-americas-most-promising-artificial-intelligence-companies/#53699d8a565c" rel="noopener noreferrer" target="_blank">Forbes AI 50: America’s Most Promising Artificial Intelligence Companies</a>. Three years ago, we started Scale to tackle the most critical bottleneck in modern AI development: generating robust training data. It was a problem born out of our own frustrations building AI models—the majority of our time was spent on producing high quality data rather than building the model itself. This first-hand experience inspired us to solve the data problem for everybody. All companies recognized on the Forbes list leverage artificial intelligence as a core part of their business model and future success. A panel of experts selected the list based on the technology, business model, customers and financials. We are big believers in the transformational impact of AI over the coming decades, and we are proud of our work in enabling our customers to build incredible AI products more quickly. There has never been a better time to join Scale. If you’re interested in working with us, take a look at all of our <a href="https://scale.com/careers?location=All&amp;team=All#jobs" rel="noopener noreferrer" target="_blank">open positions</a>!

Scale AI selected as one of Forbes' Most Promising Artificial Intelligence Companies

Scale AI Selected as One of America's Most Promising AI Companies

Scale AI is pleased to be selected as one of <a href="http://lnkd.in/USTopStartups2019" rel="noopener noreferrer" target="_blank">LinkedIn’s 2019 Top Startups</a> in the U.S. Our mission is to accelerate the development of AI applications. Labeled data is the key bottleneck to the growth of the machine learning industry, and by building the infrastructure for machine learning, we are enabling a new age of software. There are three broad reasons why someone should join Scale AI: <ul><li> </li><li>Our Work:</li><li> </li><li> </li><li>Machine learning (ML) is likely the most significant technological shift happening today. Powered by a dynamic research community, computers can now recognize images and audio, translate languages, generate realistic text, and even beat humans at games. But aside from a few innovative products from large tech companies, ML has not yet made a tremendous impact. </li><li> </li><li> </li><li>Labeled data is the key bottleneck to the growth of the machine learning industry. Scale AI not only provides high quality labeled data for a wide range of AI applications, but our team is working to build the infrastructure needed to realize ML’s potential impact. </li><li> </li><li>Timing:</li><li> </li><li> </li><li>In the last year, Scale AI has grown from 20 employees to just north of 100. We have experienced phenomenal revenue growth and have also tripled the number of enterprise customers year on year. Customers also quickly realize the quality, scalability and value of leveraging Scale AI, and increase their usage on the platform rapidly. Scale AI is positioned to continue growing at an incredibly fast pace. </li><li> </li><li> </li><li>Now is a great time to join Scale AI, because as we continue to expand into new use cases and new verticals, employees (we like to call ourselves Scaliens) have the ability to make an incredible impact with their work. By joining now, you have the ability to move and execute on projects quickly, have the steep upside of joining early, but with significantly less risk. </li><li> </li><li>Culture:</li><li> </li><li> </li><li>Scale AI succeeds or fails as a team, and we make an effort to constantly trust and invest in each other to build a robust, scalable company. Scaliens include people who helped scale Cruise, Dropbox, Facebook, Lyft and Uber as well as new graduates just starting their careers. We believe that having diverse perspectives is crucial to build products to accelerate the development of AI, and we strive to create an environment where employees can give their all at work, while still being able to give their all at home. </li><li> </li></ul>There has never been a better time to join Scale. If you’re interested in working with us, take a look at all of our <a href="https://scale.com/careers?location=All&amp;team=All#jobs" rel="noopener noreferrer" target="_blank">open positions</a>!

Scale AI selected as one of LinkedIn’s 2019 Top Startups in the U.S..

Scale AI Selected as One of LinkedIn’s 2019 Top Startups in the U.S.

We at Scale AI felt inspired to use our computer vision training tools to put the two self driving methods to the test. 

Is Elon Musk wrong about LiDAR? We put the two self driving methods to the test.

Is Elon Wrong About LiDAR?

Our mission at Scale is to accelerate the development of AI applications. We’re proud of what we’ve built over the last three years, and today we’re announcing our Series C funding round to support our continued work against that mission. Scale has raised $100m at a valuation of over $1B. <a href="https://foundersfund.com/" rel="noopener noreferrer" target="_blank">Founders Fund</a> led the round, supported by <a href="https://accel.com/" rel="noopener noreferrer" target="_blank">Accel</a>, <a href="https://www.coatue.com/" rel="noopener noreferrer" target="_blank">Coatue Management</a>, <a href="https://www.indexventures.com/" rel="noopener noreferrer" target="_blank">Index Ventures</a>, <a href="http://www.sparkcapital.com/" rel="noopener noreferrer" target="_blank">Spark Capital</a>, <a href="https://www.thrivecap.com/" rel="noopener noreferrer" target="_blank">Thrive Capital</a>, <a href="https://en.wikipedia.org/wiki/Kevin_Systrom" rel="noopener noreferrer" target="_blank">Kevin Systrom</a>, <a href="https://en.wikipedia.org/wiki/Mike_Krieger" rel="noopener noreferrer" target="_blank">Mike Krieger</a> and <a href="https://en.wikipedia.org/wiki/Adam_D%27Angelo" rel="noopener noreferrer" target="_blank">Adam d’Angelo</a>. We are big believers in the transformational impact of AI over the coming decades. Machine learning is likely the most important technological shift of our time, and the overall benefits to the world will be comparable to those of the internet. What we noticed after working on AI at some of the most advanced organizations in the world was that building machine learning systems was challenging due to a lack of mature infrastructure. In particular, we noticed that the critical bottleneck to further progress today was data—in particular, labeled datasets. Many engineers will tell you that getting labeled data is the hardest part of building a machine learning model. Building AI represents a fundamentally different paradigm than building traditional software. The performance of an AI system depends more on the data than the algorithm—in most cases, without labeled data, there is no model. Similarly, for a model to improve and adapt, it requires more data rather than simply more code. This paradigm has been true since the very beginning of deep learning; the modern deep learning age was ignited by the launch of the ImageNet dataset by Fei-Fei Li’s lab at Stanford in 2009.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F8c80d06e32aa42c689836dbda1f7fd09" alt="AI DEV CYCLE"><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb0518c76600f4b9fa58189d4bfe750ba" alt="TRADITIONAL DEV CICLE"> We set out to solve this problem by building a platform that enables organizations with large high-quality datasets needed to build AI systems. Especially now, as AI is being applied in more and more ways, this matters because:<ul><li> </li><li> Safe, accurate and unbiased AI systems depend on large volumes of high</li><li> quality training data.</li><li> </li><li> </li><li> The process to acquire, label, and verify training data is slow, manual and</li><li> expensive.</li><li> </li><li> </li><li> Today’s training data bottleneck limits AI's impact to a small group of</li><li> well-funded technology companies.</li><li> </li><li> </li></ul> Scale is a solution to these problems. Developers use our API to send their raw data to our platform. We then use a combination of machine learning and human insight to label and annotate this data. We pass this high-quality ground truth data back to our customers, who are then enabled to build world-class models to solve real-world problems. Scale is now used by world leading machine learning teams at companies like OpenAI, Airbnb, Nuro, and Lyft. Our platform is being used to provide high-quality training data to an ever growing number of machine learning teams and industries, including:<ul><li>Autonomous Vehicles: Advancements in AI for perception,</li><li> prediction, and planning have enabled the development of autonomy, which in</li><li> turn relies on data.</li><li> </li><li>Natural Language and Content Understanding: More than ever</li><li> before, systems can understand and drive decision-making on large corpuses</li><li> of speech, text, images, and other forms of content.</li><li> </li><li>E-commerce and Search: Powerful search systems that connect</li><li> consumers to specific content, images, or products increasingly rely on AI</li><li> systems to drive understanding of this content. Our platform helps engineers</li><li> train those systems.</li><li> </li><li>Mapping: Mapping using satellite, drone, and street-level</li><li> imagery for use cases from agriculture to insurance rely on high quality</li><li> training data to ensure accuracy.</li><li> </li><li>Robotics: Robots in factory or warehouse settings need to</li><li> be able to understand their environments, safely interact with humans and</li><li> recognize products or components.</li><li> </li><li>Augmented Reality and Virtual Reality: AR and VR systems</li><li> used in gaming, real estate or manufacturing rely on being able to perceive</li><li> the 3D environment around a camera, headset or user.</li><li> </li><li>Offline retail: Cashierless checkouts, inventory management</li><li> systems and footfall analysis all depend on perception systems, which in</li><li> turn depend on training data.</li><li> </li><li> </li></ul> We’re proud of the progress we’ve made building the Scale platform for our customers. It’s been an incredibly quick 3 years since I started the company, and the ability we’ve had to accelerate progress in machine learning is palpable. But, as we hope is always the case, we’re far more excited about what we will do in the future. There is a lack of mature infrastructure for building AI, forcing teams to build their own technology stack from scratch before they can have a meaningful impact. Per our mission, we know there are many problems beyond the training data bottleneck for us to solve to continue accelerating AI development. Despite our excitement for the potential of AI, we’re closer to the beginning of its history than we are to the end. We’re grateful to our customers for building world-leading AI products with Scale, and we’re looking forward to accelerating the development of machine learning in an even broader set of industries and technologies as we grow. If you are interested in being part of creating the infrastructure machine learning needs, take a look at our <a href="https://scale.com/careers?location=All&amp;team=All" rel="noopener noreferrer" target="_blank">open positions</a>. If you're interested in partnering with us, please <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">reach out</a>.

Scale AI’s Series C: Building the data platform for ML

Scale AI’s Series C: Building the data platform to accelerate machine learning

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

basecamp-agenda

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

research

leaderboards

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog