experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

<h2>Update:</h2>
 We have received an overwhelming response to our announcement of free, AI-ready datasets in support of Ukraine. Nearly a hundred AI companies, researchers, developers, and GIS practitioners have offered to support the initiative with Scale &ndash; an inspiring global community of AI Developers for good.
Given the high demand, we will provide the datasets in phases starting with requestors with the greatest need, followed by a broader community. At this time, our focus is on providing Structure Damage Assessments in Kyiv, Kharkiv, and Dnipro.
The path forward is now two-pronged &ndash; our top priority is the delivery of damage assessments to those with urgent need.
The second prong of our effort is to provide AI-ready training datasets for Damage Assessment to the broader AI community, starting with a structure recognition training dataset.
Today, we are sharing an initial dataset with the broader AI community: <a href="https://drive.google.com/uc?id=17lpMFngiqeegIm1U0RzUYfB-9GjmI7fk&amp;export=download" target="_blank" rel="noopener noreferrer">link to download dataset</a>.
Note: the corresponding ZIP file's SHA256 Checksum is: 379c394dab6a4d7f1358625d086ebea4b39aaa124548e8c079bee16fb574bc53.
We appreciate your patience as we assess and prioritize requests for data and data development. We look forward to continuing to support Ukraine alongside other AI Developers leveraging critically important open-source and commercially available data for this important work.
Sincerely,
Scale AI Team
&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;
At Scale, we believe in using AI technology in support of democratic values. That&rsquo;s why starting today, Scale will be providing a series of AI-Ready datasets that algorithm developers can use to rapidly train and deploy AI in support of Ukrainian operations. By providing these datasets at no cost to national security practitioners, we hope to support a diplomatic solution and swift end to this conflict.
Diplomacy is fueled by insights; foundational to that is quality, trusted data. We have recently created strategic partnerships with commercial satellite imagery providers in order to develop these datasets, which include Aircraft Detections, Structure Damage Assessments, Change Detection, and the JSON and GeoJSON annotations. We encourage AI Developers to contact detections@scale.com to receive this data free of charge.
To better understand how these datasets can be used, Scale presents a contemporary use case of our partnerships with national security leaders. We deliver an AI-powered change detection solution capable of identifying and tracking military formations across the globe.
The video below demonstrates continuous monitoring and improvements by Scale&rsquo;s ML engineers and deployment strategists. We provide timely and accurate insights through our AI Data Readiness platform, our Test and Evaluation platform and our continuous ML Ops tooling.
Leveraging Scale's AI/ML ops platform, we continuously develop and hone deep learning models to describe scenes and detect objects across a range of inputs from commercial satellite imagery providers.
<iframe class="ql-video" style="width: 564px; height: 282px;" src="https://scaleai.wistia.com/embed/medias/hphfibwbn6" width="100%" height="400" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
We are now delivering insights to U.S. national security customers from this full-stack, multi-intelligence, human-teaming product, including on Russian formations across Europe and Eurasia.
Here, on Russian air and ground order of battle (OOB) data, we successively identified OOB changes across the EUCOM area of responsibility (AOR) over the past several months. We used high-resolution synthetic aperture RADAR(SAR) imagery from Capella Space and high-resolution optical imagery to deliver high-accuracy, time-sensitive insights.
&ldquo;In these intricate and time-sensitive scenarios, it is critical that we deliver insights from multiple, disparate datasets at the speed of relevance,&rdquo; said Mark Valentine, Head of Federal at Scale. &ldquo;We are proud to demonstrate the power of our platform, and our approach to human-machine teaming to support national security leaders and practitioners.&rdquo;
Monitoring the Russian OOB on the border of Ukraine using commercial satellite imagery, and other non-traditional Intelligence, Surveillance and Reconnaissance (ISR) sources requires adaptation to a variety of challenges in a constantly-evolving field.
The challenges in using AI to analyze OOB changes include: exigent scenarios presented by weather, disparate image varieties and quality, and the inclusion of relatively rare military objects such as transporter erector launchers (TELs), for which there are limited real-world observations.
The depth of ontology can also present complications: the presence of multi-generational aircraft, ground vehicles, RADAR and TELs mean that models need to be trained against tens of thousands of observations in a range of environmental conditions.
If you are an AI Developer who would like to receive this data free of charge, please contact <a href="mailto:detections@scale.com">detections@scale.com</a>.

ae82457827ce45793eefc6460623a063.jpeg

Scale AI provides free datasets for national security partners

Scale AI providing free datasets to help national security partners gain current insights into Russia-Ukraine conflict

Scale Rapid is a quick and easy way to get clean, high-quality labeled data for your next ML project. We used it to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model</a> in a previous post - and we’re back now to show you how to use Scale Rapid to train an unsupervised model! In this blog, we are going to use <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener noreferrer" target="_blank">CycleGAN</a> (<a href="https://arxiv.org/pdf/1703.10593.pdf" rel="noopener noreferrer" target="_blank">paper</a>) to anime-fy images with Scale Rapid. CycleGAN is trained in an unsupervised manner, meaning each X input does not need to correspond 1:1 to a y output in order to train the net. Instead of using Scale Rapid to label training data, we used it to clean data. We separated a group of images scraped from the internet into categories. This classification allows us to make the two inputs necessary for CycleGAN - a class A (real-life images) and class B (Japanese anime-style images). To start, we downloaded 10k images from Google images from a variety of different searches. At a glance, the corpus had all sorts of images. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/27b06dda-1661-4805-c721-0ec70ee20300/public" alt="Sample images from Google">Class A contains real-life images and Class B contains anime-style images. Image 3 belongs to Class A, and Image 2 to Class B. Images 1 and 4 do not belong in the dataset as they are neither real-life nor anime-style images. The ask for Scale Rapid is to discriminate between these three types of images: a real-world, realistically edited photograph, a Japanese-style anime scene, or neither. Rapid responses are then used as ground truth to establish which images should and shouldn’t be in the dataset we train CycleGAN on. Rapid’s categorization endpoint supports image categorization - and it's only 8 cents per image! Because CycleGAN is trained on unsupervised data, we don’t need to pair class A and class B images - we can simply separate them from one another and train. <h2>Using Scale Rapid</h2> Scale Rapid works in two phases- first, the user sets up the project &amp; instructions, and runs small calibration batches to ensure the initial results are as expected. Once quality on those looks good, larger production batches may be uploaded for labeling. For this application, we set up a categorization project. The taxonomy is just one multiple-choice question: What type of image is this? <ul><li> </li><li>Anime (japanese animation) image</li><li> </li><li> </li><li>Photorealistic (real-life) image</li><li> </li><li> </li><li>Neither</li><li> </li></ul> Scale Rapid also has a feature called long hints, which allows us to prompt labelers with specific things to think about upon choice selection via a tooltip. For the first choice (Anime (japanese animation) image) I clarified with the following long hint: "Japanese animation refers specifically to anime, as in the instructions. Other cartoon-style or generated images belong in the neither category." We begin by uploading a calibration batch, and follow up with production batches. The labeling results are easily parseable via JSON. <h2>Scale Rapid Results:</h2> A random audit of tasks found just one mistake out of 200 randomly selected labeled images. A few examples of correctly labeled images are below, organized by class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d6e522b3-b00e-4e91-47f5-3f709d144300/public" alt="Examples of correctly labeled images by class">With the help of Scale Rapid, raw data from the web was transformed into cleaned and confirmed data and separated into class A, class B, and neither. <h2>Model</h2> CycleGAN produces mappings from class A to class B and vice versa using two different types of losses- adversarial loss and cycle consistency loss. Adversarial loss incentivizes the mapping to generate images that look similar to the target set. We achieve this by optimizing against an adversary D–a discriminator that tries to separate the generated images for the target class from actual data from the target class. Cycle consistency loss incentivizes the mapping to produce images that look like they fit the distribution of the target class.This occurs by minimizing the difference between original data and that same data transformed into the same target class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/cdcd9177-a511-4863-70fa-6a9c1194dd00/public" alt="Cycle consistency loss">Cycle consistency loss is calculated by the difference between the untransformed class A image and the double-transformed class A image. The mapping aims to minimize this.We trained CycleGAN with our Scale Rapid sorted dataset.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3c9b78d0-1aaa-4d49-ad1d-e7d91e37ca00/public" alt="Loss Function Graph"><h2>Model Results:</h2> We are now able to anime-fy images! A few examples:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5a438327-e4b7-41e5-a19b-2cd457c94d00/public" alt="CycleGAN Images"><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Try Scale Rapid</a> for your own project today!

In this blog, we explore combining Scale Rapid Labeling with CycleGAN to anime-fy images.

Using Scale Rapid to train an unsupervised model.

Using CycleGAN to Train Unsupervised ML Models

Using Rapid and Nucleus together will get you better results than using Rapid alone.

Use Scale Nucleus with Rapid to Uncover Errors Smartly

Peer-Reviewed Doesn’t Mean Perfect Data

Mission critical industries including logistics, financial services, government, and healthcare rely on high quality document data extraction in order to optimize business workflows, save money, and improve customer experience. In this article, we explain the conventional approaches for document processing, deep dive into drawbacks with these solutions, and introduce a new approach that enables higher accuracy, even across the most challenging document types. <h3>What is OCR?</h3> Optical character recognition (OCR) is a class of document processing technologies that, as the name suggests, reads characters from text on a page. Originating in the 1920s, OCR has since evolved to support use cases involving data extraction from digitized documents. There are two kinds of OCR technologies: First, generic OCR, which pulls all of the text on a digitized document (image, PDF, etc.) and outputs a raw list of the words on a page. With a number of improvements in character recognition technology, generic OCR has become suitable for distinguishing words and reading complex font types. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/98229f9a-1e6c-4a80-7416-5a38dfbd1d00/public" alt="Generic OCR: extraction/digitization but no field classification"> Second, intelligent OCR, which has become a popular alternative since these systems can automate data extraction for key fields of interest rather than just receiving a blob of all the text on a page. Intelligent OCR is built on generic OCR, but enables a user to define a specific template for a document, manually specify all the fields they need extracted, and automatically get only the structured data they need for all similar-looking documents. <h3>Problems with OCR</h3> There are clear challenges with both of these conventional OCR approaches that reduce the quality of data extraction. Generic OCR simply outputs all the text on a page and doesn’t extract only the key information you might need for downstream workflows. These solutions are typically paired with complex regex or hard-coded heuristics that could require a team of nearly 4-5 engineers just to support parsing out required fields and maintaining the infrastructure. Even then, the variability in text layout and document format ultimately results in poor data extraction accuracy (~50-70% for simple documents, 10%-60% for complex unstructured documents). Intelligent OCR can extract fields you need in a document, but requires manually creating templates and rules to define the location of those fields for every document layout. This approach can quickly become painstaking and incredibly difficult to scale as organizations expand use cases. These solutions also fail when it doesn’t find a matching template on new layouts or when you need to add new fields to extract—common in real-world scenarios that have high volumes or complex, unstructured document types. While intelligent OCR approaches might get up to 70-80% data extraction accuracy with heuristics, it still requires significant engineering and manual operations support to ensure errors don’t fall through the cracks. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6eb13fad-3577-422f-8cc9-51fc0a1eb100/public" alt="Intelligent OCR: extraction and digitization, classification from templates only"> <h3>To summarize:</h3> <ul><li> </li><li>Generic OCR simply extracts text but doesn’t identify the specific fields you need extracted. This approach requires significant engineering effort to maintain the infrastructure and yields low accuracy data.</li><li> </li><li> </li><li>Intelligent OCR can extract your desired fields but requires manually defining templates and hard-coded rules. This approach can get to higher accuracy levels than generic OCR but still results in many errors and doesn’t scale to large volumes and high variability use cases. </li><li> </li></ul><h3>Why use Adaptive ML Document Processing instead?</h3> Running OCR is the first of many steps needed to have a robust document processing solution that can extract data at the highest quality. While these solutions may get you part of the way there, they don’t deliver a complete solution because OCR still requires significant engineering and operations support in order to pull out key fields and fix errors. While we hear about machine learning frequently in moonshot applications like autonomous driving and Artificial General Intelligence, the technology can be significantly useful today in enterprise use cases. Leveraging machine learning is key to getting a high quality and truly automated document processing solution even across highly variable and challenging document types. Rather than relying on off-the-shelf methods or ones that need manually defined templates, an end-to-end machine learning based approach can enable a robust solution with higher accuracy results for highly variable unstructured and structured documents. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8d041be1-a4c2-4e5b-7641-785b4c115000/public" alt="Scale Document AI Adaptive Document Processing: the Best of All Worlds"> However, setting up these systems in-house requires significant investment across the machine learning stack in order to deploy a production system, including data labeling, model training, and continuous monitoring. Choosing the right partner to do this at scale can help you get higher accuracy, lower latency, and save time and money to spend on your company’s core competencies. <h3>Why Scale?</h3> Scale Document AI supports customers with quickly deploying a complete document processing solution with guaranteed quality SLAs for any of their document types. We use our own in-house OCR engine for identifying the text on a page, but don’t stop there. What’s unique about our approach is that we leverage base models built on state of the art Computer Vision and Natural Language Processing research. These models are trained on millions of data points and we further refine them for each customer use case. This fine-tuning uses Scale’s proprietary data labeling and ML training infrastructure that we’ve developed over the past few years to deliver a solution that consistently performs at high accuracy – even across unique edge cases, customer requirements, or variable document types. Scale provides Adaptive ML document processing for many enterprises and organizations across financial services, logistics, manufacturing, healthcare, real estate and government. For example, Scale delivers fast extraction from messy invoices <a href="http://scale.com/customers/brex" rel="noopener noreferrer" target="_blank">for Brex</a>, accurate and timely data from complicated shipping paperwork <a href="http://scale.com/customers/flexport" rel="noopener noreferrer" target="_blank">for Flexport</a>, and essential data out of titles, mortgage applications, and deeds <a href="https://scale.com/customers/states-title" rel="noopener noreferrer" target="_blank">for Doma</a> in real estate transaction processing. If you’re looking for a solution that’s higher quality, lower latency, and free of the hassle of creating and maintaining templates, <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">learn more and contact us</a>.

OCR and even Intelligent Document Processing fall short, while AI-based extraction delivers.

With adaptive ML models performing extraction, Scale Document AI goes far beyond OCR.

Comparing Scale Document AI to Traditional OCR

<h1>Q&amp;A with Scale's Synthetic team</h1><h1>Why did you join Scale?</h1>Vivek: Choosing a new role is a consequential and high-impact decision. I have been following the fantastic work by the Scale team for a few years now. I was intrigued by their ambition and impact in a relatively short window. Eventually, my brain caught up to my gut. Once I spoke to the team at Scale, joining turned out to be a relatively simple decision.Joel: I have spent the last decade working at the intersection of computer graphics and computer vision. I have never been more excited about the future. Never before have so many profound technological breakthroughs converged at the same time - unprecedented compute power, large amounts of visual data, powerful generative models, and scalable AI infrastructure. Together, these advancements enable many new AI applications that we could only dream of in the past. I am incredibly excited to join Scale to help accelerate that future.<h1>What aspects of AI and ML are you excited to work on at Scale?</h1>Vivek: We are still very early in the technology arc of transitioning from traditional software systems to AI. Building infrastructure to get there quickly and responsibly has fueled my passion for several years. I believe Scale will play a significant role in shaping that future.Although we have seen significant investment in model architectures over the past decade, we need a fundamental shift in data practices to unleash AI’s full power. Model code and data form the foundation of an AI system. Scale’s leading end-to-end solutions for the ML lifecycle based on real-world data will continue to set the bar for the data-centric movement.Joel: When I completed my PhD in 2015, the deep learning revolution was just getting going. My thesis focused on physically based rendering of synthetic objects in real environments. Fast forward to 2022, and deep learning has fundamentally changed how most visual computing problems are approached.The practical success of these models is often attributable to their capacity to scale to large amounts of data - a fact that I came to appreciate first hand during my time at Apple.<h1>What are the biggest challenges to implementing synthetic data?</h1>Vivek: The physical world continues to be a limitation for data collection. Enter: digital twins. Digital twins are virtual clones that recreate the real world. This synthetic world will give developers access to unlimited data quickly. But, more importantly, it will take us a step further in addressing the most pressing ethical issues in the AI world today - privacy, data bias, and the democratization of data. Scale Synthetic is our next addition towards accelerating the AI future.Joel: Most recently as Head of Machine Learning at Nines, I came to appreciate that often the fastest way to improve the performance of modern ML models is not to tweak the model to work better for less data, but to focus one’s efforts on understanding, collecting and labeling more high quality data.This insight also presents a challenge to applications where data is scarce, or difficult to collect or label. Instances of this include when there are privacy considerations, dataset bias, or rare events. Hence, the need for synthetic data.<h1>In your view, what sets Scale apart from other AI and ML companies?</h1>Vivek: Scale’s credos resonate with my core beliefs as tenets for successful teams.Two credos that stand out for me are “ambition shapes reality” and “results speak loudest”. Raising the bar, and regularly and objectively measuring progress, is foundational to great products. Once I spent time with the team, it was evident that the credos were not symbolic, but formed the day-to-day backbone.Building, and scaling a culture alongside a product is tough. However, reading Alex and Brad’s posts gave me confidence that I am joining a company where the founder and exec team cares immensely about culture. Solid culture is a necessary condition to build a generational company–one which I hope to contribute to and shape with the team.Joel: The data-centric approach is also what got me excited about Scale. Scale enables any ML developer to build better AI models, faster, by building solutions to manage the entire ML lifecycle. Scale started with high quality data annotation, and is now a clear market leader. Scale has since continued to innovate and has built an impressive suite of leading products around AI operations and infrastructure

Scale AI Welcomes New Synthetic Team Leadership

Why we joined Scale Synthetic

Today, we are thrilled to announce the Early Access Availability of Scale Synthetic, the most efficient way to enhance ML performance with data that complements real-world datasets. With Scale Synthetic, ML teams can combine the benefits of real-world data with diverse and realistic synthetic data to achieve desired data distributions.
&nbsp;
Real-world data remains indispensable to developing performant models. ML teams, however, have encountered several challenges with relying on real-world data alone. Data privacy issues can prevent collected data from being used, collected datasets also inherit the biases of conditions under which data was collected, and collecting enough examples of edge cases and rare scenarios can be both costly and time-consuming.
&nbsp;
With Scale Synthetic, ML teams can:
<ol>
<li>Overcome the challenges of real-world datasets while maximizing the return on investment (ROI) of real-world data by realistically augmenting real-world data or generating fully synthetic data from scratch.</li>
<li>Bridge the sim-to-real reality gap by leveraging Scale&rsquo;s global, human-in-the-loop network that ensures objects in augmented or fully synthetic scenes are realistically placed, and follow a logical path.</li>
<li>Seamlessly integrate with Scale&rsquo;s full ecosystem of products for near-zero friction to add augmented or fully synthetic data into existing ML data pipelines.</li>
</ol>
Scale Synthetic currently supports the augmentation or generation of image, video, or 3D point cloud data but custom requests are also encouraged.
&nbsp;
To date, Scale Synthetic is trusted by leading ML teams such as Kodiak Robotics as well as the U.S. Department of Defense to deliver high-quality synthetic data. Additionally, Scale Synthetic has been recognized in the 2022 Gartner&reg; Innovation Insight for Synthetic Data* report as a Representative Provider in Sample Synthetic Data.
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/13cf537b-0b89-4b55-885f-069505c8c900/public" alt="Kodiak Robotics Quote">
To continue to enhance and build on the foundations of Scale Synthetic, we are also pleased to welcome Joel Kronander as the new Head of Synthetic Data and Vivek Raju Muppalla as the Director of Synthetic Services. Joel joins Scale from Nines, a startup that applied ML and AI expertise to the field of medicine where he served as the Head of Machine Learning. Vivek joins Scale from Unity Technologies, where he was the Director of Engineering, AI and Simulation.
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9281bfec-9ad9-49d8-d658-c16381bb0c00/public" alt="Joel Kronander and Vivek Muppalla Quotes">
If you would like to get early access for Scale Synthetic, <a href="https://scale.com/demo" target="_blank" rel="noopener">contact us today</a>.
&nbsp;
&nbsp;
*Gartner, &ldquo;Innovation Insight for Synthetic Data&rdquo;, Arun Chandrasekaran, Alexander Linden, Anthony Mullen, Farhan Choudhary, February 7, 2022.
&nbsp;
&nbsp;
Disclaimer- GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner&rsquo;s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Scale Synthetic boosts ML performance by complementing real-world datasets with efficient, high-quality synthetic data.

Enhancing Real-World Datasets with Diverse Synthetic Data

Introducing: Scale Synthetic - Now Available for Early Access 

<h1>Field Engineering at Scale</h1>Scale’s Field Engineering is a global team of creative engineers who collaborate with clients to understand their challenges and architect solutions. Our solutions are rooted in our technical skill, product expertise, and a thorough understanding of our customers’ complex needs. Whether the answer is a bespoke solution to solve a unique problem, or a new developer tool that can scale across Scale, everything we do aims to ensure our customers benefit from the full potential of Scale products and expertise. Our work is at the intersection of customer success, product, and engineering, helping Scale’s customers succeed and reach their business goals with Scale by unblocking them technically. Field Engineers are subject matter experts who know all of the products and technologies Scale has inside and out, and work closely with product managers to influence new product features. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2893973b-04ee-4e9e-9c7e-6304d5b38a00/public" alt="Field Engineering Map"><h1>A Day in the Life</h1> Field Engineers start each day with a team standup. During the standup, we align on daily priorities and check in on our longer-term projects. There’s almost always an urgent customer issue to work on, but when we have downtime, we tackle longer term projects that will help the company move faster in the future. We’ll go over some of these longer term projects further down. One of the more fun parts of our daily standup is ending with the “Word of the Day”. As half of our team is located in Buenos Aires, Argentina, each day someone will share a word that they would like to know how to say in English or Spanish and we’ll share our own take, the different definitions, and any fun stories that go along with it. After the standup, each day can look pretty different. Our days will often include at least a few customer meetings, where we will consult with our existing customers (think: the world’s leading ML companies) on technical aspects of their projects, any challenges they may be facing, and new initiatives coming up. Field Engineering is the customer’s technical contact to ensure all goes well with their work. Covid-permitting, we use lunchtime to socialize with colleagues in the office while we eat all the amazing food that is provided for us. It’s a nice time to decompress and connect. After customer meetings and any urgent issues have been addressed, we focus on our longer-term projects. We’re focused on making the lives of our customers, our internal partners, and ourselves as streamlined and scalable as we can. So, we spend significant time on internal tools, customer SDks, documentation and anything else we can think of to improve Scale’s developer experience. <h1>Field Engineering Initiatives</h1> <h2>Introducing a new and improved Scale Python SDK</h2> 2021 saw the introduction of a new version of our Python SDK. Rebuilt from the ground up to deliver new features faster, this new version includes a set of new ways to interact with the Scale API and improve infrastructure for a smoother developer experience. The tool was built on a new foundation that includes novel abstractions, so you can get more out of the Scale API with less effort. This new foundation allows us to add new functionality faster and better than ever before, so expect more new features from Scale to show up in the SDK with improved documentation and release notes. <h2>Shipping a Lidar Toolkit to make sense of multi-sensor datasets.</h2> The act of fusing multiple cameras, lidar sensors, and radars from a vehicle or robot into one cohesive scene that can be annotated is no easy feat. Historically, helping customers work through the nuances of their data and how to format it for Scale’s API-based workflows was one of the most time-consuming parts of our team’s day. In 2021 we overhauled our Lidar Toolkit and introduced a set of resources, code samples, and debugging tools to enable maximum efficiency for developers working with Lidar data. The toolkit, built on Open3d and Scale’s Python SDK, is full of helper functions that make working with the geometry, camera projections, and coordinate systems much easier to understand and implement than doing it from scratch. The Lidar Toolkit our team has published is now the de facto way all customers share and upload their data to Scale today. <h2>Embracing the Operational Nuances at Scale</h2> Scale is in the business of human-in-the-loop data annotation, and humans + loads of data means a lot of operational overhead comes with the territory. We don’t want to rely on running our business based on custom scripts day in and day out. So, we continually develop, iterate and ship internal tools to make the lives of our customer delivery team as seamless as possible. Some of the tools make facilitating customer requests easier, like moving tasks (tasks are what we call the unit of work we are annotating, like an image) between customer projects, re-sending callbacks to our customer’s endpoints, or canceling tasks en masse. Other sets of tools ensure high-quality data is coming out the other side of our task pipelines. A few years ago, had we wanted to programmatically identify potential annotation errors, a custom script would need to be written, tailored to that client’s specific projects. Over the past 18 months, Field Engineering has developed a framework that can be configured by our non-technical ops counterparts to flag any erroneous tasks in their pipelines. And, better yet, this framework can proactively prevent these tasks by flagging warnings directly in the labeling interface—before the annotations are even submitted. This has become one of our most powerful tools to ensure reasonable data is consistently delivered to our clients. <h1>Could Field Engineering be a fit for you?</h1> If you enjoy leveraging both sides of your brain, the people skills side, and the deeply technical one, Field Engineering may be a promising place to build and Scale your career. We are consistently hiring and growing the team (the team has doubled in size each year) and are interested in candidates with a variety of technical backgrounds and skills. Bonus points are given to candidates who possess a background in Computer Vision or API building and troubleshooting. If you're interested, <a href="https://scale.com/careers?location=All&amp;team=Engineering" rel="noopener noreferrer" target="_blank">explore our open positions today</a>!

A Day in the Life of a Field Engineer at Scale

Field Engineering at Scale

Field Engineering at Scale | A Day in the Life

In an exciting announcement to close out the year, Scale has been featured in two of Gartner, Inc.’s 2021 Hype Cycles. We are honored to have been featured in Hype Cycles for both Artificial Intelligence and Data Science and Machine Learning. These reports detail emerging technologies in AI, data science and machine learning, and name Scale as a representative vendor for Data Labeling and Annotation Services. The reports describe an urgency for data labeling and annotation services (DLA) to become widely adopted. According to Gartner analyst Anthony Mullen’s analysis, the demand for labeled and annotated data has risen in order to remove bottlenecks in industry-specific use cases. And, due to the fact that most companies do not have in-house skills and processes for DLA, companies like Scale are the best option to provide DLA to yield optimal AI results. Why good data matters Better data leads to more performant models, and performant models lead to faster deployment. With better data, Scale helps customers deliver more value. An increased diversity of use cases in DLA will drive both AI and industry-specific innovation, according to Gartner analyst Anthony Mullen. Vendors in the market have specific offerings across industries including commerce, robotics, autonomous vehicles, retail, GiS/maps, AR/VR, agriculture, finance, transportation and more. Scale is proud to accelerate innovation for customers including PayPal, iRobot, Square, Brex, Pinterest, Etsy, SAP, Airbnb, Instacart, NVIDIA, OpenAI, Samsung, Toyota, GM, the U.S. Air Force, and more. How does Scale solve the bad data problem? Scale is your AI readiness partner. We got our start in data labeling and annotation for autonomous vehicles. Now, Scale provides a data-centric, end-to-end solution to manage the entire ML lifecycle. That means that Scale can meet you wherever you are in your data process. ML-powered, human-in-the loop data Scale’s strength is the combined power of our ML platform and expert human insights to help our customers develop their data strategy with high quality data from the ground up. According to Mullen, there are AI solutions which would be impossible without human-in-the-loop labeling, including in cases of search engine tuning and image recognition. Furthermore, DLA allows data scientists to allocate and prioritize time for higher-order tasks, instead of taking time and effort to annotate data themselves. With Scale Rapid, any machine learning developer can achieve production-quality labels with no data minimums. Receive quality labels in just a matter of hours, with detailed insights and identified edge cases. Scale Rapid can even handle subjective tasks with the utmost precision and quality. But what comes after labeled data? Gartner Inc., has made the case for why data matters to AI development. After data labeling, what’s next? Data management. Whether you have prepared the data, or we label and annotate the data for you, we’re your premiere data management partner. Scale Nucleus is the command and control for your ML dataset. Visualize, curate and debug your dataset to improve your ML model performance with Nucleus. Scale’s advantage: AI development, from data labeling, through the entire ML lifecycle. Scale’s mission is to accelerate the development of AI. We do this by providing a data-centric, end-to-end solution to annotate, manage, automate, evaluate, collect and generate your data. Our products and services meet you wherever you are in the ML lifecycle, whether you need data labeled and annotated, or need an ML engineer to help explore and refine how AI can help your business.

Gartner Hype Cycles for AI and Data Science and Machine Learning

Scale highlighted in Gartner Inc.’s 2021 AI Hype Cycles

Scale Highlighted in Gartner Inc.’s 2021 AI Hype Cycles

Innovations in the field of Machine Learning (ML) continue to be driven by a dynamic research community. The ML team at Scale is proud to contribute to the field through its own research, dataset partnerships with leading Universities such as Oxford (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: Reddit 50K Dataset</a>) and the University of Waterloo and the University of Toronto (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">CADC Dataset</a>), and collaborations with Stanford University, the University of California, Berkeley, and the University of Washington. As <a href="https://neurips.cc/" rel="noopener noreferrer" target="_blank">NeurIPS 2021</a>, gets underway, we are pleased to share details on three papers the ML team have published as part of the conference: <ol><li> </li><li>DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</li><li>Authors: John Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, Doyne Farmer</li><li> </li><li>Natural Adversarial Objects</li><li>Authors: Felix Lau, Sasha Harrison, Nishant Subramani, Aerin Kim, Elliot Branson, Rosanne Liu</li><li> </li><li> </li><li>Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</li><li>Authors: Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, Ludwig Schmidt</li><li> </li></ol><h2><a href="https://openreview.net/forum?id=udVUN__gFO" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</a></h2> <h3>Paper Summary:</h3> DEBAGREEMENT is a dataset including 42,894 comment-reply pairings from the popular debate website Reddit, each of which has been labeled with agree, neutral, or disagree. We gathered interactions from five different forums: r/BlackLivesMatter, r/Brexit, r/Climate, r/Democrats, and r/Republican. Comment pairings for each forum were chosen in such a way that they generate a user interaction graph when taken as a whole. DEBAGREEMENT offers a challenge for Natural Language Processing (NLP) systems because it includes slang, irony, and topic-specific humor, all of which are common in online discussions. We compared the performance of state-of-the-art language models on a (dis)agreement detection task, and looked at the usage of contextual information that is accessible to the models during training (graph, authorship, and temporal information). DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect agreements as well as disagreements online, in light of recent research showing that context, such as social context or knowledge graph information, enables language models to perform better on downstream NLP tasks. <h3>Key Takeaways:</h3> <ol><li> </li><li>Language models trained on existing datasets underperform when run on real world data. This discrepancy is exposed when evaluating the models using the new DEBAGREEMENT dataset.</li><li> </li><li> </li><li>DEBAGREEMENT presents new opportunities for modeling diverse online interactions with text and context (authorship, graph, temporal information). Its graph structure allows for the combination of text-based machine learning (ML) with graph representation learning (GRL) approaches.</li><li> </li><li> </li><li>By modeling online discussion forums as graphs of user interactions, researchers can: 1) transform the agreement/disagreement detection problem into a sign link prediction task, and 2) use existing signed graph embedding techniques evaluated on publicly accessible signed graphs such as Epinions and Slashdot.</li><li>The sentiment and polarization of topics is not static in time and for the first time, this dataset exposes that and shows how realistic topic sentiment evolves over time.</li><li> </li></ol><h2><a href="https://arxiv.org/abs/2111.04204" rel="noopener noreferrer" target="_blank">Natural Adversarial Objects</a></h2> <h3>Paper Summary:</h3> Although cutting-edge object identification approaches have shown impressive performance, models are often vulnerable to adversarial assaults and out-of-distribution data. Natural Adversarial Objects (NAO) is a novel dataset for assessing the robustness of object identification techniques. NAO comprises 7,934 images and 9,943 objects that are unaltered and depict real-world circumstances, yet cause cutting-edge detection methods to misclassify with high confidence. When compared to the conventional MS-COCO validation set, the mean average precision (mAP) of EfficientDet-D7 lowers by 74.5 percent when tested on NAO. Furthermore, when evaluating a range of object detection architectures, we discovered that improved performance on the MS-COCO validation set does not always transfer to better performance on the NAO, implying that robustness cannot be gained merely by training a more accurate model. We look at why NAO cases are tough to identify and classify. Experiments using image patch shuffles demonstrate that models are too sensitive to local texture. Furthermore, when predicting class labels with integrated gradients and background replacement, we discovered that the detection model is dependent on pixel information inside the bounding box and insensitive to background context. <h3>Key Takeaways:</h3> <ol><li> </li><li>The NAO dataset presents a difficult robustness assessment challenge for detection models trained on MS-COCO. As part of our evaluation, we tested seven state-of-the-art detection models from a variety of families, and found that they consistently failed to perform accurately on NAO when compared to the MS-COCO validation set, which included both in-distribution and out-of-distribution objects, among other things.</li><li> </li><li> </li><li>We also described the process of building such a dataset yourself, which may be used to create similar datasets for any target dataset or pre-trained model.</li><li> </li><li> </li><li>The popular MS-COCO dataset has "blind-spots" that make it challenging to appropriately categorize these naturally adversarial objects. </li><li> </li><li> </li><li>Additionally, we show that detection models are overly sensitive to local texture but insensitive to background change, which makes them vulnerable to natural adversarial objects.</li><li> </li></ol><h2><a href="https://openreview.net/pdf?id=mPducS1MsEK" rel="noopener noreferrer" target="_blank">Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</a></h2> <h3>Paper Summary:</h3> Evaluation issues often undermine the validity of results in machine learning research. In collaboration with researchers from Stanford University, the University of California, Berkeley, and the University of Washington, we conducted a meta-review of 100+ survey papers to identify common benchmark evaluation problems across subfields. In some cases, several years’ worth of progress in certain fields may be misstated. Our meta-review surveys evaluation papers reporting on a broad range of subfields, ranging from computer vision to deep reinforcement learning, to recommender systems and natural language processing, and more. We found a consistent set of failure modes, which we organized into a systematic taxonomy. This taxonomy of failure modes helps researchers understand where different types of failures are common and provides a framework for future research. Our taxonomy distinguishes between “internal” and “external” validity failures. Internal validity failures occur within the context of a single benchmark and raise questions about the consistency or reliability of measurements on that specific benchmark. For example, running baseline methods with less hyperparameter tuning than a newly proposed method can underestimate the performance of existing methods and hence lead to overly optimistic claims of performance improvements. External validity failures, in contrast, describe failures that occur when dealing with multiple benchmarks. External validity failures occur when results from one benchmark do not generalize to other settings. One example is dataset misalignment: if data collected for a benchmark is not representative of the setting where models will be used, then results from that benchmark may not reliably predict performance in the real world. <h3>Key Takeaways:</h3> <ol><li> </li><li>Evaluation failures pose a risk to the validity of research findings in machine learning. Our meta-review of 100+ survey papers observed common failure modes across a wide range of subfields. </li><li> </li><li> </li><li>While some internal validity failures can be identified in the context of a single benchmark, other external validity failures only appear when looking at multiple datasets.</li><li> </li></ol><h2>Conclusion</h2> As we look ahead to 2022, we look forward to supporting more university research teams and driving more research initiatives within Scale. To learn more about our work and the work of many of our partners, our team hosts regular TechTalks on <a href="https://exchange.scale.com/public/events" rel="noopener noreferrer" target="_blank">Scale Exchange</a>, our community of ML practitioners. Join the community to learn more and register for upcoming talks. If you’re interested in joining our growing ML team, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for open positions.

An Overview of ML Research Papers Accepted At NeurIPS 2021

Research Papers Accepted At NeurIPS 2021

Machine Learning Papers at NeurIPS 2021

Artificial Intelligence (AI) is now a part of our everyday lives and has the potential to massively impact the future of our society. Leaders in the private sector are making a real-world impact with AI today - from autonomous vehicles, medical diagnostics, vaccine distribution, and eCommerce. We also recognize the importance of the federal government, academia, and the private sector to work collaboratively to drive innovation and ensure continued U.S. leadership in technology. In early October, we hosted our second virtual conference: TransformX. Over the two days, we brought together more than 22,000 researchers, practitioners, and executives to hear from leaders across industries, academia, and government. We’re excited to share some key insights and takeaways that the foremost government leaders have on the future of AI. <h1>Fireside Chats With Key Leaders</h1>Mark “Drifter” Valentine, Head of Federal at Scale AI hosted fireside discussions with key government and private sector leaders including Lieutenant General Kirk S. Pierce, Commander, Continental U.S. North American Aerospace Defense Command Region-1st Air Force, Rear Admiral Lorin Selby, Chief of Naval Research, and Steven Escaravage, Senior Vice President at Booz Allen Hamilton. <h2>Lieutenant General Kirk S. Pierce</h2> LTG Pierce spoke with Mark on operationalizing AI for National Security. For Pierce, homeland defense is a chief interest. To work effectively and with prudence, “We have this grand, sweeping concept of joint all domain command and control.” To achieve this, Pierce says interoperability is key. The key to interoperability? Data. “We cannot continue to have data stovepipes,” he said. “Depending on wherever that data comes from—whether it's a sensor, above water, below water, in space, whatever it is, we want to have the data to be able to make decisions.” He also said that he sees AI and ML operability as “an opportunity to reset, fundamentally, how we look at information, how we aggregate information and be able to evaluate and assess the validity of the information.” “The companies that are working on that are gonna change how we fundamentally, as senior leaders, make decisions.” <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/vgxbm41ptn"></iframe> <h2>Rear Admiral Lorin Selby</h2> RADM Selby discussed the importance of fostering an AI innovation ecosystem. As a leader who is positioned at the cross-section of government, academia and industry, Selby is optimistic about the utility of AI to help humans reach their potential, and the Navy achieve its goals. He foresees a future where AI can “unburden the mind” from tedious bureaucratic work, “So we can actually focus on the things the human mind is really good at focusing on— which is kind of more strategic thinking, more abstract thinking.” “This is a pivotal moment in history,” he said of AI and ML advancements. “I think we’ll look back on this time as a key transition phase in history.” Selby said that building confidence in AI systems is essential to innovation, as is remaining anchored in the strength of human decision making. “We have to develop trust,” he said, while emphasizing the need to keep humans in the loop of critical decisions. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/h3hikuszel"></iframe> <h2>Steve Escaravage</h2> Steve Escaravage shared his insights on developing realistic approaches to deploying machine learning (ML) in federal environments. Escaravage said that for advancements in ML, the private sector is leading, and the government has to follow. “The innovation is coming from outside—it’s coming from the private sector, from commercial use cases” he said. “For the public sector—for the government to be successful, we simply cannot ignore that investment and we need to channel that into the public sector.” He said that although there has been real progress and momentum since 2017, there are pieces of the puzzle that are missing. Friction points include: access to real world data assets and data sets to train AI, approved computing environments that are configurable and scalable and systems with authorization to operate (ATO) accreditation. “We all have a responsibility as citizens of our governments to try to get them to better outcomes,” he remarked.<iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/0b1pm4gn25"></iframe> <h2>Expert Panel:</h2> In addition to the fireside chats, Mark also led an expert panel discussion on The State of AI for National Security Mission Applications featuring Michael “Rabbi” Harasimowicz and Rachael Martin. Martin said that she hopes that in the future, AI will be a seamless part of her agency’s workflow, and as commonplace, critical and functional as something like email. “The new warfighting demand is time,” she said. “When you have machine speed, you need to have trust and confidence that what the machine is telling you is correct.” As the NGA relies on overhead imagery and data to compel decision-making, she said it’s important to have a systematic data workflow from input to warfighter. “We do think that AI and automation is really the best way of accomplishing that,” Martin said. She added that she sees the time investment and the leadership investment in bringing AI to the mission space as heartening. “You have to change culture by getting people involved in the change,” she reflected. Harasimowicz said that if our government doesn’t update its systems, the systems and the country are at risk. “Our national security challenges are really getting to the point where they're beyond human scale,” he said. “And I say that because the amount of data that is flowing in and around all of our domains is just insurmountable for just a commander.” The level of support required to manage the data at a tactical and strategic level drives us to have a level of assistance, he said. “So artificial intelligence can provide that backdrop.” “Our adversaries are challenging themselves in aggressive ways,” Harasimowicz said. “And we have to be as aggressive, if not more so aggressive…We have to go faster. We have to find ways to solve solutions in weeks, as opposed to years.”<iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/2yn8oyur22"></iframe>

Highlights from TransformX 2021: U.S. Government Related Sessions

Highlights from TransformX 2021: U.S. Government Sessions

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

basecamp-speakers-events

basecamp-agenda

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog