experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

<h1>Background</h1> At Scale AI, we’re always looking for ways to apply machine learning (ML) in our internal systems to deliver the best results for our customers. We’ve previously written about Scale’s <a href="https://scale.com/blog/how-we-scale-machine-learning" rel="noopener noreferrer" target="_blank">ML-powered pre-labeling</a> as well as our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">active tooling, like Autosegment</a>, which helps our Taskers label faster and at higher quality. In this piece, we discuss how we think about and incorporate some basic tenets of design thinking to develop ML systems that make a difference for our labeling pipeline. <a href="http://web.mit.edu/jrankin/www/engin_as_lib_art/Design_thinking.pdf" rel="noopener noreferrer" target="_blank">Design thinking</a> is a mindset and set of practices which foster the collaboration required to solve problems in human-centered ways and was notably popularized by organizations such as IDEO. It’s useful for machine learning engineers who need to understand the full scope of how the system they are building will be used and are responsible for both business and operational outcomes. We’ll describe how design thinking techniques can be applied to scoping machine learning problems and introduce some concepts to help business stakeholders communicate more effectively with ML engineers.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a41ed7a-4287-491b-6202-1f80a4c29a00/public" alt="Phases of Developing an ML System">Idealized phases of developing an ML system. In practice, it’s not this linear.In our experience, the different phases of developing an ML system are: <ol><li>Discovery - Locate a business or operational pain point</li><li> </li><li> Design - Map the different ways a new or existing model could fit into the</li><li> extant pipeline</li><li> </li><li> </li><li> Decision &amp; Alignment - Align with operational and engineering</li><li> stakeholders on scope and resourcing</li><li> </li><li> </li><li> Development, Integration, Deployment - Deliver the system. Build data</li><li> pipelines, train models, integrate into the pipeline, etc.</li><li> </li><li> </li><li> Measurement &amp; Experimentation - Identify the effect of the ML system as</li><li> an intervention on the existing pipeline.</li><li> </li><li> </li><li> Continuous improvement - Monitor and retrain to account for distribution</li><li> shifts and new edge cases</li><li> </li><li> </li></ol> This is a descriptive mental model, not a prescriptive blueprint. In reality, a project is always moving between different phases and can often be in multiple phases at the same time. For example, measurement and development form a tight iteration loop. But, it’s still useful to think about these phases in isolation. Concepts from design thinking are mostly employed in the Discovery and Design phases when the problem statements are the most inchoate and ill-formed. In these phases, the need or specific outcome is communicated to an ML engineer and an initial strategy to address that need or deliver that outcome is designed. We focus on the Discovery and Design phases in this blog and cover the remaining phases briefly.<h1>Phase 1: Discovery</h1> An ML system gains life when a need is discovered. By need, we distinguish between a request for a specific feature, such as a linter (a tool we use to ensure data quality), and the underlying operational or business pain point or problem - such as a new labelling capability requiring ML support. It’s not uncommon for operational or business teams to stumble into the <a href="https://en.wikipedia.org/wiki/XY_problem" rel="noopener noreferrer" target="_blank">XY problem</a> by pattern matching and requesting a clone of an existing feature, without realizing that their underlying problem is actually quite different. Or, they might ask for something which is incredibly difficult, unaware that a slightly modified version would save literally months of development time. So an ML engineer should always hunt down the true problem that needs to be solved. Pain points and problems can be project-specific or exist horizontally across business teams or units. Historically, many of the problems that the ML team at Scale have tackled are project-specific and brought to us by our cross-functional stakeholders. For example, a project manager running into issues with a labeling project or a product manager looking to launch a new feature will surface these project-specific issues to our team. Tackling these isolated problems can result in marooned solutions which aren’t readily reusable. Solving horizontal issues, in comparison, maximizes our leverage, since the ML systems we build will then scale with our business. However, cross-business issues that can be solved with ML are often harder to identify because of the sheer operational complexity. These issues require a more active discovery process and take more time and deeper alignment with partners. The problems with the highest stakes are not obvious to ML engineers, who don’t get day to day exposure to them, even as these frictions impose significant pain on the operations or business side. To get a sense for this, imagine explaining the pain of managing multiple python environments and package versions to an operational lead - it can be incredibly painful for you, yet esoteric, even cryptic, to them. As another example, while working on an image semantic segmentation project, we learned that pre-labels generated by a model specific to that project were sometimes ignored by taskers, but the root causes were unclear. A close investigation identified a number of culprits, ranging from the inflexibility of pre-labels to having optimized for the wrong metric during development. <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, developed by Sean Li, Saleh Hamadeh, and the two of us, aims to tackle these problems. Autosegment solves a challenge for all image segmentation projects and isn’t restricted to a particular project. Learning about this challenge and building the appropriate solution required input from numerous stakeholders - from the project leads, who wanted to accelerate taskers; the taskers themselves, who used our tools and knew what worked and what didn’t; to engineers, who were familiar with the implementation of image segmentation in our interface and understood the kinds of tradeoffs that different tools would be need to make.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7866cd69-bd81-40f7-4e44-6094a84f2c00/public" alt="Stakeholder concerns">Example overlap of concerns between different stakeholders for a new ML-assisted tool. Identifying the pain points of the end user always requires talking to the end user. Instead of theorizing what could be a problem for them, it’s always more effective to speak to users directly and observe them as they encounter the problem. When building systems to accelerate labeling operations, ML engineers need to build user empathy. A simple way to build user empathy is to spend time tasking and dogfooding our own platform. There is no substitute for first hand experience for understanding what makes tasking hard.<h1>Phase II: Design</h1> After Discovery, the project enters a design phase where an initial approach is sketched out - what kind of model to build, what sort of data is needed, the timelines, and so on. Designing effective ML systems at Scale requires two things: a clear need, which we’ve discussed above, and knowing the many different ways a model can fit into the labelling pipeline. This last point is worth further elaboration. The same machine learning model can be used for a variety of operational purposes. For example, a model that predicts bounding boxes for cars can be used to:<ol><li>Estimate the number of boxes in a task (task difficulty estimation)</li><li>Prepopulate the scene with bounding boxes (pre-labelling)</li><li> </li><li> Suggest fixes to bounding boxes made in real-time by a tasker (active</li><li> tooling)</li><li> </li><li> </li><li> Identify missing boxes or unexpected boxes in a completed scene (linting)</li><li> For each of these, the inference API could have the exact same request and</li><li> return formats. However, each of the four approaches above require different</li><li> operational adjustments, not all of which may be feasible. Task difficulty</li><li> estimation makes assumptions about what makes a task difficult or easy for</li><li> humans and implicitly maps these to legible subpopulations of taskers.</li><li> Active tooling can require nontrivial operational investment in teaching</li><li> taskers how to use the new tool and engineering lift to build new frontend</li><li> components. Building a useful ML system means designing how an ML model will</li><li> interface with the rest of the operational and engineering systems, in</li><li> conjunction with actually training the ML model.</li><li> </li><li> </li></ol><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8e6d0b0b-5185-4a7c-227e-03350b1bdd00/public" alt="One model, many uses">The same model may be applied in different ways to an existing pipeline, requiring different accommodations by engineering or operational teams. We’ve noticed two particular traps for ML engineers which can lead them on wild goose chases during the design phase. The first is misalignment between the machine learning problem and the business problem. The second is confusing the customer and the end user of a system. ML engineers are trained to use models to solve machine learning tasks, like box detection or text classification. For a box detection dataset, you’d use a box detection model. For a text classification dataset, a text classification model. When the business problem looks like an image classification problem, you might reach for an image classification model, only to discover that what you really need to solve is tabular regression. Faced with using ML to assist, say, labelling semantic segmentation training data, a common reflex is to train a semantic segmentation model to do the labelling. But if you already had a capable model, you wouldn’t need to collect training data for it! The mistake here is to apply the familiar task paradigm - semantic segmentation - to solve the business problem - accelerating semantic segmentation. Many operational questions take the form “Is something X or not X?” To an ML engineer, this sounds like a classification problem, and it’s easy to reach for a classification model which directly answers the question. Putting aside whether modeling X directly is feasible, it’s often the case that we already have an operational or engineering system which could answer this question more effectively with some additional information. In this situation, it may be faster to figure out how to extract the augmenting information rather than replicate the existing system altogether; a strong reason to apply a modeling scheme that looks very different from the modeling question.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3505a43e-5c45-4274-cdd3-2e27242e0600/public" alt="Leveraging existing systems">Modeling the business problem to solve doesn’t always mean reaching for the most obvious machine learning task paradigm. The second trap is mistaking the end user of a system with the people who request or manage the system. When we build a tool like <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, even though the end user of the active tooling is the Tasker, the internal customer is actually the Delivery Operations team. What we build has to meet the customer’s expectations, so even if Taskers enjoy using our tool, we need to make sure to move the needle for the Delivery Operations team. But the expectations and needs of the end user may differ or conflict from the internal customer (e.g. latency tradeoff with cost). In order to avoid falling into these two traps, ML engineers have to be deliberate in how they transport the business problem into a machine learning context; and always keep in mind the different stakeholders.<h1>Phases III - VI: Decision and Alignment Through Continuous Improvement</h1> After the initial Discovery and Design phases, the project needs a go / no-go decision by both the ML team as well as cross-functional teams. In this phase, all teams must decide whether to allocate the necessary resources to the project. If the ML team decides to dedicate resources but the cross-functional team cannot commit the time or spare the bandwidth, the effort will ultimately fall short. Some factors teams must consider when making this decision include: prioritization, level of effort needed, and available bandwidth. Once all teams commit to the project, we can develop the ML-specific components of the ML system. This is when we build data pipelines, train models, and ship an endpoint. Once the model is developed, the project nearly always needs engineering and operational integration. For example, if we are developing a new pre-labeling model, the model requires integration with our labeling pipelines to attach the pre-label to the task. Now the project is actually deployed. Deployment should also follow best practices: meaning the feature should only be enabled for a small percent of tasks or users at the beginning, making it easier to revert if there are any issues. The system should then be constantly measured. Especially in the first few weeks, close attention should be paid to ensure no issues arise. We then use the metrics and qualitative feedback gathered in the measurement step to continuously improve the model. Models often need to be retrained to account for new edge cases. As we maintain the production model, we often encounter data distribution shifts or taxonomic changes which must be addressed.<h1>Conclusion:</h1> We hope this post provided some insight on how we design our ML systems and how you and your teams can effectively partner with horizontal teams to deploy models that serve your business needs. Throughout the phases of ML development, we strive to ensure alignment with our cross-functional stakeholders. While gaining and maintaining alignment takes time and deliberate effort, it allows us to unlock capabilities on our platform that wouldn’t otherwise be possible.

Understanding operational and business context is critical to shipping ML systems that work. 

How we leverage concepts from design thinking to design ML systems in our internal processes to deliver results for customers.

Design Thinking for ML Systems at Scale

As a member of the U.S. Air Force Rescue and #AFWERX, I have been working to modernize the tools used by Search and Rescue to increase the odds of survival of those lost at sea. Every year, hundreds of individuals find themselves isolated and at the mercy of the ocean, creating a dangerous scenario which is impossible to navigate without the best technological tools and deep expertise. Being in the ocean is one of the more extreme environments to be isolated. The conditions that make it hard to survive also make it difficult to locate and rescue the isolated; rescue is rarely a simple mission. Attempting to modernize the Search and Rescue model is arguably just as difficult. Systems are out of date: the “how” hasn’t changed in over 20 years. And changes that have been adopted, have not been implemented at the speed of relevancy. Five years ago, I began exploring different approaches to solving Search and Rescue’s toughest problem statements. During this discovery process, two significant experiences led me to become a Fellow at Scale. In my first experience, I was deployed as a United States Special Operations Command (SOCOM) Tech Advisor as a Personnel Recovery Subject Matter Expert (SME). I had the opportunity to share these problem statements with Josh Wolfe, co-founder of Lux Capital, and the rest of the SOCOM Tech Advisory Panel team. Josh shared his opinion on how the DoD could use a page out of the venture capital playbook: to move faster with sustainable speed in developing solutions. Seeing how Josh was investing in startups that invent the future inspired me. I began to engage with early-stage startups to find new solutions that would save lives. The second instance was at a Hacking for Defense session at Stanford University. This time, I was working with young college students learning the lean startup methodology to solve DoD problems. Steve Newell, the founder and CEO of BMNT, introduced the Search and Rescue problem statement to the class. He presented it in such an elegant way—I could physically see his inspirational words produce passion on the young data scientists’ faces. At this moment, I realized that AI developed to save lives would be the catalyst for large-scale DoD adoption. A few years later, and several AFVentures Small business Innovation Research contracts later, Forrest Underwood introduced me to Shift.org. This was the opportunity I couldn't pass up: the chance to work inside a startup for 8 weeks. Shift was able to recognize a critical flaw in my strategy to solve problems. How can you get the DoD to adopt industry AI solutions without possessing a well-developed ML pipeline? For this reason, I think Shift matched me with Scale AI. Scale AI is the best-in-class at providing the entire ML lifestyle to its customers so they can focus on the real, pressing problems at hand. This is exactly what's needed when a DoD customer asks for an AI solution. It is a misunderstanding of the requirements surrounding ML that can be difficult when attempting field AI solutions. Without a mature ML pipeline, it’s very difficult to have your model perform correctly and continue to improve as the extremely dynamic world evolves. On the first day on the job, I attended #TransformX, a virtual conference with the CEO Alex Wang, a community of leaders, visionaries, practitioners, and researchers across industries. This was probably one of the best onboarding or familiarization phases that I've experienced in 20 years. Spending 2 days listening in on thought leaders discussing the art of what's possible in a long forum was extremely insightful. Throughout the entire fellowship experience, I compared how Scale operates against Air Force Institutions of which I’ve been a part. One observation that stood out to me was the holistically flat organizational rank structure of Scale. After seeing the level of engagement that Alex Wang had during TransformX and the scope of his responsibilities, it was easy to draw parallels between Alex and a Geographic Combatant Commander. One welcoming attribute of Scale was the inclusion of ambitious ideas. It is understood that the company is at the edge of the AI frontier, and new ideas are required to move forward quickly. This is what has allowed Scale to grow at an incredible rate; it’s also the attribute that the Air Force is trying to re-install into its ranks. Inside Scale, I had the opportunity to work for the Federal division. The level of talent and prior experience of the team was truly awe-inspiring. I had a false assumption that I would be working with individuals without experience in the Armed Forces. I was surprised when I learned that there was a rich mix of individuals from both the DoD and Silicon Valley. At first, it felt like I walked into a Special Operations Group with Senior Instructor Pilots, Chief of Intel, and the Ground Forces Commander discussing an Operations Order. Except, everyone was in business casual and there were lead engineers in the conversation. For Scale to recognize the talent pool coming out of the DoD was exciting to see. The method of mixing operators, strategists, and engineers inside is not common practice for the Air Force. The standard Air Force approach is to silo these different functions on a day-to-day basis. Even my two cents were considered as the new guy. That says a lot about an organization. The art of creating what's possible is something that Scale was always discussing with potential customers. This wasn't a "fake it until you make it” model. The conversation was followed up with things that were deployed in that space to demonstrate their success. My gained understanding of what's possible is what I will take back to the Air Force. Thank you Scale.

c63a1be0661f1594644b740c08bede0e.jpeg

Bon Strout details his experience as a Scale Fellow

Scale Fellow Spotlight on Bon Strout

Shift Ventures Fellow Spotlight: Bon Strout

Scale board member William Hockey details why he believes AI/ML has the fundamental ability to restructure the way that systems and businesses are built.

Scale welcomes William Hockey as a board member

Why I Joined Scale

<h2>Update:</h2>
 We have received an overwhelming response to our announcement of free, AI-ready datasets in support of Ukraine. Nearly a hundred AI companies, researchers, developers, and GIS practitioners have offered to support the initiative with Scale &ndash; an inspiring global community of AI Developers for good.
Given the high demand, we will provide the datasets in phases starting with requestors with the greatest need, followed by a broader community. At this time, our focus is on providing Structure Damage Assessments in Kyiv, Kharkiv, and Dnipro.
The path forward is now two-pronged &ndash; our top priority is the delivery of damage assessments to those with urgent need.
The second prong of our effort is to provide AI-ready training datasets for Damage Assessment to the broader AI community, starting with a structure recognition training dataset.
Today, we are sharing an initial dataset with the broader AI community: <a href="https://drive.google.com/uc?id=17lpMFngiqeegIm1U0RzUYfB-9GjmI7fk&amp;export=download" target="_blank" rel="noopener noreferrer">link to download dataset</a>.
Note: the corresponding ZIP file's SHA256 Checksum is: 379c394dab6a4d7f1358625d086ebea4b39aaa124548e8c079bee16fb574bc53.
We appreciate your patience as we assess and prioritize requests for data and data development. We look forward to continuing to support Ukraine alongside other AI Developers leveraging critically important open-source and commercially available data for this important work.
Sincerely,
Scale AI Team
&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;&mdash;
At Scale, we believe in using AI technology in support of democratic values. That&rsquo;s why starting today, Scale will be providing a series of AI-Ready datasets that algorithm developers can use to rapidly train and deploy AI in support of Ukrainian operations. By providing these datasets at no cost to national security practitioners, we hope to support a diplomatic solution and swift end to this conflict.
Diplomacy is fueled by insights; foundational to that is quality, trusted data. We have recently created strategic partnerships with commercial satellite imagery providers in order to develop these datasets, which include Aircraft Detections, Structure Damage Assessments, Change Detection, and the JSON and GeoJSON annotations. We encourage AI Developers to contact detections@scale.com to receive this data free of charge.
To better understand how these datasets can be used, Scale presents a contemporary use case of our partnerships with national security leaders. We deliver an AI-powered change detection solution capable of identifying and tracking military formations across the globe.
The video below demonstrates continuous monitoring and improvements by Scale&rsquo;s ML engineers and deployment strategists. We provide timely and accurate insights through our AI Data Readiness platform, our Test and Evaluation platform and our continuous ML Ops tooling.
Leveraging Scale's AI/ML ops platform, we continuously develop and hone deep learning models to describe scenes and detect objects across a range of inputs from commercial satellite imagery providers.
<iframe class="ql-video" style="width: 564px; height: 282px;" src="https://scaleai.wistia.com/embed/medias/hphfibwbn6" width="100%" height="400" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
We are now delivering insights to U.S. national security customers from this full-stack, multi-intelligence, human-teaming product, including on Russian formations across Europe and Eurasia.
Here, on Russian air and ground order of battle (OOB) data, we successively identified OOB changes across the EUCOM area of responsibility (AOR) over the past several months. We used high-resolution synthetic aperture RADAR(SAR) imagery from Capella Space and high-resolution optical imagery to deliver high-accuracy, time-sensitive insights.
&ldquo;In these intricate and time-sensitive scenarios, it is critical that we deliver insights from multiple, disparate datasets at the speed of relevance,&rdquo; said Mark Valentine, Head of Federal at Scale. &ldquo;We are proud to demonstrate the power of our platform, and our approach to human-machine teaming to support national security leaders and practitioners.&rdquo;
Monitoring the Russian OOB on the border of Ukraine using commercial satellite imagery, and other non-traditional Intelligence, Surveillance and Reconnaissance (ISR) sources requires adaptation to a variety of challenges in a constantly-evolving field.
The challenges in using AI to analyze OOB changes include: exigent scenarios presented by weather, disparate image varieties and quality, and the inclusion of relatively rare military objects such as transporter erector launchers (TELs), for which there are limited real-world observations.
The depth of ontology can also present complications: the presence of multi-generational aircraft, ground vehicles, RADAR and TELs mean that models need to be trained against tens of thousands of observations in a range of environmental conditions.
If you are an AI Developer who would like to receive this data free of charge, please contact <a href="mailto:detections@scale.com">detections@scale.com</a>.

ae82457827ce45793eefc6460623a063.jpeg

Scale AI provides free datasets for national security partners

Scale AI providing free datasets to help national security partners gain current insights into Russia-Ukraine conflict

Scale Rapid is a quick and easy way to get clean, high-quality labeled data for your next ML project. We used it to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model</a> in a previous post - and we’re back now to show you how to use Scale Rapid to train an unsupervised model! In this blog, we are going to use <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener noreferrer" target="_blank">CycleGAN</a> (<a href="https://arxiv.org/pdf/1703.10593.pdf" rel="noopener noreferrer" target="_blank">paper</a>) to anime-fy images with Scale Rapid. CycleGAN is trained in an unsupervised manner, meaning each X input does not need to correspond 1:1 to a y output in order to train the net. Instead of using Scale Rapid to label training data, we used it to clean data. We separated a group of images scraped from the internet into categories. This classification allows us to make the two inputs necessary for CycleGAN - a class A (real-life images) and class B (Japanese anime-style images). To start, we downloaded 10k images from Google images from a variety of different searches. At a glance, the corpus had all sorts of images. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/27b06dda-1661-4805-c721-0ec70ee20300/public" alt="Sample images from Google">Class A contains real-life images and Class B contains anime-style images. Image 3 belongs to Class A, and Image 2 to Class B. Images 1 and 4 do not belong in the dataset as they are neither real-life nor anime-style images. The ask for Scale Rapid is to discriminate between these three types of images: a real-world, realistically edited photograph, a Japanese-style anime scene, or neither. Rapid responses are then used as ground truth to establish which images should and shouldn’t be in the dataset we train CycleGAN on. Rapid’s categorization endpoint supports image categorization - and it's only 8 cents per image! Because CycleGAN is trained on unsupervised data, we don’t need to pair class A and class B images - we can simply separate them from one another and train. <h2>Using Scale Rapid</h2> Scale Rapid works in two phases- first, the user sets up the project &amp; instructions, and runs small calibration batches to ensure the initial results are as expected. Once quality on those looks good, larger production batches may be uploaded for labeling. For this application, we set up a categorization project. The taxonomy is just one multiple-choice question: What type of image is this? <ul><li> </li><li>Anime (japanese animation) image</li><li> </li><li> </li><li>Photorealistic (real-life) image</li><li> </li><li> </li><li>Neither</li><li> </li></ul> Scale Rapid also has a feature called long hints, which allows us to prompt labelers with specific things to think about upon choice selection via a tooltip. For the first choice (Anime (japanese animation) image) I clarified with the following long hint: "Japanese animation refers specifically to anime, as in the instructions. Other cartoon-style or generated images belong in the neither category." We begin by uploading a calibration batch, and follow up with production batches. The labeling results are easily parseable via JSON. <h2>Scale Rapid Results:</h2> A random audit of tasks found just one mistake out of 200 randomly selected labeled images. A few examples of correctly labeled images are below, organized by class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d6e522b3-b00e-4e91-47f5-3f709d144300/public" alt="Examples of correctly labeled images by class">With the help of Scale Rapid, raw data from the web was transformed into cleaned and confirmed data and separated into class A, class B, and neither. <h2>Model</h2> CycleGAN produces mappings from class A to class B and vice versa using two different types of losses- adversarial loss and cycle consistency loss. Adversarial loss incentivizes the mapping to generate images that look similar to the target set. We achieve this by optimizing against an adversary D–a discriminator that tries to separate the generated images for the target class from actual data from the target class. Cycle consistency loss incentivizes the mapping to produce images that look like they fit the distribution of the target class.This occurs by minimizing the difference between original data and that same data transformed into the same target class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/cdcd9177-a511-4863-70fa-6a9c1194dd00/public" alt="Cycle consistency loss">Cycle consistency loss is calculated by the difference between the untransformed class A image and the double-transformed class A image. The mapping aims to minimize this.We trained CycleGAN with our Scale Rapid sorted dataset.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3c9b78d0-1aaa-4d49-ad1d-e7d91e37ca00/public" alt="Loss Function Graph"><h2>Model Results:</h2> We are now able to anime-fy images! A few examples:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5a438327-e4b7-41e5-a19b-2cd457c94d00/public" alt="CycleGAN Images"><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Try Scale Rapid</a> for your own project today!

In this blog, we explore combining Scale Rapid Labeling with CycleGAN to anime-fy images.

Using Scale Rapid to train an unsupervised model.

Using CycleGAN to Train Unsupervised ML Models

Using Rapid and Nucleus together will get you better results than using Rapid alone.

 Using Scale Nucleus with Rapid to Unearth Errors—Label Smart, not Hard

Peer-Reviewed Doesn’t Mean Perfect Data

Mission critical industries including logistics, financial services, government, and healthcare rely on high quality document data extraction in order to optimize business workflows, save money, and improve customer experience. In this article, we explain the conventional approaches for document processing, deep dive into drawbacks with these solutions, and introduce a new approach that enables higher accuracy, even across the most challenging document types. <h3>What is OCR?</h3> Optical character recognition (OCR) is a class of document processing technologies that, as the name suggests, reads characters from text on a page. Originating in the 1920s, OCR has since evolved to support use cases involving data extraction from digitized documents. There are two kinds of OCR technologies: First, generic OCR, which pulls all of the text on a digitized document (image, PDF, etc.) and outputs a raw list of the words on a page. With a number of improvements in character recognition technology, generic OCR has become suitable for distinguishing words and reading complex font types. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/98229f9a-1e6c-4a80-7416-5a38dfbd1d00/public" alt="Generic OCR: extraction/digitization but no field classification"> Second, intelligent OCR, which has become a popular alternative since these systems can automate data extraction for key fields of interest rather than just receiving a blob of all the text on a page. Intelligent OCR is built on generic OCR, but enables a user to define a specific template for a document, manually specify all the fields they need extracted, and automatically get only the structured data they need for all similar-looking documents. <h3>Problems with OCR</h3> There are clear challenges with both of these conventional OCR approaches that reduce the quality of data extraction. Generic OCR simply outputs all the text on a page and doesn’t extract only the key information you might need for downstream workflows. These solutions are typically paired with complex regex or hard-coded heuristics that could require a team of nearly 4-5 engineers just to support parsing out required fields and maintaining the infrastructure. Even then, the variability in text layout and document format ultimately results in poor data extraction accuracy (~50-70% for simple documents, 10%-60% for complex unstructured documents). Intelligent OCR can extract fields you need in a document, but requires manually creating templates and rules to define the location of those fields for every document layout. This approach can quickly become painstaking and incredibly difficult to scale as organizations expand use cases. These solutions also fail when it doesn’t find a matching template on new layouts or when you need to add new fields to extract—common in real-world scenarios that have high volumes or complex, unstructured document types. While intelligent OCR approaches might get up to 70-80% data extraction accuracy with heuristics, it still requires significant engineering and manual operations support to ensure errors don’t fall through the cracks. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6eb13fad-3577-422f-8cc9-51fc0a1eb100/public" alt="Intelligent OCR: extraction and digitization, classification from templates only"> <h3>To summarize:</h3> <ul><li> </li><li>Generic OCR simply extracts text but doesn’t identify the specific fields you need extracted. This approach requires significant engineering effort to maintain the infrastructure and yields low accuracy data.</li><li> </li><li> </li><li>Intelligent OCR can extract your desired fields but requires manually defining templates and hard-coded rules. This approach can get to higher accuracy levels than generic OCR but still results in many errors and doesn’t scale to large volumes and high variability use cases. </li><li> </li></ul><h3>Why use Adaptive ML Document Processing instead?</h3> Running OCR is the first of many steps needed to have a robust document processing solution that can extract data at the highest quality. While these solutions may get you part of the way there, they don’t deliver a complete solution because OCR still requires significant engineering and operations support in order to pull out key fields and fix errors. While we hear about machine learning frequently in moonshot applications like autonomous driving and Artificial General Intelligence, the technology can be significantly useful today in enterprise use cases. Leveraging machine learning is key to getting a high quality and truly automated document processing solution even across highly variable and challenging document types. Rather than relying on off-the-shelf methods or ones that need manually defined templates, an end-to-end machine learning based approach can enable a robust solution with higher accuracy results for highly variable unstructured and structured documents. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8d041be1-a4c2-4e5b-7641-785b4c115000/public" alt="Scale Document AI Adaptive Document Processing: the Best of All Worlds"> However, setting up these systems in-house requires significant investment across the machine learning stack in order to deploy a production system, including data labeling, model training, and continuous monitoring. Choosing the right partner to do this at scale can help you get higher accuracy, lower latency, and save time and money to spend on your company’s core competencies. <h3>Why Scale?</h3> Scale Document AI supports customers with quickly deploying a complete document processing solution with guaranteed quality SLAs for any of their document types. We use our own in-house OCR engine for identifying the text on a page, but don’t stop there. What’s unique about our approach is that we leverage base models built on state of the art Computer Vision and Natural Language Processing research. These models are trained on millions of data points and we further refine them for each customer use case. This fine-tuning uses Scale’s proprietary data labeling and ML training infrastructure that we’ve developed over the past few years to deliver a solution that consistently performs at high accuracy – even across unique edge cases, customer requirements, or variable document types. Scale provides Adaptive ML document processing for many enterprises and organizations across financial services, logistics, manufacturing, healthcare, real estate and government. For example, Scale delivers fast extraction from messy invoices <a href="http://scale.com/customers/brex" rel="noopener noreferrer" target="_blank">for Brex</a>, accurate and timely data from complicated shipping paperwork <a href="http://scale.com/customers/flexport" rel="noopener noreferrer" target="_blank">for Flexport</a>, and essential data out of titles, mortgage applications, and deeds <a href="https://scale.com/customers/states-title" rel="noopener noreferrer" target="_blank">for Doma</a> in real estate transaction processing. If you’re looking for a solution that’s higher quality, lower latency, and free of the hassle of creating and maintaining templates, <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">learn more and contact us</a>.

OCR and even Intelligent Document Processing fall short, while AI-based extraction delivers.

With adaptive ML models performing extraction, Scale Document AI goes far beyond OCR.

Comparing Scale Document AI to Traditional OCR

<h1>Q&amp;A with Scale's Synthetic team</h1><h1>Why did you join Scale?</h1>Vivek: Choosing a new role is a consequential and high-impact decision. I have been following the fantastic work by the Scale team for a few years now. I was intrigued by their ambition and impact in a relatively short window. Eventually, my brain caught up to my gut. Once I spoke to the team at Scale, joining turned out to be a relatively simple decision.Joel: I have spent the last decade working at the intersection of computer graphics and computer vision. I have never been more excited about the future. Never before have so many profound technological breakthroughs converged at the same time - unprecedented compute power, large amounts of visual data, powerful generative models, and scalable AI infrastructure. Together, these advancements enable many new AI applications that we could only dream of in the past. I am incredibly excited to join Scale to help accelerate that future.<h1>What aspects of AI and ML are you excited to work on at Scale?</h1>Vivek: We are still very early in the technology arc of transitioning from traditional software systems to AI. Building infrastructure to get there quickly and responsibly has fueled my passion for several years. I believe Scale will play a significant role in shaping that future.Although we have seen significant investment in model architectures over the past decade, we need a fundamental shift in data practices to unleash AI’s full power. Model code and data form the foundation of an AI system. Scale’s leading end-to-end solutions for the ML lifecycle based on real-world data will continue to set the bar for the data-centric movement.Joel: When I completed my PhD in 2015, the deep learning revolution was just getting going. My thesis focused on physically based rendering of synthetic objects in real environments. Fast forward to 2022, and deep learning has fundamentally changed how most visual computing problems are approached.The practical success of these models is often attributable to their capacity to scale to large amounts of data - a fact that I came to appreciate first hand during my time at Apple.<h1>What are the biggest challenges to implementing synthetic data?</h1>Vivek: The physical world continues to be a limitation for data collection. Enter: digital twins. Digital twins are virtual clones that recreate the real world. This synthetic world will give developers access to unlimited data quickly. But, more importantly, it will take us a step further in addressing the most pressing ethical issues in the AI world today - privacy, data bias, and the democratization of data. Scale Synthetic is our next addition towards accelerating the AI future.Joel: Most recently as Head of Machine Learning at Nines, I came to appreciate that often the fastest way to improve the performance of modern ML models is not to tweak the model to work better for less data, but to focus one’s efforts on understanding, collecting and labeling more high quality data.This insight also presents a challenge to applications where data is scarce, or difficult to collect or label. Instances of this include when there are privacy considerations, dataset bias, or rare events. Hence, the need for synthetic data.<h1>In your view, what sets Scale apart from other AI and ML companies?</h1>Vivek: Scale’s credos resonate with my core beliefs as tenets for successful teams.Two credos that stand out for me are “ambition shapes reality” and “results speak loudest”. Raising the bar, and regularly and objectively measuring progress, is foundational to great products. Once I spent time with the team, it was evident that the credos were not symbolic, but formed the day-to-day backbone.Building, and scaling a culture alongside a product is tough. However, reading Alex and Brad’s posts gave me confidence that I am joining a company where the founder and exec team cares immensely about culture. Solid culture is a necessary condition to build a generational company–one which I hope to contribute to and shape with the team.Joel: The data-centric approach is also what got me excited about Scale. Scale enables any ML developer to build better AI models, faster, by building solutions to manage the entire ML lifecycle. Scale started with high quality data annotation, and is now a clear market leader. Scale has since continued to innovate and has built an impressive suite of leading products around AI operations and infrastructure

Scale AI Welcomes New Synthetic Team Leadership

Why we joined Scale Synthetic

Today, we are thrilled to announce the Early Access Availability of <a href="https://scale.com/synthetic" rel="noopener noreferrer" target="_blank">Scale Synthetic</a>, the most efficient way to enhance ML performance with data that complements real-world datasets. With Scale Synthetic, ML teams can combine the benefits of real-world data with diverse and realistic synthetic data to achieve desired data distributions. Real-world data remains indispensable to developing performant models. ML teams, however, have encountered several challenges with relying on real-world data alone. Data privacy issues can prevent collected data from being used, collected datasets also inherit the biases of conditions under which data was collected, and collecting enough examples of edge cases and rare scenarios can be both costly and time-consuming. With Scale Synthetic, ML teams can: <ol><li> </li><li>Overcome the challenges of real-world datasets while maximizing the return on investment (ROI) of real-world data by realistically augmenting real-world data or generating fully synthetic data from scratch.</li><li> </li><li> </li><li>Bridge the sim-to-real reality gap by leveraging Scale’s global, human-in-the-loop network that ensures objects in augmented or fully synthetic scenes are realistically placed, and follow a logical path.</li><li> </li><li> </li><li>Seamlessly integrate with Scale’s full ecosystem of products for near-zero friction to add augmented or fully synthetic data into existing ML data pipelines. </li><li> </li></ol>Scale Synthetic currently supports the augmentation or generation of image, video, or 3D point cloud data but custom requests are also encouraged. To date, Scale Synthetic is trusted by leading ML teams such as Kodiak Robotics as well as the U.S. Department of Defense to deliver high-quality synthetic data. Additionally, Scale Synthetic has been recognized in the 2022 Gartner® Innovation Insight for Synthetic Data* report as a Representative Provider in Sample Synthetic Data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/13cf537b-0b89-4b55-885f-069505c8c900/public" alt="Kodiak Robotics Quote">To continue to enhance and build on the foundations of Scale Synthetic, we are also pleased to welcome Joel Kronander as the new Head of Synthetic Data and Vivek Raju Muppalla as the Director of Synthetic Services. Joel joins Scale from Nines, a startup that applied ML and AI expertise to the field of medicine where he served as the Head of Machine Learning. Vivek joins Scale from Unity Technologies, where he was the Director of Engineering, AI and Simulation.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9281bfec-9ad9-49d8-d658-c16381bb0c00/public" alt="Joel Kronander and Vivek Muppalla Quotes">If you would like to get early access for Scale Synthetic, <a href="https://scale.com/synthetic" rel="noopener noreferrer" target="_blank">contact us today</a>. *Gartner, “Innovation Insight for Synthetic Data”, Arun Chandrasekaran, Alexander Linden, Anthony Mullen, Farhan Choudhary, February 7, 2022. Disclaimer- GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Scale Synthetic is the most efficient way to enhance ML performance with synthetic data that complements real-world datasets.

Complementing Real-World Datasets with Diverse and Realistic Synthetic Data

Introducing: Scale Synthetic - Now Available for Early Access 

<h1>Field Engineering at Scale</h1>Scale’s Field Engineering is a global team of creative engineers who collaborate with clients to understand their challenges and architect solutions. Our solutions are rooted in our technical skill, product expertise, and a thorough understanding of our customers’ complex needs. Whether the answer is a bespoke solution to solve a unique problem, or a new developer tool that can scale across Scale, everything we do aims to ensure our customers benefit from the full potential of Scale products and expertise. Our work is at the intersection of customer success, product, and engineering, helping Scale’s customers succeed and reach their business goals with Scale by unblocking them technically. Field Engineers are subject matter experts who know all of the products and technologies Scale has inside and out, and work closely with product managers to influence new product features. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2893973b-04ee-4e9e-9c7e-6304d5b38a00/public" alt="Field Engineering Map"><h1>A Day in the Life</h1> Field Engineers start each day with a team standup. During the standup, we align on daily priorities and check in on our longer-term projects. There’s almost always an urgent customer issue to work on, but when we have downtime, we tackle longer term projects that will help the company move faster in the future. We’ll go over some of these longer term projects further down. One of the more fun parts of our daily standup is ending with the “Word of the Day”. As half of our team is located in Buenos Aires, Argentina, each day someone will share a word that they would like to know how to say in English or Spanish and we’ll share our own take, the different definitions, and any fun stories that go along with it. After the standup, each day can look pretty different. Our days will often include at least a few customer meetings, where we will consult with our existing customers (think: the world’s leading ML companies) on technical aspects of their projects, any challenges they may be facing, and new initiatives coming up. Field Engineering is the customer’s technical contact to ensure all goes well with their work. Covid-permitting, we use lunchtime to socialize with colleagues in the office while we eat all the amazing food that is provided for us. It’s a nice time to decompress and connect. After customer meetings and any urgent issues have been addressed, we focus on our longer-term projects. We’re focused on making the lives of our customers, our internal partners, and ourselves as streamlined and scalable as we can. So, we spend significant time on internal tools, customer SDks, documentation and anything else we can think of to improve Scale’s developer experience. <h1>Field Engineering Initiatives</h1> <h2>Introducing a new and improved Scale Python SDK</h2> 2021 saw the introduction of a new version of our Python SDK. Rebuilt from the ground up to deliver new features faster, this new version includes a set of new ways to interact with the Scale API and improve infrastructure for a smoother developer experience. The tool was built on a new foundation that includes novel abstractions, so you can get more out of the Scale API with less effort. This new foundation allows us to add new functionality faster and better than ever before, so expect more new features from Scale to show up in the SDK with improved documentation and release notes. <h2>Shipping a Lidar Toolkit to make sense of multi-sensor datasets.</h2> The act of fusing multiple cameras, lidar sensors, and radars from a vehicle or robot into one cohesive scene that can be annotated is no easy feat. Historically, helping customers work through the nuances of their data and how to format it for Scale’s API-based workflows was one of the most time-consuming parts of our team’s day. In 2021 we overhauled our Lidar Toolkit and introduced a set of resources, code samples, and debugging tools to enable maximum efficiency for developers working with Lidar data. The toolkit, built on Open3d and Scale’s Python SDK, is full of helper functions that make working with the geometry, camera projections, and coordinate systems much easier to understand and implement than doing it from scratch. The Lidar Toolkit our team has published is now the de facto way all customers share and upload their data to Scale today. <h2>Embracing the Operational Nuances at Scale</h2> Scale is in the business of human-in-the-loop data annotation, and humans + loads of data means a lot of operational overhead comes with the territory. We don’t want to rely on running our business based on custom scripts day in and day out. So, we continually develop, iterate and ship internal tools to make the lives of our customer delivery team as seamless as possible. Some of the tools make facilitating customer requests easier, like moving tasks (tasks are what we call the unit of work we are annotating, like an image) between customer projects, re-sending callbacks to our customer’s endpoints, or canceling tasks en masse. Other sets of tools ensure high-quality data is coming out the other side of our task pipelines. A few years ago, had we wanted to programmatically identify potential annotation errors, a custom script would need to be written, tailored to that client’s specific projects. Over the past 18 months, Field Engineering has developed a framework that can be configured by our non-technical ops counterparts to flag any erroneous tasks in their pipelines. And, better yet, this framework can proactively prevent these tasks by flagging warnings directly in the labeling interface—before the annotations are even submitted. This has become one of our most powerful tools to ensure reasonable data is consistently delivered to our clients. <h1>Could Field Engineering be a fit for you?</h1> If you enjoy leveraging both sides of your brain, the people skills side, and the deeply technical one, Field Engineering may be a promising place to build and Scale your career. We are consistently hiring and growing the team (the team has doubled in size each year) and are interested in candidates with a variety of technical backgrounds and skills. Bonus points are given to candidates who possess a background in Computer Vision or API building and troubleshooting. If you're interested, <a href="https://scale.com/careers?location=All&amp;team=Engineering" rel="noopener noreferrer" target="_blank">explore our open positions today</a>!

A Day in the Life of a Field Engineer at Scale

Field Engineering at Scale

Field Engineering at Scale | A Day in the Life

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

basecamp-agenda

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

research

leaderboards

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog