experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

Engineering

For anyone following AI or machine learning trends in recent weeks, text-to-image generation has taken the spotlight by force. A never-ending stream of synthetically-generated (yes, algorithm-generated!) images is populating social media feeds. This was very different just a couple of months ago, before an invite only beta version of <a href="https://openai.com/dall-e-2/" rel="noopener noreferrer" target="_blank">Open AI’s DALL-E 2</a> was opened to the public in July. With DALL-E available via signup and waitlist, and now with <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">Stable Diffusion</a> released to all as public open-source software, developers have been prolific and building and hosting an array of online image generators. Perhaps the users of these image generators are even more prolific than their authors—libraries of millions of generated images are swiftly gaining popularity, making it easy to browse other people’s creative ideas and find inspiration. We’ve put together a list of the 5 best, openly available online AI image generators using <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">stable diffusion</a> to help you jump right into the mix.<h3><a href="https://beta.dreamstudio.ai/dream" rel="noopener noreferrer" target="_blank">Dream Studio</a></h3>This is the foundational tool released by the company that created Stable Diffusion, Stability AI. The basic version requires a login, but is free to use. The generator offers a ton of settings, including the model version, Cfg scale (how close the image will be to your prompt), steps, number of outputs, image dimensions and more. You can start both from scratch or with a seed image of your choice. Conveniently, Dream Studio keeps a history of all your creations and lets you download the full resolution image. There is also a guide on how to write the best prompts so that the generator achieves the best results.<h3><a href="https://replicate.com/stability-ai/stable-diffusion" rel="noopener noreferrer" target="_blank">Replicate</a></h3>Replicate is a small startup on a mission to make machine learning models easier to use for everyone. They provide a service that helps you use previously uploaded models or upload your own model to run inference on demand, instead of spinning up your own server. With the wave of interest in stable diffusion, they have created a useful web app to use the model in order to generate images as well. Signup is required, but otherwise it is free to use. To date, the model has generated more than 5 million images!<h3><a href="https://creator.nightcafe.studio/stable-diffusion-image-generator" rel="noopener noreferrer" target="_blank">NightCafe Creator</a></h3>NightCafe is a company whose mission is to democratize the creation of art using technology. They have been going strong since 2019, but have recently gained huge momentum with the advancements of text-to-image models. A simple text field invites you to enter your prompt, but advanced customizations are available. You can generate up to 5 images for free, but have to buy credits afterwards. NightCafe also offers other creation models besides stable diffusion, such as VQGAN+CLIP, which was their most popular before 2022. They offer a huge collection of artworks created by other users, a Discord server and a Subreddit for discussions.<h3><a href="https://www.phase.art/" rel="noopener noreferrer" target="_blank">Phase.art</a></h3>This project is a simple text-to-image generator using stable diffusion, with only a simple input for a prompt. There are suggested categories and tokens that can be used to compose the prompt and get inspiration. Creation requires login, which gives you 10 credits (equivalent to 10 images generated) and also enables you to revisit your previously generated images. After the first 10, you have to buy additional credits to keep creating. Phase.art offers a Discord server to connect with fellow text-to-image artists.<h3><a href="https://www.craiyon.com/" rel="noopener noreferrer" target="_blank">Craiyon</a></h3>This AI image generator was previously known as DALL-E mini, and became wildly popular as the original DALL-E by OpenAI is not yet openly available to everyone. The authors, Boris Dayma and Pedro Cuenca, attempted to implement a similar image generator, but using an open source model. This is actually not a stable diffusion model and the results are sometimes a bit less polished (especially for faces) when compared to other image generators. However, the service is entirely free and enables you to download your creations, even if only as a screenshot and not in full resolution.<h4>Not sure how to get started or what prompts to use?</h4>Fortunately, with web apps like <a href="https://lexica.art/" rel="noopener noreferrer" target="_blank">Lexica.art</a> and <a href="https://arthub.ai/" rel="noopener noreferrer" target="_blank">ArtHub.ai</a> there are several fast growing libraries of images that other people have created using stable diffusion including the relevant prompt they used in the AI image generator. If you’re in lack of ideas, go ahead and browse the prompts that other users submitted to get started!If you'd like to learn more about Stable Diffusion and other generative image models, check out our <a href="https://scale.com/guides/diffusion-models-guide" rel="noopener noreferrer" target="_blank">Guide on Diffusion Models here</a>.

image.jpg

5 tools to assist you in generating beautiful art from text prompts with ML models.

5 open, freely available AI image generators to get you off the ground.

How to Make Your Own AI-Generated Images

<a href="http://wwww.scale.com/" rel="noopener noreferrer" target="_blank">Scale AI</a>, the data infrastructure for AI, today announced the achievement of the “In Process” designation for its AI Data platform by the Federal Risk and Authorization Management Program (FedRAMP), a crucial milestone in the FedRAMP certification process. This designation indicates that Scale is on track to achieving a FedRAMP authorized product within the next 12 months.&nbsp;The AI Data platform is seeking authorization at the FedRAMP High Impact Level, enabling Scale AI to support the U.S. government’s most sensitive data and missions.&nbsp; Scale serves customers of all sizes across a wide range of industries. Reaching this milestone on the path to authorization is invaluable to Scale’s ability to enable federal, state and local governments in the adaptation of AI applications.&nbsp; FedRAMP is the U.S. Government’s standardized approach to providing security authorizations for the adoption and use of cloud services by the federal government. The government-wide program’s goal is to accelerate the use of secure cloud technologies by government agencies. The “In Process” designation is an important step towards Scale AI’s full FedRAMP authorization at the FedRAMP High Impact Level. Scale’s commitment to security is one reason why leading ML teams from private sector organizations and the U.S. Government rely on Scale AI to accelerate the development of their AI applications. If you’d like further details, please contact us at fedramp@scale.com. &nbsp;

Scale is on track to achieving a FedRAMP authorized product within the next 12 months

Scale achieved a crucial milestone in the FedRAMP certification process

Scale Achieves FedRAMP ‘In Process’ Designation for its AI Data Platform

Delivering a great buying experience makes all the difference in whether or not a user will convert to a customer. A core part of the eCommerce buying experience is creating an exceptional user journey from initial discovery to purchase. To learn more about accelerating the customer experience, we spoke to Nick Beighton, former CEO of ASOS, about his experience bringing a data-driven approach to online retail.&nbsp;<h2>Data Attribution as the Foundation for the Customer Experience&nbsp;</h2>For a brand to be successful, it needs to have agility in its business and technology to scale quickly to improve shopping experiences based on customer needs. At the core of agility for eCommerce companies, digitization is vital on both the front and back end. Nick explains,“Digitization is really important on the front end, but it’s also important on the back end in terms of how you digitize your processes, how you use data to drive greater personalization, and how you make experiences more efficient, faster, and intuitive.”&nbsp;Nick explains that digitizing the front end encompasses creating intuitive and personalized experiences. As customers engage with the digital offer, the experience becomes more customized, from ads to product recommendations. Scaling operational processes on the back end centers on strong data discipline. A key part of this is maintaining a large catalog of product attributes, which encompasses everything from physical characteristics such as size, color, and material to brand and product descriptions.“Digitizing the back end can only be done at scale through strong data attribution, strong control of your product master files, and the attributes within it. To attribute that data correctly, you can then turn your customer care from a cost center, to a profit center, by using the data to add on more sales and think differently about that interaction with the consumer.”By starting with strong data attribution, eCommerce companies can better connect marketing and SEO investments to the entire customer experience.<h2>Investing in Product Catalog Data at ASOS</h2>Nick applied a data-focused approach to scaling ASOS by first focusing on the search journey on the ASOS website. A core area that the ASOS team investigated was the differing conversion rate between customers who used the search bar to find a product and customers who used the website navigation: customers who used the search bar had double the conversion rate. As a result, the team explored how to maximize the search journey. One discovery was how product attribute data impacts the customer search journey.“We realized that the search journey didn’t have all the right data attributes to serve the most relevant search. So if you’re searching for men’s shoes, you might end up with women’s shoes too. We had to clean up the data on our main website to improve the search journey, and the recommendations served to the customers. That was one of the early ways we started investing in data attribution and product catalogs through the journey.”Nick spearheaded an ML team that maximized product attribution data, and the investment proved successful. By improving product catalog data, conversion moved from 2% to over 7% on average and quadrupled the number of major SKUs added to the site. Nick explains, “Data attributes and attribution of product information was critical to maintaining and improving that conversion while simultaneously adding more and more products.”&nbsp;Investing in product attributes proved instrumental at ASOS. The team connected robust product catalog data to maximize ROI and focus on conversion and customer experience. By building strong AI operational processes, the ASOS team optimized how they served customers.&nbsp;<h2>How eCommerce Companies Can Get Started with a Data-Driven Approach</h2>How do eCommerce companies get started with having strong data discipline? What are some of the best ways to implement these practices across an organization? Nick explains the critical areas for teams to focus on are investing in data attribution early, experimentation, and establishing product goals:<ol><li>Invest in data attribution: Strong product data attribution is at the core of improving the customer journey. ML teams can address long-tail product and content issues by investing in data attribution.&nbsp;&nbsp;</li><li>Experiment to get started: Don’t limit experimentation with tech solutions that only have strong business cases and ROI. You may not know what experiment will give you an exponential return, so it’s essential to have multiple experiments and tests in parallel and review the data to understand the impact.</li><li>Align on product goals: Investing in product catalog data and tying the data attribution goals to product performance metrics is crucial to transforming the business. By working closely with product teams, teams working in eCommerce can provide a direct correlation with internal metrics.&nbsp;</li></ol>Nick emphasizes the importance of data discipline is all about creating the best end experience for the customer,“A customer will rarely say, ‘I love your data attribution,’ right? She will say, ‘I feel the site gets me and gives me all the things I want when I’m searching for it.’ That’s the consumer mindset goal, and how to achieve it is through digitalization and data discipline.”&nbsp;At Scale, we believe investing in data is the key to unlocking success for<a href="https://scale.com/retail" rel="noopener noreferrer" target="_blank"> eCommerce companies</a>. If you’re interested in learning more about boosting your business’s conversion, speak to our team today.

Nick Beighton, former CEO of ASOS, discusses how he transformed the company from 200 million to 4 billion in revenue by investing in product catalog data. 

How Nick Beighton Transformed ASOS by Investing in Product Catalog Data

Warm-Start Your Vision Project: 10 Robust Datasets for Pre-Fine-Tuning

Warm-Start Vision Projects with Robust Pre-Fine-Tuning Data

The 10 Best Public Datasets for Object Detection in 2022

&ldquo;This is the Unix philosophy: write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.&rdquo;
&mdash;<a href="https://www.azquotes.com/author/50799-Douglas_McIlroy" target="_blank" rel="noopener noreferrer">Douglas McIlroy</a>
Computer software came to maturity in the late 1960s. While most programs to date had been written monolithically, mostly from scratch, uniquely built for building-sized mainframes, several pioneers were starting to standardize a new way of thinking. Some of the programs they wrote have familiar names (e.g. <code style="background-color: rgb(240, 240, 240);">grep</code>, <code style="background-color: rgb(240, 240, 240);">diff</code>) and live on to this day as a toolset for computer programmers handed down through the mists of time, their origins unclear but their utility unquestionable. Their continued success is a microcosm of the continued success of the UNIX philosophy, originated by the titans of computer software Doug McIlroy and Ken Thompson in the early 70s. They noted the power of composable tools that operate on and modify streams of text.
A text stream program comes with automatic ease of collaboration and organization: individuals can change the internals of different programs without worrying about breaking the work of others. After all, the interface is standard. Text stream programs also come with extensibility built in for free. When the rudimentary data format of I/O never changes, you can string together long sequences of nested shell invocations to build ornate software <a href="http://www.catb.org/~esr/writings/cathedral-bazaar/" target="_blank" rel="noopener noreferrer">cathedrals</a> just by wiring simple programs together that modify each other&rsquo;s behavior. Text became the most common lever by which mankind would work computers to do their bidding.
Decades later, researchers began to conquer many previously unsolvable language and text tasks in unison. In February 2019, a precocious rising talent in AI research named Alec Radford came out with a seminal paper on language modeling that made him and his coauthors instant legends not just in the AI community but among a broad audience of technologists. Their patron institution, OpenAI, decided that the powerful model software created for the paper was the first AI plausibly too dangerous to simply release into the wild for fear of what malign actors might do with it. With the slow, eventual release of this model, <a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" target="_blank" rel="noopener noreferrer">GPT-2</a>, they ushered in the age of what we might call large language models (LLMs).
In a previous iteration of the machine learning paradigm, researchers were obsessed with cleaning their datasets and ensuring that every data point seen by their models is pristine, gold-standard, and does not disturb the fragile learning process of billions of parameters finding their home in model space.&nbsp;Many began to realize that data scale trumps most other priorities in the deep learning world; utilizing general methods that allow models to scale in tandem with the complexity of the data is a superior approach. Now, in the era of LLMs, researchers tend to dump whole mountains of barely filtered, mostly unedited scrapes of the internet into the eager maw of a hungry model.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F6654d3839c574311b10ba2104db8de45">
The resultant model displays alarming signs of general intelligence &mdash; it&rsquo;s able to perform many sorts of tasks that can be represented as text! Because, for example, chess games are commonly serialized into a standard format describing the board history and included in web scrapes, it turns out large language models <a href="https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-game/" target="_blank" rel="noopener noreferrer">can play chess</a>. Noticing these strange emergent properties, other researchers have pushed this general text processing faculty to the limits: a recent&nbsp;<a href="https://say-can.github.io/" target="_blank" rel="noopener noreferrer">paper</a>&nbsp;utilizes a large language model's common sense reasoning ability to help a robot make decisions. They simply submit a list of potential moves as a text stream and acquire the approval or lack thereof of the AI. The text model knows, for example, that finding a sponge before washing the dishes is more reasonable than the other way around and commands the robot as such. Seeing these quite disparate tasks being tamed under one unlikely roof, we have to ask &ndash; what other difficult problems can simply be transcribed into text and asked to an oracular software intelligence? McIlroy must be smiling somewhere.
Several are venturing down this intellectual avenue, asking in the spirit of the UNIX pioneers if the general intelligence of large language models can become the core modular cognitive engine for their various software programs and businesses. These intrepid entrepreneurs and engineers who require that the world&rsquo;s best artificial intelligences behave in a certain way to find value for their users are giving birth to the brand new discipline of prompt engineering. Dennis Xu, a founder of <a href="http://mem.ai/" target="_blank" rel="noopener noreferrer">mem.ai</a>, is one such frontiersman pounding down the doors of the future: his company works on a self-organizing note-taking app that works as an intelligent knowledge store, with an AI that curates, tags, and connects different notes.
Dennis&rsquo;s team knows it doesn&rsquo;t make any product sense to train their own model to solve their core cognitive task. Trying to understand the breadth of human knowledge from scratch in order to write note summaries and mark thematic tags would be an absurdity. Rather, why not use one of humanity&rsquo;s most powerful robotic minds readily available to the public over an API? Where collecting one's own dataset for a downstream application like a note-taking app will require a massive pre-existing user base, <a href="https://openai.com/blog/gpt-3-apps/" target="_blank" rel="noopener noreferrer">GPT-3</a> (codename: &ldquo;davinci&rdquo;), perhaps the world&rsquo;s most celebrated language model, is out there and has been trained on half a trillion tokens encoding the breadth of common sense reasoning and language task understanding. We can simply ask it to perform tasks like &ldquo;summarize this document,&rdquo; &ldquo;list the topics this note covers,&rdquo; and others. Of course, it&rsquo;s never so simple. We have to provide a few examples of notes and their corresponding summaries or topic tags to ensure the model is thinking in the right direction.The basics of the prompt generally consist of a call to action and a few examples that define the space of the task, but this barely scratches the surface.
For anyone who works closely with these models, it becomes clear that the vast and comprehensive training gauntlet that creates these technological leviathans embeds some difficult behaviors. While the breadth of example material spawns broadly intelligent digital creatures capable of working on a vast range of text tasks, it can require significant prodding, cajoling, and pleading to get these models into the right mood for the particular task at hand. In the early days after the release of GPT-3, many noticed that the AI seems to become more cogent if you start the completion prompt with something along the lines of &ldquo;this is a conversation between a human and an extremely intelligent artificial intelligence&rdquo; versus not including the modifier. Sometimes, GPT seems to be hiding the true depth of its intelligence for the sake of <a href="https://twitter.com/ESYudkowsky/status/1285333002252247040?s=20&amp;t=3NsHEt34xW8qvgnrWTXLIA" target="_blank" rel="noopener noreferrer">the character it is playing</a> at the moment. The prompt to GPT may even make the difference between being able to solve <a href="https://twitter.com/arankomatsuzaki/status/1529278580189908993" target="_blank" rel="noopener noreferrer">arithmetic puzzles or not</a>. In another insane quirk of language modeling, text-to-image models seem to create higher quality art if you specify that it&rsquo;s from the &ldquo;<a href="https://www.unrealengine.com/en-US" target="_blank" rel="noopener noreferrer">unreal engine</a>&rdquo; game studio, known for its photorealistic style. A self-supervised language model can be thought of as a vast ensemble of many models that play many characters with different personalities due to the breadth of the document distribution and dizzying variety of writing styles seen in the half a trillion internet-scraped tokens that it studiously observes at training time.&nbsp;
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3f7768f3370144b48eb9f190af54d0e1">
Every prompt a user feeds to a large foundation model instantiates a new model, one more limited in scope: it enters into a <a href="https://arxiv.org/abs/2207.10342" target="_blank" rel="noopener noreferrer">different conditional probability distribution</a>, or in other words, a different mood. A language model&rsquo;s mood gives birth to a new piece of composable modular software that takes in a text token stream and leaves another as output. The elegant power of UNIX-style software comes from the simple but incredibly powerful primitive known as the pipe. It transfers the output of one program to become the input or modifier for another. The output of one prompted query, which asks for a summary of a note, can easily be aggregated and sent to another prompted model that asks to order the summaries by relevance for a given search topic.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=e1c179bb6169ac2f09a28522a29860f4.png&amp;f=webp&amp;q=75&amp;w=3840">
The pure energy of modular software creation is buzzing in the prompt engineering war room, turning human will into repeatable programs with far reduced effort: find a prompt, don&rsquo;t write a program. Dennis and his team work hard to find the right incantation to coax GPT-3 into the right mood to perform the task at hand. Like most engineering, it&rsquo;s an ongoing optimization process. The structure of OpenAI&rsquo;s pricing model demands that the end users have to take care to use as few tokens for the prompt as possible since that&rsquo;s the basic unit of cost in the GPT-3 API. A prompt that uses only 100 characters to get GPT-3 doing the right thing is preferable to one that takes 400. Prompt engineers must play a delicate game of asking the model to think well while shortening their incantations.
These resourceful engineers are adapting some of the methods of the machine learning world to the prompt engineering world: an old bag of tricks under a new layer of abstraction. For example, they create tools to store &ldquo;test sets&rdquo; of difficult task examples to assess the quality of a new prompt (or an old prompt after a model change). Let&rsquo;s say that a prompt engineer has an idea for shortening the text prompt that asks the machine for document summaries. It could potentially save thousands a month on API fees. She tries it on a gauntlet of twenty examples (&ldquo;the test set&rdquo;) that the team has concluded are interesting and duly illustrate the performance of the model. After checking that all twenty examples are producing successful results (or at least the performance is not worse than before), she can roll out her change in an automated way into the product. Using these methods to assure safety and speed up experimentation, the mem.ai team creates such innovations constantly. They may even cleverly compose several tasks in the same query (multi-task inference!), further decreasing costs. All input is text, so there are several obvious ways to recombine and reroute queries into different prompts. All output is text, so human reviewers can quickly examine a few test examples and check for correctness and performance.
It is easy to bet against new paradigms in their beginning stages: the Copernican heliocentric model of cosmology was originally less predictive of observed orbits than the intricate looping geocentric competitor. It is simple to play around with a large language model for a bit, watch it make some very discouraging errors, and throw in the towel on the LLM paradigm. But the inexorable <a href="https://arxiv.org/abs/2001.08361" target="_blank" rel="noopener noreferrer">scaling laws</a> of deep learning models work in its favor. Language models become more intelligent like clockwork due to the tireless work of the brilliant AI researchers and engineers concentrated in a few Silicon Valley companies to make both the model and the dataset larger. OpenAI&rsquo;s new model available in beta (codename: davinci2) is dramatically smarter than the old one unveiled just two years ago. Like a precocious child, a more intelligent model requires less prompting to do the same job better. Prompt engineers can do more with less effort over time. Soon, prompting may not look like &ldquo;engineering&rdquo; at all but a simple dialogue with the machine. We see that the gradient points in the right direction: prompting becomes easier, language models become smarter, and the new universal computing interface begins to look inevitable.
Sometimes, despite our best efforts, we must leave the realm of text. There are endless forms of computation that operate on other media &mdash; much of the digital content that exists is photos, videos, and graphical interfaces, and users adore all of it. We can already see that the power of natural language is being leveraged to ply and manipulate these as well. The rapidly advancing line of models called <a href="https://openai.com/dall-e-2/" target="_blank" rel="noopener noreferrer">DALL-E</a> and <a href="https://github.com/CompVis/stable-diffusion#stable-diffusion-v1" target="_blank" rel="noopener noreferrer">Stable Diffusion</a> are proliferating on the internet, generating incredible social media buzz about the advent of AI art and what it means for humanity. These tools allow us to take snapshots of our imaginations, communicate them only through the versatile interface of text, and command powerful intelligences to reify that vision into pixels. This technology is remarkable &mdash; even the ancients conceived of a gift that could take a picture of the mind, and now it's available to anyone that can type.
Others are pursuing even more audacious efforts to command other modalities under text. The team at <a href="https://www.adept.ai/post/introducing-adept" target="_blank" rel="noopener noreferrer">Adept.AI</a>, which includes the <a href="https://www.linkedin.com/in/ashish-vaswani-99892181/" target="_blank" rel="noopener noreferrer">legendary</a> <a href="https://www.linkedin.com/in/nikiparmar/" target="_blank" rel="noopener noreferrer">authors</a> of the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" target="_blank" rel="noopener noreferrer">seminal paper</a> introducing the Transformer architecture used in all modern language models, notes that LLMs, for all their generalized intelligence, cannot take seamless action in the graphical interface world and are aiming to fix that discrepancy posthaste. An example listed on their blog: you tell the computer, in text, to &ldquo;generate our monthly compliance report&rdquo; and watch as your will unfolds on screen, as a computer mind converts text into action on an Excel or Word interface. In this paradigm, art, graphics, and GUIs themselves will be commanded under the modality of text.
Slowly but surely, we can see a new extension to the UNIX credo being born. Those who truly understand the promise of large language models, prompt engineering, and text as a universal interface are retraining themselves to think in a new way. They start with the question of how any new business process or engineering problem can be represented as a text stream. What is the input? What is the output? Which series of prompts do we have to run to get there? Perhaps we notice that the person in charge of the corporate Twitter account is painstakingly transforming GitHub changelogs into tweet threads every week. There&rsquo;s a prompt somewhere that solves this business challenge and a language model mood corresponding to it. With a smart enough model and a good enough prompt, this may be true of every business challenge. Where textual output alone is truly not enough, we train a joint embedding model such as DALL-E that translates text input into other domains.
The most complicated reasoning programs in the world can be defined as a textual I/O stream to a leviathan living on some technology company&rsquo;s servers. Engineers can work on improving the quality and cost of these programs. They can be modular, recombined, and, unlike typical UNIX shell programs, are able to recover from user errors. Like shell programs living on through the ages and becoming more powerful as underlying hardware gets better, prompted models become smarter and more on task as the underlying language model becomes smarter. It&rsquo;s possible that in the near future all computer interfaces that require bespoke negotiations will pay a small tax to the gatekeeper of a large language model for the sheer leverage it gives an operator: a new bicycle for the mind. Even today, many of us find ourselves already reliant on tools like <a href="https://github.com/features/copilot" target="_blank" rel="noopener noreferrer">GitHub Copilot</a>, despite their very recent invention, to read, suggest, and modify text for the creation of our more traditional software programs. Thousands or millions of well-crafted language model moods will proliferate, interconnected, each handling a small cognitive task in the tapestry of human civilization.
"The language model works with text. The language model remains the best interface I've ever used. It's user-friendly, composable, and available everywhere. It's easy to automate and easy to extend."
Follow the author&nbsp;<a href="https://twitter.com/tszzl" target="_blank" rel="noopener noreferrer">@tszzl</a>&nbsp;on Twitter.

cover_ Text Is the Universal Interface.png

Roon guest-authors a post on large language models and the future of computing.

Text became the most common lever by which mankind would later induce computers to do their bidding.

Text Is the Universal Interface

Aerin is an engineering manager at Scale AI, leading a Catalog ML team. Below are Aerin's stories about what she does and why she loves working at Scale.
<h2>What brought you to Scale?</h2>
Before I joined Scale, I worked on a project called NL2SQL which stands for "Natural Language to Structured Query Language" at Microsoft. NL2SQL is a translation problem that automatically converts a user's spoken-language query into SQL.
Just a few months after my team had started working on it, we were beating the SOTA (state-of-the-art) performance on research benchmarks. We achieved this by evaluating different ML model architectures and improving them. In other words, we somewhat solved the NL2SQL problem in research after only a few months of work.
However, in production, we weren&rsquo;t breaking the state of the art. The model still made a few mistakes on the user's input. And it became clear to me that changing the model architectures would only get us so far, and that the most crucial thing we could do to improve the performance of NL2SQL was to improve the training data. This was done by generating edge cases synthetically and therefore increasing the training data coverage.
So, we started synthetically generating training data (NL-SQL pairs) by following syntax requirements, snowflake schema, etc. However, the data that we have artificially created couldn't possibly cover every possible scenario or edge case. Because the true edge cases are ones that you can't predict or plan for, like the questions that people have typed into our search query bar.&nbsp;Another way to increase the coverage of the training data was to label the user inputs and feed them back to the training data. Taking care of that lifecycle, also called an auto-labeling pipeline, was quite an operation. Even though I was given a lot of resources to manage the auto labeling pipeline, I couldn't undertake any other engineering work because it required so much overhead.
So I started looking for the best third-party training data provider in the market, and that's how I met Scale, and the rest is history.
<h2>What do you enjoy most about working at Scale?</h2>
I like that I get to work on problems that are important and challenging enough to customers that they are willing to pay to have them solved. This means that the problems are real.
I also like that Scale's work is always about data, which is my favorite subject. For every business, I believe data is the most valuable and private asset. Say if you're developing a machine learning model, the one and only input that determines how well your model will perform is your data. The fact that customers trust us with their data and allow us to manage it energizes me, and I feel a great deal of commitment and responsibility. Because the data we provide with them is being used as a bloodline for their business, this relationship just can't work without trust. Having a close relationship with customers is also a rewarding part of my job.
Another highlight of my time here at Scale is the everyday connection with my team. When the engineering team works well, it feels like a well-oiled machine, and you will notice tangible results from that. We've done some seemingly impossible jobs together, and it feels amazing to see the whole team pulling together to complete a challenging project. When working on projects, you learn a great deal by openly debating and choosing the best technical solution with your team. If you're an engineer, I believe that seeing and interacting with other engineers is one of the best ways to learn and grow. That's also something I enjoy at Scale.
<h2>What makes working for a startup unique from other jobs?</h2>
I've worked at both a big tech company and a startup, and they are indeed quite different, and each one is unique in its own way.
At a startup, leading a project feels like driving a sports car. You're close to the ground and can feel the movements you make. You can turn quickly, and you can observe fast motion while you are steering. You can see the results of your team's work fairly quickly and clearly. For example, you complete a pilot project successfully, and the customer signs a large contract. There is also a good chance of crashing - for example, you fail to deliver a project and lose a customer. At a startup, you will see the results of what your team does right away.
At a big company, running a project is more like flying a spaceship (I can only imagine this). You&rsquo;re farther from the ground, it's harder to make quick decisions because you are not the only stakeholder, and the course has been set long in advance. There are also very well-built manuals and processes to guide you along the way.
So far, I've enjoyed seeing the immediate, direct effects of my team's work at Scale.&nbsp;
<h2>What's your favorite part about being an engineering manager at Scale?</h2>
One of the things I like about being an engineering manager is that I get to work on large-scale projects with a strong, cohesive team, which results in a scalable solution and a greater overall team output. Apart from the collaborative and multiplier nature of the job, I also enjoy problem solving at Scale.
People think that a software engineer's job is to write code, but I don't think that's a complete picture of an engineer. A more accurate job description would be "a problem solver", since our main goal is to build things that help people solve problems. You won't be able to solve problems if you blindly write code, especially from the wrong angle.
I believe the most important thing to do in order to solve problems is to accurately diagnose them. So the ability to diagnose problems accurately is, in my opinion, the most important skill for an engineer to have. If your diagnosis is wrong, you will rarely be effective in solving actual problems. And you can only make the right diagnosis if you have a full, concrete understanding of the problem, the data, and the use cases.&nbsp;
While surgeons can rely on other professionals like physicians, x-ray technicians, to aid with diagnosis, ML/SW engineers have to diagnose problems on their own. We have to carefully examine the available data, debug the code thoroughly, then choose the best solution and also implement. The ML/SWE profession comes with a significant amount of both power and responsibility.
If you are someone who enjoys doing things like this (proper diagnosis and implementation), you will do well in the field of engineering.
<h2>Tell me about interesting projects that you ran at Scale?</h2>
Every project at Scale is interesting and unique in its own way, as each project has different data modality (3D Lidar, 2D imagery, NLP, tabular, etc.) I enjoy that I get to work on diverse problems and apply the peculiarities of each data modality to ML model development. For example, 3D data often comes with the new axes - yaw, pitch, roll, unlike 2D data and the sensor's effect on the LIDAR data is significant. NLP also has its own quirks, such as different types of tokenizers, vector representations, etc.
These days, my team is working on multi-modal (Language + Vision) problems. For example, given an image of a model wearing clothes, can we describe what she&rsquo;s wearing in detail (e.g. short sleeve, polka-dots, long boots, etc.) in natural language? To this end, one popular research/exploratory approach is to train a multi-task (e.g., Visual Question Answering, Image Captioning, Text-to-Image Generation, Visual Grounding, etc.) model and use it for attribute prediction.
Although I can't talk about the details of the projects I work on for clients, I do make an intentional effort to make my team's work more universally applicable, scientifically relevant, and therefore publishable. The attribute prediction work (link) that was just accepted at CVPR 2022 is a great example. One of the most common requests we get from clients is to assign visual characteristics such as color, shape, or motion to detected objects. But there aren't many researchers working on this topic publicly (though there are a lot of practitioners working on it in industry), and there aren't many benchmark datasets for attribute prediction. So we did in-depth research on this topic, presented our findings at CVPR, and received overwhelmingly positive feedback from the ML community.
The Fitzpatrick 17k project with MIT is also a good example. The Fitzpatrick scale is a 6-point scale that tells how dark your skin tone is. And this project&rsquo;s motives are quite fascinating. Because medical imaging doesn't have much data on darker skin, ML models trained on these datasets have trouble recognizing dermatological illnesses on darker skin.&nbsp;We wanted to bring attention to this problem and encourage more inclusion of images with darker skin tones in datasets. So we annotated 17k clinical images in DermaAmin and Atlas Dermatologico, the two major dermatology atlases, using the Fitzpatrick Scale.&nbsp;
We trained a few models on this new dataset and then looked at how well they work on images with different skin tones. And yes, adding more diverse data to the training set was an effective way to reduce bias.&nbsp;
Because of this work, researchers and practitioners&nbsp;are now a little more likely to take into account and examine biases in training datasets before developing ML models in the healthcare sector. Ultimately, we want to make sure that ML models trained on imbalanced datasets don't make healthcare inequalities worse by accident. With this study, we received an honorable mention award at the CVPR skin imaging workshop.
&hellip;&hellip;..
If you are interested in any of the topics that I have said above or in working to solve intriguing ML/SWE challenges, email your resume at <a href="mailto:aerin.kim@scale.com" target="_blank" rel="noopener noreferrer">aerin.kim@scale.com</a>!

Read Aerin's stories about what she does and why she loves working at Scale

Aerin is an engineering manager at Scale AI, leading a Catalog ML team

How Nick Beighton transformed ASOS into a 4 billion dollar business by investing in product catalog data

We've had an incredible group of people join Scale—and we’re still <a href="https://scale.com/careers?office=All&amp;department=All#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! Meet the engineers who are accelerating the development of AI applications:<h3> </h3><h3>Albert Sheu</h3>Albert is a fifteen-year veteran of the tech and startup industry, including having worked as the second engineer at Quora and the Head of Engineering at Patreon. His experience runs up and down the stack, working on distributed systems, developer experience, WebGL interfaces, and all areas in the frontend and backend. At Scale he developed the early 3D labeling pipeline, and has also worked on Scale’s Nucleus and Synthetic Data products. <h3>Jack Wu</h3>Jack Wu, previously was a manager at Niantic where he built the Platform Infrastructure team from scratch and led the team to rebuild the Niantic million QPS, planet scale Augmented Reality platform on an alternate Cloud provider. Prior to transitioning to management, Jack was an early engineer at Uber and LinkedIn where he spent most of time building large volume data storage, data analytics and data security systems and saw both companies grow more than 10X during his tenure.<h3> </h3><h3>Yuting Duan</h3>Yuting worked at Google as a Staff Software Engineer and Engineering Manager and studied at CMU. She worked on some of the most large-scale consumer-facing products: Google Search and Google Maps and founded 2 full-stack teams in Google to help users find food wherever they are, whenever they are. <h3>Shah Xia</h3>Shah has previously worked at Bytedance where she was the Chief of Staff to the CEO. She was part of the early team to incubate the TikTok app, and later directed its global expansion strategy. She is receiving her MBA from Wharton. <h3>Sam Clearman</h3>Sam Clearman received his PhD in Mathematics at Lehigh for his work in algebraic combinatorics.&nbsp;He worked at Google using statistical methods to improve Android battery life before joining Scale in 2018.&nbsp;At Scale, he has worked on pretty much everything, but he has always returned to marketplace management and incentive design.&nbsp;In his spare time, he flies planes (sometimes upside down), and enjoys making games such as 2etris [<a href="https://2etris.com/" rel="noopener noreferrer" target="_blank">https://2etris.com/</a>], Down for a Block [<a href="http://d4ab.com/" rel="noopener noreferrer" target="_blank">http://d4ab.com/</a>], and Up for a Camel [<a href="https://upforacamel.com/" rel="noopener noreferrer" target="_blank">https://upforacamel.com/</a>].&nbsp; <h3>Sasha Harrison</h3>Sasha Harrison received a Bachelor's Degree in Mathematical and Computational Sciences, and a Master's Degree in Computer Science from Stanford University, where her concentration was in Deep Learning. As a student, she conducted research projects related to algorithmic fairness in computer vision and NLP, and was an intern at Google and Microsoft. At Scale, Sasha has worked across new products and ML, in particular, helping to develop scalable batch inference infrastructure as well as deep learning models for application to the content understanding data annotation vertical. <h3>Calvin Huang</h3>Calvin Huang met Alexandr Wang at the US Physics Camp just before starting to study electrical engineering and computer science at MIT.&nbsp;While at MIT, Calvin did research in SMT solving and natural language processing. After graduation, Alex recruited Calvin to join Scale as an early engineer, where he built out a large portion of the infrastructure stack and developer tooling, and helped the core systems scale as Scale did. <h3>Vivek Muppalla</h3>Vivek is an industry veteran who has worked in and lead intraprenual efforts across startups and large companies. His experience is leading full stack engineering and product teams. Most recently he was an early member of Unity’s AI &amp; simulation team, and helped Unity become a leader in the Synthetic data and Robotics simulation space. At Scale, he is focused on building our Simulation based products.

Meet Scale's incredible engineering talent 

Scale's Talented Engineering Team

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;
That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;
Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.
<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a>
For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;
While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:
<h4>Aggregate evaluation metrics miss critical regressions</h4>
A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.
<h4>Coverage for ML model testing is not well defined</h4>
While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;
Testing pipelines are static and don&rsquo;t evolve with the models
Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.
Accordingly, we built Scale Validate with the following capabilities:
<h3>Scenario Tests</h3>
Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).
<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI">
<h3>Model Comparison</h3>
Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.
<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8">
<h3>Slice Discovery</h3>
Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.
<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">
We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.
As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:
<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">
Getting started with Validate only takes a few steps!&nbsp;
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.

Introducing Scale Validate for Model Testing

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

Within the field of computer vision, semantic segmentation is a close approximation to describing the visual world the way humans perceive it: entities with precise boundaries. For use cases where nuanced scene understanding is mission-critical, like autonomous vehicles and intelligent robots, building models requires massive amounts of highly accurate segmentation masks with pixel-level precision.The exactness required near object edges that makes semantic segmentation valuable also makes it incredibly time-consuming for humans to annotate.&nbsp;Given taskers annotate thousands of images per day, we wanted to accelerate the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotation</a> process without losing fine-grain precision. To accomplish this, we developed a feature called Autosegment. An annotator draws a box around an object and a segmentation mask is automatically generated. The annotator can then adjust the mask before marking it as complete. Under the hood, the subset of the image is sent through a model, which automatically identifies and segments the object of interest.<a href="https://scaleai.wistia.com/medias/lvnzhzfl7b" rel="noopener noreferrer" target="_blank">video</a>The Autosegment functionality was originally developed for autonomous vehicle data, which is a core customer use case at Scale. (You can read more about our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">original Autosegment tool</a> in our previous blog post.) When it was first launched, we saw a 30% decrease in the time it took to produce segmentation masks. However, the use cases for Autosegment have expanded greatly since its introduction, with AV data now constituting only one of many domains for which Scale produces segmentation data. While the underlying model did generalize beyond the autonomous vehicle domain, we observed that it performed noticeably better on objects like street signs, vehicles, and pedestrians than on other types of classes.This article contains a technical deep dive into the machine learning model that powers Autosegment. We’ll walk through some of the choices we made during the development process as well as our recent effort to overhaul the model for improved generalizability. In particular, we will discuss the data-centric approach we took to boost the usefulness of Autosegment across a wider range of use cases.<h2>Model Anatomy</h2>The segmentation model developed to power this tool addresses a problem known as salient object detection wherein the goal is to identify and segment the most “salient” or prominent object in an image. There is subjectivity in this task because an image can easily contain more than one prominent item. The intention behind Autosegment is that the user draws an approximate bounding box around the object of interest. Based on this we can firmly make two assumptions:<ol><li>The target object will be relatively centered within the box</li><li>The target object will extend nearly to the edges of the box, with some nonzero padding</li></ol>We encode these assumptions into the training process. Specifically, images in the training set are cropped in such a way that the target object is always centered with some randomly added padding. For any training image containing multiple segmented objects, each object is cropped separately to produce an individual training observation. <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fe23c12266e384e7ca60a35f93258ba3e">The architecture we chose to implement for this purpose is based on U2-Net proposed by <a href="https://arxiv.org/pdf/2005.09007.pdf" rel="noopener noreferrer" target="_blank">Qin et al.</a> for this type of problem. U2-Net provides two concrete benefits over other architectures we considered.First, it is designed to more effectively take into account both local and global contrast features of the image. Most of the popular feature extractors (VGG, ResNet, etc.) were designed and optimized for image classification. While they can work well for segmentation tasks, they do so by identifying features specific to the semantic classes they are trying to predict. The consequence of this is that they tend not to generalize as well to classes absent in the training set. On the other hand, U2-Net is designed to make use of image features that indicate object saliency. Because of this, while U2-Net may not perform as well at outright classification as more traditional architectures (we did not test this), in our testing it does a good job at identifying salient objects pertaining to obscure classes. This is important for Scale’s customers who often need to annotate data for niche use cases not represented in the training data.Second, we want to carefully avoid the need to stretch or downsample images. Because the goal is pixel-perfect accuracy, the model needs to output a high-resolution segmentation mask. Downsampling, stretching, or otherwise modifying the image in any information-reducing manner before feeding it to the model will typically result in blurrier segmentation masks than desired. To this end, utilizing a fully convolutional network allows for the processing of any image crop size or shape as long as it is large enough to be processed by the encoder phase of the U-Net (and small enough to fit within GPU memory). In addition, unlike some other architectures, U2-Net does not excessively downsample the input with convolution strides or pooling. This helps propagate enough information about the original image through the network to produce high-resolution mask outputs.<h2>Improving Generalizability</h2>Neural networks have a tendency to try and “cheat” by taking shortcuts when learning to solve a problem. Faced with the task of salient object detection, one shortcut is to simply learn how to segment the specific classes present in the training data. For example, if trained exclusively on autonomous vehicle camera data, the Autosegment model only needs to memorize what certain object classes look like, such as cars, trucks, and pedestrians. The resulting model may work for annotating vehicle scenes, but it will not perform well when applied to new object classes because it hasn’t truly learned the concept of “saliency.”<a href="https://scaleai.wistia.com/medias/dhzm8kaohk" rel="noopener noreferrer" target="_blank">Video</a>To overcome this challenge, the model needs to be given training examples for which this shortcut does not work. In the case of Autosegment, we expanded the training dataset to include hundreds of new object types. Neural networks have a finite number of parameters,&nbsp;meaning that there is a limit to how much “knowledge” they can hold. By providing more classes than the network can effectively memorize, the hope is that it will be forced to learn a simpler strategy that can detect saliency across a more general set of images.Our first step towards generalization was to incorporate the open-source <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">Common Objects In Context (COCO)</a> segmentation dataset into the training process. However, while COCO contains a wide variety of object classes, it only contains about 200 thousand annotated images, which is a small fraction of the training data from autonomous vehicle cameras. In addition, COCO segmentation masks are often somewhat imprecise.We then sought to leverage the vast segmentation data that has been produced using Rapid, our annotation platform for rapid experimentation. Rapid supports both full-scene segmentation and partial-scene segmentation. In full-scene semantic segmentation, every pixel in the image is classified into one of multiple classes. In partial-scene segmentation, pixels may be categorized into zero or more classes. Hundreds of thousands of images have been annotated using Rapid.<a href="https://scaleai.wistia.com/medias/15nwn1o6ff" rel="noopener noreferrer" target="_blank">Video</a>Full-scene segmentation often contains classes that are not beneficial to train on. For example, in outdoor scenes the sky is often a distinct class in the segmentation mask but is typically considered less salient than all other objects in the image. While we want to train on a diverse set of object classes, we do not want to train on “background classes” or objects that do not match a reasonable definition of saliency. Neural networks are generally robust to occasional adverse training samples, but in large quantities they can negatively affect training. In order to filter out these undesirable classes, we have taken the time to construct a mapping between a common taxonomy and the customer-specific classes. For example, this mapping helps us recognize that what one customer calls a “bicycle” is the same as what another customer calls a “bike.” From this, we filter out segmentations that do not match a list of approved classes. Partial-scene segmentation is the more common project type on Rapid and mostly avoids this problem of extraneous classes.<h2>Validation</h2>An important consideration for us was that because Autosegment is already in use across a large number of projects and use cases, changing the underlying model risks unknowingly worsening its accuracy on specific categories of images. In theory this type of accuracy loss could occur due to the finite information capacity of the model architecture. As mentioned earlier, the network needs to encode its problem-solving process using a limited set of parameters. If the number of parameters is too small, underfitting will occur and impede accuracy. We combat this possibility in two ways.First, we ensure that the training data is well-balanced. If the overall composition of the dataset is skewed in favor of any particular domain, then the model may tend to overemphasize training samples from that domain. Not only can this result in lower accuracy in other domains, but it could impede our goal of forcing the model to learn a generalized solution to the salient object detection problem.Second, in addition to benchmarking accuracy on the overall collection of images, we also benchmark the model’s performance on subdomains (such as autonomous vehicle data) to ensure it performs well within each of the strata. This essentially serves as “regression testing” for the model. Regression testing is a software development practice to ensure that code changes do not break existing known use cases. To accomplish this, we validate the model on independent test sets representing distinct customer use cases. The crucial use case we monitored during these improvements was autonomous vehicle data:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdb0470f15fa945cda7b946e690837b34">Beyond standard pixel classification metrics, we also calculate the precision and recall of pixels within a small neighborhood of the segmentation edges. These metrics help us assess how well this tooling will actually benefit annotators for whom the most time-consuming aspect of semantic segmentation is the object boundary pixels.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fac64ee3d7a814bc0852260ed4e5721f9">To calculate these edge metrics, we first start with the predicted and ground truth masks. Keeping only the single-pixel boundary of each mask, a <a href="https://en.wikipedia.org/wiki/Dilation_(morphology)" rel="noopener noreferrer" target="_blank">dilation operation</a> is applied with some small threshold. Metrics are then calculated from the dilated edges. The above image visualizes this process.<h2>The feedback loop</h2>Beyond the initial training of the model, data annotated with Scale Rapid continues to get fed back into the training process. This creates a feedback loop: Autosegment helps users to more quickly produce higher quality annotations, and in turn, those annotations serve to continuously improve the automatic segmentation accuracy. Having high-quality training annotations is an indispensable prerequisite to training a high-performing model. This <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> lifecycle is a unique characteristic of Scale’s annotation platform and fundamental to how we achieve best-in-class annotation speed and accuracy.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F03a3cce06e0e4cfba0dd090356f2669d"><h2>How you can use Autosegment</h2>Autosegment is one of the multiple Active Tooling capabilities that exist in Scale’s products. Active Tools provide interactive ML-powered assistance to data annotators. By letting humans make changes to the model inputs in real time, model outputs can be quickly dialed in to match the situation precisely and minimize time spent on the task.You can supercharge your data annotation using <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a>, which offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.

How Scale AI’s computer vision annotation tooling (CVAT) uses machine learning-assisted tooling for faster and more accurate semantic segmentation.

Machine Learning-Assisted Image Semantic Segmentation

Machine Learning Assisted Image Segmentation

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

basecamp-speakers-events

basecamp-agenda

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog