experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

Engineering

Any ML developer who has trained an ML model multiple times e.g. with different hyperparameters, and tried to figure out which iteration is better, is already familiar with the basics of model comparison. Before her model can deploy to production, an ML developer needs to be confident that it would perform better than the previous model. However, this is easier said than done. Comparing models with the intent of figuring out which one is better is a complex task. Most ML developers take a rather rudimentary approach, which creates a significant risk of shipping the wrong model.In this blog, we will go through the core challenges associated with model comparison and discuss strategies to mitigate them. Before we do, though let’s take a quick look at the basic approach adopted my most ML developers:<h3 class="ql-align-justify">The basic approach</h3>Let’s say we have recently trained two versions of an object detection model with a different learning rate and now want to find out which one is better. Here’s what that process looks like for most ML engineers:<ol><li class="ql-align-justify">Choose a standard aggregate metric e.g. mAP (mean average precision), <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-1</a> score, precision or recall&nbsp;</li><li class="ql-align-justify">Apply <a href="https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c" rel="noopener noreferrer" target="_blank">non-max suppression</a> to reduce the number of candidate predictions, ideally leaving the best one accessible and viewable to the user</li><li class="ql-align-justify">Match each of the resulting predictions with ground truth using intersection over union</li><li class="ql-align-justify">Set minimum IoU and confidence scores thresholds to obtain class confusions</li><li class="ql-align-justify">Using these confusions—True Positives (TPs) &amp; False Positives (FPs)—calculate each model’s aggregate metric&nbsp;&nbsp;</li><li class="ql-align-justify">The candidate model with the highest aggregate metric score is best and the one you’ll want to deploy&nbsp;&nbsp;</li></ol> <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F17406fbd379046e08c2882ab8590acb4"> <img src="https://lh5.googleusercontent.com/AYGlxcGUVLCh48j-UXIyixhm5qtp4hZTrGtycJQm6ejAuV-Qx3z_ibyj_nWS-ACIEsTmLJ4hS2xBAoQy1Xgjrndgzoxud3o2wVT3gO7c4NYBrAgpq1tYASskbRH_0oZtzTTr-NHC1Bf_lS62mulaY_u3Snf2KrxtQ03wIFubYFLhbVku1hPB9sqOZA">While such a one-dimensional and aggregate comparison is helpful thanks to its simplicity, on its own it’s definitely not sufficient to identify the best model. An aggregate approach obfuscates performance over critical rare cases by assuming all data is equally important. Finding the best model is a complex multidimensional process and more often than not relies on the developer’s subjective opinion. Over the past years, we have worked closely with many ML teams on this topic and have identified strategies to systematically find the best model. It all starts with finding the right metric: <h3 class="ql-align-justify">💡Deciding on the right metric&nbsp;</h3>More often than not, ML engineers choose the standard metrics—for example mAP (referenced above) for object detection. However, metric selection is one of the most important decisions when evaluating and comparing models. The metric must capture the nature of the application and the business expectations from the model. For example, if you are working on a cancer detection problem, you would want to detect as many cancer cases as possible even if it means in a few scenarios, you classify a healthy person as having cancer. So “Recall” would be a great metric. However, if you are performing email spam detection then it’s acceptable to miss a few spam emails but not okay to classify a legit email as spam. So “Precision” would be the metric of choice. Typically, you’ll need to make a trade-off between precision and recall. In these situations, tools like the PR (precision-recall) curve or <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-scores</a> can help better capture model performance.&nbsp; To get a feel for these metrics, try computing a sample metric on a slice of data in <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=best_model" rel="noopener noreferrer" target="_blank">Scale Validate</a>.<a href="https://scaleai.wistia.com/medias/9sp4zf58r3" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/9sp4zf58r3</a> <h3 class="ql-align-justify">🔎 Finding the optimal operating point</h3>Operating point is just another name for input controls which determine confusion calculations e.g. for object detection the operating point is a combination of minimum IoU and Confidence thresholds, if the thresholds are 0.5 then the evaluation logic will only consider objects with &gt;0.5 confidence and &gt;0.5 IoU. So you would want to ensure that these thresholds are set such that your model is giving the lowest number of confusions and the highest performance e.g. the highest F1 score. However, the extent to which you can reduce or increase these thresholds depends on your application. For example, you don’t want a low IoU threshold for object detection because that would mean large errors in location &amp; size of the detected objects. Therefore, you need to find the application constraints first and then find the optimal point for each candidate model within those constraints. To find the optimal point you can traverse over each sensible range of these input controls and look at the resulting performance metric. Once you have the optimal operating points you can do an apples to apples comparison of the aggregate scores. Try varying the IoU &amp; Confidence thresholds of this <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">sample model on Scale Nucleus</a> and see the effects on the confusion matrix and PR curve.&nbsp;<a href="https://scaleai.wistia.com/medias/x2yaorc7f0" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/x2yaorc7f0</a> <h3 class="ql-align-justify">🧮 Complexity of multiple metrics</h3>In most instances it is hard to capture model performance with a single metric. For example, for obstacle detection you don’t just want to detect an object but also be very accurate with the location and size. Therefore, on top of an accuracy metric such as recall, you would also want to track average IoU to check for the tightness of the predictions being produced. The challenge of doing model comparison with multiple metrics is that no model might perform the best on all metrics. To tackle this problem, the simplest method is to do a weighted aggregation of these metrics and obtain a single score. This is exactly what the F-scores do for the metrics of precision &amp; recall. If you want to assign equal importance to both metrics you can compute F1 score, otherwise you can use F0.5 or F2 scores to weigh one metric more than the other. You can read more about F-beta scores <a href="https://machinelearningmastery.com/fbeta-measure-for-machine-learning/" rel="noopener noreferrer" target="_blank">here</a>. Try <a href="https://dashboard.scale.com/nucleus/model-comparison?analyze_dataset_id=ds_bwkezj6g5c4g05gqp1eg&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;analyze_compare_model_id=run_cabt1b5e3xf005r21sxg" rel="noopener noreferrer" target="_blank">model comparison over multiple metrics</a> on Scale Validate.<a href="https://scaleai.wistia.com/medias/g5ie6sy2bl" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/g5ie6sy2bl</a> <h3 class="ql-align-justify">📊 Dealing with data inequality&nbsp;</h3>Real world data is never equally distributed or equally relevant. Some subsets or slices are just more important than others for the model in question. And more often than not these high importance subsets are underrepresented and hence small in size. The way aggregate metrics are calculated, they assume that all data is equally important and obscure performance on rare important samples. For example, a pedestrian detection model can have an mAP of 0.9. But that 0.9 can be very misleading if the examples it fails on are the ones where pedestrians are on a crosswalk! Similarly when comparing two models, the aggregate performance might go up but that doesn’t mean that the new model is better. To address this it is important to first identify these longtail situations and curate small subsets which act as data unit tests similar to unit tests for software. Then models can be compared with regards to how well they perform over these unit tests in addition to the aggregate metric. For example, model B is only better than model A if it improves on the aggregate and doesn’t regress on any of the data unit tests. Read more about data unit tests <a href="https://scale.com/blog/testing-models-validate" rel="noopener noreferrer" target="_blank">here</a> and try to compare models on specific unit tests in Scale Validate.<a href="https://scaleai.wistia.com/medias/0kekfm8stb" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/0kekfm8stb</a> <h3 class="ql-align-justify">To summarize:&nbsp;&nbsp;</h3><ol><li class="ql-align-justify">Choose metrics which are representative for your application</li><li class="ql-align-justify">Carefully analyze and define thresholds in your models</li><li class="ql-align-justify">Ensure that critical subsets of data have an impact on the evaluation score</li></ol>We have built Scale Validate with these strategies in mind. You can try it for free on your data or an open source dataset <a href="http://scale.com/validate?utm=n" rel="noopener noreferrer" target="_blank">here</a>.

image.jpg

 Model comparison can be hard—here is how you can find your best model!

Finding Your Best Model Is Tricky—Here's How to Do It!

For anyone following AI or machine learning trends in recent weeks, text-to-image generation has taken the spotlight by force. A never-ending stream of synthetically-generated (yes, algorithm-generated!) images is populating social media feeds. This was very different just a couple of months ago, before an invite only beta version of <a href="https://openai.com/dall-e-2/" rel="noopener noreferrer" target="_blank">Open AI’s DALL-E 2</a> was opened to the public in July. With DALL-E available via signup and waitlist, and now with <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">Stable Diffusion</a> released to all as public open-source software, developers have been prolific and building and hosting an array of online image generators. Perhaps the users of these image generators are even more prolific than their authors—libraries of millions of generated images are swiftly gaining popularity, making it easy to browse other people’s creative ideas and find inspiration. We’ve put together a list of the 5 best, openly available online AI image generators using <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">stable diffusion</a> to help you jump right into the mix.<h3><a href="https://beta.dreamstudio.ai/dream" rel="noopener noreferrer" target="_blank">Dream Studio</a></h3>This is the foundational tool released by the company that created Stable Diffusion, Stability AI. The basic version requires a login, but is free to use. The generator offers a ton of settings, including the model version, Cfg scale (how close the image will be to your prompt), steps, number of outputs, image dimensions and more. You can start both from scratch or with a seed image of your choice. Conveniently, Dream Studio keeps a history of all your creations and lets you download the full resolution image. There is also a guide on how to write the best prompts so that the generator achieves the best results.<h3><a href="https://replicate.com/stability-ai/stable-diffusion" rel="noopener noreferrer" target="_blank">Replicate</a></h3>Replicate is a small startup on a mission to make machine learning models easier to use for everyone. They provide a service that helps you use previously uploaded models or upload your own model to run inference on demand, instead of spinning up your own server. With the wave of interest in stable diffusion, they have created a useful web app to use the model in order to generate images as well. Signup is required, but otherwise it is free to use. To date, the model has generated more than 5 million images!<h3><a href="https://creator.nightcafe.studio/stable-diffusion-image-generator" rel="noopener noreferrer" target="_blank">NightCafe Creator</a></h3>NightCafe is a company whose mission is to democratize the creation of art using technology. They have been going strong since 2019, but have recently gained huge momentum with the advancements of text-to-image models. A simple text field invites you to enter your prompt, but advanced customizations are available. You can generate up to 5 images for free, but have to buy credits afterwards. NightCafe also offers other creation models besides stable diffusion, such as VQGAN+CLIP, which was their most popular before 2022. They offer a huge collection of artworks created by other users, a Discord server and a Subreddit for discussions.<h3><a href="https://www.phase.art/" rel="noopener noreferrer" target="_blank">Phase.art</a></h3>This project is a simple text-to-image generator using stable diffusion, with only a simple input for a prompt. There are suggested categories and tokens that can be used to compose the prompt and get inspiration. Creation requires login, which gives you 10 credits (equivalent to 10 images generated) and also enables you to revisit your previously generated images. After the first 10, you have to buy additional credits to keep creating. Phase.art offers a Discord server to connect with fellow text-to-image artists.<h3><a href="https://www.craiyon.com/" rel="noopener noreferrer" target="_blank">Craiyon</a></h3>This AI image generator was previously known as DALL-E mini, and became wildly popular as the original DALL-E by OpenAI is not yet openly available to everyone. The authors, Boris Dayma and Pedro Cuenca, attempted to implement a similar image generator, but using an open source model. This is actually not a stable diffusion model and the results are sometimes a bit less polished (especially for faces) when compared to other image generators. However, the service is entirely free and enables you to download your creations, even if only as a screenshot and not in full resolution.<h4>Not sure how to get started or what prompts to use?</h4>Fortunately, with web apps like <a href="https://lexica.art/" rel="noopener noreferrer" target="_blank">Lexica.art</a> and <a href="https://arthub.ai/" rel="noopener noreferrer" target="_blank">ArtHub.ai</a> there are several fast growing libraries of images that other people have created using stable diffusion including the relevant prompt they used in the AI image generator. If you’re in lack of ideas, go ahead and browse the prompts that other users submitted to get started!If you'd like to learn more about Stable Diffusion and other generative image models, check out our <a href="https://scale.com/guides/diffusion-models-guide" rel="noopener noreferrer" target="_blank">Guide on Diffusion Models here</a>.

5 tools to assist you in generating beautiful art from text prompts with ML models.

5 open, freely available AI image generators to get you off the ground.

How to Make Your Own AI-Generated Images

Delivering a great buying experience makes all the difference in whether or not a user will convert to a customer. A core part of the eCommerce buying experience is creating an exceptional user journey from initial discovery to purchase. To learn more about accelerating the customer experience, we spoke to Nick Beighton, former CEO of ASOS, about his experience bringing a data-driven approach to online retail.&nbsp;<h2>Data Attribution as the Foundation for the Customer Experience&nbsp;</h2>For a brand to be successful, it needs to have agility in its business and technology to scale quickly to improve shopping experiences based on customer needs. At the core of agility for eCommerce companies, digitization is vital on both the front and back end. Nick explains,“Digitization is really important on the front end, but it’s also important on the back end in terms of how you digitize your processes, how you use data to drive greater personalization, and how you make experiences more efficient, faster, and intuitive.”&nbsp;Nick explains that digitizing the front end encompasses creating intuitive and personalized experiences. As customers engage with the digital offer, the experience becomes more customized, from ads to product recommendations. Scaling operational processes on the back end centers on strong data discipline. A key part of this is maintaining a large catalog of product attributes, which encompasses everything from physical characteristics such as size, color, and material to brand and product descriptions.“Digitizing the back end can only be done at scale through strong data attribution, strong control of your product master files, and the attributes within it. To attribute that data correctly, you can then turn your customer care from a cost center, to a profit center, by using the data to add on more sales and think differently about that interaction with the consumer.”By starting with strong data attribution, eCommerce companies can better connect marketing and SEO investments to the entire customer experience.<h2>Investing in Product Catalog Data at ASOS</h2>Nick applied a data-focused approach to scaling ASOS by first focusing on the search journey on the ASOS website. A core area that the ASOS team investigated was the differing conversion rate between customers who used the search bar to find a product and customers who used the website navigation: customers who used the search bar had double the conversion rate. As a result, the team explored how to maximize the search journey. One discovery was how product attribute data impacts the customer search journey.“We realized that the search journey didn’t have all the right data attributes to serve the most relevant search. So if you’re searching for men’s shoes, you might end up with women’s shoes too. We had to clean up the data on our main website to improve the search journey, and the recommendations served to the customers. That was one of the early ways we started investing in data attribution and product catalogs through the journey.”Nick spearheaded an ML team that maximized product attribution data, and the investment proved successful. By improving product catalog data, conversion moved from 2% to over 7% on average and quadrupled the number of major SKUs added to the site. Nick explains, “Data attributes and attribution of product information was critical to maintaining and improving that conversion while simultaneously adding more and more products.”&nbsp;Investing in product attributes proved instrumental at ASOS. The team connected robust product catalog data to maximize ROI and focus on conversion and customer experience. By building strong AI operational processes, the ASOS team optimized how they served customers.&nbsp;<h2>How eCommerce Companies Can Get Started with a Data-Driven Approach</h2>How do eCommerce companies get started with having strong data discipline? What are some of the best ways to implement these practices across an organization? Nick explains the critical areas for teams to focus on are investing in data attribution early, experimentation, and establishing product goals:<ol><li>Invest in data attribution: Strong product data attribution is at the core of improving the customer journey. ML teams can address long-tail product and content issues by investing in data attribution.&nbsp;&nbsp;</li><li>Experiment to get started: Don’t limit experimentation with tech solutions that only have strong business cases and ROI. You may not know what experiment will give you an exponential return, so it’s essential to have multiple experiments and tests in parallel and review the data to understand the impact.</li><li>Align on product goals: Investing in product catalog data and tying the data attribution goals to product performance metrics is crucial to transforming the business. By working closely with product teams, teams working in eCommerce can provide a direct correlation with internal metrics.&nbsp;</li></ol>Nick emphasizes the importance of data discipline is all about creating the best end experience for the customer,“A customer will rarely say, ‘I love your data attribution,’ right? She will say, ‘I feel the site gets me and gives me all the things I want when I’m searching for it.’ That’s the consumer mindset goal, and how to achieve it is through digitalization and data discipline.”&nbsp;At Scale, we believe investing in data is the key to unlocking success for<a href="https://scale.com/retail" rel="noopener noreferrer" target="_blank"> eCommerce companies</a>. If you’re interested in learning more about boosting your business’s conversion, speak to our team today.

Nick Beighton, former CEO of ASOS, discusses how he transformed the company from 200 million to 4 billion in revenue by investing in product catalog data. 

How Nick Beighton Transformed ASOS by Investing in Product Catalog Data

<a href="http://wwww.scale.com/" rel="noopener noreferrer" target="_blank">Scale AI</a>, the data infrastructure for AI, today announced the achievement of the “In Process” designation for its AI Data platform by the Federal Risk and Authorization Management Program (FedRAMP), a crucial milestone in the FedRAMP certification process. This designation indicates that Scale is on track to achieving a FedRAMP authorized product within the next 12 months.&nbsp;The AI Data platform is seeking authorization at the FedRAMP High Impact Level, enabling Scale AI to support the U.S. government’s most sensitive data and missions.&nbsp; Scale serves customers of all sizes across a wide range of industries. Reaching this milestone on the path to authorization is invaluable to Scale’s ability to enable federal, state and local governments in the adaptation of AI applications.&nbsp; FedRAMP is the U.S. Government’s standardized approach to providing security authorizations for the adoption and use of cloud services by the federal government. The government-wide program’s goal is to accelerate the use of secure cloud technologies by government agencies. The “In Process” designation is an important step towards Scale AI’s full FedRAMP authorization at the FedRAMP High Impact Level. Scale’s commitment to security is one reason why leading ML teams from private sector organizations and the U.S. Government rely on Scale AI to accelerate the development of their AI applications. If you’d like further details, please contact us at fedramp@scale.com. &nbsp;

Scale is on track to achieving a FedRAMP authorized product within the next 12 months

Scale achieved a crucial milestone in the FedRAMP certification process

Scale Achieves FedRAMP ‘In Process’ Designation for its AI Data Platform

Warm-Start Your Vision Project: 10 Robust Datasets for Pre-Fine-Tuning

Warm-Start Vision Projects with Robust Pre-Fine-Tuning Data

The 10 Best Public Datasets for Object Detection in 2022

&ldquo;This is the Unix philosophy: write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.&rdquo;
&mdash;<a href="https://www.azquotes.com/author/50799-Douglas_McIlroy" target="_blank" rel="noopener noreferrer">Douglas McIlroy</a>
Computer software came to maturity in the late 1960s. While most programs to date had been written monolithically, mostly from scratch, uniquely built for building-sized mainframes, several pioneers were starting to standardize a new way of thinking. Some of the programs they wrote have familiar names (e.g. <code style="background-color: rgb(240, 240, 240);">grep</code>, <code style="background-color: rgb(240, 240, 240);">diff</code>) and live on to this day as a toolset for computer programmers handed down through the mists of time, their origins unclear but their utility unquestionable. Their continued success is a microcosm of the continued success of the UNIX philosophy, originated by the titans of computer software Doug McIlroy and Ken Thompson in the early 70s. They noted the power of composable tools that operate on and modify streams of text.
A text stream program comes with automatic ease of collaboration and organization: individuals can change the internals of different programs without worrying about breaking the work of others. After all, the interface is standard. Text stream programs also come with extensibility built in for free. When the rudimentary data format of I/O never changes, you can string together long sequences of nested shell invocations to build ornate software <a href="http://www.catb.org/~esr/writings/cathedral-bazaar/" target="_blank" rel="noopener noreferrer">cathedrals</a> just by wiring simple programs together that modify each other&rsquo;s behavior. Text became the most common lever by which mankind would work computers to do their bidding.
Decades later, researchers began to conquer many previously unsolvable language and text tasks in unison. In February 2019, a precocious rising talent in AI research named Alec Radford came out with a seminal paper on language modeling that made him and his coauthors instant legends not just in the AI community but among a broad audience of technologists. Their patron institution, OpenAI, decided that the powerful model software created for the paper was the first AI plausibly too dangerous to simply release into the wild for fear of what malign actors might do with it. With the slow, eventual release of this model, <a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" target="_blank" rel="noopener noreferrer">GPT-2</a>, they ushered in the age of what we might call large language models (LLMs).
In a previous iteration of the machine learning paradigm, researchers were obsessed with cleaning their datasets and ensuring that every data point seen by their models is pristine, gold-standard, and does not disturb the fragile learning process of billions of parameters finding their home in model space.&nbsp;Many began to realize that data scale trumps most other priorities in the deep learning world; utilizing general methods that allow models to scale in tandem with the complexity of the data is a superior approach. Now, in the era of LLMs, researchers tend to dump whole mountains of barely filtered, mostly unedited scrapes of the internet into the eager maw of a hungry model.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=6c72512d2513baa32348274add4dd341.jpg&amp;f=webp&amp;q=75" width="1024" height="1024">
The resultant model displays alarming signs of general intelligence &mdash; it&rsquo;s able to perform many sorts of tasks that can be represented as text! Because, for example, chess games are commonly serialized into a standard format describing the board history and included in web scrapes, it turns out large language models <a href="https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-game/" target="_blank" rel="noopener noreferrer">can play chess</a>. Noticing these strange emergent properties, other researchers have pushed this general text processing faculty to the limits: a recent&nbsp;<a href="https://say-can.github.io/" target="_blank" rel="noopener noreferrer">paper</a>&nbsp;utilizes a large language model's common sense reasoning ability to help a robot make decisions. They simply submit a list of potential moves as a text stream and acquire the approval or lack thereof of the AI. The text model knows, for example, that finding a sponge before washing the dishes is more reasonable than the other way around and commands the robot as such. Seeing these quite disparate tasks being tamed under one unlikely roof, we have to ask &ndash; what other difficult problems can simply be transcribed into text and asked to an oracular software intelligence? McIlroy must be smiling somewhere.
Several are venturing down this intellectual avenue, asking in the spirit of the UNIX pioneers if the general intelligence of large language models can become the core modular cognitive engine for their various software programs and businesses. These intrepid entrepreneurs and engineers who require that the world&rsquo;s best artificial intelligences behave in a certain way to find value for their users are giving birth to the brand new discipline of prompt engineering. Dennis Xu, a founder of <a href="http://mem.ai/" target="_blank" rel="noopener noreferrer">mem.ai</a>, is one such frontiersman pounding down the doors of the future: his company works on a self-organizing note-taking app that works as an intelligent knowledge store, with an AI that curates, tags, and connects different notes.
Dennis&rsquo;s team knows it doesn&rsquo;t make any product sense to train their own model to solve their core cognitive task. Trying to understand the breadth of human knowledge from scratch in order to write note summaries and mark thematic tags would be an absurdity. Rather, why not use one of humanity&rsquo;s most powerful robotic minds readily available to the public over an API? Where collecting one's own dataset for a downstream application like a note-taking app will require a massive pre-existing user base, <a href="https://openai.com/blog/gpt-3-apps/" target="_blank" rel="noopener noreferrer">GPT-3</a> (codename: &ldquo;davinci&rdquo;), perhaps the world&rsquo;s most celebrated language model, is out there and has been trained on half a trillion tokens encoding the breadth of common sense reasoning and language task understanding. We can simply ask it to perform tasks like &ldquo;summarize this document,&rdquo; &ldquo;list the topics this note covers,&rdquo; and others. Of course, it&rsquo;s never so simple. We have to provide a few examples of notes and their corresponding summaries or topic tags to ensure the model is thinking in the right direction.The basics of the prompt generally consist of a call to action and a few examples that define the space of the task, but this barely scratches the surface.
For anyone who works closely with these models, it becomes clear that the vast and comprehensive training gauntlet that creates these technological leviathans embeds some difficult behaviors. While the breadth of example material spawns broadly intelligent digital creatures capable of working on a vast range of text tasks, it can require significant prodding, cajoling, and pleading to get these models into the right mood for the particular task at hand. In the early days after the release of GPT-3, many noticed that the AI seems to become more cogent if you start the completion prompt with something along the lines of &ldquo;this is a conversation between a human and an extremely intelligent artificial intelligence&rdquo; versus not including the modifier. Sometimes, GPT seems to be hiding the true depth of its intelligence for the sake of <a href="https://twitter.com/ESYudkowsky/status/1285333002252247040?s=20&amp;t=3NsHEt34xW8qvgnrWTXLIA" target="_blank" rel="noopener noreferrer">the character it is playing</a> at the moment. The prompt to GPT may even make the difference between being able to solve <a href="https://twitter.com/arankomatsuzaki/status/1529278580189908993" target="_blank" rel="noopener noreferrer">arithmetic puzzles or not</a>. In another insane quirk of language modeling, text-to-image models seem to create higher quality art if you specify that it&rsquo;s from the &ldquo;<a href="https://www.unrealengine.com/en-US" target="_blank" rel="noopener noreferrer">unreal engine</a>&rdquo; game studio, known for its photorealistic style. A self-supervised language model can be thought of as a vast ensemble of many models that play many characters with different personalities due to the breadth of the document distribution and dizzying variety of writing styles seen in the half a trillion internet-scraped tokens that it studiously observes at training time.&nbsp;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=2385ad67c2db7db124a7a29030278fb1.jpg&amp;f=webp&amp;q=75" width="1050" height="1390">
Every prompt a user feeds to a large foundation model instantiates a new model, one more limited in scope: it enters into a <a href="https://arxiv.org/abs/2207.10342" target="_blank" rel="noopener noreferrer">different conditional probability distribution</a>, or in other words, a different mood. A language model&rsquo;s mood gives birth to a new piece of composable modular software that takes in a text token stream and leaves another as output. The elegant power of UNIX-style software comes from the simple but incredibly powerful primitive known as the pipe. It transfers the output of one program to become the input or modifier for another. The output of one prompted query, which asks for a summary of a note, can easily be aggregated and sent to another prompted model that asks to order the summaries by relevance for a given search topic.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=e1c179bb6169ac2f09a28522a29860f4.png&amp;f=webp&amp;q=75&amp;w=3840">
The pure energy of modular software creation is buzzing in the prompt engineering war room, turning human will into repeatable programs with far reduced effort: find a prompt, don&rsquo;t write a program. Dennis and his team work hard to find the right incantation to coax GPT-3 into the right mood to perform the task at hand. Like most engineering, it&rsquo;s an ongoing optimization process. The structure of OpenAI&rsquo;s pricing model demands that the end users have to take care to use as few tokens for the prompt as possible since that&rsquo;s the basic unit of cost in the GPT-3 API. A prompt that uses only 100 characters to get GPT-3 doing the right thing is preferable to one that takes 400. Prompt engineers must play a delicate game of asking the model to think well while shortening their incantations.
These resourceful engineers are adapting some of the methods of the machine learning world to the prompt engineering world: an old bag of tricks under a new layer of abstraction. For example, they create tools to store &ldquo;test sets&rdquo; of difficult task examples to assess the quality of a new prompt (or an old prompt after a model change). Let&rsquo;s say that a prompt engineer has an idea for shortening the text prompt that asks the machine for document summaries. It could potentially save thousands a month on API fees. She tries it on a gauntlet of twenty examples (&ldquo;the test set&rdquo;) that the team has concluded are interesting and duly illustrate the performance of the model. After checking that all twenty examples are producing successful results (or at least the performance is not worse than before), she can roll out her change in an automated way into the product. Using these methods to assure safety and speed up experimentation, the mem.ai team creates such innovations constantly. They may even cleverly compose several tasks in the same query (multi-task inference!), further decreasing costs. All input is text, so there are several obvious ways to recombine and reroute queries into different prompts. All output is text, so human reviewers can quickly examine a few test examples and check for correctness and performance.
It is easy to bet against new paradigms in their beginning stages: the Copernican heliocentric model of cosmology was originally less predictive of observed orbits than the intricate looping geocentric competitor. It is simple to play around with a large language model for a bit, watch it make some very discouraging errors, and throw in the towel on the LLM paradigm. But the inexorable <a href="https://arxiv.org/abs/2001.08361" target="_blank" rel="noopener noreferrer">scaling laws</a> of deep learning models work in its favor. Language models become more intelligent like clockwork due to the tireless work of the brilliant AI researchers and engineers concentrated in a few Silicon Valley companies to make both the model and the dataset larger. OpenAI&rsquo;s new model available in beta (codename: davinci2) is dramatically smarter than the old one unveiled just two years ago. Like a precocious child, a more intelligent model requires less prompting to do the same job better. Prompt engineers can do more with less effort over time. Soon, prompting may not look like &ldquo;engineering&rdquo; at all but a simple dialogue with the machine. We see that the gradient points in the right direction: prompting becomes easier, language models become smarter, and the new universal computing interface begins to look inevitable.
Sometimes, despite our best efforts, we must leave the realm of text. There are endless forms of computation that operate on other media &mdash; much of the digital content that exists is photos, videos, and graphical interfaces, and users adore all of it. We can already see that the power of natural language is being leveraged to ply and manipulate these as well. The rapidly advancing line of models called <a href="https://openai.com/dall-e-2/" target="_blank" rel="noopener noreferrer">DALL-E</a> and <a href="https://github.com/CompVis/stable-diffusion#stable-diffusion-v1" target="_blank" rel="noopener noreferrer">Stable Diffusion</a> are proliferating on the internet, generating incredible social media buzz about the advent of AI art and what it means for humanity. These tools allow us to take snapshots of our imaginations, communicate them only through the versatile interface of text, and command powerful intelligences to reify that vision into pixels. This technology is remarkable &mdash; even the ancients conceived of a gift that could take a picture of the mind, and now it's available to anyone that can type.
Others are pursuing even more audacious efforts to command other modalities under text. The team at <a href="https://www.adept.ai/post/introducing-adept" target="_blank" rel="noopener noreferrer">Adept.AI</a>, which includes the <a href="https://www.linkedin.com/in/ashish-vaswani-99892181/" target="_blank" rel="noopener noreferrer">legendary</a> <a href="https://www.linkedin.com/in/nikiparmar/" target="_blank" rel="noopener noreferrer">authors</a> of the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" target="_blank" rel="noopener noreferrer">seminal paper</a> introducing the Transformer architecture used in all modern language models, notes that LLMs, for all their generalized intelligence, cannot take seamless action in the graphical interface world and are aiming to fix that discrepancy posthaste. An example listed on their blog: you tell the computer, in text, to &ldquo;generate our monthly compliance report&rdquo; and watch as your will unfolds on screen, as a computer mind converts text into action on an Excel or Word interface. In this paradigm, art, graphics, and GUIs themselves will be commanded under the modality of text.
Slowly but surely, we can see a new extension to the UNIX credo being born. Those who truly understand the promise of large language models, prompt engineering, and text as a universal interface are retraining themselves to think in a new way. They start with the question of how any new business process or engineering problem can be represented as a text stream. What is the input? What is the output? Which series of prompts do we have to run to get there? Perhaps we notice that the person in charge of the corporate Twitter account is painstakingly transforming GitHub changelogs into tweet threads every week. There&rsquo;s a prompt somewhere that solves this business challenge and a language model mood corresponding to it. With a smart enough model and a good enough prompt, this may be true of every business challenge. Where textual output alone is truly not enough, we train a joint embedding model such as DALL-E that translates text input into other domains.
The most complicated reasoning programs in the world can be defined as a textual I/O stream to a leviathan living on some technology company&rsquo;s servers. Engineers can work on improving the quality and cost of these programs. They can be modular, recombined, and, unlike typical UNIX shell programs, are able to recover from user errors. Like shell programs living on through the ages and becoming more powerful as underlying hardware gets better, prompted models become smarter and more on task as the underlying language model becomes smarter. It&rsquo;s possible that in the near future all computer interfaces that require bespoke negotiations will pay a small tax to the gatekeeper of a large language model for the sheer leverage it gives an operator: a new bicycle for the mind. Even today, many of us find ourselves already reliant on tools like <a href="https://github.com/features/copilot" target="_blank" rel="noopener noreferrer">GitHub Copilot</a>, despite their very recent invention, to read, suggest, and modify text for the creation of our more traditional software programs. Thousands or millions of well-crafted language model moods will proliferate, interconnected, each handling a small cognitive task in the tapestry of human civilization.
"The language model works with text. The language model remains the best interface I've ever used. It's user-friendly, composable, and available everywhere. It's easy to automate and easy to extend."
Follow the author&nbsp;<a href="https://twitter.com/tszzl" target="_blank" rel="noopener noreferrer">@tszzl</a>&nbsp;on Twitter.

cover_ Text Is the Universal Interface.png

Roon guest-authors a post on large language models and the future of computing.

Text became the main way humans commanded computers.

Text Is the Universal Interface

Aerin is an engineering manager at Scale AI, leading a Catalog ML team. Below are Aerin's stories about what she does and why she loves working at Scale.
<h2>What brought you to Scale?</h2>
Before I joined Scale, I worked on a project called NL2SQL which stands for "Natural Language to Structured Query Language" at Microsoft. NL2SQL is a translation problem that automatically converts a user's spoken-language query into SQL.
Just a few months after my team had started working on it, we were beating the SOTA (state-of-the-art) performance on research benchmarks. We achieved this by evaluating different ML model architectures and improving them. In other words, we somewhat solved the NL2SQL problem in research after only a few months of work.
However, in production, we weren&rsquo;t breaking the state of the art. The model still made a few mistakes on the user's input. And it became clear to me that changing the model architectures would only get us so far, and that the most crucial thing we could do to improve the performance of NL2SQL was to improve the training data. This was done by generating edge cases synthetically and therefore increasing the training data coverage.
So, we started synthetically generating training data (NL-SQL pairs) by following syntax requirements, snowflake schema, etc. However, the data that we have artificially created couldn't possibly cover every possible scenario or edge case. Because the true edge cases are ones that you can't predict or plan for, like the questions that people have typed into our search query bar.&nbsp;Another way to increase the coverage of the training data was to label the user inputs and feed them back to the training data. Taking care of that lifecycle, also called an auto-labeling pipeline, was quite an operation. Even though I was given a lot of resources to manage the auto labeling pipeline, I couldn't undertake any other engineering work because it required so much overhead.
So I started looking for the best third-party training data provider in the market, and that's how I met Scale, and the rest is history.
<h2>What do you enjoy most about working at Scale?</h2>
I like that I get to work on problems that are important and challenging enough to customers that they are willing to pay to have them solved. This means that the problems are real.
I also like that Scale's work is always about data, which is my favorite subject. For every business, I believe data is the most valuable and private asset. Say if you're developing a machine learning model, the one and only input that determines how well your model will perform is your data. The fact that customers trust us with their data and allow us to manage it energizes me, and I feel a great deal of commitment and responsibility. Because the data we provide with them is being used as a bloodline for their business, this relationship just can't work without trust. Having a close relationship with customers is also a rewarding part of my job.
Another highlight of my time here at Scale is the everyday connection with my team. When the engineering team works well, it feels like a well-oiled machine, and you will notice tangible results from that. We've done some seemingly impossible jobs together, and it feels amazing to see the whole team pulling together to complete a challenging project. When working on projects, you learn a great deal by openly debating and choosing the best technical solution with your team. If you're an engineer, I believe that seeing and interacting with other engineers is one of the best ways to learn and grow. That's also something I enjoy at Scale.
<h2>What makes working for a startup unique from other jobs?</h2>
I've worked at both a big tech company and a startup, and they are indeed quite different, and each one is unique in its own way.
At a startup, leading a project feels like driving a sports car. You're close to the ground and can feel the movements you make. You can turn quickly, and you can observe fast motion while you are steering. You can see the results of your team's work fairly quickly and clearly. For example, you complete a pilot project successfully, and the customer signs a large contract. There is also a good chance of crashing - for example, you fail to deliver a project and lose a customer. At a startup, you will see the results of what your team does right away.
At a big company, running a project is more like flying a spaceship (I can only imagine this). You&rsquo;re farther from the ground, it's harder to make quick decisions because you are not the only stakeholder, and the course has been set long in advance. There are also very well-built manuals and processes to guide you along the way.
So far, I've enjoyed seeing the immediate, direct effects of my team's work at Scale.&nbsp;
<h2>What's your favorite part about being an engineering manager at Scale?</h2>
One of the things I like about being an engineering manager is that I get to work on large-scale projects with a strong, cohesive team, which results in a scalable solution and a greater overall team output. Apart from the collaborative and multiplier nature of the job, I also enjoy problem solving at Scale.
People think that a software engineer's job is to write code, but I don't think that's a complete picture of an engineer. A more accurate job description would be "a problem solver", since our main goal is to build things that help people solve problems. You won't be able to solve problems if you blindly write code, especially from the wrong angle.
I believe the most important thing to do in order to solve problems is to accurately diagnose them. So the ability to diagnose problems accurately is, in my opinion, the most important skill for an engineer to have. If your diagnosis is wrong, you will rarely be effective in solving actual problems. And you can only make the right diagnosis if you have a full, concrete understanding of the problem, the data, and the use cases.&nbsp;
While surgeons can rely on other professionals like physicians, x-ray technicians, to aid with diagnosis, ML/SW engineers have to diagnose problems on their own. We have to carefully examine the available data, debug the code thoroughly, then choose the best solution and also implement. The ML/SWE profession comes with a significant amount of both power and responsibility.
If you are someone who enjoys doing things like this (proper diagnosis and implementation), you will do well in the field of engineering.
<h2>Tell me about interesting projects that you ran at Scale?</h2>
Every project at Scale is interesting and unique in its own way, as each project has different data modality (3D Lidar, 2D imagery, NLP, tabular, etc.) I enjoy that I get to work on diverse problems and apply the peculiarities of each data modality to ML model development. For example, 3D data often comes with the new axes - yaw, pitch, roll, unlike 2D data and the sensor's effect on the LIDAR data is significant. NLP also has its own quirks, such as different types of tokenizers, vector representations, etc.
These days, my team is working on multi-modal (Language + Vision) problems. For example, given an image of a model wearing clothes, can we describe what she&rsquo;s wearing in detail (e.g. short sleeve, polka-dots, long boots, etc.) in natural language? To this end, one popular research/exploratory approach is to train a multi-task (e.g., Visual Question Answering, Image Captioning, Text-to-Image Generation, Visual Grounding, etc.) model and use it for attribute prediction.
Although I can't talk about the details of the projects I work on for clients, I do make an intentional effort to make my team's work more universally applicable, scientifically relevant, and therefore publishable. The attribute prediction work (link) that was just accepted at CVPR 2022 is a great example. One of the most common requests we get from clients is to assign visual characteristics such as color, shape, or motion to detected objects. But there aren't many researchers working on this topic publicly (though there are a lot of practitioners working on it in industry), and there aren't many benchmark datasets for attribute prediction. So we did in-depth research on this topic, presented our findings at CVPR, and received overwhelmingly positive feedback from the ML community.
The Fitzpatrick 17k project with MIT is also a good example. The Fitzpatrick scale is a 6-point scale that tells how dark your skin tone is. And this project&rsquo;s motives are quite fascinating. Because medical imaging doesn't have much data on darker skin, ML models trained on these datasets have trouble recognizing dermatological illnesses on darker skin.&nbsp;We wanted to bring attention to this problem and encourage more inclusion of images with darker skin tones in datasets. So we annotated 17k clinical images in DermaAmin and Atlas Dermatologico, the two major dermatology atlases, using the Fitzpatrick Scale.&nbsp;
We trained a few models on this new dataset and then looked at how well they work on images with different skin tones. And yes, adding more diverse data to the training set was an effective way to reduce bias.&nbsp;
Because of this work, researchers and practitioners&nbsp;are now a little more likely to take into account and examine biases in training datasets before developing ML models in the healthcare sector. Ultimately, we want to make sure that ML models trained on imbalanced datasets don't make healthcare inequalities worse by accident. With this study, we received an honorable mention award at the CVPR skin imaging workshop.
&hellip;&hellip;..
If you are interested in any of the topics that I have said above or in working to solve intriguing ML/SWE challenges, email your resume at <a href="mailto:aerin.kim@scale.com" target="_blank" rel="noopener noreferrer">aerin.kim@scale.com</a>!

Read Aerin's stories about what she does and why she loves working at Scale

Aerin is an engineering manager at Scale AI, leading a Catalog ML team

How Nick Beighton transformed ASOS into a 4 billion dollar business by investing in product catalog data

We've had an incredible group of people join Scale—and we’re still <a href="https://scale.com/careers?office=All&amp;department=All#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! Meet the engineers who are accelerating the development of AI applications:<h3> </h3><h3>Albert Sheu</h3>Albert is a fifteen-year veteran of the tech and startup industry, including having worked as the second engineer at Quora and the Head of Engineering at Patreon. His experience runs up and down the stack, working on distributed systems, developer experience, WebGL interfaces, and all areas in the frontend and backend. At Scale he developed the early 3D labeling pipeline, and has also worked on Scale’s Nucleus and Synthetic Data products. <h3>Jack Wu</h3>Jack Wu, previously was a manager at Niantic where he built the Platform Infrastructure team from scratch and led the team to rebuild the Niantic million QPS, planet scale Augmented Reality platform on an alternate Cloud provider. Prior to transitioning to management, Jack was an early engineer at Uber and LinkedIn where he spent most of time building large volume data storage, data analytics and data security systems and saw both companies grow more than 10X during his tenure.<h3> </h3><h3>Yuting Duan</h3>Yuting worked at Google as a Staff Software Engineer and Engineering Manager and studied at CMU. She worked on some of the most large-scale consumer-facing products: Google Search and Google Maps and founded 2 full-stack teams in Google to help users find food wherever they are, whenever they are. <h3>Shah Xia</h3>Shah has previously worked at Bytedance where she was the Chief of Staff to the CEO. She was part of the early team to incubate the TikTok app, and later directed its global expansion strategy. She is receiving her MBA from Wharton. <h3>Sam Clearman</h3>Sam Clearman received his PhD in Mathematics at Lehigh for his work in algebraic combinatorics.&nbsp;He worked at Google using statistical methods to improve Android battery life before joining Scale in 2018.&nbsp;At Scale, he has worked on pretty much everything, but he has always returned to marketplace management and incentive design.&nbsp;In his spare time, he flies planes (sometimes upside down), and enjoys making games such as 2etris [<a href="https://2etris.com/" rel="noopener noreferrer" target="_blank">https://2etris.com/</a>], Down for a Block [<a href="http://d4ab.com/" rel="noopener noreferrer" target="_blank">http://d4ab.com/</a>], and Up for a Camel [<a href="https://upforacamel.com/" rel="noopener noreferrer" target="_blank">https://upforacamel.com/</a>].&nbsp; <h3>Sasha Harrison</h3>Sasha Harrison received a Bachelor's Degree in Mathematical and Computational Sciences, and a Master's Degree in Computer Science from Stanford University, where her concentration was in Deep Learning. As a student, she conducted research projects related to algorithmic fairness in computer vision and NLP, and was an intern at Google and Microsoft. At Scale, Sasha has worked across new products and ML, in particular, helping to develop scalable batch inference infrastructure as well as deep learning models for application to the content understanding data annotation vertical. <h3>Calvin Huang</h3>Calvin Huang met Alexandr Wang at the US Physics Camp just before starting to study electrical engineering and computer science at MIT.&nbsp;While at MIT, Calvin did research in SMT solving and natural language processing. After graduation, Alex recruited Calvin to join Scale as an early engineer, where he built out a large portion of the infrastructure stack and developer tooling, and helped the core systems scale as Scale did. <h3>Vivek Muppalla</h3>Vivek is an industry veteran who has worked in and lead intraprenual efforts across startups and large companies. His experience is leading full stack engineering and product teams. Most recently he was an early member of Unity’s AI &amp; simulation team, and helped Unity become a leader in the Synthetic data and Robotics simulation space. At Scale, he is focused on building our Simulation based products.

Meet Scale's incredible engineering talent 

Scale's Talented Engineering Team

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;
That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;
Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.
<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a>
For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;
While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:
<h4>Aggregate evaluation metrics miss critical regressions</h4>
A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.
<h4>Coverage for ML model testing is not well defined</h4>
While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;
Testing pipelines are static and don&rsquo;t evolve with the models
Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.
Accordingly, we built Scale Validate with the following capabilities:
<h3>Scenario Tests</h3>
Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).
<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI">
<h3>Model Comparison</h3>
Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.
<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8">
<h3>Slice Discovery</h3>
Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.
<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">
We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.
As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:
<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">
Getting started with Validate only takes a few steps!&nbsp;
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.

Introducing Scale Validate for Model Testing

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Resources

Customers

Enterprise →

Government →

Leaderboards →

Blog