BlogCategoryTag

Header

Footer

PreFooter

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

PaginationButton

AnnouncementContextProvider

UnnamedGlobalGroupOfVariants2

Screen

Segment

Experiment

Experiment2

scale.com (Production)

plasmic-nav

plasmic-embed-css

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

react-slick

react-quill

Product

General

People

Engineering

We've had an incredible group of people join Scale—and we’re still <a href="https://scale.com/careers?office=All&amp;department=All#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! Meet the engineers who are accelerating the development of AI applications:<h3> </h3><h3>Albert Sheu</h3>Albert is a fifteen-year veteran of the tech and startup industry, including having worked as the second engineer at Quora and the Head of Engineering at Patreon. His experience runs up and down the stack, working on distributed systems, developer experience, WebGL interfaces, and all areas in the frontend and backend. At Scale he developed the early 3D labeling pipeline, and has also worked on Scale’s Nucleus and Synthetic Data products. <h3>Jack Wu</h3>Jack Wu, previously was a manager at Niantic where he built the Platform Infrastructure team from scratch and led the team to rebuild the Niantic million QPS, planet scale Augmented Reality platform on an alternate Cloud provider. Prior to transitioning to management, Jack was an early engineer at Uber and LinkedIn where he spent most of time building large volume data storage, data analytics and data security systems and saw both companies grow more than 10X during his tenure.<h3> </h3><h3>Yuting Duan</h3>Yuting worked at Google as a Staff Software Engineer and Engineering Manager and studied at CMU. She worked on some of the most large-scale consumer-facing products: Google Search and Google Maps and founded 2 full-stack teams in Google to help users find food wherever they are, whenever they are. <h3>Shah Xia</h3>Shah has previously worked at Bytedance where she was the Chief of Staff to the CEO. She was part of the early team to incubate the TikTok app, and later directed its global expansion strategy. She is receiving her MBA from Wharton. <h3>Sam Clearman</h3>Sam Clearman received his PhD in Mathematics at Lehigh for his work in algebraic combinatorics.&nbsp;He worked at Google using statistical methods to improve Android battery life before joining Scale in 2018.&nbsp;At Scale, he has worked on pretty much everything, but he has always returned to marketplace management and incentive design.&nbsp;In his spare time, he flies planes (sometimes upside down), and enjoys making games such as 2etris [<a href="https://2etris.com/" rel="noopener noreferrer" target="_blank">https://2etris.com/</a>], Down for a Block [<a href="http://d4ab.com/" rel="noopener noreferrer" target="_blank">http://d4ab.com/</a>], and Up for a Camel [<a href="https://upforacamel.com/" rel="noopener noreferrer" target="_blank">https://upforacamel.com/</a>].&nbsp; <h3>Sasha Harrison</h3>Sasha Harrison received a Bachelor's Degree in Mathematical and Computational Sciences, and a Master's Degree in Computer Science from Stanford University, where her concentration was in Deep Learning. As a student, she conducted research projects related to algorithmic fairness in computer vision and NLP, and was an intern at Google and Microsoft. At Scale, Sasha has worked across new products and ML, in particular, helping to develop scalable batch inference infrastructure as well as deep learning models for application to the content understanding data annotation vertical. <h3>Calvin Huang</h3>Calvin Huang met Alexandr Wang at the US Physics Camp just before starting to study electrical engineering and computer science at MIT.&nbsp;While at MIT, Calvin did research in SMT solving and natural language processing. After graduation, Alex recruited Calvin to join Scale as an early engineer, where he built out a large portion of the infrastructure stack and developer tooling, and helped the core systems scale as Scale did. <h3>Vivek Muppalla</h3>Vivek is an industry veteran who has worked in and lead intraprenual efforts across startups and large companies. His experience is leading full stack engineering and product teams. Most recently he was an early member of Unity’s AI &amp; simulation team, and helped Unity become a leader in the Synthetic data and Robotics simulation space. At Scale, he is focused on building our Simulation based products.

image.jpg

Meet Scale's incredible engineering talent 

Scale's Talented Engineering Team

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;
That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;
Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.
<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a>
For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;
While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:
<h4>Aggregate evaluation metrics miss critical regressions</h4>
A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.
<h4>Coverage for ML model testing is not well defined</h4>
While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;
Testing pipelines are static and don&rsquo;t evolve with the models
Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.
Accordingly, we built Scale Validate with the following capabilities:
<h3>Scenario Tests</h3>
Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).
<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI">
<h3>Model Comparison</h3>
Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.
<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8">
<h3>Slice Discovery</h3>
Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.
<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">
We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.
As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:
<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">
Getting started with Validate only takes a few steps!&nbsp;
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.

Introducing Scale Validate for Model Testing

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

Within the field of computer vision, semantic segmentation is a close approximation to describing the visual world the way humans perceive it: entities with precise boundaries. For use cases where nuanced scene understanding is mission-critical, like autonomous vehicles and intelligent robots, building models requires massive amounts of highly accurate segmentation masks with pixel-level precision.The exactness required near object edges that makes semantic segmentation valuable also makes it incredibly time-consuming for humans to annotate.&nbsp;Given taskers annotate thousands of images per day, we wanted to accelerate the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotation</a> process without losing fine-grain precision. To accomplish this, we developed a feature called Autosegment. An annotator draws a box around an object and a segmentation mask is automatically generated. The annotator can then adjust the mask before marking it as complete. Under the hood, the subset of the image is sent through a model, which automatically identifies and segments the object of interest.<a href="https://scaleai.wistia.com/medias/lvnzhzfl7b" rel="noopener noreferrer" target="_blank">video</a>The Autosegment functionality was originally developed for autonomous vehicle data, which is a core customer use case at Scale. (You can read more about our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">original Autosegment tool</a> in our previous blog post.) When it was first launched, we saw a 30% decrease in the time it took to produce segmentation masks. However, the use cases for Autosegment have expanded greatly since its introduction, with AV data now constituting only one of many domains for which Scale produces segmentation data. While the underlying model did generalize beyond the autonomous vehicle domain, we observed that it performed noticeably better on objects like street signs, vehicles, and pedestrians than on other types of classes.This article contains a technical deep dive into the machine learning model that powers Autosegment. We’ll walk through some of the choices we made during the development process as well as our recent effort to overhaul the model for improved generalizability. In particular, we will discuss the data-centric approach we took to boost the usefulness of Autosegment across a wider range of use cases.<h2>Model Anatomy</h2>The segmentation model developed to power this tool addresses a problem known as salient object detection wherein the goal is to identify and segment the most “salient” or prominent object in an image. There is subjectivity in this task because an image can easily contain more than one prominent item. The intention behind Autosegment is that the user draws an approximate bounding box around the object of interest. Based on this we can firmly make two assumptions:<ol><li>The target object will be relatively centered within the box</li><li>The target object will extend nearly to the edges of the box, with some nonzero padding</li></ol>We encode these assumptions into the training process. Specifically, images in the training set are cropped in such a way that the target object is always centered with some randomly added padding. For any training image containing multiple segmented objects, each object is cropped separately to produce an individual training observation. <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fe23c12266e384e7ca60a35f93258ba3e">The architecture we chose to implement for this purpose is based on U2-Net proposed by <a href="https://arxiv.org/pdf/2005.09007.pdf" rel="noopener noreferrer" target="_blank">Qin et al.</a> for this type of problem. U2-Net provides two concrete benefits over other architectures we considered.First, it is designed to more effectively take into account both local and global contrast features of the image. Most of the popular feature extractors (VGG, ResNet, etc.) were designed and optimized for image classification. While they can work well for segmentation tasks, they do so by identifying features specific to the semantic classes they are trying to predict. The consequence of this is that they tend not to generalize as well to classes absent in the training set. On the other hand, U2-Net is designed to make use of image features that indicate object saliency. Because of this, while U2-Net may not perform as well at outright classification as more traditional architectures (we did not test this), in our testing it does a good job at identifying salient objects pertaining to obscure classes. This is important for Scale’s customers who often need to annotate data for niche use cases not represented in the training data.Second, we want to carefully avoid the need to stretch or downsample images. Because the goal is pixel-perfect accuracy, the model needs to output a high-resolution segmentation mask. Downsampling, stretching, or otherwise modifying the image in any information-reducing manner before feeding it to the model will typically result in blurrier segmentation masks than desired. To this end, utilizing a fully convolutional network allows for the processing of any image crop size or shape as long as it is large enough to be processed by the encoder phase of the U-Net (and small enough to fit within GPU memory). In addition, unlike some other architectures, U2-Net does not excessively downsample the input with convolution strides or pooling. This helps propagate enough information about the original image through the network to produce high-resolution mask outputs.<h2>Improving Generalizability</h2>Neural networks have a tendency to try and “cheat” by taking shortcuts when learning to solve a problem. Faced with the task of salient object detection, one shortcut is to simply learn how to segment the specific classes present in the training data. For example, if trained exclusively on autonomous vehicle camera data, the Autosegment model only needs to memorize what certain object classes look like, such as cars, trucks, and pedestrians. The resulting model may work for annotating vehicle scenes, but it will not perform well when applied to new object classes because it hasn’t truly learned the concept of “saliency.”<a href="https://scaleai.wistia.com/medias/dhzm8kaohk" rel="noopener noreferrer" target="_blank">Video</a>To overcome this challenge, the model needs to be given training examples for which this shortcut does not work. In the case of Autosegment, we expanded the training dataset to include hundreds of new object types. Neural networks have a finite number of parameters,&nbsp;meaning that there is a limit to how much “knowledge” they can hold. By providing more classes than the network can effectively memorize, the hope is that it will be forced to learn a simpler strategy that can detect saliency across a more general set of images.Our first step towards generalization was to incorporate the open-source <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">Common Objects In Context (COCO)</a> segmentation dataset into the training process. However, while COCO contains a wide variety of object classes, it only contains about 200 thousand annotated images, which is a small fraction of the training data from autonomous vehicle cameras. In addition, COCO segmentation masks are often somewhat imprecise.We then sought to leverage the vast segmentation data that has been produced using Rapid, our annotation platform for rapid experimentation. Rapid supports both full-scene segmentation and partial-scene segmentation. In full-scene semantic segmentation, every pixel in the image is classified into one of multiple classes. In partial-scene segmentation, pixels may be categorized into zero or more classes. Hundreds of thousands of images have been annotated using Rapid.<a href="https://scaleai.wistia.com/medias/15nwn1o6ff" rel="noopener noreferrer" target="_blank">Video</a>Full-scene segmentation often contains classes that are not beneficial to train on. For example, in outdoor scenes the sky is often a distinct class in the segmentation mask but is typically considered less salient than all other objects in the image. While we want to train on a diverse set of object classes, we do not want to train on “background classes” or objects that do not match a reasonable definition of saliency. Neural networks are generally robust to occasional adverse training samples, but in large quantities they can negatively affect training. In order to filter out these undesirable classes, we have taken the time to construct a mapping between a common taxonomy and the customer-specific classes. For example, this mapping helps us recognize that what one customer calls a “bicycle” is the same as what another customer calls a “bike.” From this, we filter out segmentations that do not match a list of approved classes. Partial-scene segmentation is the more common project type on Rapid and mostly avoids this problem of extraneous classes.<h2>Validation</h2>An important consideration for us was that because Autosegment is already in use across a large number of projects and use cases, changing the underlying model risks unknowingly worsening its accuracy on specific categories of images. In theory this type of accuracy loss could occur due to the finite information capacity of the model architecture. As mentioned earlier, the network needs to encode its problem-solving process using a limited set of parameters. If the number of parameters is too small, underfitting will occur and impede accuracy. We combat this possibility in two ways.First, we ensure that the training data is well-balanced. If the overall composition of the dataset is skewed in favor of any particular domain, then the model may tend to overemphasize training samples from that domain. Not only can this result in lower accuracy in other domains, but it could impede our goal of forcing the model to learn a generalized solution to the salient object detection problem.Second, in addition to benchmarking accuracy on the overall collection of images, we also benchmark the model’s performance on subdomains (such as autonomous vehicle data) to ensure it performs well within each of the strata. This essentially serves as “regression testing” for the model. Regression testing is a software development practice to ensure that code changes do not break existing known use cases. To accomplish this, we validate the model on independent test sets representing distinct customer use cases. The crucial use case we monitored during these improvements was autonomous vehicle data:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdb0470f15fa945cda7b946e690837b34">Beyond standard pixel classification metrics, we also calculate the precision and recall of pixels within a small neighborhood of the segmentation edges. These metrics help us assess how well this tooling will actually benefit annotators for whom the most time-consuming aspect of semantic segmentation is the object boundary pixels.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fac64ee3d7a814bc0852260ed4e5721f9">To calculate these edge metrics, we first start with the predicted and ground truth masks. Keeping only the single-pixel boundary of each mask, a <a href="https://en.wikipedia.org/wiki/Dilation_(morphology)" rel="noopener noreferrer" target="_blank">dilation operation</a> is applied with some small threshold. Metrics are then calculated from the dilated edges. The above image visualizes this process.<h2>The feedback loop</h2>Beyond the initial training of the model, data annotated with Scale Rapid continues to get fed back into the training process. This creates a feedback loop: Autosegment helps users to more quickly produce higher quality annotations, and in turn, those annotations serve to continuously improve the automatic segmentation accuracy. Having high-quality training annotations is an indispensable prerequisite to training a high-performing model. This <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> lifecycle is a unique characteristic of Scale’s annotation platform and fundamental to how we achieve best-in-class annotation speed and accuracy.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F03a3cce06e0e4cfba0dd090356f2669d"><h2>How you can use Autosegment</h2>Autosegment is one of the multiple Active Tooling capabilities that exist in Scale’s products. Active Tools provide interactive ML-powered assistance to data annotators. By letting humans make changes to the model inputs in real time, model outputs can be quickly dialed in to match the situation precisely and minimize time spent on the task.You can supercharge your data annotation using <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a>, which offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.

How Scale AI’s computer vision annotation tooling (CVAT) uses machine learning-assisted tooling for faster and more accurate semantic segmentation.

Machine Learning-Assisted Image Semantic Segmentation

Machine Learning Assisted Image Segmentation

<h2>About me:&nbsp;</h2>My name is Sophia Luo and I am currently an engineering manager and product manager on Scale Rapid, our self-serve and rapid experimentation data annotation product. During my time at Scale, I have been an engineer and product manager on several different teams; including eCommerce, Marketplace, and Platform. In four years, I graduated with a Masters of Engineering in Computer Science, Bachelors of Sciences in Computer Science, and Bachelors of Sciences in Mathematical Economics from the Massachusetts Institute of Technology. In my free time, I like to dance, draw, eat good food, and plan fun and new experiences with friends.&nbsp; <h2>Top Slacker at Scale AI&nbsp;</h2>When I first graduated college, I kept telling my friends how confused I was trying to decide where to go for my first job as a software engineer.&nbsp;Fortunately, I had the opportunity to choose a path that was best suited for me, as I intentionally reviewed multiple offers. This was a major life decision, the biggest one I had to make since deciding which college to attend four years prior.&nbsp;When making this decision, I accumulated 18 different company traits, including growth opportunity, work life balance, mentorship, and more that I would evaluate to see if the position would be a good fit. In hindsight, the whole process was very heavy handed. In the end, I realized that there were only a few traits that really mattered to me and it became clear that Scale was the best choice:<ol><li>Growth opportunity: Scale was quickly growing, and I knew that there would be many organic career opportunities. It also became apparent that the company was open to providing opportunities on important projects to those willing to take them on– regardless of tenure or experience. This was very important to me as someone starting their career. I wanted to be able to dive straight into the deep end and learn by doing.</li><li>Talent: The high concentration of talent at Scale was awe-inspiring. Just from talking to the employees and searching their profiles on LinkedIn, I was both intimidated and excited to join a group of people who were so talented. I wanted to surround myself with and learn from some of the most talented people in the industry.&nbsp;</li><li>Culture: After spending time with Scale employees and also hearing about their experiences, I found that there was a very tight-knit community of friendship and trust. As a fast growing startup, there were ups and inevitable downs, but it was clear that the Scale people had a very tight bond that helped them get through even the toughest of times.&nbsp;&nbsp;</li></ol>Fast forward about half a year and — boom — the pandemic and quarantine. The last semester of my college career was cut abruptly short, and I found myself back at home feeling sad, bored, and lonely. As a lonely new grad starting at a company where I couldn’t see the faces of my coworkers by default, I regularly sent cold messages to people across the company on Slack and asked to schedule 1:1 meetings to get to know them.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa8d872b8d3a54833b1932f565b02172e">As the number of people I became friendly acquaintances with grew, so did the number of Slack messages I sent. In fact, I was sending Slack messages outside of my normal work duties so much that I found myself added to our #top-slack-users channel after a few months of starting at Scale.As I was getting to know people through these Slack DMs and 1:1s, I also found myself working with many of them through my first large-scale project. Within my first six months, I was tasked with developing Scale’s first marketplace model as the third engineer on our Marketplace team. Developing this model from 0 to 1 involved applying many econometrics and linear programming concepts I learned in school. I needed to compute annotation pipeline capacity requirements and derive other useful variables. Furthermore, I was learning about data orchestration technologies and systems design techniques to be able to compute across all live annotation pipelines. The implementation and rollout of the model also involved more cross-functional work than I could ever imagine; I was interfacing with product managers, program managers, operators, engagement managers, and engineers across the organization who all wanted to learn about and integrate with my work.&nbsp;For being so early in my career, I felt blessed with the opportunity: there was just so much to learn and I was excited to be owning such a large surface area of work so early on in my career. I felt like I was learning and growing very quickly. In addition to improving my raw technical ability, I was learning how to become a more effective engineer by learning how to program manage, communicate with stakeholders, prioritize feature requests, and drive outcomes.&nbsp;As I was working on the marketplace model, I also became heavily involved in our university recruiting program. I was very excited to write, test, and calibrate new engineering interview questions, and it was Scale’s first time courting candidates completely virtually. Instead of hosting in-person events for candidates to learn more about Scale and the culture, we had to be more creative. I ended up organizing and playing countless virtual Among Us games with candidates, and we also threw together a Trivia Night where we invited candidates and Scale employees to compete for fun prizes together.&nbsp;After quite the eventful first year at Scale, I found myself with yet another career growth opportunity. I had transitioned from the Marketplace team to become the third engineer on the eCommerce engineering team where we were building a new product from 0 to 1 to develop AI-enabled data infrastructure for eCommerce.&nbsp;Being on the eCommerce team was quite the rollercoaster ride. As a team of three engineers, we first worked on a proof of concept for a customer who is still one of our largest today. This was a two week sprint where we built everything from scratch. With the correct licensing and legal practices, we needed to collect raw catalog data, extract and transform the data to standardize information across all catalogs, enrich the data with human-in-the-loop annotation pipelines, and build quality assurance tools and processes in order to ensure high quality delivery of data.&nbsp;Given the specification and deadline, it did not seem possible to meet the customer goals. But, after some long nights of sprinting, problem solving, and teamwork, we exceeded our own and the customer’s expectations – so much so that the proof of concept exploded into a very high volume contract. The two week sprint then became a two month marathon to meet the next customer deadline. We had to productionize all our scrappy processes from the proof of concept in order to scale to more than 1000x the volumes.&nbsp;The team grew from three engineers to around twenty in a short amount of time. I found myself not only executing on my own engineering work but also onboarding and mentoring countless others. Although my title was still officially “software engineer,” I was working on building out parts of the eCommerce product as a product manager and also became the acting engineering manager for some of the onboarded engineers. I was rapidly growing into more leadership-like roles within the team; I owned large surface areas within the product, and led pods of engineers to drive outcomes within those areas. Later, I ultimately stepped up to take on the responsibility of becoming the single threaded owner who led and owned the success of the entire production contract, which involved leading not only all the engineers on the team but also all cross-functional stakeholders.&nbsp;After my work on the eCommerce team, I transitioned to becoming the first product manager and engineering manager on Rapid, one of Scale’s fastest growing self-serve products after having been launched only one year prior. Now, I spearhead numerous workstreams to push us towards the overall goal of being the default annotation platform for all machine learning and data use cases. Every day, I find myself hopping from thread to thread across engineering, product, design, finance, marketing, and sales. I also still code from time to time to increase engineering bandwidth on the team so that we can push new features for our customers and patch inevitable bugs as they come up.&nbsp;&nbsp;The challenges on Rapid are unlike those that I have previously experienced at Scale. Instead of building for a specific annotation use case, we need to build generalizable and scalable product solutions in order to support all data types and annotation use cases. This involves being able to support images, videos, text, audio, and other types of data. We also need to develop the taxonomy and tooling to support all associated annotation types, including classification, transcription, semantic segmentation, and geometries.&nbsp;&nbsp;Given that customers want high quality data as quickly as possible, we also optimize our work to redefine the data quality and turnaround time frontier. To ensure high quality data annotation, we innovate in areas, such as generalizable worker training infrastructure, and ML-assisted validators to flag annotation issues and edge cases. To make sure the data is annotated as quickly as possible without detracting from quality, we also invest in better tooling, ML-assisted annotation (e.g. <a href="https://scale.com/blog/automatic-segmentation" rel="noopener noreferrer" target="_blank">auto semantic segmentation</a>), and pipeline improvements to break one larger annotation task into multiple smaller ones so that multiple people can work at the same time (e.g. our <a href="https://scale.com/docs/video-annotation-with-rapid" rel="noopener noreferrer" target="_blank">video stitching pipeline</a>).&nbsp;While driving all these product improvements and leading some parts of the technical solution, I also partner with our marketing, sales, and finance teams. I experiment with different growth hacking initiatives, such as hosting our <a href="https://scale.com/kaggle" rel="noopener noreferrer" target="_blank">first Kaggle competition</a> and <a href="https://scale.com/rapid/university" rel="noopener noreferrer" target="_blank">partnering with universities</a>. Furthermore, I work closely with our sales team to make sure that we are growing existing accounts and customer segments as well as targeting untapped market opportunities. For every sale opportunity, I am also looped in on pricing decisions in order to make sure that we drive revenue growth while balancing our costs and margins.&nbsp;&nbsp;From building upon my raw technical ability to building a diverse suite of skills as a product and engineering manager, I am grateful everyday for the opportunities that Scale has given me to build my career. I will never forget the memories I have made and the skills that I have grown on the Marketplace, eCommerce, and Rapid teams as well as through my heavy involvement in university recruiting and event planning.&nbsp;The tight-knit community is a constant reminder that we can accomplish great things together while making lifelong memories along the way. Recently, I was somehow able to convince our company leadership (including our very own CEO!) to learn a dance that I taught them for at most thirty minutes. Then, they performed this dance together in front of the entire company during our first company-wide in-person celebration after quarantine.&nbsp;Scale is a great community with some of the most hardworking, talented, and fun people in the Silicon Valley tech scene. I will always be thankful for the highly talented group of people who were with me through the ups and downs. These past two years have been quite the rollercoaster ride, but my reign as the top Slacker of Scale has been quite consistent.&nbsp;

How a new grad grew her career at Scale

Why I Joined Scale

Learn how we extended our infrastructure by sharding our MongoDB cluster.

From MVP to Serving Enterprise Scale

How We Scaled Our MongoDB Instance Gracefully

Large-scale datasets are at the center of model development and evaluation. Quality of training data is the dominant factor for model performance. Anyone can create an excellent model based on great data, but even the best ML engineers can’t design good models with bad data.&nbsp;If you’ve ever used popular public datasets such as ImageNet or MNIST, you would assume that the class and attribute <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labels</a> are correct. Shockingly, ImageNet, the most widely used dataset for popular image recognition systems, has a 6% error rate[6]. When models are fed bad data, such as incorrect classes, incorrect features, or missing <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a>, they often have severe classification problems, which can destabilize ML benchmarks and contribute to representational harms in datasets.&nbsp;Examples of representational harms[1] in datasets include under-representation of darker skinned subjects compared with lighter-skinned subjects within prominent facial analysis datasets[2,3], and millions of images of people in ImageNet labeled with offensive categories, including racial slurs and derogatory phrases[5].&nbsp;Even in leading applied ML research groups, approaches for characterizing and fixing labeling errors in massive datasets are limited.&nbsp;One approach to combat noise in a dataset is Confident Learning (CL), which is based on pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. However, CL entirely ignores the case where one or more true labels are missing. This limitation occurs because CL relies on building a pairwise class-conditional distribution between the given and true labels[6]. Other techniques include sorting by model loss, which surfaces the examples that are most ambiguous and difficult to learn, and sorting by confidence where model and ground truth disagree (quite effective!).What is the most principled option after amassing a collection of bad labels? In practice, it would be fixing your dataset by going through the images and fixing the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> one by one.&nbsp;This is both laborious and impractical.That’s why (1.), a way to visualize, triage, and organize your data, and (2.) a way to fix your annotations, are critical to your data annotation project.Scale provides an integrated dataset and workforce management platform that solves these problems.Let’s walk through an example of how you can use two of Scale’s products: Scale Nucleus and Scale Studio to identify and fix bad labels leveraging model predictions. For this example, we’ll be using the public <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a>.&nbsp;<img src="https://lh3.googleusercontent.com/hYW3_C_Yz4vuaZKQppVBRJB273v9Nz2b9bgHBDbl-WUrdKjfHg31Ic6E2j34WJZSuXSufVkBgfWf0Q7ThfnZ-l7CxScpOViGECQXXHBhk3_VkMqYT0t-GpgGL_1J_O97Vg7wmCjTPh-PSodX-Zm7o-c">Starting on the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">“Charts” page in Scale Nucleus</a>, a simple click into the confusion matrix shows 2,277 cases in which models predicted “person” but the ground truth label is “car”.<img src="https://lh5.googleusercontent.com/e-ElAIxlS6X1koxM2vOVAHv-MUG_IC_trcCX6K1YXyvDDPZbO4RnNOXBXRUgUaK6Apf6c9AM5-NHq67tounEbnyEqoQbMxKoxnRndm3FFaMh4CNkTbrbR7qNg_W1p5JWwvD9DuNViLj2BUt1NfahoZg">After sorting the results by descending model confidence, a brief scan of the corresponding objects makes it clear that these annotations are wrong.This may be surprising given that the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a> is being used by thousands of researchers and models, but almost all datasets have these kinds of issues.<img src="https://lh4.googleusercontent.com/iPxyYxuBrw-c2817ZQiLrCxCU_xVEyC4aHDq5044UO30_qAEeXOXc3H9PnsXxiaD9BLRI4reFCGERZ1eI4KTjVRmoCifT41hOh8mGIvOOBMwwMJhG8oQuEE2aE5iiExfkkoRRu2LLbFxiDO3OkoWYeo">Now that you have discovered some errors with the data labels, you might want to correct them.With one click, you can send the bad labels over to Scale Studio, our self-service annotation platform, where your workforce can review and fix the errors. The annotation results will be automatically imported back into Scale Nucleus, overwriting the previous, flawed set of labels.<img src="https://lh3.googleusercontent.com/JWVK3_yCVI-Z1udBltlndb0T3qWfKboWTrHFaW2IFYnIoggE2PojskkIvJfQgBecKtkJtzf5f2ti7QeYrS7Cog2mPdnhHItx2-e31Uh3ZV3HM3fqcBTBQSpSrUTcCH9cTqvZphpSoO2ulpwyOLhB-o4"><h2>What’s next:</h2>At Scale, we are perpetually focused on both data and label quality. That’s why we build tools, products, and platforms to ensure that you’re getting the best results you possibly can from your ML models. But building high-performance models starts with error-free datasets. As you are <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">exploring your dataset in Nucleus</a>, verifying label accuracy, identifying edge cases, and collecting mismatches between predictions and ground truth, you may find that you and your teammates want to collaborate on adjusting or improving your ground truth labels. If you want to make these edits in house, in the most efficient way possible, <a href="https://dashboard.scale.com/signup?product=studio" rel="noopener noreferrer" target="_blank">sign up for Scale Studio today</a>.<h2>References</h2><ol><li>Crawford, Kate. "The trouble with bias." Conference on Neural Information Processing Systems, invited speaker. 2017.</li><li>Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." Conference on fairness, accountability and transparency. PMLR, 2018.</li><li>Raji, Inioluwa Deborah, and Genevieve Fried. "About face: A survey of facial recognition evaluation." arXiv preprint arXiv:2102.00813 (2021).</li><li>De Vries, Terrance, et al. "Does object recognition work for everyone?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.</li><li>Crawford, Kate, Trevor Paglen, and A. I. Excavating. "The politics of images in machine learning training sets." Excavating AI.</li><li>Northcutt, Curtis, Lu Jiang, and Isaac Chuang. "Confident learning: Estimating uncertainty in dataset labels." Journal of Artificial Intelligence Research 70 (2021): 1373-1411.</li></ol>

Nucleus helps you curate data to train better models; Studio helps you use internal resources to improve your dataset in ways that Nucleus flags.

Sometimes it's helpful to bring your own labelers domain expertise to assist in both data and model quality.

Building Great Models with Better Data on Nucleus and Studio

Nucleus is the centralized, scalable, and collaborative control center for your machine learning (ML) model’s training data. Nucleus helps you update missing or erroneous labels, and identify model failure cases so that you can direct your data curation efforts with the express goal of training better models.With simple integrations for model integration, testing, validation, and deployment, Nucleus can serve as your one stop shop to continuously iterate on your dataset for superior model quality, even in the face of new edge cases, model drift, and unexpected changes in environmental variables. Nucleus helps you tackle the long tail, the one certain path to training models that can address new scenarios and eliminate failure modes.Read on to learn the essentials—how Nucleus can help you manage your dataset efficiently.<h2>The basics:</h2><h3>Addressing model failures:</h3><ul><li>Where is my model currently failing and how can I fix these issues with the next round of labeling data?</li></ul>Once you’ve uploaded ground truth labels and at least one set of predictions for your dataset, you can click into the “Objects” view in Nucleus to identify False Positives, as shown in the below video. You can also identify other simple failure modes like False Negatives or low Intersection-over-Union (IoU) scores for image data.<a href="https://scaleai.wistia.com/medias/511bisntyb" rel="noopener noreferrer" target="_blank">this video</a><h3>Addressing label quality issues:</h3><ul><li>How can I easily check and improve the quality of existing labels?</li></ul>With your ground truth labels already in Nucleus along with one set of model predictions, the IoU slider and the “sort by” drop down menu lets you sort classifications by confidence from low to high. You can then choose if you want to label or fix the lowest-confidence images, with Rapid or if you’re an enterprise customer, your labeling pipeline.<a href="https://scaleai.wistia.com/medias/bgmcf03b9u" rel="noopener noreferrer" target="_blank">this video</a><h2>Breaking down your data in new ways:</h2><h3>Finding images without labels:</h3><ul><li>How can I find images with certain properties for which no ground truth label is available?</li></ul>If no label exists for the query you’ve conceptualized, you can search your dataset with simple, natural language text. Using CLIP embeddings under the hood, Natural Language Search lets you search for images with English phrases, instead of the structured query language you use for labels and metadata.&nbsp;<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">this video</a><h3>Finding similar images:</h3><ul><li>I’ve identified an example image: how can I find all similar images in my dataset?</li></ul>If no label is present for a class of images, but you’ve found a sample image, you can use Object or Image Autotag to build up a Slice of your dataset that consists of similar images. A step-by-step process will show you how to provide positive and negative examples, in multiple rounds, to fine-tune your search for like images. When you proceed to query your Autotag, you can set a threshold to broaden or narrow the scope of your search.<a href="https://scaleai.wistia.com/medias/uuavyfeapq" rel="noopener noreferrer" target="_blank">this video</a><h2>Debug your models:</h2><h3>Find classification failures:</h3><ul><li>How can I find examples of classes in which my model and my labels disagree?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can find a Confusion Matrix on the Charts page. This grid lets you find color-highlighted cases in which the labels and model disagree on a classification. You can click on any box to filter your entire dataset on just those examples, to further explore where your model fails and where it succeeds.<a href="https://scaleai.wistia.com/medias/ic8y27jymv" rel="noopener noreferrer" target="_blank">this video</a><h3>Sorting and filtering both failures and successes:</h3><ul><li>How can I find the best and worst classification examples?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can sort all of your data objects by confidence or intersection-over-union (IOU), or bound your images with specific minimum and maximum thresholds for these score values. You can also filter by specific types of confusions such as False Positives and False Negatives.<a href="https://scaleai.wistia.com/medias/c2f14217wn" rel="noopener noreferrer" target="_blank">this video</a><h3>Avoiding regressions:</h3><ul><li>How can I ensure I'm not breaking anything that worked with a previous model or dataset?</li></ul>As you improve and expand your dataset, then proceed to train new models, you’ll want to ensure that your new models don’t fail on cases that your old models solved successfully. Through Scale Validate, Nucleus helps you track metrics such as mean average precision (mAP), precision, and recall on specific scenarios in your dataset. Or, you can establish and write your own metrics to track over time. This process ensures that every label addition and dataset modification you make only yields improving outcomes rather than regressions. You can think of this as a continuous integration (CI) or regression detection process for your models.<a href="https://scaleai.wistia.com/medias/ak5lqayith" rel="noopener noreferrer" target="_blank">this video</a> Ready to try Nucleus? <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Click Here</a> to set up your first project.

Nucleus facilitates data curation and dataset management.

Learn how Nucleus facilitates dataset curation, to help you and your team train and debug better models.

Nucleus Simplified—Dataset Management for Everyone

All companies, from the earliest startups to the largest Fortune 100 corporations, face a decision as they scale up their artificial intelligence (AI) and machine learning (ML) efforts: How much of it should be built in-house?&nbsp;That’s a particularly thorny question when it comes to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>. It’s important to get the approach right from the start; studies have shown that an abundance of high-quality annotated data is typically the most significant driver of model performance. But how it’s done, across all steps of the data process — from selecting a <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> platform, to writing high-quality instructions and taxonomies, and doing the annotation itself — must be based on what works best for each unique organization.&nbsp;Here we explore what to keep in mind — the benefits and hidden pitfalls at each step of the process — as you try to solve your own data annotation build vs. buy equation.<h2>Selecting a Data Annotation Platform</h2>Choosing the right data annotation platform from the beginning of the process can minimize complications arising from changing approaches or platforms in the middle of a project.As you think about a software “home” for your data annotation activities, there are a few key capabilities you should consider:&nbsp;<ul><li>How easy is it to set up a new project or pipeline for data annotation?</li><li>How quickly and easily can annotators add high-quality, accurate annotations to your data?&nbsp;</li><li>How simple is the process to add annotators to the platform?&nbsp;</li><li>Can you analyze the annotation workforce’s performance across dimensions such as throughput, quality, and time spent?&nbsp;</li><li>What are the guardrails or in-product capabilities to ensure the data annotation output is as high-quality as possible?&nbsp;</li><li>Can you import your model output as “hypothesis annotations” to give your annotators a head-start on their work?</li></ul>Indexing on these capabilities can help guide your decision on whether to build a platform in-house or purchase a software solution. Once you’ve established your criteria in each of these dimensions, the build-vs.-buy equation comes down to a few ROI calculations:<ol><li>Customizability: If you’re looking for a platform tailor-made to suit your specific annotation needs, there will be no platform that is more customized than one that you and your team build yourself. This is doubly true if you have annotation needs that are non-standard and have minimal (if any) support from annotation platforms on the market today. The fact of the matter is, commercial tooling caters to the most common use cases, workflows, and annotation types, which may suit the majority of situations, but may not work for you. Some people customize an open source solution (such as <a href="https://github.com/openvinotoolkit/cvat" rel="noopener noreferrer" target="_blank">CVAT</a>) in order to suit their needs, and others build their own from scratch.&nbsp;</li><li>Predictability: Are your AI/ML efforts predictable over the next few years, or is it likely that you’ll experiment with different data types or annotation volumes? If you expect the scope of any AI/ML project to evolve rapidly, buying a commercial platform can be a prescient choice, since you’ll be able to spin up new data types or annotation types more quickly, provided that they’re supported. If you have a sense of data types you’ll likely experiment with in the future, it’s smart to check if they are supported in whatever annotation platform you consider</li><li>Time to market: If speed to production is an important consideration, buying an annotation platform will almost always get you there faster than building one in-house. The one exception is if your data and annotation type are simple, and supported by an open-source solution out-of-the-box. Otherwise, building a custom in-house platform can take months to get the first project in production. That’s an opportunity cost that is often neglected, but which may pose one of the greatest implicit expenses (or loss of revenue) for your organization.</li><li>Resources for maintenance: Building an in-house solution will require engineers to maintain and update it to accommodate new experiments or pipelines, and keep up with new standards and approaches. That said, even if you choose a commercial platform, you’ll still likely need to budget some engineering time to set up your pipelines – albeit less than if you’re starting from scratch – especially if you are utilizing API capabilities to import data, create projects, and/or export data.&nbsp;</li></ol>In your overall consideration, keep an eye on both short-term and long-term costs. For some smaller organizations, building an in-house tool or customizing an open-source solution “feels” cheaper because there’s no explicit cash outlay. However, you should consider whether the time and expense involved in establishing and maintaining your own data annotation tool is worth it. Future budget demands are more likely to go toward developing and training models than toward maintaining your in-house data annotation platform.For the largest organizations with custom annotation needs, the calculus becomes more nuanced. In these cases, where the team has a need for steady, large-scale annotation, where having a team of engineers dedicated to building and maintaining an in-house platform could start to make financial sense. That said, if you anticipate annotation needs evolving and quick time-to-market is important, be sure to still consider a commercial solution – at least initially, with an eye towards a potential eventual transition to in-house.<h2>Developing High-Quality Instructions &amp; Taxonomies</h2>In addition to selecting the right platform, you will also need high-quality instructions and taxonomies if you want to train high-quality models. Whether you use your in-house annotation team or outsource to a third party, explaining exactly what you want annotated (along with examples of what to do vs. not do) is critical to getting high-quality data back.&nbsp;This is an area where it makes sense to keep in-house as much as possible. You and your team will have the best, most intimate knowledge of what exactly you need annotated, as the team developing the models. By keeping this in-house, you’ll have a tight feedback loop as you scale up. You’d be surprised at how certain instructions are interpreted — and by having your ML team stay tightly involved in workshopping the instructions, providing examples, and adjusting the taxonomies, you can be confident that you're getting the best possible data back.<h2>Choosing a Workforce for Data Annotation</h2>Most teams start annotating data with internal employees who already are familiar with the data and who can start the process part-time. But as the data volume ramps up, be prepared to either recruit new full-time members to the annotation team or to outsource the work entirely to a third party.Things to consider when making this decision:<ul><li>Cost: Paying the annotation team is not the only consideration here. For a more realistic assessment of the cost of outsourcing data annotation, factor in the time and effort involved in managing an in-house team.</li><li>Flexibility: Will your data annotation needs ramp up and down quickly? Some third-party providers are prepared to handle such surges with minimal disruptions. An in-house team will need to be kept prepared to jump back in during a scaleback, while still appropriately occupied.</li><li>Subject matter expertise: While some third-party providers offer teams with specific expertise, most do not. If your annotation work is highly specialized, it may be best to recruit and train your own data annotation experts in-house.</li></ul>Some organizations choose to utilize a blend of in-house as well as external data annotators. One common workflow is using the external team to do the ‘first attempt’ of annotation, while the internal team reviews the attempts and provides an additional layer of quality assurance. Others utilize the external team to annotate steady production pipelines, and the internal team to work on data for short, bursty experiments.<h2>Conclusion: Buy Some, Customize Others</h2>Ultimately, answering the build vs. buy question doesn’t have to be an all-or-nothing proposition. If your data annotation needs are particularly custom, your budget for in-house development relatively unrestricted, and any future needs relatively predictable, you may be able to develop a great annotation solution in-house. For other teams, commercial software is likely a good choice: it can be deployed rapidly, and you can choose how and when to use its various parts. Depending on your needs and scope, third-party providers can work with your in-house teams to build or modify features and extensions as your AI/ML projects evolve.&nbsp;If you’ve considered your options and decided that you’d like to try an annotation platform built by annotation experts, <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.

Choosing the right data annotation approach can minimize complications down the line. What are the factors you should consider in making this build vs buy decision?

Should you build or buy a data annotation solution?

Build vs. Buy: What Makes Sense for Data Annotation?

<h3>It all started with a breakdancing video.</h3>Now, on the face of it, that shouldn’t have been too surprising, back in 2010. YouTube was already well on its way to assembling a massive collection of the human experience. We had run programs like Life in a Day - asking viewers around the world to submit videos of their daily life on a single day - to document everyday life in a way no-one had parsed before. Breakdancing and hip-hop culture were undeniably a part of that global experience - maybe surprisingly, in countries like France, where communities like <a href="https://fr.wikipedia.org/wiki/Juste_debout" rel="noopener noreferrer" target="_blank">Juste Debout </a>had started to document a thriving scene.&nbsp;But at YouTube, we had a big problem making sense of those videos. Viewers from around the world had uploaded a huge range of content, with little in the way of titles or tags to make sense of it all. For many uploaders, it was their first time uploading a video to the internet - ever. In spite of that, what made YouTube successful, and what my team focused on, was a vision to drive viewers to videos they didn’t expect, and that hadn’t amassed thousands of views - through related videos, through search, and through trends.So you can imagine my surprise in 2010 when my team showed me a 15 second video of a man spinning on his back with his sneakers in the air, on a street somewhere in Europe. The video had a random title and the uploader had no followers. Under normal circumstances, this would be a very difficult video for someone to find - just a random selection out of hundreds of thousands of videos that morning. But that day, we tried a new approach to understanding videos, based on what were called Convolutional Neural Networks, with the capacity and training horsepower to work across the whole range of the YouTube catalog.Sure enough, this time, bubbles appeared next to the video, saying “breakdancing”, “spinning”, even “<a href="https://www.google.com/search?q=breakdancing+munchmill&amp;rlz=1C5GCEM_enUS996US996&amp;ei=hCuWYtHoM7m2uvQPy9eCcA&amp;ved=0ahUKEwjRuq_M_on4AhU5m44IHcurAA4Q4dUDCA4&amp;uact=5&amp;oq=breakdancing+munchmill&amp;gs_lcp=Cgdnd3Mtd2l6EAM6BwgAEEcQsANKBAhBGABKBAhGGABQ9QJYpA1gkRFoAXABeACAAWyIAfYGkgEDNC41mAEAoAEByAEIwAEB&amp;sclient=gws-wiz" rel="noopener noreferrer" target="_blank">munchmill</a>” . Trained on tens of millions of videos, deep learning had discovered the connections between this human movement across a few frames of video, and an entire global culture built around hip-hop.&nbsp;For me, those connections launched the next decade of my career. The combination of powerful, expressive machine learning models, with new approaches to understanding the diversity and complexity of content shared on the web, was an incredible opportunity to build new experiences. My time launching a startup with Steve Chen highlighted for me how those techniques could be brought to bear to live and immersive content, while my time at Apple helped me to understand how “data centric” AI could also be used to build new hardware and software experiences with a “privacy-first” mindset.I recently joined Scale to run our engineering organization, in part because of a similar credo that Scale has in our culture - that Ambition Shapes Reality. The next decade of AI achievement is going to be built by teams that take on outsized ambitions, and understand that it is not always a straight line to get there.&nbsp;<h3 class="ql-align-center">Operational&nbsp; &amp; Data Centric AI</h3>&nbsp;The market for AI developer tools and infrastructure is currently small, and limited by the number of companies who are actually AI capable and have teams capable of using ML to drive business goals. But in the long run this market will be an industry-defining set of capabilities, and we have a golden opportunity to own the primitives that build this market, given our leadership position in autonomous vehicles and perception data. Those capabilities will include the ability to drive operational improvements through a counterfactual view of complex systems, being able to leverage multi-modal and Foundation models to work across visual and language data, and building insights in real-time as data distributions shift.Building out these capabilities for ML teams not only gives us a call option on this really exciting future, but it also gets us early and close access to the small pool of AI practitioners who are pioneering that future today. That partnership between forward-looking researchers and product leaders and engineers characterizes the fastest-moving ML teams today.<h3 class="ql-align-center">Running Through Walls</h3>When I was thinking about joining Scale, I realized I had a lot of vision alignment with Alex, Scale’s CEO. In an internal memo, Alex recently wrote, “I am tremendously excited about what’s beginning to really work here, probably more than I’ve ever been in the history of Scale.” It was obvious to me that Scale is unique, in part due to the massive opportunities in the rapidly growing emergent field of AI, and also in part by the mission upheld by leadership.&nbsp;Alex truly understands the nuances of AI. He recently did a <a href="https://www.youtube.com/watch?v=diBoo0pQ8c8" rel="noopener noreferrer" target="_blank">TED Talk at Berkeley </a>where he reiterated that AI needs humans to teach it individual values, nudge it to find thoughtful outcomes, and ensure that human intentions and values are aligned with the AI. He talked about how machines make mistakes and how they have to be taught to tell the truth.&nbsp;This alignment was a grounding factor for me. Alex understands that “AI is a supercharger for humans - when AI is better than humans, it makes humans better. AI will enable us to be even more creative and more idea driven. It allows us to embrace the generative aspect of human nature so we can run faster with our ideas and build better, more powerful solutions to the world’s biggest problems”. And building solutions to the world’s biggest problems is a focal point of Scale’s culture and formed the company’s Credos.Our credos provide a framework for us to make decisions and work effectively as a team. The one that has resonated with me the most is “run through walls.’ We are undaunted by barriers and race towards overcoming them. We step outside of our comfort zone, test our ideas, and do the hard work to get to the right solution to a hard problem. I have seen this credo in action with every one of my team members.&nbsp;<h3 class="ql-align-center">Become a Scalien!</h3>Scale is in an exciting place where growth opportunities for AI organically grow your career too. Being at Scale allows you to work with high visibility customers - often impacting millions of users.You’d have the opportunity to ideate with some of the world’s leading ML and Data Science teams at our partner companies, and find creative ways to adapt new ML techniques like Generative Models, Diffusion Models,&nbsp; and Large Language Models, or Online Optimization techniques applied to expressive embedding spaces, to building a new product.It’s been a long and rewarding journey since that first breakdancing video. I know AI and its innovation will continue to surprise me and I am excited to experience that here at Scale. I hope that you will join me here too.&nbsp;

How a breakdancing video launched a career in AI

Why I joined Scale

open-dataset-nucleus

slugs must be all lowercase. ie: ‘my-slug’

events-banner

This entry represents both interview quote highlight and interview chat highlight because of Builder limitations. Choose the fields that matches either of them.

interview-highlight

If it's selected, the author and quote fields will be taken, otherwise the chat fields. The default is for chat.

resources-download-model

transform-session

CTA to open the session. Leave blank if you don't want a cta for that session

Link for the CTA. Usually the link to the exchange session

blog-filter-type

event-transform-x-ticket

solution

homepage-quotes

logos

homepage-content

builder-debug

transfrom-x-in-person

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

guides-cta

summit-2023-partners

sales

z[deprecated] page

quotes

z[deprecated] compare

product

blog-category-model

announcement-banner

accelerate

Vercel Powered and Edge Computed OG Image. Preferred over og img file upload. If this field is non empty, it will take precedent over the uploaded file

summit-2023-speakers

homepage-resources

customers

Display order on the page /customers. Higher values show first

page-cross-product-webinar

interview

event-transform

Event model for new marketing events page

event-2-0

homepage-event

custom-fonts

transform-x-speaker

Be careful to not add any trailing white spaces to the slug. Slugs must be all lowercase. ie: ‘my-slug’

Order of the featured speakers displayed on the home page for transformX

z[deprecated] document-ai

summit-2023-agenda

event-converge

hosts

author

transform-x

Page: /In-Person. Section: Hero. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Card Body

Page: /In-Person. Section: Hero. Text: Tagline

open-dataset

z[deprecated] customers

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

legal

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

z[deprecated] guides-old

z[deprecated] guides-lp

partners

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Government

Solutions

Resources

Customers

Pricing →

Blog

The future of your industry starts here.