experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

image.jpg

Learn how we extended our infrastructure by sharding our MongoDB cluster.

From MVP to Serving Enterprise Scale

How We Scaled Our MongoDB Instance Gracefully

<p>Large-scale datasets are at the center of model development and evaluation. Quality of training data is the dominant factor for model performance. Anyone can create an excellent model based on great data, but even the best ML engineers can’t design good models with bad data.&nbsp;</p><p>If you’ve ever used popular public datasets such as ImageNet or MNIST, you would assume that the class and attribute <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labels</a> are correct. Shockingly, ImageNet, the most widely used dataset for popular image recognition systems, has a 6% error rate[6]. When models are fed bad data, such as incorrect classes, incorrect features, or missing <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a>, they often have severe classification problems, which can destabilize ML benchmarks and contribute to representational harms in datasets.&nbsp;</p><p>Examples of representational harms[1] in datasets include under-representation of darker skinned subjects compared with lighter-skinned subjects within prominent facial analysis datasets[2,3], and millions of images of people in ImageNet labeled with offensive categories, including racial slurs and derogatory phrases[5].&nbsp;</p><p>Even in leading applied ML research groups, approaches for characterizing and fixing labeling errors in massive datasets are limited.&nbsp;</p><p>One approach to combat noise in a dataset is Confident Learning (CL), which is based on pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. However, CL entirely ignores the case where one or more true labels are missing. This limitation occurs because CL relies on building a pairwise class-conditional distribution between the given and true labels[6]. Other techniques include sorting by model loss, which surfaces the examples that are most ambiguous and difficult to learn, and sorting by confidence where model and ground truth disagree (quite effective!).</p><p>What is the most principled option after amassing a collection of bad labels? In practice, it would be fixing your dataset by going through the images and fixing the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> one by one.&nbsp;</p><p>This is both laborious and impractical.</p><p>That’s why (1.), a way to visualize, triage, and organize your data, and (2.) a way to fix your annotations, are critical to your data annotation project.</p><p>Scale provides an integrated dataset and workforce management platform that solves these problems.</p><p>Let’s walk through an example of how you can use two of Scale’s products: Scale Nucleus and Scale Studio to identify and fix bad labels leveraging model predictions. For this example, we’ll be using the public <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a>.&nbsp;</p><p><img src="https://lh3.googleusercontent.com/hYW3_C_Yz4vuaZKQppVBRJB273v9Nz2b9bgHBDbl-WUrdKjfHg31Ic6E2j34WJZSuXSufVkBgfWf0Q7ThfnZ-l7CxScpOViGECQXXHBhk3_VkMqYT0t-GpgGL_1J_O97Vg7wmCjTPh-PSodX-Zm7o-c"></p><p>Starting on the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">“Charts” page in Scale Nucleus</a>, a simple click into the confusion matrix shows 2,277 cases in which models predicted “person” but the ground truth label is “car”.</p><p><img src="https://lh5.googleusercontent.com/e-ElAIxlS6X1koxM2vOVAHv-MUG_IC_trcCX6K1YXyvDDPZbO4RnNOXBXRUgUaK6Apf6c9AM5-NHq67tounEbnyEqoQbMxKoxnRndm3FFaMh4CNkTbrbR7qNg_W1p5JWwvD9DuNViLj2BUt1NfahoZg"></p><p>After sorting the results by descending model confidence, a brief scan of the corresponding objects makes it clear that these annotations are wrong.</p><p>This may be surprising given that the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a> is being used by thousands of researchers and models, but almost all datasets have these kinds of issues.</p><p><img src="https://lh4.googleusercontent.com/iPxyYxuBrw-c2817ZQiLrCxCU_xVEyC4aHDq5044UO30_qAEeXOXc3H9PnsXxiaD9BLRI4reFCGERZ1eI4KTjVRmoCifT41hOh8mGIvOOBMwwMJhG8oQuEE2aE5iiExfkkoRRu2LLbFxiDO3OkoWYeo"></p><p>Now that you have discovered some errors with the data labels, you might want to correct them.</p><p>With one click, you can send the bad labels over to Scale Studio, our self-service annotation platform, where your workforce can review and fix the errors. The annotation results will be automatically imported back into Scale Nucleus, overwriting the previous, flawed set of labels.</p><p><img src="https://lh3.googleusercontent.com/JWVK3_yCVI-Z1udBltlndb0T3qWfKboWTrHFaW2IFYnIoggE2PojskkIvJfQgBecKtkJtzf5f2ti7QeYrS7Cog2mPdnhHItx2-e31Uh3ZV3HM3fqcBTBQSpSrUTcCH9cTqvZphpSoO2ulpwyOLhB-o4"></p><h2>What’s next:</h2><p>At Scale, we are perpetually focused on both data and label quality. That’s why we build tools, products, and platforms to ensure that you’re getting the best results you possibly can from your ML models. But building high-performance models starts with error-free datasets. As you are <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">exploring your dataset in Nucleus</a>, verifying label accuracy, identifying edge cases, and collecting mismatches between predictions and ground truth, you may find that you and your teammates want to collaborate on adjusting or improving your ground truth labels. If you want to make these edits in house, in the most efficient way possible, <a href="https://dashboard.scale.com/signup?product=studio" rel="noopener noreferrer" target="_blank">sign up for Scale Studio today</a>.</p><h2>References</h2><ol><li>Crawford, Kate. "The trouble with bias." <em>Conference on Neural Information Processing Systems, invited speaker</em>. 2017.</li><li>Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." <em>Conference on fairness, accountability and transparency</em>. PMLR, 2018.</li><li>Raji, Inioluwa Deborah, and Genevieve Fried. "About face: A survey of facial recognition evaluation." <em>arXiv preprint arXiv:2102.00813</em> (2021).</li><li>De Vries, Terrance, et al. "Does object recognition work for everyone?." <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops</em>. 2019.</li><li>Crawford, Kate, Trevor Paglen, and A. I. Excavating. "The politics of images in machine learning training sets." <em>Excavating AI</em>.</li><li>Northcutt, Curtis, Lu Jiang, and Isaac Chuang. "Confident learning: Estimating uncertainty in dataset labels." Journal of Artificial Intelligence Research 70 (2021): 1373-1411.</li></ol><p><br></p>

Nucleus helps you curate data to train better models; Studio helps you use internal resources to improve your dataset in ways that Nucleus flags.

Sometimes it's helpful to bring your own labelers domain expertise to assist in both data and model quality.

Building Great Models with Better Data on Nucleus and Studio

<p>Nucleus is the centralized, scalable, and collaborative control center for your machine learning (ML) model’s training data. Nucleus helps you update missing or erroneous labels, and identify model failure cases so that you can direct your data curation efforts with the express goal of training better models.</p><p>With simple integrations for model integration, testing, validation, and deployment, Nucleus can serve as your one stop shop to continuously iterate on your dataset for superior model quality, even in the face of new edge cases, model drift, and unexpected changes in environmental variables. Nucleus helps you tackle the long tail, the one certain path to training models that can address new scenarios and eliminate failure modes.</p><p>Read on to learn the essentials—how Nucleus can help you manage your dataset efficiently.</p><h2>The basics:</h2><h3>Addressing model failures:</h3><ul><li><em>Where is my model currently failing and how can I fix these issues with the next round of labeling data?</em></li></ul><p>Once you’ve uploaded ground truth labels and at least one set of predictions for your dataset, you can click into the “Objects” view in Nucleus to identify False Positives, as shown in the below video. You can also identify other simple failure modes like False Negatives or low Intersection-over-Union (IoU) scores for image data.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/511bisntyb" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Addressing label quality issues:</h3><ul><li><em>How can I easily check and improve the quality of existing labels?</em></li></ul><p>With your ground truth labels already in Nucleus along with one set of model predictions, the IoU slider and the “sort by” drop down menu lets you sort classifications by confidence from low to high. You can then choose if you want to label or fix the lowest-confidence images, with Rapid or if you’re an enterprise customer, your labeling pipeline.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/bgmcf03b9u" rel="noopener noreferrer" target="_blank">this video</a></p><h2>Breaking down your data in new ways:</h2><h3>Finding images without labels:</h3><ul><li><em>How can I find images with certain properties for which no ground truth label is available?</em></li></ul><p>If no label exists for the query you’ve conceptualized, you can search your dataset with simple, natural language text. Using CLIP embeddings under the hood, Natural Language Search lets you search for images with English phrases, instead of the structured query language you use for labels and metadata.&nbsp;</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Finding similar images:</h3><ul><li><em>I’ve identified an example image: how can I find all similar images in my dataset?</em></li></ul><p>If no label is present for a class of images, but you’ve found a sample image, you can use Object or Image Autotag to build up a Slice of your dataset that consists of similar images. A step-by-step process will show you how to provide positive and negative examples, in multiple rounds, to fine-tune your search for like images. When you proceed to query your Autotag, you can set a threshold to broaden or narrow the scope of your search.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/uuavyfeapq" rel="noopener noreferrer" target="_blank">this video</a></p><h2>Debug your models:</h2><h3>Find classification failures:</h3><ul><li><em>How can I find examples of classes in which my model and my labels disagree?</em></li></ul><p>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can find a Confusion Matrix on the Charts page. This grid lets you find color-highlighted cases in which the labels and model disagree on a classification. You can click on any box to filter your entire dataset on just those examples, to further explore where your model fails and where it succeeds.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/ic8y27jymv" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Sorting and filtering both failures and successes:</h3><ul><li><em>How can I find the best and worst classification examples?</em></li></ul><p>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can sort all of your data objects by confidence or intersection-over-union (IOU), or bound your images with specific minimum and maximum thresholds for these score values. You can also filter by specific types of confusions such as False Positives and False Negatives.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/c2f14217wn" rel="noopener noreferrer" target="_blank">this video</a></p><h3>Avoiding regressions:</h3><ul><li><em>How can I ensure I'm not breaking anything that worked with a previous model or dataset?</em></li></ul><p>As you improve and expand your dataset, then proceed to train new models, you’ll want to ensure that your new models don’t fail on cases that your old models solved successfully. Through Scale Validate, Nucleus helps you track metrics such as mean average precision (mAP), precision, and recall on specific scenarios in your dataset. Or, you can establish and write your own metrics to track over time. This process ensures that every label addition and dataset modification you make only yields improving outcomes rather than regressions. You can think of this as a continuous integration (CI) or regression detection process for your models.</p><p class="ql-align-center"><a href="https://scaleai.wistia.com/medias/ak5lqayith" rel="noopener noreferrer" target="_blank">this video</a></p><p><br></p><p>Ready to try Nucleus? <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Click Here</a> to set up your first project.</p>

Nucleus facilitates data curation and dataset management.

Learn how Nucleus facilitates dataset curation, to help you and your team train and debug better models.

Nucleus Simplified—Dataset Management for Everyone

<p>All companies, from the earliest startups to the largest Fortune 100 corporations, face a decision as they scale up their artificial intelligence (AI) and machine learning (ML) efforts: How much of it should be built in-house?&nbsp;</p><p>That’s a particularly thorny question when it comes to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>. It’s important to get the approach right from the start; studies have shown that an abundance of high-quality annotated data is typically the most significant driver of model performance. But <em>how</em> it’s done, across all steps of the data process — from selecting a <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> platform, to writing high-quality instructions and taxonomies, and doing the annotation itself — must be based on what works best for each unique organization.&nbsp;</p><p>Here we explore what to keep in mind — the benefits and hidden pitfalls at each step of the process — as you try to solve your own data annotation build vs. buy equation.</p><h2>Selecting a Data Annotation Platform</h2><p>Choosing the right data annotation platform from the beginning of the process can minimize complications arising from changing approaches or platforms in the middle of a project.</p><p>As you think about a software “home” for your data annotation activities, there are a few key capabilities you should consider:&nbsp;</p><ul><li>How easy is it to set up a new project or pipeline for data annotation?</li><li>How quickly and easily can annotators add high-quality, accurate annotations to your data?&nbsp;</li><li>How simple is the process to add annotators to the platform?&nbsp;</li><li>Can you analyze the annotation workforce’s performance across dimensions such as throughput, quality, and time spent?&nbsp;</li><li>What are the guardrails or in-product capabilities to ensure the data annotation output is as high-quality as possible?&nbsp;</li><li>Can you import your model output as “hypothesis annotations” to give your annotators a head-start on their work?</li></ul><p>Indexing on these capabilities can help guide your decision on whether to build a platform in-house or purchase a software solution. Once you’ve established your criteria in each of these dimensions, the build-vs.-buy equation comes down to a few ROI calculations:</p><ol><li>Customizability: If you’re looking for a platform tailor-made to suit your specific annotation needs, there will be no platform that is more customized than one that you and your team build yourself. This is doubly true if you have annotation needs that are non-standard and have minimal (if any) support from annotation platforms on the market today. The fact of the matter is, commercial tooling caters to the most common use cases, workflows, and annotation types, which may suit the majority of situations, but may not work for you. Some people customize an open source solution (such as <a href="https://github.com/openvinotoolkit/cvat" rel="noopener noreferrer" target="_blank">CVAT</a>) in order to suit their needs, and others build their own from scratch.&nbsp;</li><li>Predictability: Are your AI/ML efforts predictable over the next few years, or is it likely that you’ll experiment with different data types or annotation volumes? If you expect the scope of any AI/ML project to evolve rapidly, buying a commercial platform can be a prescient choice, since you’ll be able to spin up new data types or annotation types more quickly, provided that they’re supported. If you have a sense of data types you’ll likely experiment with in the future, it’s smart to check if they are supported in whatever annotation platform you consider</li><li>Time to market: If speed to production is an important consideration, buying an annotation platform will almost always get you there faster than building one in-house. The one exception is if your data and annotation type are simple, and supported by an open-source solution out-of-the-box. Otherwise, building a custom in-house platform can take months to get the first project in production. That’s an opportunity cost that is often neglected, but which may pose one of the greatest implicit expenses (or loss of revenue) for your organization.</li><li>Resources for maintenance: Building an in-house solution will require engineers to maintain and update it to accommodate new experiments or pipelines, and keep up with new standards and approaches. That said, even if you choose a commercial platform, you’ll still likely need to budget some engineering time to set up your pipelines – albeit less than if you’re starting from scratch – especially if you are utilizing API capabilities to import data, create projects, and/or export data.&nbsp;</li></ol><p>In your overall consideration, keep an eye on both short-term and long-term costs. For some smaller organizations, building an in-house tool or customizing an open-source solution “feels” cheaper because there’s no explicit cash outlay. However, you should consider whether the time and expense involved in establishing and maintaining your own data annotation tool is worth it. Future budget demands are more likely to go toward developing and training models than toward maintaining your in-house data annotation platform.</p><p>For the largest organizations with custom annotation needs, the calculus becomes more nuanced. In these cases, where the team has a need for steady, large-scale annotation, where having a team of engineers dedicated to building and maintaining an in-house platform could start to make financial sense. That said, if you anticipate annotation needs evolving and quick time-to-market is important, be sure to still consider a commercial solution – at least initially, with an eye towards a potential eventual transition to in-house.</p><h2>Developing High-Quality Instructions &amp; Taxonomies</h2><p>In addition to selecting the right platform, you will also need high-quality instructions and taxonomies if you want to train high-quality models. Whether you use your in-house annotation team or outsource to a third party, explaining exactly what you want annotated (along with examples of what to do vs. not do) is critical to getting high-quality data back.&nbsp;</p><p>This is an area where it makes sense to keep in-house as much as possible. You and your team will have the best, most intimate knowledge of what exactly you need annotated, as the team developing the models. By keeping this in-house, you’ll have a tight feedback loop as you scale up. You’d be surprised at how certain instructions are interpreted — and by having your ML team stay tightly involved in workshopping the instructions, providing examples, and adjusting the taxonomies, you can be confident that you're getting the best possible data back.</p><h2>Choosing a Workforce for Data Annotation</h2><p>Most teams start annotating data with internal employees who already are familiar with the data and who can start the process part-time. But as the data volume ramps up, be prepared to either recruit new full-time members to the annotation team or to outsource the work entirely to a third party.</p><p>Things to consider when making this decision:</p><ul><li>Cost: Paying the annotation team is not the only consideration here. For a more realistic assessment of the cost of outsourcing data annotation, factor in the time and effort involved in managing an in-house team.</li><li>Flexibility: Will your data annotation needs ramp up and down quickly? Some third-party providers are prepared to handle such surges with minimal disruptions. An in-house team will need to be kept prepared to jump back in during a scaleback, while still appropriately occupied.</li><li>Subject matter expertise: While some third-party providers offer teams with specific expertise, most do not. If your annotation work is highly specialized, it may be best to recruit and train your own data annotation experts in-house.</li></ul><p>Some organizations choose to utilize a blend of in-house as well as external data annotators. One common workflow is using the external team to do the ‘first attempt’ of annotation, while the internal team reviews the attempts and provides an additional layer of quality assurance. Others utilize the external team to annotate steady production pipelines, and the internal team to work on data for short, bursty experiments.</p><h2>Conclusion: Buy Some, Customize Others</h2><p>Ultimately, answering the build vs. buy question doesn’t have to be an all-or-nothing proposition. If your data annotation needs are particularly custom, your budget for in-house development relatively unrestricted, and any future needs relatively predictable, you may be able to develop a great annotation solution in-house. For other teams, commercial software is likely a good choice: it can be deployed rapidly, and you can choose how and when to use its various parts. Depending on your needs and scope, third-party providers can work with your in-house teams to build or modify features and extensions as your AI/ML projects evolve.&nbsp;</p><p>If you’ve considered your options and decided that you’d like to try an annotation platform built by annotation experts, <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.</p>

Choosing the right data annotation approach can minimize complications down the line. What are the factors you should consider in making this build vs buy decision?

Should you build or buy a data annotation solution?

Build vs. Buy: What Makes Sense for Data Annotation?

<h3>It all started with a breakdancing video.</h3><p>Now, on the face of it, that shouldn’t have been too surprising, back in 2010. YouTube was already well on its way to assembling a massive collection of the human experience. We had run programs like Life in a Day - asking viewers around the world to submit videos of their daily life on a single day - to document everyday life in a way no-one had parsed before. Breakdancing and hip-hop culture were undeniably a part of that global experience - maybe surprisingly, in countries like France, where communities like <a href="https://fr.wikipedia.org/wiki/Juste_debout" rel="noopener noreferrer" target="_blank">Juste Debout </a>had started to document a thriving scene.&nbsp;</p><p>But at YouTube, we had a big problem making sense of those videos. Viewers from around the world had uploaded a huge range of content, with little in the way of titles or tags to make sense of it all. For many uploaders, it was their first time uploading a video to the internet - <em>ever</em>. In spite of that, what made YouTube successful, and what my team focused on, was a vision to drive viewers to videos they didn’t expect, and that hadn’t amassed thousands of views - through related videos, through search, and through trends.</p><p>So you can imagine my surprise in 2010 when my team showed me a 15 second video of a man spinning on his back with his sneakers in the air, on a street somewhere in Europe. The video had a random title and the uploader had no followers. Under normal circumstances, this would be a very difficult video for someone to find - just a random selection out of hundreds of thousands of videos that morning. But that day, we tried a new approach to understanding videos, based on what were called Convolutional Neural Networks, with the capacity and training horsepower to work across the whole range of the YouTube catalog.</p><p>Sure enough, this time, bubbles appeared next to the video, saying “breakdancing”, “spinning”, even “<a href="https://www.google.com/search?q=breakdancing+munchmill&amp;rlz=1C5GCEM_enUS996US996&amp;ei=hCuWYtHoM7m2uvQPy9eCcA&amp;ved=0ahUKEwjRuq_M_on4AhU5m44IHcurAA4Q4dUDCA4&amp;uact=5&amp;oq=breakdancing+munchmill&amp;gs_lcp=Cgdnd3Mtd2l6EAM6BwgAEEcQsANKBAhBGABKBAhGGABQ9QJYpA1gkRFoAXABeACAAWyIAfYGkgEDNC41mAEAoAEByAEIwAEB&amp;sclient=gws-wiz" rel="noopener noreferrer" target="_blank">munchmill</a>” . Trained on tens of millions of videos, deep learning had discovered the connections between this human movement across a few frames of video, and an entire global culture built around hip-hop.&nbsp;</p><p>For me, those connections launched the next decade of my career. The combination of powerful, expressive machine learning models, with new approaches to understanding the diversity and complexity of content shared on the web, was an incredible opportunity to build new experiences. My time launching a startup with Steve Chen highlighted for me how those techniques could be brought to bear to live and immersive content, while my time at Apple helped me to understand how “data centric” AI could also be used to build new hardware and software experiences with a “privacy-first” mindset.</p><p>I recently joined Scale to run our engineering organization, in part because of a similar credo that Scale has in our culture - that Ambition Shapes Reality. The next decade of AI achievement is going to be built by teams that take on outsized ambitions, and understand that it is not always a straight line to get there.&nbsp;</p><h3 class="ql-align-center">Operational&nbsp; &amp; Data Centric AI</h3><p>&nbsp;The market for AI developer tools and infrastructure is currently small, and limited by the number of companies who are actually AI capable and have teams capable of using ML to drive business goals. But in the long run this market will be an industry-defining set of capabilities, and we have a golden opportunity to own the primitives that build this market, given our leadership position in autonomous vehicles and perception data. Those capabilities will include the ability to drive operational improvements through a counterfactual view of complex systems, being able to leverage multi-modal and Foundation models to work across visual and language data, and building insights in real-time as data distributions shift.</p><p>Building out these capabilities for ML teams not only gives us a call option on this really exciting future, but it also gets us early and close access to the small pool of AI practitioners who are pioneering that future today. That partnership between forward-looking researchers and product leaders and engineers characterizes the fastest-moving ML teams today.</p><h3 class="ql-align-center">Running Through Walls</h3><p>When I was thinking about joining Scale, I realized I had a lot of vision alignment with Alex, Scale’s CEO. In an internal memo, Alex recently wrote, “I am tremendously excited about what’s beginning to really work here, probably more than I’ve ever been in the history of Scale.” It was obvious to me that Scale is unique, in part due to the massive opportunities in the rapidly growing emergent field of AI, and also in part by the mission upheld by leadership.&nbsp;</p><p>Alex truly understands the nuances of AI. He recently did a <a href="https://www.youtube.com/watch?v=diBoo0pQ8c8" rel="noopener noreferrer" target="_blank">TED Talk at Berkeley </a>where he reiterated that AI needs humans to teach it individual values, nudge it to find thoughtful outcomes, and ensure that human intentions and values are aligned with the AI. He talked about how machines make mistakes and how they have to be taught to tell the truth.&nbsp;</p><p>This alignment was a grounding factor for me. Alex understands that “AI is a supercharger for humans - when AI is better than humans, it makes humans better. AI will enable us to be even more creative and more idea driven. It allows us to embrace the generative aspect of human nature so we can run faster with our ideas and build better, more powerful solutions to the world’s biggest problems”. And building solutions to the world’s biggest problems is a focal point of Scale’s culture and formed the company’s Credos.</p><p>Our credos provide a framework for us to make decisions and work effectively as a team. The one that has resonated with me the most is “run through walls.’ We are undaunted by barriers and race towards overcoming them. We step outside of our comfort zone, test our ideas, and do the hard work to get to the right solution to a hard problem. I have seen this credo in action with every one of my team members.&nbsp;</p><h3 class="ql-align-center">Become a Scalien!</h3><p>Scale is in an exciting place where growth opportunities for AI organically grow your career too. Being at Scale allows you to work with high visibility customers - often impacting millions of users.</p><p>You’d have the opportunity to ideate with some of the world’s leading ML and Data Science teams at our partner companies, and find creative ways to adapt new ML techniques like Generative Models, Diffusion Models,&nbsp; and Large Language Models, or Online Optimization techniques applied to expressive embedding spaces, to building a new product.</p><p>It’s been a long and rewarding journey since that first breakdancing video. I know AI and its innovation will continue to surprise me and I am excited to experience that here at Scale. I hope that you will join me here too.&nbsp;</p>

How a breakdancing video launched a career in AI

Why I joined Scale

<p>Data is the lifeblood of today’s enterprise, but as many organizations are discovering, it’s increasingly the quality of that data that matters more than the quantity. Highly trustworthy training data is crucial for the success of machine learning initiatives. Organizations that attempt to train models based on lower quality data are finding that accuracy ultimately suffers. In fact, with even a small amount of bad, inaccurate, or outdated data, these models are never able to become fully optimized and useful.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F830b8674f2f84bea90e7d9e6d818b766"></p><p>To use an analogy, consider what happens when adults try to learn a new language. We’ve been “trained” on our native tongues for decades, and as such we’ve probably picked up a lot of preconceived notions about how language should work. Because we are ingrained with this biased training data, it’s notoriously difficult for adults to learn a new language, as we subconsciously attempt to force our old models on the process. Conversely, children who are raised bilingually have fewer of these issues; they are being “trained” with a bias-free, more balanced, dataset.</p><p>We’ve seen the consequences of inaccurate and biased training data time and time again in the industry. Consider <a href="https://www.wsj.com/articles/zillow-offers-real-estate-algorithm-homes-ibuyer-11637159261" rel="noopener noreferrer" target="_blank">Zillow’s misguided home-buying algorithms</a>, which underestimated the accuracy of its home value estimates, causing it to overpay for properties or the racial bias that crept into the <a href="https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm" rel="noopener noreferrer" target="_blank">COMPAS recidivism algorithm</a>.</p><h2>The Importance of Data Annotation</h2><p>Many algorithmic problems that occur are due to poor data quality. One way in which data quality can be improved for ML algorithms is through <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>, the process of tagging data with certain properties or characteristics. For example, an archive of photos of fruits could be manually annotated as apple, pear, watermelon, and so on, giving an algorithm a head-start when it comes to predicting the identity of other, unlabeled objects. <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">Data annotation</a> can be a tedious, manual chore, but as datasets become very large and complex, annotation can become crucial.</p><p>The consequences of poorly annotated data can be not only annoying but also costly, as models must be redesigned, retrained, and run over and over again. In some cases, the consequences can be devastating if an organization makes decisions based on a badly trained ML model, such as a medical algorithm designed to spot signs of cancer in an x-ray.</p><h2>How Good Is Your Training Data?</h2><p>So, let’s say your data is already annotated. Is it any good? How can you find out?</p><p>Start by assessing the performance of your model. It’s often straightforward to discern whether results are in line with expectations. If your fruit-detection algorithm is coming up with nothing but bananas, you might have a data problem, and some deeper investigation is probably in order.</p><p>A good next step (either way) is to take a random sample from your dataset and meticulously audit its annotations. How accurate are the annotations? Are annotations missing? This kind of effort can give you a good sense of how accurate your data will be for a specific task as well as a look at the overall health of the project – assuming the sample you took is truly random. If you are looking for more automated ways of debugging your dataset, tools like <a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> can help identify errors and gaps in your existing data.</p><h2>How to Improve Data Annotation Quality</h2><p>If you’ve uncovered a problem with the quality of your data annotation, you can remediate the problem via a number of tactics, including:</p><p><br></p><ol><li>Make sure your annotation instructions are well-understood. Are annotators incorrectly including the stem of the fruit or the container in which the apples are in as part of the annotation? If they aren’t following instructions perfectly, your annotation efforts will be wasted – and your data may be worse off when the task is finished than when you began. Start slowly by giving annotators a small number of easier tasks, allow them to ask questions, and ensure they understand what makes for a high quality annotation. As you work through the training process, revise your instructions to be as clear and precise as possible, which will help you with the next annotator.</li><li>Add a review cycle. Add a second layer of personnel responsible for overseeing the first-round of annotators. (This second layer of reviewers can be chosen from those annotators who prove themselves adept at high-quality annotations.) This second level does not create new annotations from scratch but rather monitors annotation work, corrects any errors as they are spotted, and adds annotations that might have been forgotten somewhere in the process. This extra check layer can increase the overall quality of your data considerably.</li><li>Try a consensus pipeline. One solid quality tactic is to have multiple people annotate the same data, then use consensus to determine which annotation is correct. If four people tag a fruit as “apple” and one tags it as “orange,” it’s highly likely that the three apple tags are correct and the orange tag can be discarded. If this pipeline also considers the overall accuracy level of each individual annotator, you can build an even higher confidence in your data quality.&nbsp;</li><li>Add a quality screen for annotators. If annotators want to work for you, put them through a quality test first. They have to achieve a set accuracy – 99 percent, perhaps – or else they aren’t allowed to start working on your queue. This step saves you from having to rigorously monitor quality and ensures that you’re starting with high-quality annotations.</li><li>Add evaluation tasks throughout the queue and evaluate your annotated data against this benchmark. Evaluation tasks are tasks that have known correct answers. This benchmark serves as a kind of answer key against which an algorithm can be tested, allowing you to determine the quality of your data without having to manually check every piece of the dataset. When data is being annotated, each annotator should be directed to complete the same set of benchmarks, giving you an ongoing understanding of your dataset’s average quality level. By periodically benchmarking your data, you can ensure nothing has gone off the rails and that annotators are still on target, even after the initial training period has passed. If annotators’ evaluation scores are dropping, you may want to provide more in-depth oversight and training.</li></ol><p>To get your data annotated with high quality, you can either leverage your own team of annotators - or you can have Scale’s team of trained annotators do it for you.&nbsp;</p><ul><li><a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> is a data annotation platform for you to use with your own workforce, and it comes with all the tools to implement the best practices listed above, learned from providing billions of high-quality annotations for our customers. If you are training and managing your own team of annotators, Studio provides the annotation tools and the management systems to achieve a high-quality data set.&nbsp;</li><li><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> allows you to send your raw data to Scale, and we’ll return high-quality data quickly using the best practices listed above. Simply define the parameters of your project, and send your data out for annotation. There’s no need to reinvent the wheel when you can simply pay as you go and receive production-level data in hours instead of weeks.</li></ul><p><br></p>

Training data is the lifeblood of machine learning. Learn how to assess the quality of your annotated data and methods to develop high-quality training data.

Understand and improve the quality of your training data. 

How to Understand and Improve the Quality of Annotated Data

<p>When setting up a new machine learning (ML) system, collecting and annotating data can be a time-consuming process that requires a significant upfront investment. Since data quality is strongly correlated with ML model performance, it’s crucial to invest in high-quality solutions for data annotation from the onset of your project. The process of annotating data for an ML project can be surprisingly complicated – There are a variety of factors you’ll need to consider, from how you’ll manage your approach to annotation to what tooling and automation approaches you’ll use.</p><h2>How to Build Your Data Annotation Pipeline</h2><p>Annotating data at high quality requires significant upfront and continued investment over the lifespan of your ML system. While there are automated solutions to speed up and streamline the data annotation process, having humans-in-the-loop is key to producing high-quality results. When setting up a data annotation pipeline, your company will need to decide how you’ll find and train annotators, as well as how you’ll measure their performance. To ensure annotators produce high-quality annotations, you’ll need to provide them with performance incentives that align with your most important annotation metrics, whether that’s throughput or quality, or a combination of both. You’ll also need to provide annotation tools for your workers, as well as design a pipeline that encourages your annotators to output consistent, high-quality annotations.</p><h3>Hiring Annotators While Minimizing Overhead</h3><p>Finding annotators to complete your task is one of the first challenges you’ll face in the data annotation process. You’ll need to identify workers who are well-suited to annotate your data, whether that involves selecting people who speak a certain language, are from a specific location, or have a particular type of expertise. You’ll also need to decide how to approach the hiring process. Your choice of workforce influences whether you have access to a large selection of annotators who work fewer hours, or a small selection of annotators who work longer hours and generate more consistent results. If you expect a continuous flow of production-quality data, you’ll also need to potentially hire workers across time zones to meet the demands of your project.&nbsp;</p><p>You should decide whether to hire a larger pool of diverse annotators versus a smaller set of consistent annotators based on your use case. If you anticipate that you’ll need annotations for small, irregular projects, for example, you’ll likely want to rely on a larger selection of annotators that can accommodate short-term surges in need. Alternatively, if you’re pursuing a long-term production project with consistent annotation needs, a smaller team of annotators with regular annotation hours may deliver better results.</p><p>Once you’ve hired annotators, you’ll also need to devote resources to communication and expectation setting to ensure that your annotators provide you high-quality results. In short, just finding workers to annotate your data and keeping them on your platform can be a time-consuming process with logistical overhead! These challenges are often why many ML teams opt to outsource the entire process to an external company.</p><h3>Training Annotators and Monitoring Performance</h3><p>Once you’ve hired your annotators, your work is not yet over – You need to provide training to teach them to accurately annotate your data across different use cases, task types, and annotation types. To ensure high-quality annotations, you’ll need to provide workers with clear and comprehensive instructions, so that different workers can annotate data consistently. You may also find over time that there are several edge cases within your data that your instructions do not cover, as it can be difficult to write non-ambiguous and comprehensive instructions that cover all potential edge cases.</p><p>As the administrator of the data pipeline, you’ll need to create a “golden dataset” that has accurate annotations for a subset of your data. You can use this dataset to check that workers understand the instructions, as well as to periodically re-evaluate their performance against a gold standard. You should also regularly update this dataset with new examples to provide annotators with up-to-date examples that reflect real-world data.&nbsp;</p><p>Suppose your goal is to achieve 95%+ quality in your overall dataset. If all your annotators consistently achieve 95% quality, you should be able to achieve this goal. However, if there is a particular edge case that only appears 1% of the time, your annotators can get 100% of that 1% wrong and still average 95% quality overall. While you are still hitting your goal of 95%+ quality, this is not ideal, as your model will not be able to account for this edge case if trained on this dataset.</p><p>To mitigate this challenge, you will need to add examples to your golden dataset to ensure that it remains well-balanced across all classes and edge cases. By updating and adding new examples to your dataset, you can effectively assess annotator performance over time, and achieve a holistic view of their performance across various classes and edge cases.</p><h3>Reviewing Annotations for Quality</h3><p>After your annotators complete tasks, you’ll need to evaluate the final output. To do this, there are two main methods for evaluating annotation quality. For classification or text-based tasks where there is more subjectivity, you can implement a consensus algorithm. Suppose you want to develop a sentiment analysis model, you can have multiple people determine if a person in an image looks happy or sad and the option the majority selected will win. If there isn’t enough confidence in the voted result, you can have a higher-level reviewer determine the final result. In some cases, you may also not want to weigh every person’s vote the same. If you have higher confidence in some people’s votes over others, you may want to implement a weighted voting system that biases the final response more towards the votes of people you have higher confidence in.&nbsp;</p><p>For image-based or other computer vision tasks, you can have one preliminary attempter annotate the task, then have a higher-level reviewer either review the work and accept the results or review the work and fix annotations.&nbsp;You can select reviewers from past annotators who consistently produce high-quality results. Selecting reviewers, however, can be complicated, since someone’s annotation performance can fluctuate over time. You’ll need to constantly identify reviewers who are no longer performing well so that you can demote them to attempts, while also identifying well-performing annotators so that you can promote them to reviewers. Since this approach removes high-quality annotators from the labeling pool, it will naturally reduce your throughput, so you’ll want to find a balance between quality and throughput.</p><p>Whatever approach you follow for evaluating label quality, you’ll need to be on the lookout for people who consistently underperform and label data poorly. You should automatically identify underperforming annotators, and either retrain them so they produce higher-quality labels, or remove them from the project entirely.</p><h3>Selecting the Best Annotation Tool For Your Task</h3><p>Another big decision you’ll need to make is on what labeling tools your workers will use. You can either build your own tool set or use available commercial tools. While you can specialize in-house tools for your specific task, they can be difficult and time-consuming to build, especially if they need to support different annotation types. For example, image, audio, and text data all require different annotation approaches – For image annotation, you’ll need a system to draw boxes, polygons, or other geometries around identified objects; whereas, for audio annotation, your system will need to support transcription and associating timestamps with event attributes. For classification tasks, on the other hand, your system will need to permit multiple-choice selections and potentially large taxonomies. For video annotation tasks, video files can be very large in size, making them technically challenging to support, especially if your workers have less powerful laptops or slow internet connections.&nbsp;&nbsp;</p><p>Compared to in-house tooling, commercial data tools make it easier to jump into labeling your data, although they may be less specialized for your use case. Another consideration when selecting the right annotation tool is ML-assisted tooling. Some tools may have ML-assisted tooling, like our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosemseg tool</a>, to improve annotation efficiency and speed while others do not. Building and selecting the appropriate labeling tools is vital for optimizing labeler efficiency, so you should ensure the tools you select are well-suited for your use case.</p><h2>What Next?</h2><p>Needless to say, getting data annotation right can be a challenging and time-consuming process. You’ll need to consider a wide range of factors, from how you’ll manage your labeling workforce to what automation approaches you’ll use. Your data labeling pipeline is a process that’s incredibly important to get right since ML system performance is correlated with data quality.</p><p>Instead of building a data pipeline from scratch, you can leverage a product like <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> or <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> to quickly create production-level ratings. It’s designed so that you can set up your projects in minutes and get high-quality labels within hours. Trusted by AI teams like OpenAI, Square, and Pinterest, this tool can help you hit the ground running on your next ML project!</p>

Data annotation, or data labeling, is a more challenging problem than most teams realize but investing in annotation is key to unlocking the power of AI.

Scale AI’s How To Guide to Data Annotation/Data Labeling

How to Setup Data Annotation Pipelines

<p>At first glance, human-in-the-loop processes seem like a relic of older times. Before technological advancements, humans effectively performed functions completed by computers today. For example, NASA hired large teams of arithmetic processors- men and women whose only job was to evaluate numerical expressions quickly and accurately. These calculators were pivotal to enabling space exploration.</p><p>At Scale, our labeling teams are just as critical to achieving our mission. We’ve found we’re able to effectively apply the principles behind generative adversarial networks to our human-in-the-loop processes to deliver the highest quality annotations to customers.</p><h2>How does this work, exactly?</h2><p>A GAN, or generative adversarial network, is given an unlabeled dataset and learns to produce new examples that are not already in the dataset but are indistinguishable from the items in the dataset. For example, a net meant to produce new images of flowers would be fed as input images of real flowers. To achieve this goal, the net has to achieve two things:</p><ol><li>The generated flowers must not be distinguishable from real flowers. That is to say, if given a set of both generated and real flower images, one should not be able to tell which is which.</li><li>In achieving the above, the GAN must not simply produce copies of the real flower images.</li></ol><p>Therefore, the net must be trained to generate new flowers, discriminate between real and fake flowers, and avoid simply replicating training data. The training loop jointly optimizes the generator (which produces new images that are not copies of the training set), and the discriminator (which attempts to discriminate between training data and generated images). The net is successful when the generator becomes accurate enough to fool the discriminator, i.e. the discriminator cannot outperform random choice when classifying generated vs non-generated examples.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F018ccca0549b448bb3686f7d5703244e"></p><p>In the context of human-in-the loop processes, we can apply this same principle to produce quality labels on subjective tasks. Let’s say the pipeline we’ve chosen has a minimum of two annotators seeing each task, one attempter and one reviewer.</p><p>Example Task:</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F4a26d7e4818f49dfa8acd7616a2bf50c"></p><p>Instructions: Please describe the image as accurately and descriptively as possible with a short phrase.</p><p>Possible attempter responses: </p><ul><li>Attempter 1: ‘asdf’</li><li>Attempter 2: ‘trees’</li><li>Attempter 3: ‘green hills’</li><li>Attempter 4: ‘grassy hills with mountains, trees, houses’</li><li>Attempter 5: ‘grassy hills with trees and 3 houses, mountains and clouds in background’</li><li>Attempter 6: ‘tall trees on green grassy hill, small red and gray houses, blue mountains in background’</li></ul><p>The attempters act as generators, and can generate any one of the above responses for this task. Attempters 1-3 have written low-quality responses. On a regular rapid project, these attempters would be disabled via evaluation tasks. However, on a generative task like the above, exact-match evaluation tasks would not guarantee quality. For this image, both Attempters 5 and 6 provided accurate and descriptive responses. Therefore, we turn instead to the discriminator to filter out the better responses.</p><p>Our discriminators are reviewers. A reviewer is the final person to look at a task and is able to see the attempters’ responses. Reviewers are the final stage before tasks are returned to the customer If we allow reviewers to edit an attempter’s response directly and write a caption themselves, we run into the same problem as with attempter evaluation tasks.&nbsp;</p><p>However, once we move more into thinking of the reviewer as a GAN discriminator, the solution becomes clear. Reviewers must discriminate between good and bad responses, but they need not write any captions themselves. Although there is no 1:1 mapping between a task and a certain correct answer, we do know upon encountering a caption whether it is acceptable or unacceptable. So we give the reviewers the choice of either accepting or rejecting an attempter’s response, but we do not give them the option to change it. Now the process of evaluating their quality is simple- create evaluation tasks with pre-generated responses that are expected to be accepted or rejected appropriately.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8be702f8ef547cba93bf016ee11d500"></p><p>The two parts of a GAN, generator and discriminator, work together to pass information between them. In our human network, reviewers may provide feedback upon rejecting an attempt, to help the generator learn. The discriminator, composed of both the reviewers and the reviewer benchmarks managed by the customer, learns in turn from the generator. From a variety of generated data, the customer and in turn the reviewers are able to converge on a similar understanding of the desired result.</p><p>Now that we are sure the discriminator is accurately discriminating between low and high-quality responses, we are able to train the generator. What this means is that attempters are now disabled based on how many of their responses are rejected. Over time, this leads to a state where the attempters are effective, high-quality generators- essentially a trained GAN.</p><p>This pipeline is available via Rapid’s <a href="https://docs.scale.com/reference/text-collection" rel="noopener noreferrer" target="_blank">text collection</a> and <a href="https://docs.scale.com/reference/categorization" rel="noopener noreferrer" target="_blank">categorization</a> projects. <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Get started with Scale Rapid</a> today!</p>

Applying the principles behind generative adversarial networks (GANs) to our human-in-the-loop data annotation processes.

Using generative adversarial networks (GANs) to align human metrics

Using GANs to Align Human Metrics

<p>At<a href="https://scale.com/defense" rel="noopener noreferrer" target="_blank"> Scale</a>, we are committed to partnering with the U.S. government and technology leaders to ensure U.S. investments in AI solve critical challenges. It’s imperative the U.S. government has access to, and is deploying, the best quality data and AI technology available.&nbsp;</p><p>That's why we are excited to announce Scale has partnered with <a href="https://www2.deloitte.com/us/en.html" rel="noopener noreferrer" target="_blank">Deloitte</a>, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID) Basic Ordering Agreement (BOA). The JAIC DRAID is a five-year, $241 million BOA that enables Scale and Deloitte to support AI data preparation for the U.S. government. This partnership brings nearly $500M in&nbsp;<a href="https://www.businesswire.com/news/home/20220131005304/en/Scale-AI-Selected-by-U.S.-Department-of-Defense-to-Accelerate-Government%E2%80%99s-AI-Capabilities" rel="noopener noreferrer" target="_blank">total JAIC ceiling value</a>&nbsp;to Scale. Scale’s technology will be readily available to all federal agencies and will empower the government to feel confident in leveraging AI and ML in their most important workflows.</p><p>With this partnership, Scale will provide services focused on data management, AI modeling and innovative development including:</p><ul><li>High quality data integration, prep, labeling and management</li><li>Synthetic data generation&nbsp;</li><li>NLP-enabled processing</li><li>Algorithm management</li><li>Innovative AI application development</li></ul><p>Scale’s data-centric approach to its end-to-end AI platform provides the U.S. government with the support needed to manage the entire AI lifecycle, all while ensuring the highest quality data is incorporated. Using our technology, analysts and operators will be able to understand, visualize, curate, and collaborate on their data through process automation, enhanced decision support, and insight generation, faster and with greater accuracy.</p>

Scale has partnered with Deloitte, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID)

Scale AI Partners with Deloitte

Scale partners with Deloitte to accelerate AI adoption across the U.S. government

<p>I spent the last 6 years of my career at Uber across both Finance and Business leadership roles. In this time, I witnessed first hand how AI and ML drove improved customer experiences, more efficient operations, and will be instrumental in unlocking a future with self-driving cars and trucks.&nbsp;</p><p>As I considered joining Scale AI, I got deeper into the latest trends and potential of AI. I have become even more excited about the many applications of AI and AI’s ability to improve decision making, drive productivity leaps, as well as deliver improved customer experiences and outcomes across every industry and domain on the planet.&nbsp;</p><p>What I found in Alex and his team at Scale is a vision to help all companies and the U.S. government realize the potential of AI. Scale’s mission is to accelerate the development of AI applications and to do it by first tackling one of the hardest problems in AI, better data. Scale started by supporting the autonomous vehicle industry but has since expanded across many industries and applications. They have built products and solutions that support customers at the leading edge of AI as well as companies that are just getting started. In short, this convinced me that the opportunity and potential are unparalleled.&nbsp;</p><p>As I spent more time getting to know Alex, we talked about what he was looking for in a CFO. We agreed that Scale needs a business-minded finance partner to foster continued, rapid growth of the Finance function that enables the company through increased strategic alignment, business model evolution, and increased operational discipline.&nbsp;</p><p>Today, I am excited to announce that I am joining Scale as their first CFO to help the team deliver on their vision of building and scaling what will be one of the most impactful enterprise companies in the world by accelerating the adoption of AI. I will be leading Finance, Accounting, Corporate Development, and Strategic Partnerships. We are hiring across the team. Come join us!&nbsp;</p><p><br></p>

Scale Welcomes Dennis Cinelli as First CFO

Why I Joined Scale

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog

How We Scaled Our MongoDB Instance Gracefully

Building Great Models with Better Data on Nucleus and Studio

Nucleus Simplified—Dataset Management for Everyone

Build vs. Buy: What Makes Sense for Data Annotation?

Why I joined Scale

How to Understand and Improve the Quality of Annotated Data

How to Setup Data Annotation Pipelines

Using GANs to Align Human Metrics

Scale partners with Deloitte to accelerate AI adoption across the U.S. government

Why I Joined Scale

The future of your industry starts here

Blog

How We Scaled Our MongoDB Instance Gracefully

Building Great Models with Better Data on Nucleus and Studio

Nucleus Simplified—Dataset Management for Everyone

Build vs. Buy: What Makes Sense for Data Annotation?

Why I joined Scale

How to Understand and Improve the Quality of Annotated Data

How to Setup Data Annotation Pipelines

Using GANs to Align Human Metrics

Scale partners with Deloitte to accelerate AI adoption across the U.S. government

Why I Joined Scale

The future of your industry starts here