GuidesPage

Header

Footer

PreFooter

ProseBlock

NavSidebar

SocialLinks

Prose

AnnouncementContextProvider

UnnamedGlobalGroupOfVariants2

Screen

Segment

Experiment

Experiment2

scale.com (Production)

plasmic-nav

plasmic-embed-css

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

react-slick

react-quill

Michael Fagundo

Machine Learning Research Engineer

Data labeling is one of the most critical activities in the machine learning lifecycle, though it is often overlooked in its importance. Powered by enormous amounts of data, machine learning algorithms are incredibly good at learning and detecting patterns in data and making useful predictions, all without being explicitly programmed to do so. Data labeling is necessary to make this data understandable to machine learning models.

<p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F2631e40f8ac847f6b85532fe22cee3c4"></p><p>Machine learning has revolutionized our approach to solving problems in computer vision and natural language processing. Powered by enormous amounts of data, machine learning algorithms are incredibly good at learning and detecting patterns in data and making useful predictions, all without being explicitly programmed to do so.</p><p>Trained on large amounts of image data, computer models can predict objects with very high accuracy. They can recognize faces, cars, and fruit, all without requiring a human to write software programs explicitly dictating how to identify them.</p><p>Similarly, natural language processing models power modern voice assistants and chatbots we interact with daily. Trained on enormous amounts of audio and text data, these models can recognize speech, understand the context of written content, and translate between different languages.</p><p>Instead of engineers attempting to hand-code these capabilities into software, machine learning engineers program these models with a large amount of relevant, clean data. Data needs to be labeled to help models make these valuable predictions. Data labeling is one of machine learning's most critical and overlooked activities.</p><p>This guide aims to provide a comprehensive reference for data labeling and to share practical best practices derived from Scale's extensive experience in addressing the most significant problems in data labeling.</p>

Data Labeling for Machine Learning

<p>Data labeling is the activity of assigning context or meaning to data so that machine learning algorithms can learn from the labels to achieve the desired result.</p><p>To better understand data labeling, we will first review the types of machine learning and the different types of data to be labeled. Machine learning has three broad categories: supervised, unsupervised, and reinforcement learning. We will go into more detail about each type of machine learning in <a href="https://scale.com/guides/data-labeling-authoritative-guide#why-is-data-annotation-important?" rel="noopener noreferrer" target="_blank">Why is Data Annotation Important</a>?</p><p>Supervised machine learning algorithms leverage large amounts of labeled data to “train” neural networks or models to recognize patterns in the data that are useful for a given application. Data labelers define ground truth annotations to data, and machine learning engineers feed that data into a machine learning algorithm. For example, data labelers will label all cars in a given scene for an autonomous vehicle object recognition model. The machine learning model will then learn to identify patterns across the labeled dataset. These models then make predictions on never before seen data.</p><h3>Types of Data</h3><h4>Structured vs. Unstructured Data</h4><p>Structured data is highly organized, such as information in a relational database (RDBMS) or spreadsheet. Customer information, phone numbers, social security numbers, revenue, serial numbers, and product descriptions are structured data.</p><p>Unstructured data is data that is not structured via predefined schemas and includes things like images, videos, LiDAR, Radar, some text data, and audio data.</p><h4>Images</h4><p>Camera sensors output data initially in raw format and then converted to .png or preferably .jpg files, which are compressed and take up less storage than .png, which is a serious consideration when dealing with the large amounts of data needed to train machine learning models. Image data is also scraped from the internet or collected by 3rd party services. Image data powers many applications, from face recognition to manufacturing defect detection to diagnostic imaging.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa06de25cfc1445ba9e1969b6ea0f5a28"></p><h4>Videos</h4><p>Video data also come from camera sensors in raw format and consist of a series of frames stored as .mp4, .mov, or other video file formats. MP4 is a standard in machine learning applications due to its smaller file size, similar to .jpg for image data. Video data enables applications like autonomous vehicles and fitness apps.</p><p><br></p><p><a href="https://scaleai.wistia.com/medias/xm37nn8yoo" rel="noopener noreferrer" target="_blank">video</a></p><p><br></p><h4>3D Data (LiDAR, Radar)</h4><p>3D data helps models overcome the lack of depth information from 2D data such as traditional RGB camera sensors, helping machine learning models get a deeper understanding of a scene.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F7f26bab6210c443c9dfee87f80075b57"></p><p><strong>LiDAR (Light Detection and Ranging)</strong> is a remote sensing method that uses light to generate precise 3D images of scenes. LiDAR data is stored as point clouds in raw format and the .las file format and are often converted to JSON file format for processing by machine learning applications.</p><p><strong>Radar (Radio Detection and Ranging)</strong> is a remote sensing method that uses radio waves to determine an object's distance, angle, and radial velocity relative to the radar source.</p><h4>Audio</h4><p>Typically stored as .mp3 or .wav file formats, audio data enables speech recognition for your favorite smart assistant and real-time multilingual machine translation.</p><p><br></p><p><a href="https://scale.com/static/audio/transcription.mp3" rel="noopener noreferrer" target="_blank">audio</a></p><p><br></p><h4>Text</h4><p>Text data made of characters representing information, often stored in .txt, .docx, or .html files. Text powers Natural Language Processing (NLP) applications such as virtual assistants when they answer your questions, automated translation, text-to-speech, speech-to-text, and document information extraction.</p>

What is Data Labeling?

<p>Machine learning powers revolutionary applications made possible by vast amounts of high-quality data. To better understand the importance of data labeling, it is critical to understand the different types of machine learning: supervised, unsupervised, and reinforcement learning.</p><p><strong>Reinforcement Learning</strong> leverages algorithms to take actions in an environment to maximize a reward. For instance, Deepmind’s AlphaGo used reinforcement learning to play games against itself to master the game of GO and become the strongest player in history. <a href="https://exchange.scale.com/home/blogs/build-better-reinforcement-learning-with-world-models-and-self-attention?__hstc=159041573.e3252e209316c8bf013d9f4d646f1fdd.1660697429166.1660697429166.1660697429166.1&amp;__hssc=159041573.6.1660697429166&amp;__hsfp=1562363153" rel="noopener noreferrer" target="_blank">Reinforcement learning</a> does not rely on labeled data but instead maximizes a reward function to achieve a goal.</p><h3>Supervised Learning vs. Unsupervised Learning</h3><p><strong>Supervised learning</strong> is behind the most common and powerful machine learning applications, from spam detection to enabling self-driving cars to detect people, cars, and other obstacles. Supervised learning uses a large amount of&nbsp;<strong>labeled data</strong>&nbsp;to train a model to accurately classify data or predict outcomes.</p><p><strong>Unsupervised learning</strong>&nbsp;helps analyze and cluster&nbsp;<strong>unlabeled data</strong>, driving systems like recommendation engines. These models learn from features of the dataset itself, without any labeled data to "teach" the algorithm the expected outputs. A common approach is K-means clustering, which aims to partition n observations into k clusters and assign each observation to the nearest mean.</p><p>While there are many fantastic applications for unsupervised learning, supervised learning has driven the most high-impact applications due to its high accuracy and predictive capabilities.</p><p>Machine learning practitioners have turned their attention away from model improvement to improving data, coining a new paradigm: data-centric ai. Only a tiny fraction of real-world ML systems are composed of ML code. More high-quality data and accurate data labels are necessary to power better AI. As the methods to create better machine learning models shift to data-centricity, it is essential to understand the entire process of a well-defined data pipeline, from data collection methods to data labeling to data curation.</p><p>This guide focuses on the most common types of data labels and the best practices for high quality so that you can get the most out of your data and therefore get the most out of your models.</p>

Why is Data Annotation Important?

<p>To create high-quality supervised learning models, you need a large volume of data with high-quality labels. So, how do you label data? First, you will need to determine who will label your data. There are several different approaches to building labeling teams, and each has its benefits, drawbacks, and considerations. Let's first consider whether it is best to involve humans in the labeling process, rely entirely on automated data labeling, or combine the two approaches.</p><h3>1. Choose Between Humans vs. Machines</h3><h4>Automated Data Labeling</h4><p>For large datasets consisting of well-known objects, it is possible to automate or partially automate data labeling. Custom Machine Learning models trained to label specific data types will automatically apply labels to the dataset.</p><p>Establishing high-quality ground-truth datasets early on, and only then can you leverage automated data labeling. Even with high-quality ground truth, it can be challenging to account for all edge cases and to fully trust automated data labeling to provide the highest quality labels.</p><h4>Human Only Labeling</h4><p>Humans are exceptionally skilled at tasks for many modalities we care about for machine learning applications, such as vision and natural language processing. Humans provide higher quality labels than automated data labeling in many domains.</p><p>However, human experiences can be subjective to varying degrees and training humans to label the same data consistently is a challenge. Furthermore, humans are significantly slower and can be more expensive than automated labeling for a given task.</p><h4>Human in the Loop (HITL) Labeling</h4><p>Human-in-the-loop labeling leverages the highly specialized capabilities of humans to help augment automated data labeling. HITL data labeling can come in the form of automatically labeled data audited by humans or from active tooling that makes labeling more efficient and improves quality. The combination of automated labeling plus <a href="https://arxiv.org/abs/2109.00574" rel="noopener noreferrer" target="_blank">human in the loop</a> nearly always outpaces the accuracy and efficiency of either alone.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F63f7f64666d643a8994f519bdfbac90b"></p><h3><br></h3><h3>2. Assemble Your Labeling Workforce</h3><p>If you choose to leverage humans in your data labeling workforce, which we highly recommend, you will need to figure out how to source your labeling workforce. Will you hire an in-house team, convince your friends and family to label your data for free, or scale up to a 3rd Party labeling company? We provide a framework to help you make this decision below.</p><h4>In-House Teams</h4><p>Small startups may not have the capital to afford significant investments in data labeling, so they may end up having all the team members, including the CEO, label data themselves. For a small prototype, this approach may work but is not a scalable solution.</p><p>Large, well-funded organizations may choose to keep in-house labeling teams to keep control over the entire data pipeline. This approach allows for much control and flexibility, but it is expensive and much work to manage.</p><p>Companies with privacy concerns or sensitive data may choose in-house labeling teams. While a perfectly valid approach, this can be difficult to scale.</p><p>Pros: Subject matter expertise, tight control over data pipelines</p><p>Cons: Expensive, overhead in training and managing labelers</p><h4>Crowdsourcing</h4><p>Crowdsourcing platforms provide a quick and easy way to quickly complete a wide array of tasks by a large pool of people. These platforms are fine for labeling data with no privacy concerns, such as open datasets with basic annotations and instructions. However, if more complex labels are needed or sensitive data is involved, the untrained resource pool from crowdsource platforms is a poor choice. Resources found on crowdsourcing platforms are not trained well and lack domain expertise, often leading to poor quality labeling.</p><p>Pros: Access to a larger pool of labelers</p><p>Cons: Quality is suspect; significant overhead in training and managing labelers</p><h4>3rd Party Data Labeling Partners</h4><p>3rd Party data labeling companies provide high-quality data labels efficiently and often have deep machine learning expertise. These companies can act as technical partners to advise you on best practices for the entire machine learning lifecycle, including how to best collect, curate, and label your data. With highly trained resource pools and state-of-the-art automated data labeling workflows and toolsets, these companies offer high-quality labels for a minimal cost.</p><p>To achieve extremely high quality (99%+) on a large dataset requires a large workforce (1,000+ data labelers on any given project). Scaling to this volume at high quality is difficult with in-house teams and crowdsourcing platforms. However, these companies can also be expensive and, if they are not acting as a trusted advisor, can convince you to label more data than you may need for a given application.</p><p>Pros: Technical expertise, minimal cost, high quality; The top data labeling companies have&nbsp;<a href="https://scale.com/blog/soc2-hipaa?__hstc=159041573.0e8306a6623d8cb5a8cda8466e0bc1d7.1659467210061.1660627648464.1660655610758.51&amp;__hssc=159041573.1.1660655610758&amp;__hsfp=1562363153" rel="noopener noreferrer" target="_blank">domain-relevant certifications</a>&nbsp;such as SOC2 and HIPAA.</p><p>Cons: Relinquish control of the labeling process; Need a trusted partner with proper certifications to handle sensitive data</p><h3>3. Select Your Data Labeling Platform</h3><p>Once you have determined who will label your data, you need to find a data labeling platform. There are many options here, from building in-house, using open source tools, or leveraging commercial labeling platforms.</p><h4>Open Source Tools</h4><p>These tools are free to use by anyone, with some limitations for commercial use. These tools are great for learning and developing machine learning and AI, personal projects, or testing early commercial applications of AI. While free, the tradeoff is that these tools are not as scalable or sophisticated as some commercial platforms. Some label types discussed in this guide may not be available in these open-source tools.</p><p>The list below is meant to be representative, but not exhaustive so many great open source alternatives may not be included.</p><ul><li><strong>CVAT</strong>: Originally developed by Intel, CVAT is a free, open-source web-based data labeling platform. CVAT supports many standard label types, including rectangles, polygons, and cuboids. CVAT is a collaborative tool and is excellent for introductory or smaller projects. However, web users are limited to 500 MB of data and only ten tasks per user, reducing the appeal of the collaboration features on the web version. CVAT is available locally to avoid these data constraints.</li><li><strong>LabelMe</strong>: Created by CSAIL, LabelMe is a free, open-source data-labeling platform supporting community collaboration on datasets for computer vision research. You can contribute to other projects by labeling open datasets and label your data by downloading the tool. Labelme is quite limited compared to CVAT, and the web version no longer accepts new accounts.</li><li><strong>Stanford Core NLP: </strong>A fully featured NLP labeling and natural language processing platform, Stanford's CoreNLP is a robust open source tool offering Named Entity Recognition (NER), linking, text processing, and more.</li></ul><p><strong>In-house Tools</strong></p><p><a href="https://scale.com/blog/build-vs-buy" rel="noopener noreferrer" target="_blank">Building in-house tools</a> is an option selected by some large organizations that want tighter control over their ML pipelines. You have direct control over which features to build, support your desired use cases, and address your specific challenges. However, this approach is costly, and these tools will need to be maintained and updated to keep up with the state-of-the-art.</p><p><strong>Commercial Platforms</strong></p><p>Commercial platforms offer high-quality tooling, dedicated support, and experienced labeling workforces to help you scale and can also provide guidance on best practices for labeling and machine learning. Supporting many customers improves the quality of the platforms for all customers, so you get access to state-of-the-art functionality that you may not see with in-house or open-source labeling platforms.</p><p><a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> is the industry-leading commercial platform, providing best-in-class labeling infrastructure to accelerate your team, with labeling tools to support any use case and orchestration to optimize the performance of your workforce. Easily annotate, monitor, and improve the quality of your data.</p>

How to Annotate Data

<p>Whatever annotation platform you use, maximizing the quality of your data labels is critical to getting the most out of your machine learning applications.</p><p>The classic computer science axiom "Garbage in Garbage Out" is especially acute in machine learning, as data is the primary input to the learning process. With poor quality data or labels, you will have poor results. We aim to provide you with the most critical quality metrics and discuss best practices to ensure that you are maximizing your labeling quality.</p><h3>Different Ways to Measure Quality</h3><p>We cover some of the most critical quality metrics and then discuss best practices to ensure quality in your labeling processes.</p><h4>Label Accuracy</h4><p>It is essential to analyze how closely labels follow your instructions and match your expectations. For instance, say you have assigned tasks to data labelers to annotate pedestrians. In your instructions, you have specified labels should include anything carried (i.e., a phone or backpack), but not anything that is pushed or pulled. When you review sample tasks, are the instructions followed, or are strollers (pushed) and luggage (pulled) included in the annotations?</p><p>How accurate are labelers on benchmark tasks? These are test tasks to determine overall labeler accuracy and give you more confidence that other labeled data will also be correct. Is labeling consistent across labelers or types of data? If label accuracy is inconsistent across different labelers, this may indicate that your instructions are unclear or that you need to provide more training to your labelers.</p><h4>Model Performance Improvement</h4><p>How accurate is your model at its specified task? This output metric is not solely dependent on labeling quality, the quantity and quality of data play a prominent role, but labeling quality is a significant factor to consider.</p><p>Let's review some of the most critical model performance metrics.</p><h4>Precision</h4><p>Precision defines what proportion of positive identifications were correct and is calculated as follows:</p><p>Precision = True Positives / True Positives + False Positives</p><p>A model that produces no false positives has a recall of 1.0</p><h4>Recall</h4><p>Recall defines what proportion of actual positives were identified correctly by the model, and is calculated as follows:</p><p>Precision = True Positives / True Positives + False Negatives</p><p>A model that produces no false negatives has a recall of 1.0</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fea8a770f88d1410c9ebf94d1ea8266c2"></p><h4>Precision Recall Curve</h4><p>A model with high recall but low precision returns many results, but many of the predicted labels are incorrect compared to the ground truth labels. On the other hand, a model with high precision but low recall is just the opposite, returning very few results, but most of its predicted labels are correct when compared to the ground truth labels. An ideal model with high precision and high recall will return many results, with all results labeled correctly.</p><p>Precision recall curves provide a more nuanced understanding of model performance than any single metric. Precision and recall are a tradeoff; the exact desired values depend on your model and its application.</p><p>For instance, for a diagnostic imaging application tasked to detect cancerous tumors, higher recall is desired as it is better to predict that a non-cancerous tumor is cancerous than the alternative of labeling a cancerous tumor as non-cancerous.</p><p>Alternatively, applications like SPAM filters require high precision so that important emails are not incorrectly flagged SPAM, even though this may allow more actual SPAM to enter our inbox.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc67fbabc7a424f2887d5c01418929ddd"></p><h4>Intersection Over Union (IoU)</h4><p>An indirect method of confirming label quality in computer vision applications is an evaluation metric called Intersection over Union (IoU).</p><p>IoU compares the accuracy of a predicted label over the ground truth label by measuring the ratio of the area of overlap of the predicted label to the area of the union of both the predicted label and the ground truth label.</p><p>The closer this ratio is to 1, the better trained the model is.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F2af1167ef5354c89b244eccf5a2b60e2"></p><p>As we discussed earlier, the essential factors for model training are high-quality data and labels. So, IoU is an indirect measure of the quality of data labels. You may have a different threshold of quality that you are focused on depending on the class of object. For instance, if you are building an augmented reality application focused on human interaction, you may need your IoU at 0.95 for identifying human faces but only need an IoU of 0.70 for identifying dogs.</p><p>Again, it is critical to remember that other data-related factors can influence IoU, such as bias in the dataset or an insufficient quantity of data.</p><h4>Confusion Matrices</h4><p>Confusion matrices are a simple but powerful tool to understand class confusion in your model better. A confusion matrix is a grid of classes comparing predicted and actual (ground truth) classifications. By examining the confusion matrix, you can quickly understand misclassifications, such as your model predicting a traffic sign when the ground truth is a train.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb85bcc13de5f4234a250b087469ead67"></p><p>Combining these confusions with confidence scores can provide an easy way to prioritize the object confusion, looking at those instances where the model is highly confident in an incorrect prediction. Class confusions are likely due to an issue with incorrect or missing labels or with an inadequate quantity of data containing the traffic sign and train classes.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F4c36ef7ded33424f8efbe244f60132ba"></p><h4>Best Practices for Achieving High-Quality Labels:</h4><ol><li><strong>Collect the best data possible: </strong>Your data should be high quality and as consistent as possible while avoiding the bias that may reduce your model's usefulness. Ideally, your data collection pipeline is integrated into your labeling pipeline to increase efficiency and minimize turnaround times.</li><li><strong>Hire the right labelers for the job: </strong>Ensure that your labelers speak the right language, are from a specific region, or are familiar with a particular domain. Also, ensure that your labelers are properly incentivized to provide high-quality labels.</li><li><strong>Combine humans and machines:</strong> use ML-powered labeling tooling with humans in the loop (HITL) for the highest accuracy labels.<strong><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fd4dd0824a3a249fc9bf05bce4510b925"></strong></li><li><strong>Provide clear and comprehensive instructions: </strong>This will help to ensure that different labelers will label data consistently.<img src="https://files.readme.io/dd238a1-2022-06-10_18.59.51.gif"></li><li><strong>Curate Your data: </strong>As you look to improve your model performance, you will also want to curate your data. Use a data curation tool such as <a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> to explore your data and identify data that is completely missing labels or improperly labeled. Review your dataset's IoU, ROC curve, and confusion matrix to understand poor model performance better. The best data curation tools will also allow you to visually interact with these charts to visually inspect data related to a specific confusion and even send it to your labeling team to correct. You may also discover that you are missing data, in which case you will need to collect more data to label.</li><li><strong>Benchmark tasks and screening:</strong> Collect high-confidence responses to a subset of labeling tasks and use these tasks to <a href="https://scale.com/blog/how-to-label-1m-data-points-week?__hstc=159041573.0e8306a6623d8cb5a8cda8466e0bc1d7.1659467210061.1660593380730.1660617126251.49&amp;__hssc=159041573.4.1660617126251&amp;__hsfp=1562363153" rel="noopener noreferrer" target="_blank">estimate the quality of your labelers</a><a href="https://scale.com/blog/how-to-label-1m-data-points-week" rel="noopener noreferrer" target="_blank">.</a> Mix these benchmark tasks into other tasks. Use the performance on benchmark tasks to determine if an individual labeler understands your instructions and is capable of providing your desired quality. You can screen labelers who do not pass your benchmark tasks and either retrain them or exclude them from your project.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F65a7888d90bb498ca344811165994e56"></li><li><strong>Inspect common answers for specific data: </strong>Looking at common answers for labels can help you identify trends in labeling errors. If all data labelers are incorrectly classifying a particular object or mislabeling a piece of data, then maybe the issue is not with the labeler but lies somewhere else. Reevaluate your ground truth, instructions, and training processes to ensure that your expectations have been clearly communicated. Once identified, add common mistakes to your instructions to avoid these issues in the future.</li><li><strong>Update your instructions and golden datasets as you encounter edge cases</strong></li><li><strong>Create calibration batches</strong> to ensure that your instructions are clear and that quality is high on a small sample of your data before scaling up your labeling tasks.</li><li><strong>Establish a consensus pipeline</strong>: Implement a&nbsp;<a href="https://scale.com/blog/ml-human-consensus?__hstc=159041573.0e8306a6623d8cb5a8cda8466e0bc1d7.1659467210061.1660627648464.1660655610758.51&amp;__hssc=159041573.1.1660655610758&amp;__hsfp=1562363153" rel="noopener noreferrer" target="_blank">consensus pipeline</a>&nbsp;for classification or text-based tasks with more subjectivity. Use a majority vote or a hierarchical approach based on the experience or proven quality of an individual or group of data labelers.</li><li><strong>Establish layers of review:</strong> Establish a hierarchical review structure for computer vision tasks to ensure that the labels are as accurate as possible.</li><li><strong>Randomly sample labeled data for manual auditing:</strong> Randomly sample your labeled data and audit it yourself to confirm the quality of the sample. This approach will not guarantee that the entire dataset is labeled accurately but can give you a sense of general performance on labeling tasks.</li><li><strong>Retrain or remove poor annotators: </strong>If an annotator's performance does not improve over time and with retraining, then you may need to remove them from your project.</li><li><strong>Measure your model performance: </strong>Whether your model performance improves or degrades can often be directly reflected in the quality of your data labels. Use model validation tools such as <a href="https://scale.com/validate" rel="noopener noreferrer" target="_blank">Scale validate</a> to critically evaluate precision, recall, intersection over union, and any other metrics critical to your model performance.</li></ol>

High-Quality Data Annotations

Data Labeling for Computer Vision

<p>Labeling text enables natural language processing algorithms to understand, interact with, and generate text for various applications ranging from chatbots to machine translation to product review sentiment analysis.</p><p>Like computer vision, there is a wide variety of text label types, and we will cover the most common labels in this guide.</p><h3>1. Part of Speech Tagging (POS)</h3><p>Categorizing words in a text (corpus) with a particular part of speech depending on the word's definition and context. This basic tagging enables machine learning models to understand natural language better.</p><p>Labeling parts of speech enables chatbots and virtual assistants to have more relevant conversations and better understand the content with which it interacts.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F378e14fd36c941eabfdec166b042640f"></p><p>quote source:&nbsp;<a href="https://scale.com/ai-readiness-report/data-challenges" rel="noopener noreferrer" target="_blank">Scale Zeitgeist</a>, Eric Schmidt, Schmidt Ventures</p><h3>2. Named Entity Recognition (NER)</h3><p>Named Entity recognition is similar to speech tagging but focuses on classifying text into predefined categories such as person names, organizations, locations, and time expressions. Linking entities to establish contextual relationships (i.e., Ada Lovelace is the child of Lord and Lady Byron) adds another layer of depth to your model's understanding of the text.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F2ab1107bc09b4d93869f5f43891d258e"></p><h4>Applications</h4><ul><li>Improve search terms</li><li>Ad serving models</li><li>Identify terms in customer interactions (i.e., support threads, chatbots, social media posts) to map to specific topics, brands, etc.</li></ul><h3>3. Classification</h3><p>Classify text into predefined categories, such as customer sentiment or chatbots to accurately monitor brand equity, trends, etc.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F87a1a379e37e46ba8c3598d92400fdd5"></p><h4>Applications</h4><ul><li>Customer sentiment</li><li>GPT-3 Fine Tuning</li><li>Intent on social media or chatbots</li><li>Active monitoring of brand equity</li></ul><h3>4. Audio</h3><p>Transcribe audio data into text for natural language models to make sense of the data. In this case, the text data becomes the label for the audio data. Add further depth to text data with named entity recognition or classification.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc3b2e293f0de4977a7c29ef31cd3b026"></p><h3>5. Best Practices for labeling Text</h3><ul><li>Use native speakers, ideally those with a cultural understanding that mirrors the source of the text.</li><li>Provide clear instructions on the parts of speech to be labeled and train your labelers on the task.</li><li>Set up benchmark tasks and build a consensus pipeline to ensure quality and avoid bias.</li><li>Leverage rule-based tagging/heuristics to automatically label known named entities (i.e., "Eiffel Tower") and combine this with humans in the loop to improve efficiency and avoid subtle errors for critical cases.</li><li>Deduplicate data to reduce labeling overhead.</li><li>Leverage native speakers and labelers with relevant cultural experience to your use case to avoid confusion around subtle ambiguities in language. For example, Greeks will associate the color "purple" with sadness. At the same time, those from China and Germany will consider purple emotionally ambivalent, and those from the UK may think purple to be positive.</li></ul>

NLP Data Labeling

<p>Hopefully, you have found this guide helpful as you progress in your understanding of machine learning and that you can apply these practical insights to improve your data labeling pipelines. We wanted to share the best practices we learned from providing billions of annotations to some of the largest companies in the world.&nbsp;</p><p>As the machine learning market grows, there is an ever greater need for high-quality data and labels. We suggest you take a holistic approach to machine learning, from identifying your core business challenges to collecting and labeling your data.&nbsp;</p><p>This guide aims to equip you with the knowledge you need to set up high-quality data annotation pipelines. If you are using your own labeling workforce, check out&nbsp;<a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a>, which offers a best-in-class annotation infrastructure built by expert annotators. Alternatively,&nbsp;<a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a>&nbsp;provides an easy way to offload your data labeling and ramp to production volumes if you need labeling resources. If you have any other questions, please reach out, and we will be happy to help.</p>

Conclusion

<p>1.&nbsp;<a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank" style="color: inherit;">Scale Studio</a></p><p>Scale's labeling infrastructure, just bring your own labeling workforce.</p><p>2.&nbsp;<a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank" style="color: inherit;">Scale Rapid</a></p><p>Scale up to production workloads with Scale's expert labeling workforce.</p><p>3.&nbsp;<a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank" style="color: inherit;">Scale Nucleus</a></p><p>Curate your data and identify opportunities to improve annotations.</p><p>4.&nbsp;<a href="https://arxiv.org/pdf/2108.00114.pdf" rel="noopener noreferrer" target="_blank" style="color: inherit;">On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models</a></p><p>Dive deeper into the importance of human annotations for deep learning models in this paper written by Scale's own Zeyad Emam, Andrew Kondrich, Sasha Harrison, Felix Lau, Yushi Wang, Aerin Kim, and Elliot Branson.</p><p>5.<a href="https://scale.com/blog/data-annotation-how-to" rel="noopener noreferrer" target="_blank" style="color: inherit;">&nbsp;</a><a href="https://arxiv.org/pdf/2108.00114.pdf" rel="noopener noreferrer" target="_blank" style="color: inherit;">How to Set Up Data Annotation Pipelines</a></p><p>Important factors to consider when setting up you data annotation pipelines, from tooling to automation approaches.</p>

Additional Resources

The success of your ML models is dependent on data and label quality. This is the guide you need to ensure you get the highest quality labels possible.

Data Labeling: The Authoritative Guide

open-dataset-nucleus

slugs must be all lowercase. ie: ‘my-slug’

events-banner

This entry represents both interview quote highlight and interview chat highlight because of Builder limitations. Choose the fields that matches either of them.

interview-highlight

If it's selected, the author and quote fields will be taken, otherwise the chat fields. The default is for chat.

resources-download-model

transform-session

CTA to open the session. Leave blank if you don't want a cta for that session

Link for the CTA. Usually the link to the exchange session

blog-filter-type

event-transform-x-ticket

solution

homepage-quotes

logos

homepage-content

builder-debug

transfrom-x-in-person

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

guides-cta

summit-2023-partners

sales

z[deprecated] page

quotes

z[deprecated] compare

product

blog-category-model

announcement-banner

accelerate

Vercel Powered and Edge Computed OG Image. Preferred over og img file upload. If this field is non empty, it will take precedent over the uploaded file

summit-2023-speakers

homepage-resources

customers

Display order on the page /customers. Higher values show first

page-cross-product-webinar

interview

event-transform

Event model for new marketing events page

event-2-0

homepage-event

custom-fonts

transform-x-speaker

Be careful to not add any trailing white spaces to the slug. Slugs must be all lowercase. ie: ‘my-slug’

Order of the featured speakers displayed on the home page for transformX

z[deprecated] document-ai

summit-2023-agenda

event-converge

hosts

author

transform-x

Page: /In-Person. Section: Hero. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Card Body

Page: /In-Person. Section: Hero. Text: Tagline

open-dataset

z[deprecated] customers

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

legal

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

z[deprecated] guides-old

z[deprecated] guides-lp

partners

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Government

Solutions

Resources

Customers

Pricing →