image.jpg

Clemens Viernickel

Nucleus Business Lead

Our machine learning team at Scale has recently fine-tuned GPT-4 to achieve state-of-the-art performance (84% accuracy) for generalized text-to-SQL translation on one of the most popular benchmark datasets, the <a href="https://yale-lily.github.io/spider">SpiderDev Set</a>. In this blog post, we will discuss why text2sql is an important use case, why it is hard in practice, where fine-tuning can help, how we implemented a real-world solution, and finally, what our results were.
<img src="https://img.plasmic.app/img-optimizer/v1/img/755250530a13af64c4ad33b755d28164.png?f=webp&amp;q=75" alt="" width="1785" height="982">
<h3>Why is Text2SQL important?</h3>
Most business decisions today are data-driven decisions. This means that organizations collect, aggregate, and interpret large amounts of available information about their business or the market environment with a set of tools and processes that are often summarized as business intelligence or BI. However, obtaining the relevant pieces of information from the vast amounts of available data typically requires analytical expertise (SQL or similar) and knowledge of the relevant databases, dashboards, or related tools. This often creates a massive bottleneck and reliance on data analysts to build these tools, which then proliferate and become hard to navigate. Multi-billion dollar industries have emerged to provide generalist or highly specialized analytics tools to bridge this gap.
The advent of large language models is poised to change this paradigm with the ability to generate SQL queries directly from natural language questions such as &ldquo;How many vehicles did we sell last year?&rdquo;. Building generative models that can robustly generate SQL queries for any given set of databases hence has the potential to disrupt an entire industry and truly democratize access to structured data at large.
<img src="https://img.plasmic.app/img-optimizer/v1/img/9e54c6dce3400ff9a8cb63b712b06e51.png?f=webp&amp;q=75" alt="" width="1454" height="213">
<h3 dir="ltr">Why are LLMs still bad at SQL in the real world?</h3>
Running some basic tests using models like OpenAI&rsquo;s ChatGPT provides very promising results:
<img src="https://img.plasmic.app/img-optimizer/v1/img/d03d28d863f45674f2362c3621c9da2e.png?f=webp&amp;q=75" alt="" width="518" height="362">
Also, looking at the leaderboard of benchmark datasets like the infamous SpiderDev makes it appear that the problem is pretty much solved:
<img src="https://img.plasmic.app/img-optimizer/v1/img/c906e003eec7e9f6002a4e8d41b34daf.png?f=webp&amp;q=75" alt="" width="1785" height="1569">
However, despite the impressive code generation capabilities of state-of-the-art language models like GPT-4, they are not immediately useful for generating queries that run on custom, real-world databases. First of all, the LLMs do not know the schema of the databases in question out of the box. The most obvious solution is to provide the schema to the model in addition to the prompt.
However, in many cases, real-world databases will have hundreds of columns with custom names. The schema might not fit into the context window of the prompt and even if it does, the model still does not understand the meaning of the column names and how they relate to each other. For example, does a &ldquo;date&rdquo; column in a vehicle sales database record the time the sale was recorded or the time the sale took place? A very robust understanding of typical business terms for the given databases and the column contents is essential, especially to correctly apply aggregation and window functions. The relationships between multiple table schemas are also difficult to convey in the prompt, but this is required for slightly more complex operations like JOINs.
<h3 dir="ltr">How can fine-tuning and retrieval help to resolve these challenges?</h3>
At Scale, we are working with enterprise customers across many industries to build customized Generative AI solutions for their respective use cases. In most of these applications, we fine-tune an underlying base model to solve the relevant business problems at the required accuracy level. Fine-tuning not only can improve the performance of a model for a given task, but can also drive model safety and alignment, ensuring a certain tone and behavior. It is also a good way to improve ROI as it can be used to teach smaller (and cheaper) models a very specific skill and eventually even outperform much bigger, generalized models at this task.
<img src="https://img.plasmic.app/img-optimizer/v1/img/7d078ffc87627fe7fe355e0d4a203836.png?f=webp&amp;q=75&amp;w=3840" alt="" width="1785" height="837">
Fine-tuning is an effective way to improve the specificity of a certain skill that the model is capable of performing but has not yet mastered. It can be used to teach a model highly specific terms and instructions and improve its capabilities. A good way to figure out if fine-tuning is going to work is by experimenting with prompt engineering. As a rule of thumb, if prompt engineering shows promising results, then fine-tuning will likely be effective for the given task.
<img src="https://img.plasmic.app/img-optimizer/v1/img/5d40ce3bb89bf54d983266c129db7fd0.png?f=webp&amp;q=75" alt="" width="1785" height="1512">
Conversely, fine-tuning is not a good way to add new data or knowledge, such as the database schema or even detailed explanations of columns and their relationships to the model. Instead, this type of context information is best infused into a model using Retrieval Augmented Generation or RAG (see this <a href="https://scale.com/blog/retrieval-augmented-generation-to-enhance-llms">recent blog post</a> for a deep dive on RAG). Hence, a real-world solution will likely have to include both fine-tuning and RAG to achieve acceptable results.
<h3 dir="ltr">How did we implement our solution?</h3>
Our solution reflects a system intended for enterprise customers. Accordingly, we benchmarked multiple techniques that could be used for real-world use cases against a baseline:&nbsp;
<ul>
<li dir="ltr" aria-level="1">
Full database schema with off-the-shelf model (Baseline)
</li>
<li dir="ltr" aria-level="1">
Schema RAG with In-context-learning (ICL)
</li>
<li dir="ltr" aria-level="1">
Fine Tuned model against Schema RAG and ICL
</li>
</ul>
We&rsquo;ll now walk through each of these in more detail.
Database Schema Retrieval
<img src="https://img.plasmic.app/img-optimizer/v1/img/d830672e9c198cfd081bca63a0571278.png?f=webp&amp;q=75&amp;w=3840" alt="" width="1785" height="678">
The Spider dataset is the standard benchmark for comparing natural language to SQL models and methods. However, real-world enterprise SQL databases differ from Spider in both size and complexity. Whereas 90% of the databases in Spider&rsquo;s Train and Dev datasets contain fewer than 50 columns, enterprise databases contain up to and beyond 1000 unique columns. This discrepancy renders the common approach of providing the entire database schema in the prompt infeasible for real-world use cases, given token limit constraints and the &ldquo;lost in the middle problem.&rdquo;
As many SQL queries require only a fraction of all columns, we solve the above dilemma with a fine-tuned retrieval system, which retrieves those database features relevant to a user&rsquo;s question. Given a customer&rsquo;s database schema, we can fine-tune a model to learn the unique ways customers refer to their database. Once the embedding model is deployed into the backend of our Enterprise Generative AI Platform (EGP), we can easily create, populate, and query the retrieval Knowledge Base.
<pre class="language-python"><code>from scale_egp.sdk.client import EGPClient
from scale_egp.sdk.models import S3DataSourceConfig, CharacterChunkingStrategyConfig

# Instantiate client
client = EGPClient()

# This is Pseudocode
embedding_model = client.models().create(
 model_type="embedding",
 model_template="sentence_transformers_embedding_model",
 model_template_config={
 weights_uri: "s3://model_weights/presigned_url"
 }
)

# Create a Knowledge Base
knowledge_base = client.knowledge_bases().create(
 name="my-knowledge-base" 
 embedding_model_id=embedding_model.id,
)

# Configure knowledge base and uplaod
data_source = S3DataSourceConfig(
 s3_bucket="my-bucket",
 s3_prefix="my-schema",
)

chunking_strategy_config = CharacterChunkingStrategyConfig()

upload = client.knowledge_bases().uploads().create_remote_upload(
 knowledge_base=knowledge_base,
 data_source_config=data_source
 chunking_strategy_config=chunking_strategy_config
 )

# Query schema for a given question
query = "What was last month's total expense for service provider X?"
retrieved_schema = client.knowledge_bases().query(
 knowledge_base=knowledge_base, 
 query=query, 
 top_k=20, 
)</code></pre>
In context learning
With the schema retrieval providing a lower token count, we can supply more relevant context to address misalignment between business terms and the database schema. On initial deployment, we collect terms and information relevant to SQL logic. For example, &ldquo;the term MPG refers to miles per gallon&rdquo;, or &ldquo;stock means filter on asset_type=&rsquo;equity&rsquo;&rdquo;. This initial retrieval mechanism is the primer in the engine of our data flywheel. As users then interact with the tool, we collect real-world samples that can be retrieved for in-context learning. With this additional corpus, we provide queries from similar questions in the context window provided to the LLM.
Fine Tuning
Tying together prompt engineering and the above retrieval mechanisms, we optimize the density of information available to the out-of-the-box model. To get the final accuracy boost, we turn to fine-tuning. For customers, this means collecting real user data and fine-tuning an LLM to learn the nuances of their data and terminology. The fine-tuning not only fills in gaps in the retrieval data but hones in on trends or relationships unknown to users. Once the model is complete, each step is seamlessly integrated with the EGP SDK.
To prove out the viability of this system, we leveraged&nbsp;<a href="https://platform.openai.com/docs/guides/fine-tuning/create-a-fine-tuned-model">OpenAI&rsquo;s Fine-tuning API</a> to train GPT-3.5 and GPT-4. For each question-query pair in the Spider training set, we used the above methodology to create prompts with at most 20 features and five relevant question, SQL query pairs. For each generated prompt, we simply set the target to the respective SQL query. We use the same approach to generate a validation set of question, query pairs from the Spider dev set. After packaging up the train and validation sets into files, uploading the data, fine tuning models and generating validation predictions was completely handled by OpenAI&rsquo;s robust APIs.
<pre class="language-python"><code># Create a custom LLM
LLM_MODEL = client.models().create(
 model_type="llm",
 model_template="llm_engine_model",
 model_template_config={
 weights_uri: "s3://model_weights/presigned_url"
 }
)


class Text2SQLApplication:

 name = "Text2SQL"
 description = "Natural language to SQL queries for My Company"
 llm_model = LLM_MODEL.id

 def __init__(self, schema_knowledge_base_id: str,
 icl_knowledge_base_id: str):
 self.schema_kb_id = schema_knowledge_base_id
 self.icl_kb_id = icl_knowledge_base_id

 def create_prompt(self, question: str, schema_chunks: List[str], icl_chunks: 
 List[str]) -&gt; str:
 ...
 return rag_prompt

 def generate(self, question: str, schema_k: int, icl_k: int) -&gt; str:

 # Retreive relevant schema information
 schema_chunks = client.knowledge_bases().query(
 knowledge_base=self.schema_kb_id,
 query=question, 
 top_k=schema_k
 )

 # Retrieve in-context-learning samples
 icl_chunks = client.knowledge_bases().query(
 knowledge_base=self.icl_kb_id,
 query=question, 
 top_k=icl_k
 )

 # Generate prompt from retrieval
 rag_prompt = self.create_prompt(question, schema_chunks, icl_chunks)
 
 # Generate SQL
 generate_response = client.completions().create(
 model=self.llm_model,
 prompt=rag_prompt
 )

 return generate_response.completion.text</code></pre>
Validating Against Spider
We validate the system and benchmark performance using the Spider dataset. For schema retrieval, we fine-tuned a Sentence Transformer to match questions with their relevant database columns and achieved 97% recall@20. With respect to in-context learning, we leverage an out-of-the-box Sentence Transformer. For both GPT-3.5 and GPT-4, we measure the execution accuracy of generated SQL queries for a baseline (prompt with the entire database), RAG (schema retrieval and in-context-learning), and finally the respective model fine-tuned on the RAG prompts. We observe performance improvements at each stage, which results in a best execution accuracy of 83.6% on the Spider Dev set. Thus, we not only achieve state-of-the-art level performance, but have a system optimized to provide enterprise customers with the best commercially available natural language to SQL available.
<h3 dir="ltr">What are the results?</h3>
Baseline: Entire Schema in the Prompt
With the Spider validation data, we can calculate a baseline execution accuracy. For each question-query pair, we pack the entire schema of the respective SQL database (respective to the query and question) into the prompt and ask GPT-4 to answer the question given that context. From this simple baseline, GPT-4 achieves an execution accuracy of 70% (a D+).&nbsp;
Adding Schema RAG and In-context learning (ICL)
Using a structured way to find the relevant parts of the DB schema with RAG shows consistent improvements in performance across models. For GPT-3.5, we see 7 ppts improvement from 60% to 66% and for GPT-4 a slightly smaller bump from 70% to 73%.
Adding Schema RAG, ICL, and fine-tuning
When additionally fine-tuning the model with specific prompt-response pairs, we see consistent further performance improvements both for GPT-3.5 and GPT-4. The final, fine-tuned GPT-4 with schema RAG and ICL achieves 84% accuracy on Spider, up from the 70% in the baseline version, which marks an impressive 14 ppts improvement. For GPT-3.5 the increase is even more pronounced, reaching 82% (almost as good as GPT-4) with RAG and fine-tuning, which is up 22 ppts from the baseline of using only prompt engineering. For GPT-3.5, the biggest increase is from fine-tuning itself, pushing performance from 66% to 82% with this technique alone.
Below is a comparison of the performance across the three different approaches for both GPT-3.5 and GPT-4.
<img src="https://img.plasmic.app/img-optimizer/v1/img/6144a12443945c2d0464b7dd81b04199.png?f=webp&amp;q=75" alt="" width="1785" height="1171">
Let&rsquo;s look at a practical query example to show the difference between using GPT-4 out of the box versus the RAG and fine-tuned version.
<img src="https://img.plasmic.app/img-optimizer/v1/img/3af09a34221fa56b5202729019a74ee3.png?f=webp&amp;q=75" alt="" width="1785" height="914">
We can see that the fine-tuned model displayed on the right-hand side not only interprets the terms for the natural language query correctly but also applies a better and more efficient query structure, using a subquery instead of a left join.
<h3 dir="ltr">What&rsquo;s next?</h3>
Our solution is not quite on top of the SpiderDev leaderboard, as most of the submitted architectures rely on prompt engineering and data pre-processing that is extremely tailored to this benchmark. However, our model does achieve top 5 performance and crucially demonstrates a comparable accuracy even when deployed in much more complex, real-world contexts and databases.
If you&rsquo;re interested in using our Text2SQL model for your business use case or want to learn more about our solutions in fine-tuning and RAG, book a demo below.

Scale teamed up with OpenAI to fine-tune GPT-4 and achieved state-of-the-art performance for generalized text-to-SQL translation on the SpiderDev Set! 

We Fine-Tuned GPT-4 to Beat the Industry Standard for Text2SQL

RAG expands Generative AI's applications by injecting external info into prompts, aiding context-specific output generation for diverse data sources.

Enhancing LLMs with Retrieval Augmented Generation

How Large Language Models are Trained and Tuned using Reinforcement Learning with Human Feedback (RLHF).

Why is ChatGPT so good?

For anyone following AI or machine learning trends in recent weeks, text-to-image generation has taken the spotlight by force. A never-ending stream of synthetically-generated (yes, algorithm-generated!) images is populating social media feeds. This was very different just a couple of months ago, before an invite only beta version of <a href="https://openai.com/dall-e-2/" rel="noopener noreferrer" target="_blank">Open AI’s DALL-E 2</a> was opened to the public in July. With DALL-E available via signup and waitlist, and now with <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">Stable Diffusion</a> released to all as public open-source software, developers have been prolific and building and hosting an array of online image generators. Perhaps the users of these image generators are even more prolific than their authors—libraries of millions of generated images are swiftly gaining popularity, making it easy to browse other people’s creative ideas and find inspiration. We’ve put together a list of the 5 best, openly available online AI image generators using <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">stable diffusion</a> to help you jump right into the mix.<h3><a href="https://beta.dreamstudio.ai/dream" rel="noopener noreferrer" target="_blank">Dream Studio</a></h3>This is the foundational tool released by the company that created Stable Diffusion, Stability AI. The basic version requires a login, but is free to use. The generator offers a ton of settings, including the model version, Cfg scale (how close the image will be to your prompt), steps, number of outputs, image dimensions and more. You can start both from scratch or with a seed image of your choice. Conveniently, Dream Studio keeps a history of all your creations and lets you download the full resolution image. There is also a guide on how to write the best prompts so that the generator achieves the best results.<h3><a href="https://replicate.com/stability-ai/stable-diffusion" rel="noopener noreferrer" target="_blank">Replicate</a></h3>Replicate is a small startup on a mission to make machine learning models easier to use for everyone. They provide a service that helps you use previously uploaded models or upload your own model to run inference on demand, instead of spinning up your own server. With the wave of interest in stable diffusion, they have created a useful web app to use the model in order to generate images as well. Signup is required, but otherwise it is free to use. To date, the model has generated more than 5 million images!<h3><a href="https://creator.nightcafe.studio/stable-diffusion-image-generator" rel="noopener noreferrer" target="_blank">NightCafe Creator</a></h3>NightCafe is a company whose mission is to democratize the creation of art using technology. They have been going strong since 2019, but have recently gained huge momentum with the advancements of text-to-image models. A simple text field invites you to enter your prompt, but advanced customizations are available. You can generate up to 5 images for free, but have to buy credits afterwards. NightCafe also offers other creation models besides stable diffusion, such as VQGAN+CLIP, which was their most popular before 2022. They offer a huge collection of artworks created by other users, a Discord server and a Subreddit for discussions.<h3><a href="https://www.phase.art/" rel="noopener noreferrer" target="_blank">Phase.art</a></h3>This project is a simple text-to-image generator using stable diffusion, with only a simple input for a prompt. There are suggested categories and tokens that can be used to compose the prompt and get inspiration. Creation requires login, which gives you 10 credits (equivalent to 10 images generated) and also enables you to revisit your previously generated images. After the first 10, you have to buy additional credits to keep creating. Phase.art offers a Discord server to connect with fellow text-to-image artists.<h3><a href="https://www.craiyon.com/" rel="noopener noreferrer" target="_blank">Craiyon</a></h3>This AI image generator was previously known as DALL-E mini, and became wildly popular as the original DALL-E by OpenAI is not yet openly available to everyone. The authors, Boris Dayma and Pedro Cuenca, attempted to implement a similar image generator, but using an open source model. This is actually not a stable diffusion model and the results are sometimes a bit less polished (especially for faces) when compared to other image generators. However, the service is entirely free and enables you to download your creations, even if only as a screenshot and not in full resolution.<h4>Not sure how to get started or what prompts to use?</h4>Fortunately, with web apps like <a href="https://lexica.art/" rel="noopener noreferrer" target="_blank">Lexica.art</a> and <a href="https://arthub.ai/" rel="noopener noreferrer" target="_blank">ArtHub.ai</a> there are several fast growing libraries of images that other people have created using stable diffusion including the relevant prompt they used in the AI image generator. If you’re in lack of ideas, go ahead and browse the prompts that other users submitted to get started!If you'd like to learn more about Stable Diffusion and other generative image models, check out our <a href="https://scale.com/guides/diffusion-models-guide" rel="noopener noreferrer" target="_blank">Guide on Diffusion Models here</a>.

5 tools to assist you in generating beautiful art from text prompts with ML models.

5 open, freely available AI image generators to get you off the ground.

How to Make Your Own AI-Generated Images

Warm-Start Your Vision Project: 10 Robust Datasets for Pre-Fine-Tuning

Warm-Start Vision Projects with Robust Pre-Fine-Tuning Data

The 10 Best Public Datasets for Object Detection in 2022

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>