experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

Government

People

Engineering

For the past 12 years, I've been consumed by the world of data. I've taken every chance to expand my skills, from data warehousing and analytics to cutting-edge software solutions. It's been immensely rewarding to push the boundaries of what's possible with data.&nbsp;
However, despite my love for big data platforms, I no longer felt the same level of intensity and inspiration I once had. What once was unchartered territory became business as usual.
Now, though AI, machine learning, and neural networks, there are new frontiers to explore. The nonstop learning, the unanswered big questions, the push into the unknown - that pure adrenaline rush is back. Just when I thought I'd mapped the data world, it's drawn me into its depths once again.
Machine learning and neural networks captured my imagination back in 2015 when I attended the Data Science Workshop at CERN. In the years since, I've worked on some interesting data science projects but the innovation happening at Scale is truly groundbreaking and pushes the field forward in a major way.&nbsp;
Arun Murthy, Scale&rsquo;s Chief Product &amp; Technology Officer, has been a leader I trust since my Hortonworks days. He gave me a call to explain why he joined Scale and how pivotal this company&rsquo;s role is going to be in this new era of AI and large language models. The spark was reignited immediately. I knew this opportunity was exactly what I&rsquo;ve been looking for.
It&rsquo;s only my first month here, but based on the interactions I had during the recruitment process, the first Scale meetup in Budapest, and a few more early engagements, I&rsquo;ll say the Scaliens have something about them that is truly unique and remarkable. Everyone&rsquo;s working towards the same mission: it&rsquo;s open, efficient, relentless, collaborative, and fun. You know that you&rsquo;re at the right place and the team is on a roll, when everyone&rsquo;s working super hard yet they don&rsquo;t hesitate to crack a laugh.
At the end of the day, it&rsquo;s the technology stack, product suite, and team culture that makes a career choice sticky. All that aside, Scale is the leading AI expert company: we are pushing the envelope in autonomous driving, LLMs, computer vision, and many other arenas. I&rsquo;m thrilled to embark on this new chapter and can&rsquo;t wait to contribute to Scale&rsquo;s formative work.
For those in Budapest - I hope to catch you in one of our many meetups and fun AI events coming to &ldquo;your cinema&rdquo; soon!

Why_I_joined_scale-Gergely Devenyi (1).jpg

Gergely joined Scale as the Regional Director of the Budapest Site t to accelerate the development of AI applications 

Why Gergely Devenyi joined Scale as the Regional Director of the Budapest Site 

Why I Joined Scale

As Scale works with governments and the leading companies around the world, one of the top questions that we are asked is, “How do we ensure that AI is safe to deploy?”&nbsp;Based on our years of work with the leading frontier model developers and enterprises deploying AI, we believe the best approach is to implement a risk-based test and evaluation framework to ensure that the AI is safe for its intended use case. For decades, test and evaluation has been a standard part of a product development cycle to ensure that products meet the necessary safety requirements to be brought to market or acquired by the government. While it may be most recognized in heavily regulated industries like aviation or medicine, test and evaluation takes place within every industry.For generative AI, such as large language models (LLMs), test and evaluation, especially red teaming, has the potential to help us understand bias in the models, weed out vulnerabilities, prevent algorithmic discrimination, and ensure that LLM developers and adopters understand the opportunities and limitations of their models.&nbsp;Today, given that LLMs are still relatively new, the methodologies and industry-wide standards to govern test and evaluation for generative AI are still being established. To help move the conversation forward, Scale recently <a href="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/test-and-evaluation.pdf" rel="noopener noreferrer" target="_blank">published the industry’s first comprehensive technical methodology for scalable and effective LLM test and evaluation</a> based on the best practices we have garnered from our work across the industry.Scale’s approach features a hybrid method that combines both an automated assessment with human evaluation, resulting in a comprehensive, scalable process. While there is still much work to be done to drive industry consensus around a test and evaluation approach, and codify the standards necessary for test and evaluation of generative AI, this is a critical first step to move AI safety forward.Here are the key takeaways from Scale’s methodology:<ul><li>Comprehensive test and evaluation requires testing of both the model capabilities–defined as a model’s ability to follow instructions responsibly and factually–as well as stress testing of the safety risks associated with the model.</li><li>Scalable test and evaluation can be greatly accelerated with intelligent incorporation of state-of-the-art automated methods for both capability assessment and vulnerability detection, but will still require humans in the loop to ensure accuracy in the most complex situations. We can optimize overall evaluation efficiency without sacrificing accuracy by assessing when it is safe to trust automated approaches through computing quantitative estimates of evaluation model confidence.</li><li>A test and evaluation framework needs to be flexible enough to adapt to the risk associated with the intended use case. For example, AI for medical use cases, such as those governed by the FDA, should require more rigorous test and evaluation than AI being used for generating poetry.&nbsp;</li></ul>Read the full white paper <a href="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/test-and-evaluation.pdf" rel="noopener noreferrer" target="_blank">here</a> and our vision for test and evaluation <a href="https://scale.com/guides/test-and-evaluation-vision" rel="noopener noreferrer" target="_blank">here</a>.

Test & Evaluation Whitepaper.png

Based on Scale's years of work with the leading frontier model developers and enterprises deploying AI, we believe the best approach is to implement a risk-based test and evaluation framework to ensure that the AI is safe for its intended use case. 

Test & Evaluation: The Right Approach for Safe, Secure, and Trustworthy AI

Today, Scale will sign onto the Biden-Harris Administration&rsquo;s voluntary commitments to ensure that AI is safe, secure, and trustworthy. These commitments are critical to the future of AI. The reality is that progress in frontier model capabilities must happen alongside progress in model evaluation and safety. This is not only the right thing to do, but practical. &ldquo;To advance our knowledge and understanding of large language models (LLMs)&ndash;and therefore advance their capabilities&ndash;we must understand their weaknesses as much as their strengths. It is imperative for the tech industry to take proactive measures like these given the rapid pace of technological development.&rdquo; &ndash;Alexandr Wang
The Scale team commends the White House&rsquo;s leadership in bringing together companies that are shaping the future of responsible AI development. America&rsquo;s continued technological leadership hinges on our ability to build and embrace the most cutting-edge AI across all sectors of our economy and government. This can only be achieved with collaboration between industry and government to create a broad understanding of the possibilities and limitations of the technology, so the benefits can be maximized and risks minimized.&nbsp;
Safe, secure, and trustworthy AI has always been at the core of Scale&rsquo;s mission. For more than seven years, we have built deep expertise in the entire lifecycle of developing, fine-tuning, and testing and evaluating AI. Our role as the data partner for the leading model developers has given us a unique vantage point across the AI ecosystem. It has allowed us to deeply understand the challenges that must be addressed, how various players across the industry are working to tackle them, and ultimately develop our own technology to test and evaluate LLMs.&nbsp;
As a pioneer in the AI space, Scale is at the forefront of defining what it means for an LLM to be trusted and safe.
<ul>
<li>Scale is committed to helping the broader AI community understand the benefits and safety risks and partnering with leading model builders to evaluate the safety of their models. Red teaming, in particular, is a core pillar of our approach to testing and evaluation. <a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/" target="_blank" rel="noopener noreferrer">As announced by the White House&rsquo;s Office of Science and Technology Policy</a>, Scale was selected to develop the platform to evaluate leading generative AI systems for the largest red teaming exercise, to date, at DEFCON 31.&nbsp;</li>
<li>Scale published a vision for <a href="https://scale.com/guides/test-and-evaluation-vision" target="_blank" rel="noopener noreferrer">testing and evaluation </a>and developed a platform to help establish a framework for an effective industry-wide approach, limit bias, and uphold the Responsible AI Principles, such as those outlined in the <a href="https://www.whitehouse.gov/ostp/ai-bill-of-rights/" target="_blank" rel="noopener noreferrer">Biden Administration&rsquo;s AI Bill of Rights</a>. Successful adherence to and execution of these commitments is contingent on robust foundation model testing and evaluation.&nbsp;</li>
<li>Scale is grateful to join fellow industry leaders making these voluntary commitments to ensure the responsible development of AI technology. As we forge ahead, the company is committed to working with the Administration and public, private, and civil society to provide recommendations for concrete steps that can be taken to build a robust framework to unlock safe, secure, and trustworthy AI globally.</li>
</ul>

image.jpg

Today, Scale will sign onto the Biden-Harris Administration’s voluntary commitments to ensure that AI is safe, secure, and trustworthy.

Our Commitment to Ensuring Safe, Secure, and Trustworthy AI

Our Commitments to Safe, Secure, and Trustworthy AI

As policymakers return to Washington from the annual August recess, AI regulation will dominate political discussions. Senate Majority Leader Chuck Schumer has announced multiple &ldquo;AI Insight Forums&rdquo; beginning September 13, the White House has announced plans to issue an Executive Order, and the world is getting ready for a first-of-its-kind global AI Safety Summit in the United Kingdom this November.
Since our earliest days as a company, Scale has worked across industries to help accelerate the development of AI. Today, we work with the leading frontier model developers to fine tune, red team, and test and evaluate their large language models. This work also provides us a unique vantage point to understand, create, and implement industry best practices and understand what we believe is the right balance to strike on a regulatory approach that maximizes innovation, while putting in place the proper guardrails.
In addition to our work with the leading tech and automotive companies, Scale has worked with the U.S. government since 2020, helping to ensure the United States leads the world in AI adoption. The U.S. Department of Defense (DoD) has led the deployment of AI in the federal government to date, and there are no shortage of use cases for every agency to better enhance their day-to-day operations through AI. For example, a large language model at the U.S. Department of Energy would put information at people&rsquo;s fingertips more quickly on complex research topics like grid or hydrogen capacity for electric vehicles in major cities.&nbsp;
Our collective years of working with leading companies and the DoD has uniquely positioned Scale to understand what works and provide recommendations for the federal government to efficiently and fully embrace AI.&nbsp;
This blog is the first in our series titled, &ldquo;Scale AI Policy Framework,&rdquo; and will focus on our recommendations for successful government AI adoption based on our takeaways from our work with the DoD and broader national security community. Scale strongly believes that three things are necessary for the U.S. government to successfully adopt AI:&nbsp;
<ol>
<li>Robust AI-ready data infrastructure at each Agency&nbsp;</li>
<li>Implementation of a risk-based Test and Evaluation framework prior to AI deployment&nbsp;</li>
<li>Innovative and expedited paths to AI acquisition&nbsp;</li>
</ol>
To read the full white paper click <a href="https://pub-9ba39fa3c4764cd0bb3a9579562941d6.r2.dev/gov-ai.pdf" target="_blank" rel="noopener">here.</a>

Our first "Scale AI Policy Framework" blog focuses on successful government AI adoption, drawing from our DoD work.

Scale

Scale AI Framework for Successful Government AI Adoption

OpenAI recently announced the&nbsp;<a href="https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates" target="_blank" rel="noopener">release</a> of their GPT-3.5 Turbo fine-tuning APIs. At Scale we have always believed that building <a href="https://scale.com/custom-llms">custom LLMs</a> through fine-tuning is the key to unlocking greater performance for any given organization&rsquo;s specific use case. OpenAI also named <a href="https://scale.com/blog/open-ai-scale-partnership-gpt-3-5-fine-tuning">Scale as their preferred enterprise fine-tuning partner for GPT-3.5</a>, and we have already shown early progress for the power of fine-tuning GPT-3.5 for customers like Brex. While the pre-trained GPT-3.5 model can often solve tasks with prompt engineering, the model becomes even more powerful when fine-tuned, in some cases surpassing GPT-4 in performance. Let&rsquo;s walk you through how to fine-tune GPT-3.5 in this blog post.
The first and one of the most critical steps in fine-tuning is creating the right training dataset. OpenAI recommends creating a diverse training set of conversations that are similar to the conversations that the model will see in production. These datasets can be created with Scale&rsquo;s <a href="https://scale.com/data-engine">Data Engine</a>, which is trusted by leading ML teams and enterprises to provide large volumes of high-quality data. Scale has worked with OpenAI since 2019 on powering LLMs with better data. Scale's Data Engine has powered most of the leading LLMs, and is now also powering custom LLMs for leading companies like Brex, Chegg, and Accenture. Our cost-effective operations can give you expertly labeled and diverse data at any scale, making it an indispensable asset for fine-tuning.
Fine-tuning also includes reserving the right compute resources and properly implementing the training and inference code. OpenAI&rsquo;s new fine-tuning APIs makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<h1 dir="ltr">Dataset Preparation</h1>
For this example, we will use <a href="https://scienceqa.github.io/" target="_blank" rel="noopener">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. We used <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a> to fine-tune Llama 2 on this dataset in <a href="https://scale.com/blog/fine-tune-llama-2">our previous blogpost</a>, and you can follow the data preparation steps in that post.&nbsp;
Now, let's convert this dataset into OpenAI&rsquo;s supported format, a JSONL file with lists of conversations. Each example in the dataset should be a conversation in the same format as the chat completions endpoint, specifically a list of messages where each message has a role and content (and optionally a name). The provided assistant messages in the data should be the ideal responses you want the model to provide.
<pre class="language-python"><code>def format_chat(row): 
 return json.dumps(
 {"messages": [
 {"role": "user", "content": row["prompt"]},
 {"role": "assistant", "content": str(row["response"])},
 ]}
 )
def convert_dataset(df, file_name):
 df["conversation"] = df.apply(format_chat, axis=1),
 with open(file_name, 'w') as jsonl_file:
 for example in df["conversation"]:
 jsonl_file.write(example + '\n')</code></pre>
This results in our dataset looking like this:
<pre class="language-markup"><code>{"messages:" [{"role": "user", "content": "Context: In a group of cows, ..."}, 
 {"role": "assistant", "content": "B"}]}
["messages": {"role": "user", "content": "Context: In a group of guppies..."}, 
 {"role": "assistant", "content": "A"}]}</code></pre>
The OpenAI API supports training with a training and validation dataset, and provides loss numbers on both during the course of training. These samples will be your initial signal of how much the model is improving, when compared to generations from the base model on the same test conversations.
These dataset files must be uploaded to OpenAI&rsquo;s file endpoint. Make sure to add your OpenAI API key to your system environment variables for authentication.
<pre class="language-python"><code>import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
file = openai.File.create(
 file=open("scienceqa_train.jsonl", "rb"),
 purpose='fine-tune',
)</code></pre>
You should get a response similar to the following:
<pre class="language-markup"><code>{
 "object": "file",
 "id": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "purpose": "fine-tune",
 "filename": "file",
 "bytes": 15974,
 "created_at": 1692743745,
 "status": "uploaded",
 "status_details": null
}</code></pre>
You can then check the status of the uploaded file using the provided file ID.&nbsp;
<pre class="language-markup"><code>openai.File.retrieve("file-CY0FPBluqbVcoHmuGLI80lqx")</code></pre>
You need to make sure that the file status is &ldquo;processed&rdquo; before proceeding with finetuning.
<h1 dir="ltr">Fine-tuning</h1>
Fine-tuning your model requires only one API call. Currently, only GPT-3.5 Turbo is available for fine-tuning. You just need to provide the IDs for the train and validation data files, the model name (only "gpt-3.5-turbo" is supported), and a suffix for the output model name.
<pre class="language-python"><code>train_data = "file-CY0FPBluqbVcoHmuGLI80lqx"
val_data = "file-428ptqQcofNSLoEP0sUDsU5x"
model = openai.FineTuningJob.create(
 model = "gpt-3.5-turbo", 
 training_file = train_data,
 validation_file = val_data,
 suffix = "scienceqa"
)</code></pre>
You would then get a response such as this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": null,
 "fine_tuned_model": null,
 "organization_id": "org-xxx",
 "result_files": [],
 "status": "created",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": null
}</code></pre>
There are several ways to experiment further; for example, increasing the epochs to 4 or 5 could have different results.
You can track the progress of the training job using the finetune ID.
<pre class="language-markup"><code>openai.FineTuningJob.retrieve('ftjob-NmNcHalWN6yGJfGnwealj7MB')</code></pre>
It will have the status &ldquo;running&rdquo; while it is training and &ldquo;succeeded&rdquo; after it is finished or &ldquo;failed&rdquo; if it failed. A successful training job will look like this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": 1692744189,
 "fine_tuned_model": "ft:gpt-3.5-turbo-0613:scale-ai:test:7qUNXuwT",
 "organization_id": "org-Vn7k74eir0QoobuD5SrHfwSD",
 "result_files": ["file-hNvzZShh5Phmng1oNb1Qs4sy"],
 "status": "succeeded",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": 9909
}</code></pre>
<h1 dir="ltr">Inference and Evaluation</h1>
Once you are done fine-tuning, you can view your job results. The training job will automatically upload a result file named step_metrics.csv, which contains training and test loss and accuracy for each training step.
We can use the training metrics as a rough sanity check on training stability:
<pre class="language-python"><code>from io import StringIO
def get_step_metrics(file_id):
 content = openai.File.download(file_id)
 eval_result = StringIO(result.decode())
 df = pd.read_csv(eval_result, sep=",")
 return df</code></pre>
However, inspecting the model samples gives the most relevant sense of model quality. We can run inference using our fine-tuned model using their chat completions API.
<pre class="language-python"><code>completion = openai.ChatCompletion.create(
 model="ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 messages=[
 {"role": "user", "content": "Context: A shopper is buying food at..."}
 ]
)</code></pre>
You will get a result like this:
<pre class="language-markup"><code>{
 "id": "chatcmpl-7qUvsuFrButlxKYSs2SispgL5KUrD",
 "object": "chat.completion",
 "created": 1692746316,
 "model": "ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "B"
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 81,
 "completion_tokens": 1,
 "total_tokens": 82
 }
}</code></pre>
<h1 dir="ltr">Results</h1>
From this fine-tuning job, the model achieves 84.58% accuracy. This is pretty good! Let's see how this compares with the original base GPT-3.5 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses.
<div dir="ltr" align="left">
<table style="border-collapse: collapse; width: 100%; margin-left: auto; margin-right: auto;" border="1"><colgroup><col><col><col></colgroup>
<tbody>
<tr>
<td>
ScienceQA Validation
</td>
<td>
GPT-3.5
</td>
<td>
GPT-4
</td>
</tr>
<tr>
<td>
Pre-trained (1-shot)
</td>
<td>
72.98%
</td>
<td>
74.8%
</td>
</tr>
<tr>
<td>
Fine-tuned
</td>
<td>
84.58%
</td>
<td>
-
</td>
</tr>
</tbody>
</table>
</div>
&nbsp;
As we can see, fine-tuning makes a big difference in the case of all of the language models. The performance gain from fine-tuning GPT-3.5 Turbo on ScienceQA was an 11.6% absolute difference, even outperforming GPT-4!
We also experimented with different numbers of training examples. OpenAI recommends starting with 50 - 100 examples, but this can vary based on the exact use case. We can roughly estimate the expected quality gain from doubling the training data size by fine-tuning on half or a fourth of your dataset and observing the quality gap. For this experiment, we randomly sampled between 50 to 5000 examples for our training set, and used the same validation set for all models.
<div dir="ltr" align="center">
<table style="border-collapse: collapse; width: 100%; height: 275.157px; border-width: 1px;" border="1"><colgroup><col style="width: 54.321%;" width="235"><col style="width: 45.679%;" width="235"></colgroup>
<tbody>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
# Training samples
</td>
<td style="height: 47.5938px;">
Test accuracy
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
50
</td>
<td style="height: 47.5938px;">
77.18%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
100
</td>
<td style="height: 47.5938px;">
78.23%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
500
</td>
<td style="height: 47.5938px;">
81.10%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
1000
</td>
<td style="height: 47.5938px;">
81.96%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
2500
</td>
<td style="height: 47.5938px;">
84.44%
</td>
</tr>
<tr style="height: 23.5938px;">
<td style="height: 23.5938px;">
5000
</td>
<td style="height: 23.5938px;">
84.58%
</td>
</tr>
</tbody>
</table>
</div>
<img src="https://img.plasmic.app/img-optimizer/v1/img/43f22b3d99c00c724094e06687a2a7fb.png?f=webp&amp;q=75">
 We can see that performance generally increases as the number of training samples increases but plateaus at some point. However, we sampled our dataset randomly, and adding well-crafted training examples that target remaining model issues or edge cases can better improve the model&rsquo;s performance.&nbsp;
Currently, only the number of epochs is available for customization. Other hyperparameters are set to what OpenAI believes to be good default values that should largely work across a large variety of datasets and scenarios.
If you would like higher performance, OpenAI recommends collecting higher quality and/or more data. You may need to consider the balance and diversity of your current training dataset, collect more examples that target edge cases, and ensure there are no lingering issues in the existing examples. The Scale&nbsp;<a href="https://scale.com/data-engine">Data Engine</a> can deliver large volumes of high-quality data that can maximize the performance of your fine-tuned models.
<h1 dir="ltr">Conclusion</h1>
Fine-tuning GPT-3.5 makes a huge difference in unlocking greater performance from base GPT-3.5 models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. If you are looking to optimize your model performance with even more powerful features, Scale also offers an <a href="https://scale.com/custom-llms">enterprise model customization platform</a>. Happy fine-tuning!

OpenAI announced the release of their GPT-3.5 Turbo fine-tuning APIs. 

How to Fine-Tune GPT-3.5 Turbo With OpenAI API

At Scale, we host a diverse set of deep learning models for both internal and external users. The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time, we were able to reduce cost while maintaining a stable latency SLA. We&rsquo;d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with&nbsp;<a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a>.
<h2>Why Cold Start Time Is Important</h2>
Without a fast enough cold start, users will often permanently provision GPUs for peak traffic. GPU hosting costs are significantly more expensive than the average CPU-based microservice. To formalize this idea in equations,
<img style="display: block; margin-left: auto; margin-right: auto;" title="n_warm_pods.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/153d7eae-b671-4ea1-a33d-fab7dc827ae0" alt="" width="650" height="75">
If we can cold start pods and make predictions within the latency SLA, we won&rsquo;t need any warm pods; otherwise we need to keep some warm pods based on the max amount of traffic we want to keep within SLA and the throughput per node.
<img style="display: block; margin-left: auto; margin-right: auto;" title="throughput_per_node.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/4c30cc1d-9231-4298-9b0f-410d73bc3911" alt="" width="650" height="128">
Then we could calculate compute seconds per wall clock second by dividing the total amount of requests and per pod throughput and plus number of warm pods. Lastly we could get the cost for serving all requests by multiplying compute seconds with the cost per second and duration.
As a concrete example, the chart below illustrates the hosting costs of one of our products as a function of user traffic. The X-axis shows the number of requests and the Y-axis is cost. The blue line is the cost curve for a configuration that has 3x the cold start time than the red line while all other configuration values are the same. When cold start time is high (blue line), we need to keep more warm pods to maintain latency SLA when there are more requests; however if cold start time is short enough (red line), we could spin up pods right when requests come in and only keep a small amount of warm pods for safety. This difference made a huge impact on cost.
<img style="display: block; margin-left: auto; margin-right: auto;" title="line_chart.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f92e2f99-dcf7-4516-afe6-a1707c2a11db" alt="" width="500" height="326">
<h2>Measurements</h2>
To understand where time is spent during cold start, we measured LLM endpoint cold start time for a Llama 2 7B model by looking at Kubernetes events and duration for each step that runs in the container. Here&rsquo;s the time breakdown:
<img style="display: block; margin-left: auto; margin-right: auto;" title="Cold_start_time_breakdown.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7f1f16d9-483d-4219-b02b-04b72176513d" alt="" width="700" height="395">
As shown in the chart, the actual model loading time is quite small and most of the time is spent on pulling docker images and downloading model weights.
<h2>Faster Pod Initialization</h2>
Since pulling images from repositories takes the majority of the time, we focused our efforts here first. From our prior experiences working on optimizing inference for various deep learning models like image generation models, we&rsquo;ve already encountered similar problems, but with LLMs the docker images are bigger and the problem is more prominent. We tried a few ideas and eventually optimized away this portion of time by caching images onto the nodes using Kubernetes daemonsets. We utilized the same technique for LLM images. The following chart describes the process:
<img style="display: block; margin-left: auto; margin-right: auto;" title="llm_cold_start_flow.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1086257a-01a0-4bbd-ad9e-04f15fcb5879" alt="" width="700" height="519">
A <a href="https://github.com/scaleapi/llm-engine/blob/main/model-engine/model_engine_server/infra/services/image_cache_service.py" target="_blank" rel="noopener">cacher</a> periodically scans through the database to get all &ldquo;<a href="https://github.com/scaleapi/llm-engine/blob/main/clients/python/llmengine/model.py#L51" target="_blank" rel="noopener">high priority</a>&rdquo; models, together with a full set of (GPU, docker image) pairs. We then construct and create/replace one daemonset for each type of GPU, and run each image with sleep commands like <code>/bin/sh -ec 'while : ; do sleep 30 ; done'</code>. This way we could dynamically maintain the set of images and preload them onto the nodes, and effectively eliminate docker image pulling time.
In addition to image caching, we reduced time to provision new nodes by creating&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/charts/model-engine/templates/balloon_a10_deployment.yaml" target="_blank" rel="noopener">balloon deployments</a> per GPU type to prewarm nodes. These pods have low priority and will be preempted when actual workloads are created.
<h2 dir="ltr">Faster Model Weights Loading</h2>
We utilize <a href="https://github.com/peak/s5cmd" target="_blank" rel="noopener">s5cmd</a> to download model weights which is <a href="https://joshua-robinson.medium.com/s5cmd-for-high-performance-object-storage-7071352cc09d" target="_blank" rel="noopener">much faster</a> than aws-cli. We used to put all files into a tarball for simplicity, but found that it&rsquo;s bad for concurrent download. We instead stored model files with 2GB chunks and had s5cmd download all the files in parallel. We also did some quick benchmarks for s5cmd parameters and chose 512 for --numworkers and 10 for --concurrency. With those changes we pushed the download time of the Llama 2 7B model (12.6GB) from 85s to 7s, achieving 14.4Gbps, which is close to the EBS volume bandwidth limit (16Gbps) for our host. Here is an example how we invoke s5cmd:
<pre class="language-python"><code>s5cmd --numworkers 512 cp --concurrency 10 {source_folder} {destination_folder}</code></pre>
We heavily utilize <a href="https://github.com/huggingface/text-generation-inference" target="_blank" rel="noopener">text-generation-inference</a> (TGI) to serve models. TGI prefers <a href="https://github.com/huggingface/safetensors" target="_blank" rel="noopener">safetensors</a>, a file format for storing tensors, and it loads <a href="https://huggingface.co/docs/safetensors/speed" target="_blank" rel="noopener">2 times faster</a> than raw PyTorch bin files. With safetensor files it could load the model in less than 20 seconds. Thus by default now we store only safetensors (with <code>model.save_pretrained(output_dir, safe_serialization=True</code>)) to avoid TGI having to convert PyTorch bin files on the fly.
<h2 dir="ltr">Summary</h2>
With all the optimizations applied we successfully reduced cold start time from more than 6 minutes to less than 40 seconds. This greatly improved our GPU utilization and our ability to deal with spiky traffic. Given the current 30-40s lag, it is now possible to save cost by having zero workers for sporadic workloads.
<h2 dir="ltr">Future Work</h2>
<ul>
<li dir="ltr" aria-level="1">
Though this approach conceptually works, we still need to implement (and open source) auto scaling from zero since Kubernetes&rsquo; Horizontal Pod Autoscaler does not scale from zero.
</li>
<li dir="ltr" aria-level="1">
We fine tune models with PEFT (Parameter-Efficient Fine-Tuning). The code is currently private and we will open source in the near future.
</li>
<li dir="ltr" aria-level="1">
We currently merge back PEFT trained weights onto the base model. This is cost inefficient since we need to maintain one deployment per fine-tuned model, and the cold start optimizations presented here only partially mitigate this problem. Internally, we have implemented dynamic loading / unloading for <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener">LoRA</a> and <a href="https://arxiv.org/abs/2205.05638" target="_blank" rel="noopener">IA3</a> adapters with DeepSpeed-sharded models. This way only one deployment is needed per base model, and each process would dynamically swap out adapters when serving traffic from different fine-tunes. The future work is to open-source this framework and get adapters working with continuous batching.
</li>
<li dir="ltr" aria-level="1">
To better generalize our image cache beyond just &ldquo;high-priority&rdquo; endpoints, we are investigating lazy-loading of images with projects like&nbsp;<a href="https://github.com/containerd/stargz-snapshotter/tree/main" target="_blank" rel="noopener">stargz</a>. 
</li>
</ul>
&nbsp;
<h2 dir="ltr">Next Steps</h2>
Looking to customize and serve LLMs? Try&nbsp;<a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> today!&nbsp;
Learn more about how Scale can help you <a href="https://scale.com/custom-llms">customize LLMs for your unique use case</a>.

The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time we were able to reduce cost, due to its significant impact on the number of warm pods if a stable latency SLA needs to be maintained. We’d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with LLM Engine.

How To Reduce Cold Start Times For LLM Inference

Today, we are excited to announce a strategic partnership with OpenAI, providing GPT-3.5 fine-tuning for the world&rsquo;s leading enterprises. At Scale, we believe that <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">fine-tuning</a> is the key to unlocking the performance of LLMs for any organization's most critical use cases - and we have seen both great excitement and demand for this fine-tuning capability from the market. Our partnership brings together OpenAI's advanced base model GPT-3.5 with Scale's fine-tuning expertise and industry-leading <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Data Engine</a> to help every company create custom state-of-the-art models for their specific business needs. Scale already performs fine-tuning for many commercial and open-source models. As OpenAI&rsquo;s preferred fine-tuning partner for GPT-3.5, we are excited to leverage their powerful APIs to help even more enterprises build the most powerful custom LLMs that increase efficiency while reducing costs.
&nbsp;
<h2>Fine-Tuning GPT-3.5&nbsp;</h2>
&nbsp;
Fine-tuning GPT-3.5 has already yielded performance improvements for companies like Brex. As a preferred partner to OpenAI, we were granted early access to GPT-3.5&rsquo;s fine-tuning APIs. By leveraging these APIs in combination with Scale&rsquo;s enterprise <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">Custom LLM</a> platform and high-quality data from <a href="https://www.scale.com/data-engine" target="_blank" rel="noopener noreferrer">Scale&rsquo;s Data Engine</a> we delivered initial performance improvements for Brex.
&nbsp;
"To get the most value out of our models, companies are looking to use their data to create tailored services and solutions,&rdquo; said <a href="https://openai.com/blog/openai-partners-with-scale-to-provide-support-for-enterprises-fine-tuning-models" target="_blank" rel="noopener noreferrer">Brad Lightcap, COO, OpenAI</a>. &ldquo;Scale extends our ability to bring the power of fine-tuning to more companies, building on their enterprise AI experience to help businesses better apply OpenAI models for their unique needs.&rdquo;
&nbsp;
Brex has been leveraging LLMs to generate high-quality expense memos that help ease the burden of compliance requirements for employees. Automating the generation of expense memos helps save significant amounts of employee time within the expense workflow and enables companies to close their books faster. The Brex team had previously been using GPT-4 for memo generation, but wanted to explore if they could improve cost and latency, while maintaining quality, by using a fine-tuned GPT-3.5 model.
&nbsp;
By using the GPT-3.5 fine-tuning API on Brex data annotated with Scale&rsquo;s <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Data Engine</a>, we saw that the fine-tuned GPT-3.5 model outperformed the stock GPT-3.5 turbo model 66% of the time.
&nbsp;
"At Brex, we&rsquo;ve been investing heavily in AI which is transforming how we offer financial services to customers. Our ongoing partnership with OpenAI and Scale AI position us at the cutting edge by employing state-of-the-art techniques to enhance employee compliance and help finance teams close the books faster. In particular, fine-tuning GPT-3.5 has been a game changer for us, enabling us to deliver high-quality AI experiences, comparable to GPT-4, with much lower cost and lower latency. This unlocks a whole new set of capabilities for us that were previously not viable." - Henrique Dubugras, CEO, Brex
&nbsp;
<h2>How to Get Started</h2>
&nbsp;
Our pilot project with Brex demonstrates the importance of fine-tuning LLMs to deliver outcomes.&nbsp;
&nbsp;
&ldquo;We are excited to partner with OpenAI to supercharge model performance - helping every enterprise utilize AI most effectively for their unique needs. Prompting alone&mdash;atop even the best LLMs like GPT-3.5 &mdash; is not enough model customization to produce the most accurate, efficient results. As with software, an incredible amount of value comes from fine-grained optimizations, and fine tuning is critical for that.&rdquo; - Alexandr Wang, Founder and CEO, Scale AI
&nbsp;
Scale starts by preparing and enhancing your data with the <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Scale Data Engine</a>. Then, we fine-tune GPT-3.5 with your data and further <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">customize models</a> with plugins and retrieval augmented generation, or the ability to reference and cite your proprietary documents in its responses. We then leverage our&nbsp;<a href="https://scale.com/llm-test-evaluation" target="_blank" rel="noopener noreferrer">Test and Evaluation platform</a> and trained domain experts to ensure that the model exceeds your performance expectations and safety requirements.
&nbsp;
Learn more about how Scale can help you&nbsp;<a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">customize LLMs for your unique use case</a>. 
&nbsp;
&nbsp;

cover_openai.png

Exciting news! Scale partners with OpenAI, offering GPT-3.5 fine-tuning for enterprise clients. Scale's expertise + OpenAI's model = robust solutions.

OpenAI Names Scale as Preferred Partner to Fine-Tune GPT-3.5

Today, Scale is announcing our Test &amp; Evaluation offering for LLMs, which is already being leveraged by OpenAI, and is available in early access to select customers who are at the forefront of LLM development.
Since large language models broke into the public consciousness, the conversation has largely fallen into two categories: The exciting realm of new possibilities this technology can unlock, and the risks that could come from opening Pandora&rsquo;s box.&nbsp;
While many would like to paint these as two opposite ends of the spectrum, the reality is that progress in frontier model capabilities must happen alongside progress in model evaluation and safety. This is not only a moral imperative, but a practical one. In order to advance our knowledge and understanding of LLMs &ndash; and therefore advance their capabilities &ndash; we must understand their weaknesses as much as their strengths.&nbsp;
Various studies have found popular LLMs hallucinate or provide inaccurate information ranging anywhere from 20% to 52% of the time. As AI is increasingly being leveraged by <a href="https://www.nytimes.com/2023/07/19/business/google-artificial-intelligence-news-articles.html" target="_blank" rel="noopener noreferrer">newsrooms</a>, misinformation has the potential to sway global markets and global politics. Other issues like unqualified advice have the potential to lead to real-world harms, such as providing <a href="https://mwi.westpoint.edu/disinformation-in-the-age-of-chatgpt/#:~:text=A%20study%20that%20enticed%20ChatGPT,war%20in%20Ukraine%2C%20and%20school" target="_blank" rel="noopener noreferrer">false medical information</a>, feeding into <a href="https://www.washingtonpost.com/technology/2023/08/07/ai-eating-disorders-thinspo-anorexia-bulimia/" target="_blank" rel="noopener noreferrer">mental disorders</a>, and more. LLMs can even be tricked into writing <a href="https://www.axios.com/2023/08/08/ibm-researchers-trick-chatgpt-hacking" target="_blank" rel="noopener noreferrer">malicious code</a>, opening new vectors for cybersecurity attacks.&nbsp;
For AI to reach its full potential, it&rsquo;s critical the AI industry is working to address these known challenges in parallel with other advancements to their models.&nbsp;
<h2>Our Approach to Test &amp; Evaluation</h2>
At Scale, for more than 7 years we have built up deep expertise in the entire lifecycle of developing, fine-tuning, testing and evaluating AI models. Our role as the data partner for companies like OpenAI, Meta, Microsoft, and Anthropic have given us a purview across the ecosystem that no other company has. It has allowed us to deeply understand the common challenges that must be addressed, how various players across the industry are working to address them, and ultimately develop our own technology to effectively test and evaluate LLMs.&nbsp;
Today, there is no standard for test and evaluation (T&amp;E) in AI. Our leadership in this space is why in May, the&nbsp;<a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/" target="_blank" rel="noopener noreferrer">White House announced</a> Scale was selected to develop the T&amp;E platform for DEF CON&rsquo;s public evaluation of models built by Anthropic, Cohere, Google, Hugging Face, Meta, NVIDIA, OpenAI, and Stability.&nbsp;
We&rsquo;re excited to see the industry come together for this moment, but this is just one step in the right direction. That&rsquo;s why today, we are releasing our stance on Test &amp; Evaluation to help set the standard for what an effective T&amp;E approach looks like. Like other realms of cybersecurity, we need to share best practices and methodologies and learn from each other. The industry needs to create standards that companies can follow, and be held to. We hope this is just the start of more collaboration across the industry.&nbsp;
Whether you&rsquo;re a builder or buyer, we want to enable startups, enterprises, and government agencies to test and evaluate AI for safety and security. Scale&rsquo;s methodology for a robust T&amp;E process and ecosystem spans three key pillars:
<ol>
<li>Hybrid model evaluation &ndash; conducted through a combination of automated evaluations and human expert evaluations to assess model capability and helpfulness over time.</li>
<li>Hybrid red-teaming &ndash; in-depth targeting by human experts of specific harms and techniques where a model may be weak, in order to elicit undesirable behavior. This behavior can then be cataloged, added to an adversarial test set for future tracking, and patched via additional tuning.</li>
<li>T&amp;E ecosystem &ndash;&nbsp;the paradigm by which T&amp;E should be institutionally implemented. We view this as a question of localizing the necessary ecosystem components and interaction dynamics across four institutional stakeholder groups: Frontier model developers; government; enterprises; and third-party organizations</li>
</ol>
Within this approach, we break down what T&amp;E is, how to do it, and how to implement it within the broader ecosystem.&nbsp;
Read our full approach to <a href="https://scale.com/guides/test-and-evaluation-vision" target="_blank" rel="noopener noreferrer">Test &amp; Evaluation here</a>.&nbsp;
<h2>Our Test &amp; Evaluation Offering</h2>
Enabling effective and secure generative AI will require a comprehensive T&amp;E offering, encompassing model evaluation, monitoring, and red teaming, as demonstrated in our approach to Test and Evaluation.&nbsp;
That&rsquo;s why, today, we are excited to be expanding on our critical safety and evaluation work with leading model builders and announcing early access availability for Scale&rsquo;s first T&amp;E offering, which is already being leveraged by OpenAI for red teaming their models.&nbsp;
This offering will be available to select customers who are at the forefront of LLM development, as it is critical we work together to ensure we are keeping the AI community in mind as we build.&nbsp;
Key components of our offering include:&nbsp;
<ol>
<li>Continuous Evaluation - continuously evaluate and monitor the performance of your AI systems</li>
<li>Red Teaming - identify severe risks and vulnerabilities for AI systems</li>
<li>AI System Certification - certify AI applications for safety and capability</li>
</ol>
To learn more, please visit the <a href="https://scale.com/llm-test-evaluation" target="_blank" rel="noopener noreferrer">product page</a>.&nbsp;
Together, we hope to enable safe, secure AI for all.

Scale unveils Test & Evaluation for LLMs, adopted by OpenAI, now in early access for leading LLM developers.

Setting test and evaluation standards | Scale AI

Building Trust in AI: Our vision for test and evaluation

LLM agents guide execution flow, instructing LLMs to format responses as actions, enhancing fixed prompt sequences like chained prompts.

Evaluating Performance of LLM Agents

Chegg, Inc. (NYSE: CHGG), the leading student-first online learning company, announced today its partnership with Scale AI, a leader in delivering high-quality training data for AI applications, to develop a proprietary large language model (LLM) to provide a generative learning experience for Chegg subscribers.
Scale helps customer build <a href="https://scale.com/custom-llms">custom LLMs</a> for their highest priority business use cases. Chegg&rsquo;s LLM will be specifically trained in education, leveraging the data, proprietary content, and the 150,000 subject matter experts that are unique to Chegg, creating a personalized and significantly enhanced learning experience for Chegg Study and Chegg Study Pack subscribers.&nbsp;
&ldquo;This partnership will accelerate our ability to bring generative AI tools&mdash;built for personalized learning&mdash;to all Chegg students beginning now and continuing into 2024,&rdquo; said Dan Rosensweig, President and CEO of Chegg. &ldquo;Scale is a category leader in the AI space, and they have the proven ability to help organizations scale and reach their business objectives. We are excited about the future of this partnership, and how it will impact learners.&rdquo;&nbsp;
Over the past several months, Chegg and Scale tested a successful pilot of the AI experience for students. As part of the partnership, Chegg has named category exclusivity in the educational market. With Chegg&rsquo;s new learning experience, students can access new conversational and personalized learning pathways tailored to their individual learning styles.
&ldquo;The partnership with Scale will allow us to develop new AI and machine learning models unique to Chegg and help us accelerate the annotating and categorizing of data in order to make high-quality datasets and train our AI platforms while maintaining our own IP,&rdquo; said Nathan Schultz, Chief Operating Officer of Chegg.&nbsp;
&nbsp;
Read the <a href="https://www.businesswire.com/news/home/20230807741451/en/Chegg-Partners-with-Scale-AI-to-Enhance-Learning-Experience-for-Students">full press release.</a>
Looking to build your own custom LLM?&nbsp;<a href="https://scale.com/custom-llms">Learn mo</a><a href="https://scale.com/custom-llms">re</a>.

Scale helps customer build custom LLMs for their highest priority business use cases. Chegg’s LLM will be specifically trained in education, leveraging the data, proprietary content, and the 150,000 subject matter experts that are unique to Chegg, creating a personalized and significantly enhanced learning experience for Chegg Study and Chegg Study Pack subscribers. 

Chegg Partners with Scale AI to Build Custom LLMs for an Improved Learning Experience

Chegg Partners with Scale AI to Build Custom LLM for an Improved Learning Experience

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

basecamp-speakers-events

basecamp-agenda

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog