experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

Government

Engineering

As policymakers return to Washington from the annual August recess, AI regulation will dominate political discussions. Senate Majority Leader Chuck Schumer has announced multiple &ldquo;AI Insight Forums&rdquo; beginning September 13, the White House has announced plans to issue an Executive Order, and the world is getting ready for a first-of-its-kind global AI Safety Summit in the United Kingdom this November.
Since our earliest days as a company, Scale has worked across industries to help accelerate the development of AI. Today, we work with the leading frontier model developers to fine tune, red team, and test and evaluate their large language models. This work also provides us a unique vantage point to understand, create, and implement industry best practices and understand what we believe is the right balance to strike on a regulatory approach that maximizes innovation, while putting in place the proper guardrails.
In addition to our work with the leading tech and automotive companies, Scale has worked with the U.S. government since 2020, helping to ensure the United States leads the world in AI adoption. The U.S. Department of Defense (DoD) has led the deployment of AI in the federal government to date, and there are no shortage of use cases for every agency to better enhance their day-to-day operations through AI. For example, a large language model at the U.S. Department of Energy would put information at people&rsquo;s fingertips more quickly on complex research topics like grid or hydrogen capacity for electric vehicles in major cities.&nbsp;
Our collective years of working with leading companies and the DoD has uniquely positioned Scale to understand what works and provide recommendations for the federal government to efficiently and fully embrace AI.&nbsp;
This blog is the first in our series titled, &ldquo;Scale AI Policy Framework,&rdquo; and will focus on our recommendations for successful government AI adoption based on our takeaways from our work with the DoD and broader national security community. Scale strongly believes that three things are necessary for the U.S. government to successfully adopt AI:&nbsp;
<ol>
<li>Robust AI-ready data infrastructure at each Agency&nbsp;</li>
<li>Implementation of a risk-based Test and Evaluation framework prior to AI deployment&nbsp;</li>
<li>Innovative and expedited paths to AI acquisition&nbsp;</li>
</ol>
To read the full white paper click <a href="https://pub-9ba39fa3c4764cd0bb3a9579562941d6.r2.dev/gov-ai.pdf" target="_blank" rel="noopener">here.</a>

image.jpg

Our first "Scale AI Policy Framework" blog focuses on successful government AI adoption, drawing from our DoD work.

Scale

Scale AI Framework for Successful Government AI Adoption

OpenAI recently announced the&nbsp;<a href="https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates" target="_blank" rel="noopener">release</a> of their GPT-3.5 Turbo fine-tuning APIs. At Scale we have always believed that building <a href="https://scale.com/custom-llms">custom LLMs</a> through fine-tuning is the key to unlocking greater performance for any given organization&rsquo;s specific use case. OpenAI also named <a href="https://scale.com/blog/open-ai-scale-partnership-gpt-3-5-fine-tuning">Scale as their preferred enterprise fine-tuning partner for GPT-3.5</a>, and we have already shown early progress for the power of fine-tuning GPT-3.5 for customers like Brex. While the pre-trained GPT-3.5 model can often solve tasks with prompt engineering, the model becomes even more powerful when fine-tuned, in some cases surpassing GPT-4 in performance. Let&rsquo;s walk you through how to fine-tune GPT-3.5 in this blog post.
The first and one of the most critical steps in fine-tuning is creating the right training dataset. OpenAI recommends creating a diverse training set of conversations that are similar to the conversations that the model will see in production. These datasets can be created with Scale&rsquo;s <a href="https://scale.com/data-engine">Data Engine</a>, which is trusted by leading ML teams and enterprises to provide large volumes of high-quality data. Scale has worked with OpenAI since 2019 on powering LLMs with better data. Scale's Data Engine has powered most of the leading LLMs, and is now also powering custom LLMs for leading companies like Brex, Chegg, and Accenture. Our cost-effective operations can give you expertly labeled and diverse data at any scale, making it an indispensable asset for fine-tuning.
Fine-tuning also includes reserving the right compute resources and properly implementing the training and inference code. OpenAI&rsquo;s new fine-tuning APIs makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<h1 dir="ltr">Dataset Preparation</h1>
For this example, we will use <a href="https://scienceqa.github.io/" target="_blank" rel="noopener">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. We used <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a> to fine-tune Llama 2 on this dataset in <a href="https://scale.com/blog/fine-tune-llama-2">our previous blogpost</a>, and you can follow the data preparation steps in that post.&nbsp;
Now, let's convert this dataset into OpenAI&rsquo;s supported format, a JSONL file with lists of conversations. Each example in the dataset should be a conversation in the same format as the chat completions endpoint, specifically a list of messages where each message has a role and content (and optionally a name). The provided assistant messages in the data should be the ideal responses you want the model to provide.
<pre class="language-python"><code>def format_chat(row): 
 return json.dumps(
 {"messages": [
 {"role": "user", "content": row["prompt"]},
 {"role": "assistant", "content": str(row["response"])},
 ]}
 )
def convert_dataset(df, file_name):
 df["conversation"] = df.apply(format_chat, axis=1),
 with open(file_name, 'w') as jsonl_file:
 for example in df["conversation"]:
 jsonl_file.write(example + '\n')</code></pre>
This results in our dataset looking like this:
<pre class="language-markup"><code>{"messages:" [{"role": "user", "content": "Context: In a group of cows, ..."}, 
 {"role": "assistant", "content": "B"}]}
["messages": {"role": "user", "content": "Context: In a group of guppies..."}, 
 {"role": "assistant", "content": "A"}]}</code></pre>
The OpenAI API supports training with a training and validation dataset, and provides loss numbers on both during the course of training. These samples will be your initial signal of how much the model is improving, when compared to generations from the base model on the same test conversations.
These dataset files must be uploaded to OpenAI&rsquo;s file endpoint. Make sure to add your OpenAI API key to your system environment variables for authentication.
<pre class="language-python"><code>import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
file = openai.File.create(
 file=open("scienceqa_train.jsonl", "rb"),
 purpose='fine-tune',
)</code></pre>
You should get a response similar to the following:
<pre class="language-markup"><code>{
 "object": "file",
 "id": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "purpose": "fine-tune",
 "filename": "file",
 "bytes": 15974,
 "created_at": 1692743745,
 "status": "uploaded",
 "status_details": null
}</code></pre>
You can then check the status of the uploaded file using the provided file ID.&nbsp;
<pre class="language-markup"><code>openai.File.retrieve("file-CY0FPBluqbVcoHmuGLI80lqx")</code></pre>
You need to make sure that the file status is &ldquo;processed&rdquo; before proceeding with finetuning.
<h1 dir="ltr">Fine-tuning</h1>
Fine-tuning your model requires only one API call. Currently, only GPT-3.5 Turbo is available for fine-tuning. You just need to provide the IDs for the train and validation data files, the model name (only "gpt-3.5-turbo" is supported), and a suffix for the output model name.
<pre class="language-python"><code>train_data = "file-CY0FPBluqbVcoHmuGLI80lqx"
val_data = "file-428ptqQcofNSLoEP0sUDsU5x"
model = openai.FineTuningJob.create(
 model = "gpt-3.5-turbo", 
 training_file = train_data,
 validation_file = val_data,
 suffix = "scienceqa"
)</code></pre>
You would then get a response such as this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": null,
 "fine_tuned_model": null,
 "organization_id": "org-xxx",
 "result_files": [],
 "status": "created",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": null
}</code></pre>
There are several ways to experiment further; for example, increasing the epochs to 4 or 5 could have different results.
You can track the progress of the training job using the finetune ID.
<pre class="language-markup"><code>openai.FineTuningJob.retrieve('ftjob-NmNcHalWN6yGJfGnwealj7MB')</code></pre>
It will have the status &ldquo;running&rdquo; while it is training and &ldquo;succeeded&rdquo; after it is finished or &ldquo;failed&rdquo; if it failed. A successful training job will look like this:
<pre class="language-markup"><code>{
 "object": "fine_tuning.job",
 "id": "ftjob-NmNcHalWN6yGJfGnwealj7MB",
 "model": "gpt-3.5-turbo-0613",
 "created_at": 1692743811,
 "finished_at": 1692744189,
 "fine_tuned_model": "ft:gpt-3.5-turbo-0613:scale-ai:test:7qUNXuwT",
 "organization_id": "org-Vn7k74eir0QoobuD5SrHfwSD",
 "result_files": ["file-hNvzZShh5Phmng1oNb1Qs4sy"],
 "status": "succeeded",
 "validation_file": "file-428ptqQcofNSLoEP0sUDsU5x",
 "training_file": "file-CY0FPBluqbVcoHmuGLI80lqx",
 "hyperparameters": {
 "n_epochs": 3
 },
 "trained_tokens": 9909
}</code></pre>
<h1 dir="ltr">Inference and Evaluation</h1>
Once you are done fine-tuning, you can view your job results. The training job will automatically upload a result file named step_metrics.csv, which contains training and test loss and accuracy for each training step.
We can use the training metrics as a rough sanity check on training stability:
<pre class="language-python"><code>from io import StringIO
def get_step_metrics(file_id):
 content = openai.File.download(file_id)
 eval_result = StringIO(result.decode())
 df = pd.read_csv(eval_result, sep=",")
 return df</code></pre>
However, inspecting the model samples gives the most relevant sense of model quality. We can run inference using our fine-tuned model using their chat completions API.
<pre class="language-python"><code>completion = openai.ChatCompletion.create(
 model="ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 messages=[
 {"role": "user", "content": "Context: A shopper is buying food at..."}
 ]
)</code></pre>
You will get a result like this:
<pre class="language-markup"><code>{
 "id": "chatcmpl-7qUvsuFrButlxKYSs2SispgL5KUrD",
 "object": "chat.completion",
 "created": 1692746316,
 "model": "ft:gpt-3.5-turbo-0613:scale-ai:scienceqa:7qUNXuwT",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "B"
 },
 "finish_reason": "stop"
 }
 ],
 "usage": {
 "prompt_tokens": 81,
 "completion_tokens": 1,
 "total_tokens": 82
 }
}</code></pre>
<h1 dir="ltr">Results</h1>
From this fine-tuning job, the model achieves 84.58% accuracy. This is pretty good! Let's see how this compares with the original base GPT-3.5 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses.
<div dir="ltr" align="left">
<table style="border-collapse: collapse; width: 100%; margin-left: auto; margin-right: auto;" border="1"><colgroup><col><col><col></colgroup>
<tbody>
<tr>
<td>
ScienceQA Validation
</td>
<td>
GPT-3.5
</td>
<td>
GPT-4
</td>
</tr>
<tr>
<td>
Pre-trained (1-shot)
</td>
<td>
72.98%
</td>
<td>
74.8%
</td>
</tr>
<tr>
<td>
Fine-tuned
</td>
<td>
84.58%
</td>
<td>
-
</td>
</tr>
</tbody>
</table>
</div>
&nbsp;
As we can see, fine-tuning makes a big difference in the case of all of the language models. The performance gain from fine-tuning GPT-3.5 Turbo on ScienceQA was an 11.6% absolute difference, even outperforming GPT-4!
We also experimented with different numbers of training examples. OpenAI recommends starting with 50 - 100 examples, but this can vary based on the exact use case. We can roughly estimate the expected quality gain from doubling the training data size by fine-tuning on half or a fourth of your dataset and observing the quality gap. For this experiment, we randomly sampled between 50 to 5000 examples for our training set, and used the same validation set for all models.
<div dir="ltr" align="center">
<table style="border-collapse: collapse; width: 100%; height: 275.157px; border-width: 1px;" border="1"><colgroup><col style="width: 54.321%;" width="235"><col style="width: 45.679%;" width="235"></colgroup>
<tbody>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
# Training samples
</td>
<td style="height: 47.5938px;">
Test accuracy
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
50
</td>
<td style="height: 47.5938px;">
77.18%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
100
</td>
<td style="height: 47.5938px;">
78.23%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
500
</td>
<td style="height: 47.5938px;">
81.10%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
1000
</td>
<td style="height: 47.5938px;">
81.96%
</td>
</tr>
<tr style="height: 47.5938px;">
<td style="height: 47.5938px;">
2500
</td>
<td style="height: 47.5938px;">
84.44%
</td>
</tr>
<tr style="height: 23.5938px;">
<td style="height: 23.5938px;">
5000
</td>
<td style="height: 23.5938px;">
84.58%
</td>
</tr>
</tbody>
</table>
</div>
<img src="https://img.plasmic.app/img-optimizer/v1/img/43f22b3d99c00c724094e06687a2a7fb.png?f=webp&amp;q=75">
 We can see that performance generally increases as the number of training samples increases but plateaus at some point. However, we sampled our dataset randomly, and adding well-crafted training examples that target remaining model issues or edge cases can better improve the model&rsquo;s performance.&nbsp;
Currently, only the number of epochs is available for customization. Other hyperparameters are set to what OpenAI believes to be good default values that should largely work across a large variety of datasets and scenarios.
If you would like higher performance, OpenAI recommends collecting higher quality and/or more data. You may need to consider the balance and diversity of your current training dataset, collect more examples that target edge cases, and ensure there are no lingering issues in the existing examples. The Scale&nbsp;<a href="https://scale.com/data-engine">Data Engine</a> can deliver large volumes of high-quality data that can maximize the performance of your fine-tuned models.
<h1 dir="ltr">Conclusion</h1>
Fine-tuning GPT-3.5 makes a huge difference in unlocking greater performance from base GPT-3.5 models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. If you are looking to optimize your model performance with even more powerful features, Scale also offers an <a href="https://scale.com/custom-llms">enterprise model customization platform</a>. Happy fine-tuning!

OpenAI announced the release of their GPT-3.5 Turbo fine-tuning APIs. 

How to Fine-Tune GPT-3.5 Turbo With OpenAI API

At Scale, we host a diverse set of deep learning models for both internal and external users. The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time, we were able to reduce cost while maintaining a stable latency SLA. We&rsquo;d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with&nbsp;<a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a>.
<h2>Why Cold Start Time Is Important</h2>
Without a fast enough cold start, users will often permanently provision GPUs for peak traffic. GPU hosting costs are significantly more expensive than the average CPU-based microservice. To formalize this idea in equations,
<img style="display: block; margin-left: auto; margin-right: auto;" title="n_warm_pods.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/153d7eae-b671-4ea1-a33d-fab7dc827ae0" alt="" width="650" height="75">
If we can cold start pods and make predictions within the latency SLA, we won&rsquo;t need any warm pods; otherwise we need to keep some warm pods based on the max amount of traffic we want to keep within SLA and the throughput per node.
<img style="display: block; margin-left: auto; margin-right: auto;" title="throughput_per_node.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/4c30cc1d-9231-4298-9b0f-410d73bc3911" alt="" width="650" height="128">
Then we could calculate compute seconds per wall clock second by dividing the total amount of requests and per pod throughput and plus number of warm pods. Lastly we could get the cost for serving all requests by multiplying compute seconds with the cost per second and duration.
As a concrete example, the chart below illustrates the hosting costs of one of our products as a function of user traffic. The X-axis shows the number of requests and the Y-axis is cost. The blue line is the cost curve for a configuration that has 3x the cold start time than the red line while all other configuration values are the same. When cold start time is high (blue line), we need to keep more warm pods to maintain latency SLA when there are more requests; however if cold start time is short enough (red line), we could spin up pods right when requests come in and only keep a small amount of warm pods for safety. This difference made a huge impact on cost.
<img style="display: block; margin-left: auto; margin-right: auto;" title="line_chart.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f92e2f99-dcf7-4516-afe6-a1707c2a11db" alt="" width="500" height="326">
<h2>Measurements</h2>
To understand where time is spent during cold start, we measured LLM endpoint cold start time for a Llama 2 7B model by looking at Kubernetes events and duration for each step that runs in the container. Here&rsquo;s the time breakdown:
<img style="display: block; margin-left: auto; margin-right: auto;" title="Cold_start_time_breakdown.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7f1f16d9-483d-4219-b02b-04b72176513d" alt="" width="700" height="395">
As shown in the chart, the actual model loading time is quite small and most of the time is spent on pulling docker images and downloading model weights.
<h2>Faster Pod Initialization</h2>
Since pulling images from repositories takes the majority of the time, we focused our efforts here first. From our prior experiences working on optimizing inference for various deep learning models like image generation models, we&rsquo;ve already encountered similar problems, but with LLMs the docker images are bigger and the problem is more prominent. We tried a few ideas and eventually optimized away this portion of time by caching images onto the nodes using Kubernetes daemonsets. We utilized the same technique for LLM images. The following chart describes the process:
<img style="display: block; margin-left: auto; margin-right: auto;" title="llm_cold_start_flow.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1086257a-01a0-4bbd-ad9e-04f15fcb5879" alt="" width="700" height="519">
A <a href="https://github.com/scaleapi/llm-engine/blob/main/model-engine/model_engine_server/infra/services/image_cache_service.py" target="_blank" rel="noopener">cacher</a> periodically scans through the database to get all &ldquo;<a href="https://github.com/scaleapi/llm-engine/blob/main/clients/python/llmengine/model.py#L51" target="_blank" rel="noopener">high priority</a>&rdquo; models, together with a full set of (GPU, docker image) pairs. We then construct and create/replace one daemonset for each type of GPU, and run each image with sleep commands like <code>/bin/sh -ec 'while : ; do sleep 30 ; done'</code>. This way we could dynamically maintain the set of images and preload them onto the nodes, and effectively eliminate docker image pulling time.
In addition to image caching, we reduced time to provision new nodes by creating&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/charts/model-engine/templates/balloon_a10_deployment.yaml" target="_blank" rel="noopener">balloon deployments</a> per GPU type to prewarm nodes. These pods have low priority and will be preempted when actual workloads are created.
<h2 dir="ltr">Faster Model Weights Loading</h2>
We utilize <a href="https://github.com/peak/s5cmd" target="_blank" rel="noopener">s5cmd</a> to download model weights which is <a href="https://joshua-robinson.medium.com/s5cmd-for-high-performance-object-storage-7071352cc09d" target="_blank" rel="noopener">much faster</a> than aws-cli. We used to put all files into a tarball for simplicity, but found that it&rsquo;s bad for concurrent download. We instead stored model files with 2GB chunks and had s5cmd download all the files in parallel. We also did some quick benchmarks for s5cmd parameters and chose 512 for --numworkers and 10 for --concurrency. With those changes we pushed the download time of the Llama 2 7B model (12.6GB) from 85s to 7s, achieving 14.4Gbps, which is close to the EBS volume bandwidth limit (16Gbps) for our host. Here is an example how we invoke s5cmd:
<pre class="language-python"><code>s5cmd --numworkers 512 cp --concurrency 10 {source_folder} {destination_folder}</code></pre>
We heavily utilize <a href="https://github.com/huggingface/text-generation-inference" target="_blank" rel="noopener">text-generation-inference</a> (TGI) to serve models. TGI prefers <a href="https://github.com/huggingface/safetensors" target="_blank" rel="noopener">safetensors</a>, a file format for storing tensors, and it loads <a href="https://huggingface.co/docs/safetensors/speed" target="_blank" rel="noopener">2 times faster</a> than raw PyTorch bin files. With safetensor files it could load the model in less than 20 seconds. Thus by default now we store only safetensors (with <code>model.save_pretrained(output_dir, safe_serialization=True</code>)) to avoid TGI having to convert PyTorch bin files on the fly.
<h2 dir="ltr">Summary</h2>
With all the optimizations applied we successfully reduced cold start time from more than 6 minutes to less than 40 seconds. This greatly improved our GPU utilization and our ability to deal with spiky traffic. Given the current 30-40s lag, it is now possible to save cost by having zero workers for sporadic workloads.
<h2 dir="ltr">Future Work</h2>
<ul>
<li dir="ltr" aria-level="1">
Though this approach conceptually works, we still need to implement (and open source) auto scaling from zero since Kubernetes&rsquo; Horizontal Pod Autoscaler does not scale from zero.
</li>
<li dir="ltr" aria-level="1">
We fine tune models with PEFT (Parameter-Efficient Fine-Tuning). The code is currently private and we will open source in the near future.
</li>
<li dir="ltr" aria-level="1">
We currently merge back PEFT trained weights onto the base model. This is cost inefficient since we need to maintain one deployment per fine-tuned model, and the cold start optimizations presented here only partially mitigate this problem. Internally, we have implemented dynamic loading / unloading for <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener">LoRA</a> and <a href="https://arxiv.org/abs/2205.05638" target="_blank" rel="noopener">IA3</a> adapters with DeepSpeed-sharded models. This way only one deployment is needed per base model, and each process would dynamically swap out adapters when serving traffic from different fine-tunes. The future work is to open-source this framework and get adapters working with continuous batching.
</li>
<li dir="ltr" aria-level="1">
To better generalize our image cache beyond just &ldquo;high-priority&rdquo; endpoints, we are investigating lazy-loading of images with projects like&nbsp;<a href="https://github.com/containerd/stargz-snapshotter/tree/main" target="_blank" rel="noopener">stargz</a>. 
</li>
</ul>
&nbsp;
<h2 dir="ltr">Next Steps</h2>
Looking to customize and serve LLMs? Try&nbsp;<a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> today!&nbsp;
Learn more about how Scale can help you <a href="https://scale.com/custom-llms">customize LLMs for your unique use case</a>.

The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time we were able to reduce cost, due to its significant impact on the number of warm pods if a stable latency SLA needs to be maintained. We’d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with LLM Engine.

How To Reduce Cold Start Times For LLM Inference

Today, we are excited to announce a strategic partnership with OpenAI, providing GPT-3.5 fine-tuning for the world&rsquo;s leading enterprises. At Scale, we believe that <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">fine-tuning</a> is the key to unlocking the performance of LLMs for any organization's most critical use cases - and we have seen both great excitement and demand for this fine-tuning capability from the market. Our partnership brings together OpenAI's advanced base model GPT-3.5 with Scale's fine-tuning expertise and industry-leading <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Data Engine</a> to help every company create custom state-of-the-art models for their specific business needs. Scale already performs fine-tuning for many commercial and open-source models. As OpenAI&rsquo;s preferred fine-tuning partner for GPT-3.5, we are excited to leverage their powerful APIs to help even more enterprises build the most powerful custom LLMs that increase efficiency while reducing costs.
&nbsp;
<h2>Fine-Tuning GPT-3.5&nbsp;</h2>
&nbsp;
Fine-tuning GPT-3.5 has already yielded performance improvements for companies like Brex. As a preferred partner to OpenAI, we were granted early access to GPT-3.5&rsquo;s fine-tuning APIs. By leveraging these APIs in combination with Scale&rsquo;s enterprise <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">Custom LLM</a> platform and high-quality data from <a href="https://www.scale.com/data-engine" target="_blank" rel="noopener noreferrer">Scale&rsquo;s Data Engine</a> we delivered initial performance improvements for Brex.
&nbsp;
"To get the most value out of our models, companies are looking to use their data to create tailored services and solutions,&rdquo; said <a href="https://openai.com/blog/openai-partners-with-scale-to-provide-support-for-enterprises-fine-tuning-models" target="_blank" rel="noopener noreferrer">Brad Lightcap, COO, OpenAI</a>. &ldquo;Scale extends our ability to bring the power of fine-tuning to more companies, building on their enterprise AI experience to help businesses better apply OpenAI models for their unique needs.&rdquo;
&nbsp;
Brex has been leveraging LLMs to generate high-quality expense memos that help ease the burden of compliance requirements for employees. Automating the generation of expense memos helps save significant amounts of employee time within the expense workflow and enables companies to close their books faster. The Brex team had previously been using GPT-4 for memo generation, but wanted to explore if they could improve cost and latency, while maintaining quality, by using a fine-tuned GPT-3.5 model.
&nbsp;
By using the GPT-3.5 fine-tuning API on Brex data annotated with Scale&rsquo;s <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Data Engine</a>, we saw that the fine-tuned GPT-3.5 model outperformed the stock GPT-3.5 turbo model 66% of the time.
&nbsp;
"At Brex, we&rsquo;ve been investing heavily in AI which is transforming how we offer financial services to customers. Our ongoing partnership with OpenAI and Scale AI position us at the cutting edge by employing state-of-the-art techniques to enhance employee compliance and help finance teams close the books faster. In particular, fine-tuning GPT-3.5 has been a game changer for us, enabling us to deliver high-quality AI experiences, comparable to GPT-4, with much lower cost and lower latency. This unlocks a whole new set of capabilities for us that were previously not viable." - Henrique Dubugras, CEO, Brex
&nbsp;
<h2>How to Get Started</h2>
&nbsp;
Our pilot project with Brex demonstrates the importance of fine-tuning LLMs to deliver outcomes.&nbsp;
&nbsp;
&ldquo;We are excited to partner with OpenAI to supercharge model performance - helping every enterprise utilize AI most effectively for their unique needs. Prompting alone&mdash;atop even the best LLMs like GPT-3.5 &mdash; is not enough model customization to produce the most accurate, efficient results. As with software, an incredible amount of value comes from fine-grained optimizations, and fine tuning is critical for that.&rdquo; - Alexandr Wang, Founder and CEO, Scale AI
&nbsp;
Scale starts by preparing and enhancing your data with the <a href="https://scale.com/data-engine" target="_blank" rel="noopener noreferrer">Scale Data Engine</a>. Then, we fine-tune GPT-3.5 with your data and further <a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">customize models</a> with plugins and retrieval augmented generation, or the ability to reference and cite your proprietary documents in its responses. We then leverage our&nbsp;<a href="https://scale.com/llm-test-evaluation" target="_blank" rel="noopener noreferrer">Test and Evaluation platform</a> and trained domain experts to ensure that the model exceeds your performance expectations and safety requirements.
&nbsp;
Learn more about how Scale can help you&nbsp;<a href="https://scale.com/custom-llms" target="_blank" rel="noopener noreferrer">customize LLMs for your unique use case</a>. 
&nbsp;
&nbsp;

cover_openai.png

Exciting news! Scale partners with OpenAI, offering GPT-3.5 fine-tuning for enterprise clients. Scale's expertise + OpenAI's model = robust solutions.

OpenAI Names Scale as Preferred Partner to Fine-Tune GPT-3.5

Today, Scale is announcing our Test &amp; Evaluation offering for LLMs, which is already being leveraged by OpenAI, and is available in early access to select customers who are at the forefront of LLM development.
Since large language models broke into the public consciousness, the conversation has largely fallen into two categories: The exciting realm of new possibilities this technology can unlock, and the risks that could come from opening Pandora&rsquo;s box.&nbsp;
While many would like to paint these as two opposite ends of the spectrum, the reality is that progress in frontier model capabilities must happen alongside progress in model evaluation and safety. This is not only a moral imperative, but a practical one. In order to advance our knowledge and understanding of LLMs &ndash; and therefore advance their capabilities &ndash; we must understand their weaknesses as much as their strengths.&nbsp;
Various studies have found popular LLMs hallucinate or provide inaccurate information ranging anywhere from 20% to 52% of the time. As AI is increasingly being leveraged by <a href="https://www.nytimes.com/2023/07/19/business/google-artificial-intelligence-news-articles.html" target="_blank" rel="noopener noreferrer">newsrooms</a>, misinformation has the potential to sway global markets and global politics. Other issues like unqualified advice have the potential to lead to real-world harms, such as providing <a href="https://mwi.westpoint.edu/disinformation-in-the-age-of-chatgpt/#:~:text=A%20study%20that%20enticed%20ChatGPT,war%20in%20Ukraine%2C%20and%20school" target="_blank" rel="noopener noreferrer">false medical information</a>, feeding into <a href="https://www.washingtonpost.com/technology/2023/08/07/ai-eating-disorders-thinspo-anorexia-bulimia/" target="_blank" rel="noopener noreferrer">mental disorders</a>, and more. LLMs can even be tricked into writing <a href="https://www.axios.com/2023/08/08/ibm-researchers-trick-chatgpt-hacking" target="_blank" rel="noopener noreferrer">malicious code</a>, opening new vectors for cybersecurity attacks.&nbsp;
For AI to reach its full potential, it&rsquo;s critical the AI industry is working to address these known challenges in parallel with other advancements to their models.&nbsp;
<h2>Our Approach to Test &amp; Evaluation</h2>
At Scale, for more than 7 years we have built up deep expertise in the entire lifecycle of developing, fine-tuning, testing and evaluating AI models. Our role as the data partner for companies like OpenAI, Meta, Microsoft, and Anthropic have given us a purview across the ecosystem that no other company has. It has allowed us to deeply understand the common challenges that must be addressed, how various players across the industry are working to address them, and ultimately develop our own technology to effectively test and evaluate LLMs.&nbsp;
Today, there is no standard for test and evaluation (T&amp;E) in AI. Our leadership in this space is why in May, the&nbsp;<a href="https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/" target="_blank" rel="noopener noreferrer">White House announced</a> Scale was selected to develop the T&amp;E platform for DEF CON&rsquo;s public evaluation of models built by Anthropic, Cohere, Google, Hugging Face, Meta, NVIDIA, OpenAI, and Stability.&nbsp;
We&rsquo;re excited to see the industry come together for this moment, but this is just one step in the right direction. That&rsquo;s why today, we are releasing our stance on Test &amp; Evaluation to help set the standard for what an effective T&amp;E approach looks like. Like other realms of cybersecurity, we need to share best practices and methodologies and learn from each other. The industry needs to create standards that companies can follow, and be held to. We hope this is just the start of more collaboration across the industry.&nbsp;
Whether you&rsquo;re a builder or buyer, we want to enable startups, enterprises, and government agencies to test and evaluate AI for safety and security. Scale&rsquo;s methodology for a robust T&amp;E process and ecosystem spans three key pillars:
<ol>
<li>Hybrid model evaluation &ndash; conducted through a combination of automated evaluations and human expert evaluations to assess model capability and helpfulness over time.</li>
<li>Hybrid red-teaming &ndash; in-depth targeting by human experts of specific harms and techniques where a model may be weak, in order to elicit undesirable behavior. This behavior can then be cataloged, added to an adversarial test set for future tracking, and patched via additional tuning.</li>
<li>T&amp;E ecosystem &ndash;&nbsp;the paradigm by which T&amp;E should be institutionally implemented. We view this as a question of localizing the necessary ecosystem components and interaction dynamics across four institutional stakeholder groups: Frontier model developers; government; enterprises; and third-party organizations</li>
</ol>
Within this approach, we break down what T&amp;E is, how to do it, and how to implement it within the broader ecosystem.&nbsp;
Read our full approach to <a href="https://scale.com/guides/test-and-evaluation-vision" target="_blank" rel="noopener noreferrer">Test &amp; Evaluation here</a>.&nbsp;
<h2>Our Test &amp; Evaluation Offering</h2>
Enabling effective and secure generative AI will require a comprehensive T&amp;E offering, encompassing model evaluation, monitoring, and red teaming, as demonstrated in our approach to Test and Evaluation.&nbsp;
That&rsquo;s why, today, we are excited to be expanding on our critical safety and evaluation work with leading model builders and announcing early access availability for Scale&rsquo;s first T&amp;E offering, which is already being leveraged by OpenAI for red teaming their models.&nbsp;
This offering will be available to select customers who are at the forefront of LLM development, as it is critical we work together to ensure we are keeping the AI community in mind as we build.&nbsp;
Key components of our offering include:&nbsp;
<ol>
<li>Continuous Evaluation - continuously evaluate and monitor the performance of your AI systems</li>
<li>Red Teaming - identify severe risks and vulnerabilities for AI systems</li>
<li>AI System Certification - certify AI applications for safety and capability</li>
</ol>
To learn more, please visit the <a href="https://scale.com/llm-test-evaluation" target="_blank" rel="noopener noreferrer">product page</a>.&nbsp;
Together, we hope to enable safe, secure AI for all.

Scale unveils Test & Evaluation for LLMs, adopted by OpenAI, now in early access for leading LLM developers.

Setting test and evaluation standards | Scale AI

Building Trust in AI: Our vision for test and evaluation

LLM agents guide execution flow, instructing LLMs to format responses as actions, enhancing fixed prompt sequences like chained prompts.

Evaluating Performance of LLM Agents

Chegg, Inc. (NYSE: CHGG), the leading student-first online learning company, announced today its partnership with Scale AI, a leader in delivering high-quality training data for AI applications, to develop a proprietary large language model (LLM) to provide a generative learning experience for Chegg subscribers.
Scale helps customer build <a href="https://scale.com/custom-llms">custom LLMs</a> for their highest priority business use cases. Chegg&rsquo;s LLM will be specifically trained in education, leveraging the data, proprietary content, and the 150,000 subject matter experts that are unique to Chegg, creating a personalized and significantly enhanced learning experience for Chegg Study and Chegg Study Pack subscribers.&nbsp;
&ldquo;This partnership will accelerate our ability to bring generative AI tools&mdash;built for personalized learning&mdash;to all Chegg students beginning now and continuing into 2024,&rdquo; said Dan Rosensweig, President and CEO of Chegg. &ldquo;Scale is a category leader in the AI space, and they have the proven ability to help organizations scale and reach their business objectives. We are excited about the future of this partnership, and how it will impact learners.&rdquo;&nbsp;
Over the past several months, Chegg and Scale tested a successful pilot of the AI experience for students. As part of the partnership, Chegg has named category exclusivity in the educational market. With Chegg&rsquo;s new learning experience, students can access new conversational and personalized learning pathways tailored to their individual learning styles.
&ldquo;The partnership with Scale will allow us to develop new AI and machine learning models unique to Chegg and help us accelerate the annotating and categorizing of data in order to make high-quality datasets and train our AI platforms while maintaining our own IP,&rdquo; said Nathan Schultz, Chief Operating Officer of Chegg.&nbsp;
&nbsp;
Read the <a href="https://www.businesswire.com/news/home/20230807741451/en/Chegg-Partners-with-Scale-AI-to-Enhance-Learning-Experience-for-Students">full press release.</a>
Looking to build your own custom LLM?&nbsp;<a href="https://scale.com/custom-llms">Learn mo</a><a href="https://scale.com/custom-llms">re</a>.

Scale helps customer build custom LLMs for their highest priority business use cases. Chegg’s LLM will be specifically trained in education, leveraging the data, proprietary content, and the 150,000 subject matter experts that are unique to Chegg, creating a personalized and significantly enhanced learning experience for Chegg Study and Chegg Study Pack subscribers. 

Chegg Partners with Scale AI to Build Custom LLMs for an Improved Learning Experience

Chegg Partners with Scale AI to Build Custom LLM for an Improved Learning Experience

Scale, the data infrastructure company for AI, is excited to announce that the Department of Defense (DoD) Chief Digital and Artificial Intelligence Office (CDAO) awarded us a new Other Transaction Agreement (OTA), a type of contract that will boost the DoD&rsquo;s efforts to advance artificial intelligence (AI) capabilities for the military.&nbsp;
This agreement with the CDAO will enable Scale to provide the Joint Force with data curation and annotation capabilities via Scale&rsquo;s <a href="https://scale.com/data-engine">Data Engine</a>. The capabilities will be made available across a diverse range of networks and classification levels relevant to warfighters.&nbsp;
Our partnership with the CDAO aims to build the robust AI infrastructure that is necessary to move AI pilots and prototypes out of labs and experimental environments into production for front-line teams. Moreover, the contract will help meet Combatant Commands&rsquo; crucial requirements for enhanced data to power machine learning algorithms. Notably, this capability will play a pivotal role in supporting multimodal data pipelines, such as full motion video, synthetic-aperture radar, and electro-optical imagery. We are excited to bring this capability to the Department of Defense and deploy Scale&rsquo;s suite of tools to both unclassified and classified workflows spanning projects across the Army, Marine Corps, Navy, Air Force, Space Force and Coast Guard.
Scale received the award via the CDAO&rsquo;s Tradewind Solutions Marketplace. This innovative digital repository of post-competition, readily-awardable solutions addresses the Government&rsquo;s greatest challenges in the AI/ML, digital, and data analytics space. Scale commends the CDAO and Tradewind for its partnership with commercial technology leaders to help the DoD achieve AI overmatch. At Scale, we are honored to be entrusted with this new agreement and look forward to contributing our expertise to the development of advanced AI capabilities for the DoD.
Learn more about Scale&rsquo;s Data Engine at <a href="https://scale.com/data-engine">https://scale.com/data-engine</a>, and verified .gov and .mil users can sign up for a 7-day trial of Donovan, our AI-powered decision-making platform here: <a href="https://scale.com/donovan-trial">https://scale.com/donovan-trial</a>
&nbsp;

Scale, the data infrastructure for AI, is excited to announce that the Department of Defense (DoD) Chief Digital and AI Office (CDAO) awarded us a new Other Transaction Agreement (OTA), a type of contract that will boost the DoD’s efforts to advance artificial intelligence (AI) capabilities for the military. 

Scale AI Awarded DoD Contract to Deliver Data Curation and Annotation Capabilities to Power AI Across the Joint Force

On July 15th, Scale AI hosted its second Generative AI Hackathon, bringing together talented developers, data scientists, and AI enthusiasts from around the world. The hackathon fostered creative projects, featured cute puppies, and showcased the potential of Generative AI in solving real-world challenges. Hackers created projects that leveraged the power of AI to address various domains, from code generation and bioinformatics to natural language processing and immersive experiences.​​<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/EroCZrhWWegZRf0lGRpNgoFXHyzf6LRth1PfBNUntmQHLRAe1aoLRmCC-l1DLLrA9diZC7R7vnMfjz9fwF-SroDQD4Q5L0FdGB1RzZU-4uN_5Xvy6RixAwof1KY8UPAagdu-TxYDq0XfskoedOmQffY" width="624" height="468">
<h2 dir="ltr">Winning Projects</h2>
There were a multitude of remarkable projects but the standout projects from the hackathon were:
<h3 dir="ltr">First place: YOBO (You Only Build Once)</h3>
Winners: Baladhurgesh Paramasivan, Joshua Dai, Shreyas Agarwal, and Raul Molina
YOBO, or You Only Build Once, took first place. YOBO presented an online tool that streamlines the process of converting ideas into reality. The tool requires no setup and offers a visually intuitive depiction of the code architecture, making it easy to comprehend for users. Furthermore, YOBO generates initial code snippets, enabling developers to kickstart their projects efficiently. The tool can be explored&nbsp;<a href="https://yobo-scale.onrender.com/">here</a>.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/72229cc7-f5f9-42e8-a320-941e480698c0" width="658" height="323">
<h3 dir="ltr">Second place, Best use of Chroma: Protex</h3>
Winners: Shawn Dimantha, Alex Beatson, Rohit Dilip, and Ishaan Javali
Protex, took second place as well as the best use of Chroma award. Protex wowed the judges with its search engine for proteins. Researchers could describe the protein they sought, obtain CLIP-like embeddings, and locate the closest matches. This breakthrough technology can significantly reduce the time and effort required in pharmaceutical research, potentially leading to groundbreaking medical discoveries.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/mzrSe1-PIY12igOJpomB9Y1cZTp7Pa2lg3nTrxuv7obUk2fGk09YMHSbxmpCBZlASN_PJylaIpDNHeQkELp6fIqX4zbtzdhszKUGXkU4M0BotzmHXQSmfKFZHOWviYK9oiz4Mp3Mktf7U9M7ut4mCp4" width="624" height="832">
<h3 dir="ltr">Third place, Best use of Modal: Kaboom</h3>
Winners: Divya Amirtharaj, Chloe Loughridge, Jothi Ramaswamy, Moritz Stephan, and Joshua Fang
Kaboom unleashed creativity by turning direct messages (DMs) into engaging comic strips and won third place and the Best Use of Modal award. Using Generative AI, this project transformed ordinary conversations into visually captivating narratives, offering an innovative way to communicate and share stories.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/M1CH135Q9GQxotvgzINIHZUxg77BTdwvzDl4dWy4sIHQr0bAGJlKo0AQRkhZI_CS8fcbT4s4T1LYbxkIw4RuPUzKJX0qWWtDUJ8YEUbab-5GOSlpq1Xcp6aILiR83Fnaba7j5lDdP_6BrjY9Lx1R5uE" width="624" height="416">
<h3 dir="ltr">​​Best Use of OpenAI and Audience Choice Award: Enable</h3>
Winners: Jayaditya Sethi and Ethan Jagoda
The Best Use of OpenAI prize was awarded to Enable. Enable focused on transforming natural language requests into fully functional, modular prosthetic devices to enhance accessibility and independence. The seamless integration of OpenAI's technology into this project showcased the potential of AI to create positive and transformative solutions for real-world problems. Enable also won the Audience Choice award.
<img src="https://lh4.googleusercontent.com/CGazcH_plwFscABEWZqvWR2X-koQCzwidO99OVmiosAE03-H4xjjSIXrrKmdoIvrY_YSQNATfn6-R4ikL1BiNnr2JaVN8vVxfEP4ZFs_diFq8QKj-xc692_xSO6omKJfCRH0wwBwkLy2JVkvdBEI0_A" width="624" height="367">
<img src="https://lh5.googleusercontent.com/OFw1nT-WYJTlF9RKvtfMcQxZsn6CtRp81vVpGx4FRkA3ljmH74wYCE38ol4ZOsUeR6UBWWBDLGrwmrLmsDeIPdpSdNvmx9PLje8BSPuXo44H9hOjeM5Ivd39IVKCMWNa1Y4SAw0yn6taFfOOtd_FvrY" width="624" height="416">
<h3 dir="ltr">Best Productivity Hack: Bob the Cloud Builder</h3>
Winners: Mihir Chintawar and Kiran Illindala&nbsp;
The Best Productivity Hack prize was awarded to Bob the Cloud Builder. This project revolved around Agentic Cloud deployment, where the system learns common patterns and combines them to optimize cloud-based operations. This creative use of generative AI opens up new possibilities for streamlining cloud deployment processes, saving time, and boosting overall productivity. Bob the Cloud Builder's impressive implementation stood out for its practicality and efficiency, making it a well-deserved winner in the hackathon.
<h3 dir="ltr">Best Frontend (Vercel): Agent83</h3>
Winners: Kai Hayden, Ryan Xu, and Shahan Nedadahandeh
Agent83 won the Best Frontend prize sponsored by Vercel. Agent83 is an AI-powered search engine that utilizes natural language queries to perform keyword-based searches on an LLM-generated search index. The intuitive frontend developed on Vercel showcased the team's design prowess, making the user experience engaging and effortless.&nbsp;
<h3 dir="ltr">Best Use of Spellbook: Dream with Vision Pro</h3>
Winners: Molly Cantillon, Sigil Wen, Rahul Chalamala, and Jonathan Lei
Dream with Vision Pro secured the Best Use of Spellbook prize. Dream with Vision Pro was an inspiring lucid text-to-3D tool built with the Apple VisionOS SDK. Harnessing the capabilities of multiple AI technologies, including Scale&rsquo;s Spellbook platform, OpenAI's GPT-4, and Shap-E, Modal, Replicate, and the Meta Quest 2, this project allows users to transform their imaginations into stunning immersive experiences. The clever integration of Spellbook demonstrated how AI could be leveraged to unlock new levels of creativity and enable users to bring their dreams to life in vibrant 3D environments.
<h2 dir="ltr">Prompt Engineering Part 2 with Riley Goodside</h2>
Riley Goodside, Scale&rsquo;s Staff Prompt Engineer and a leading figure in the AI and prompt engineering domain delivered an insightful session on Prompt Engineering. His presentation dove deep into advanced techniques to fine-tune AI models effectively. Attendees learned about cutting-edge strategies for prompt design and optimization, gaining valuable insights that they could apply to their own AI projects.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/ll0dNu6l8oZ2IV1uivcfoKPJjpG2YX05cmR5-L7k2XlgU6K5PiIBzdebWs5N9FQJOGa_xBnZhG8gSGKldZd8YFvCnSbPv0fsPegNsGO75GamyNCnVFEzR442bzdDg4Irj2M_ia61WzSPTaLN75vNMt4" width="624" height="416">
<h2 dir="ltr">Conclusion</h2>
Our Second Generative AI Hackathon showcased the potential impact of Generative AI across diverse fields, including prosthetics, biotech, and infrastructure. The winners, YOBO, Protex, Kaboom, and other remarkable projects, demonstrated the power of AI-driven hacks in simplifying tasks, accelerating research, and creating captivating experiences. As the AI landscape continues to evolve, Scale will continue to host events like this to bring together builders and inspire the next generation of AI enthusiasts to push the boundaries of what is possible. The Hackathon was generously sponsored by our partners Index Ventures, Chroma, Vercel, Modal, and Cometeer. Congratulations to all the participants, winners, volunteers, and sponsors for making the Second Generative AI Hackathon unforgettable!&nbsp;

In July, Scale AI hosted its second Generative AI Hackathon. This blog provides an overview of winning projects and a recap of the event. 

Scale's Second Generative AI Hackathon | Blog

Scale's Second Generative AI Hackathon

Scale is thrilled to announce that we are expanding our operations to the historic and incredibly vibrant city of Budapest. This expansion brings with it an array of exciting opportunities for professionals looking to accelerate the development of AI applications. With the addition of our Budapest office, Scale joins an up-and-coming community of technology companies and startups in Hungary.
Scale&rsquo;s Engineering arm has long been a leader in combining the power of AI and Machine Learning with technical, engineering expertise. As a full-stack solutions provider and technical expert, Scale&rsquo;s team works with companies like OpenAI, Microsoft, and Meta.&nbsp;
My experiences and professional journey in Budapest started nearly 10 years ago. In that specific instance of my many past lives, we acquired a small 10 person company and, over the course of the following 5 years, scaled the team to over 300 people. Much like the Erzsebet Bridge over the Danube, it helped me bridge over to an incredible country and wicked smart folks - many of whom I&rsquo;m proud to count as friends! I am very excited to be getting the opportunity to work with the incredible talent at Budapest through Scale AI again!
Russell Kaplan, Engineering Director at Scale, is also excited to tap into the wealth of talent in Budapest, and to continue to expand&nbsp; Scale&rsquo;s presence in Europe: &ldquo;Scale began its European expansion when we acquired <a href="https://scale.com/blog/siasearch-acquisition">SiaSearch</a>, a leading data management platform. This acquisition helped broaden our footprint in European markets and deepen the technical expertise within our company.&rdquo;
Scale&rsquo;s Engineering team is comprised of experts working on the most advanced and pressing technologies of our time. Beyond the immense scope of hard work, Scale&rsquo;s engineering culture and reputation precedes it! As the host of Generative AI Hackathons and other AI/ML community events, Scale plans to bring these events to our new European office.&nbsp;
Engineering Manager and Berlin Office Lead, Mark Pfeiffer says, "As a part of Scale AI post the acquisition of SiaSearch, I've witnessed a remarkable commitment to fostering talent in Berlin. Our growth in technical expertise and ownership over key products makes it an incredibly exciting place to work right now! I'm truly excited to take our growth in Europe to the next level and further grow the team in Budapest!"
During the first week of August, Bay-Area based Scaliens will be in Budapest to meet the team on the ground there, and to host recruiting events. If you are interested in joining an event, I will be hosting a Tech Talk, followed by the opportunity to network and exchange ideas on continued advancements and the future of AI.
Scale Software Engineering Manager, Gianluca Bobbio, says of his work, and of the engineering culture: &ldquo;At Scale, you'll find an environment that pushes boundaries, embodies the entrepreneurial spirit, yet offers the stability of an AI industry giant. We're not just following AI trends, we're setting them together. Join us on this journey to help shape the AI landscape.&rdquo;
Scale is in an exciting place where growth opportunities in the AI domain organically grow your career, too. Working at Scale allows members of every team to work with high visibility customers and contribute to solving world-wide problems. Beyond our Berlin and Budapest offices, Scale is expanding across Europe. Check out&nbsp;<a href="http://scale.com/careers">Scale.com/careers</a> for more information about roles currently hiring.

Scale joins an up-and-coming community of technology companies and startups in Hungary

Scale’s New Hub

Scale’s New Hub: Budapest

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

basecamp-speakers-events

basecamp-agenda

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog