<p>At Scale, we host a diverse set of deep learning models for both internal and external users. The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time, we were able to reduce cost while maintaining a stable latency SLA. We&rsquo;d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with&nbsp;<a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a>.</p>
<h2>Why Cold Start Time Is Important</h2>
<p>Without a fast enough cold start, users will often permanently provision GPUs for peak traffic. GPU hosting costs are significantly more expensive than the average CPU-based microservice. To formalize this idea in equations,</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" title="n_warm_pods.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/153d7eae-b671-4ea1-a33d-fab7dc827ae0" alt="" width="650" height="75"></p>
<p>If we can cold start pods and make predictions within the latency SLA, we won&rsquo;t need any warm pods; otherwise we need to keep some warm pods based on the max amount of traffic we want to keep within SLA and the throughput per node.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" title="throughput_per_node.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/4c30cc1d-9231-4298-9b0f-410d73bc3911" alt="" width="650" height="128"></p>
<p dir="ltr">Then we could calculate compute seconds per wall clock second by dividing the total amount of requests and per pod throughput and plus number of warm pods. Lastly we could get the cost for serving all requests by multiplying compute seconds with the cost per second and duration.</p>
<p dir="ltr">As a concrete example, the chart below illustrates the hosting costs of one of our products as a function of user traffic. The X-axis shows the number of requests and the Y-axis is cost. The blue line is the cost curve for a configuration that has 3x the cold start time than the red line while all other configuration values are the same. When cold start time is high (blue line), we need to keep more warm pods to maintain latency SLA when there are more requests; however if cold start time is short enough (red line), we could spin up pods right when requests come in and only keep a small amount of warm pods for safety. This difference made a huge impact on cost.</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" title="line_chart.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f92e2f99-dcf7-4516-afe6-a1707c2a11db" alt="" width="500" height="326"></p>
<h2>Measurements</h2>
<p>To understand where time is spent during cold start, we measured LLM endpoint cold start time for a Llama 2 7B model by looking at Kubernetes events and duration for each step that runs in the container. Here&rsquo;s the time breakdown:</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" title="Cold_start_time_breakdown.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7f1f16d9-483d-4219-b02b-04b72176513d" alt="" width="700" height="395"></p>
<p>As shown in the chart, the actual model loading time is quite small and most of the time is spent on pulling docker images and downloading model weights.</p>
<h2>Faster Pod Initialization</h2>
<p>Since pulling images from repositories takes the majority of the time, we focused our efforts here first. From our prior experiences working on optimizing inference for various deep learning models like image generation models, we&rsquo;ve already encountered similar problems, but with LLMs the docker images are bigger and the problem is more prominent. We tried a few ideas and eventually optimized away this portion of time by caching images onto the nodes using Kubernetes daemonsets. We utilized the same technique for LLM images. The following chart describes the process:</p>
<p><img style="display: block; margin-left: auto; margin-right: auto;" title="llm_cold_start_flow.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1086257a-01a0-4bbd-ad9e-04f15fcb5879" alt="" width="700" height="519"></p>
<p dir="ltr">A <a href="https://github.com/scaleapi/llm-engine/blob/main/model-engine/model_engine_server/infra/services/image_cache_service.py" target="_blank" rel="noopener">cacher</a> periodically scans through the database to get all &ldquo;<a href="https://github.com/scaleapi/llm-engine/blob/main/clients/python/llmengine/model.py#L51" target="_blank" rel="noopener">high priority</a>&rdquo; models, together with a full set of (GPU, docker image) pairs. We then construct and create/replace one daemonset for each type of GPU, and run each image with sleep commands like <code>/bin/sh -ec 'while : ; do sleep 30 ; done'</code>. This way we could dynamically maintain the set of images and preload them onto the nodes, and effectively eliminate docker image pulling time.</p>
<p dir="ltr">In addition to image caching, we reduced time to provision new nodes by creating&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/charts/model-engine/templates/balloon_a10_deployment.yaml" target="_blank" rel="noopener">balloon deployments</a> per GPU type to prewarm nodes. These pods have low priority and will be preempted when actual workloads are created.</p>
<h2 dir="ltr">Faster Model Weights Loading</h2>
<p dir="ltr">We utilize <a href="https://github.com/peak/s5cmd" target="_blank" rel="noopener">s5cmd</a> to download model weights which is <a href="https://joshua-robinson.medium.com/s5cmd-for-high-performance-object-storage-7071352cc09d" target="_blank" rel="noopener">much faster</a> than aws-cli. We used to put all files into a tarball for simplicity, but found that it&rsquo;s bad for concurrent download. We instead stored model files with 2GB chunks and had s5cmd download all the files in parallel. We also did some quick benchmarks for s5cmd parameters and chose 512 for --numworkers and 10 for --concurrency. With those changes we pushed the download time of the Llama 2 7B model (12.6GB) from 85s to 7s, achieving 14.4Gbps, which is close to the EBS volume bandwidth limit (16Gbps) for our host. Here is an example how we invoke s5cmd:</p>
<pre class="language-python"><code>s5cmd --numworkers 512 cp --concurrency 10 {source_folder} {destination_folder}</code></pre>
<p>We heavily utilize <a href="https://github.com/huggingface/text-generation-inference" target="_blank" rel="noopener">text-generation-inference</a> (TGI) to serve models. TGI prefers <a href="https://github.com/huggingface/safetensors" target="_blank" rel="noopener">safetensors</a>, a file format for storing tensors, and it loads <a href="https://huggingface.co/docs/safetensors/speed" target="_blank" rel="noopener">2 times faster</a> than raw PyTorch bin files. With safetensor files it could load the model in less than 20 seconds. Thus by default now we store only safetensors (with <code>model.save_pretrained(output_dir, safe_serialization=True</code>)) to avoid TGI having to convert PyTorch bin files on the fly.</p>
<h2 dir="ltr">Summary</h2>
<p>With all the optimizations applied we successfully reduced cold start time from more than 6 minutes to less than 40 seconds. This greatly improved our GPU utilization and our ability to deal with spiky traffic. Given the current 30-40s lag, it is now possible to save cost by having zero workers for sporadic workloads.</p>
<h2 dir="ltr">Future Work</h2>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Though this approach conceptually works, we still need to implement (and open source) auto scaling from zero since Kubernetes&rsquo; Horizontal Pod Autoscaler does not scale from zero.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">We fine tune models with PEFT (Parameter-Efficient Fine-Tuning). The code is currently private and we will open source in the near future.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">We currently merge back PEFT trained weights onto the base model. This is cost inefficient since we need to maintain one deployment per fine-tuned model, and the cold start optimizations presented here only partially mitigate this problem. Internally, we have implemented dynamic loading / unloading for <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener">LoRA</a> and <a href="https://arxiv.org/abs/2205.05638" target="_blank" rel="noopener">IA3</a> adapters with DeepSpeed-sharded models. This way only one deployment is needed per base model, and each process would dynamically swap out adapters when serving traffic from different fine-tunes. The future work is to open-source this framework and get adapters working with continuous batching.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">To better generalize our image cache beyond just &ldquo;high-priority&rdquo; endpoints, we are investigating lazy-loading of images with projects like&nbsp;<a href="https://github.com/containerd/stargz-snapshotter/tree/main" target="_blank" rel="noopener">stargz</a>.<strong id="docs-internal-guid-5dc6c64f-7fff-7e26-ccd3-6068424521c4"> </strong></p>
</li>
</ul>
<p dir="ltr">&nbsp;</p>
<h2 dir="ltr">Next Steps</h2>
<p dir="ltr">Looking to customize and serve LLMs? Try&nbsp;<a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> today!&nbsp;</p>
<p>Learn more about how Scale can help you <a href="https://scale.com/custom-llms">customize LLMs for your unique use case</a>.</p>

The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time we were able to reduce cost, due to its significant impact on the number of warm pods if a stable latency SLA needs to be maintained. We’d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with LLM Engine.