image.jpg

Yunfeng Bai

Software Engineer

In July 2023 we <a href="https://scale.com/blog/custom-llms">announced</a> the launch of <a href="https://llm-engine.scale.com">LLM Engine</a>, our open-source platform for customizing and serving large language models (LLMs). We decided to open-source this platform to give all enterprises the opportunity to accelerate their ability to host and fine-tune LLMs.
For customers with larger-scale hosting needs and more powerful fine-tuning features, we also built an advanced version of LLM Engine that is available on <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>. Through deploying customized LLMs for customers we have found that the details of hosting LLMs are still obscure, so we wanted to shine a light on how this works and how LLM Engine is architected to optimize each step of the process.
And we are excited to announce that coming soon, <a href="https://scale.com/genai-platform">Scale GenAI Platform</a> customers can leverage Nvidia NIM microservices through LLM Engine for optimized LLM inference. Nvidia NIM microservices provide flexibility and performance for running customized generative AI in production.
In general, deploying LLMs to the cloud is an expensive operation that requires experience in both cloud infrastructure and machine learning. While great technologies exist for general service orchestration (e.g kubernetes), there are some distinct frictions with serving deep-learning workloads:
<ul>
<li dir="ltr" aria-level="1">
Effective GPU utilization - It is well known in the industry that there is a fierce GPU supply crunch. Squeezing as many queries per second (QPS) as possible out of your GPU is essential for managing hosting costs for models.
</li>
<li dir="ltr" aria-level="1">
MLE-friendly APIs - Given the incredible rate of ML advancement, ML engineers/researchers should focus on ML models. We do not want to waste precious engineering cycles on scaling models to a high QP and large-scale cloud environment. These activities should be built into a platform and easily automated with minimal effort from ML engineers.
</li>
<li dir="ltr" aria-level="1">
Experimentation speed - In most eng orgs, teams typically only maintain a few services that serve business functions from their team. In ML orgs, a single ML engineer often rapidly deploys several models as they experiment or as they deliver models from project to project. An ML Engineer's (MLE's) overall productivity should not slow down, even when the number of endpoints they manage grows.
</li>
<li dir="ltr" aria-level="1">
Efficient auto-scaling - While QPS is a good metric for individual GPU nodes, ML teams and production applications also need the number of GPU nodes serving a model to autoscale up and down efficiently as demand fluctuates. Because of the lack of availability and cost of GPU nodes, it is also necessary to autoscale down to zero to avoid a very high minimum cost when experimenting with a large number of models.
</li>
<li dir="ltr" aria-level="1">Data privacy concerns - While many commercial models are only available via API endpoints, when possible, we wanted to host OSS models on our own infrastructure to eliminate any privacy concerns related to sending our internal proprietary data to an external service to leverage these models.</li>
</ul>
Our team found no open-source software that satisfied all these needs. Given that ML is a core competency within Scale, we did not want to lock ourselves into proprietary solutions that could not be customized.
We built LLM Engine as an open-source platform to serve and fine-tune foundation models, whether you use Scale's hosted infrastructure or your own cloud infrastructure. The advanced version of LLM Engine powers our&nbsp;<a href="https://scale.com/genai-platform">Scale GenAI Platform</a>. Today, all of our internal ML models are deployed with LLM Engine.
While the current <a href="https://github.com/scaleapi/llm-engine">public-facing documentation</a> for LLM Engine primarily focuses on the LLM-specific APIs, this blog will explore the <a href="https://scaleapi.github.io/launch-python-client/">lower-level APIs</a> that our internal MLE&rsquo;s leverage.
<h3>Usage Summary</h3>
We start with a quick overview of core API concepts and showcase how MLEs leverage LLM Engine to host their <a href="https://scale.com/blog/fine-tune-llama-2">custom models</a>. We encourage readers to check the <a href="https://scaleapi.github.io/launch-python-client/" target="_blank" rel="noopener">API docs</a> for full details. Note that code examples will have references to Launch, which is how we refer to LLM Engine internally.
First, users create a <a href="https://scaleapi.github.io/launch-python-client/concepts/model_bundles/" target="_blank" rel="noopener">ModelBundle</a>. Users can think of ModelBundles as deployable assets (code + model weights) that are capable of making &ldquo;prediction&rdquo; requests. Users must also specify an expected request/response schema for the bundle. Should users need complete customization, there is also an option for users to bring a full image as a <a href="https://scaleapi.github.io/launch-python-client/guides/custom_docker_images/" target="_blank" rel="noopener">RunnableBundle</a>. Below is a simple example:
<pre class="language-python"><code>from launch import LaunchClient
from pydantic import BaseModel

class MyRequestSchema(BaseModel):
 x: int
 y: str

class MyResponseSchema(BaseModel):
 __root__: int

def my_load_predict_fn(model):
 def returns_model_of_x_plus_len_of_y(x: int, y: str) -&gt; int:
 """MyRequestSchema -&gt; MyResponseSchema"""
 assert isinstance(x, int) and isinstance(y, str)
 return model(x) + len(y)
 return returns_model_of_x_plus_len_of_y

def my_load_model_fn():
 def my_model(x):
 return x * 2

BUNDLE_PARAMS = {
 "model_bundle_name": "test-bundle",
 "load_predict_fn": my_load_predict_fn,
 "load_model_fn": my_load_model_fn,
 "request_schema": MyRequestSchema,
 "response_schema": MyResponseSchema
 "requirements": ["pytest==7.2.1", "numpy"], # list your requirements here
 "pytorch_image_tag": "1.7.1-cuda11.0-cudnn8-runtime",
 # additional fields omitted
}

client = LaunchClient(api_key=os.getenv("LAUNCH_API_KEY"))
client.create_model_bundle_from_callable_v2(**BUNDLE_PARAMS)</code></pre>
After a ModelBundle is registered, users can now create <a href="https://scaleapi.github.io/launch-python-client/concepts/model_endpoints/" target="_blank" rel="noopener">ModelEndpoints</a>. ModelEndpoints are actual deployments of a specified model bundle that can handle production traffic. Users specify various deployment parameters, such as min/max number of workers, as well as the compute resources such as the number of GPUs, CPUs, etc. A single model bundle can be used in several ModelEndpoints, whether they are synchronous, asynchronous, or streaming:
<pre class="language-python"><code># Creating an asynchronous endpoint
ENDPOINT_PARAMS = {
 "endpoint_name": "demo-endpoint",
 "model_bundle": "test-bundle",
 "cpus": 1,
 "min_workers": 0,
 "endpoint_type": "async",
}
endpoint = client.create_model_endpoint(**ENDPOINT_PARAMS)</code></pre>
That&rsquo;s it! Once an endpoint has status READY, an MLE&rsquo;s model is ready to serve production traffic. We show an example of submitting an async request:
<pre class="language-python"><code>def predict_on_endpoint(request: MyRequestSchema) -&gt; MyResponseSchema:
 endpoint_request = EndpointRequest(args=request.dict(), return_pickled=False)
 future = endpoint.predict(request=endpoint_request)
 raw_response = future.get()
 response = MyResponseSchema.parse_raw(raw_response.result)
 return response
request = MyRequestSchema(x=5, y="hello")
response = predict_on_endpoint(request)</code></pre>
<h3>Open-Source LLM hosting</h3>
Leveraging the core API routes mentioned previously, LLM Engine includes specialized /llm/* APIs optimized for fine-tuning and hosting open-source LLM models. With LLM Engine, we host a model zoo of popular open-source LLMs (e.g Llama 2 or Mixtral 8x7B) that product teams can immediately leverage to build Generative AI applications. Under the hood, LLM Engine leverages open-source frameworks like <a href="https://github.com/vllm-project/vllm" target="_blank" rel="noopener">vLLM</a> to maximize tokens/sec.
<pre class="language-python"><code>from llmengine import Completion

response = Completion.create(
 model="mixtral-8x7b",
 prompt="Hello! ",
 max_new_tokens=10,
)
print(response.output.text)</code></pre>
Check out the <a href="https://llm-engine.scale.com/guides/completions/" target="_blank" rel="noopener">public docs</a> for usage details for LLM-focused APIs.
<h3>LLM Engine Architecture</h3>
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=60843cd603312f22b12e1ddc303e887e.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="371" height="297">
<h3>LLM Engine Gateway</h3>
The Gateway is the central API server that orchestrates all incoming user requests. All CRUD operations involving API objects such as endpoints, bundles, etc., go through the Gateway first. API object metadata is stored in a Postgres-based datastore. Endpoint prediction requests (e.g., &ldquo;Tasks&rdquo;) are routed first through the Gateway. The Gateway simply routes requests to the correct endpoint for sync or streaming tasks. For async tasks, the Gateway pushes tasks to the correct queue.
<h3>Endpoint Builder</h3>
Endpoint creation can be a long-running process depending on the Bundle provided by users. Therefore, LLM Engine separated out this process from the core Gateway for improved request stability. The Endpoint Builder is implemented as a <a href="https://docs.celeryq.dev/en/stable/userguide/workers.html" target="_blank" rel="noopener">Celery</a> worker that receives endpoint CRUD requests from a predefined queue. Endpoint creation then goes through the following steps:&nbsp;
<ol>
<li dir="ltr" aria-level="1">
Create an endpoint image - For Cloudpickle Bundles, the Endpoint Builder must first launch a <a href="https://github.com/GoogleContainerTools/kaniko" target="_blank" rel="noopener">Kaniko</a> job to bake the Bundle into a deployable container. Future Endpoint creation requests with the same bundle do not require rebuilding. For RunnableImage Bundles, this build step is skipped.
</li>
<li dir="ltr" aria-level="1">
Create Kubernetes Resources - The Endpoint Builder will translate user-specified requests (e.g resource_request, num_workers) to the correct k8s resources required to deploy the Endpoint. For the most part, one can think of an LLM Engine Endpoint corresponding 1:1 to a k8s Deployment. Several additional k8s resources (e.g PodDisruptionBudget, HorizontalPodAutoscaler) are also created, but we won&rsquo;t go into all of them in this post.
</li>
<li dir="ltr" aria-level="1">
(Async Endpoints Only) Create Queue - For async endpoints, an SQS queue is created to store EndpointRequests before they are consumed.&nbsp;
</li>
</ol>
<h3>Anatomy of an Endpoint</h3>
Pictured below is a bit more detail for an individual Endpoint worker:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=2e7294bdc04835d13148bfec18f38b30.png&amp;f=webp&amp;q=75&amp;w=3840" alt="" width="485" height="172">
Every worker for an endpoint is deployed as a k8s pod. Within the pod, there are two containers:
<ol>
<li dir="ltr" aria-level="1">
The User Container is the image built in the previously mentioned Endpoint creation process. For Cloudpickle Bundles, LLM Engine bakes the code into a Rest Framework. For RunnableImage Bundles where users essentially provide their own image, the API contract simply requires an HTTP listener at port 5005.
</li>
<li dir="ltr" aria-level="1">
The Forwarder Container contains LLM Engine system code required to forward requests to the User Container. For async endpoints, the forwarder handles the Celery logic to poll/forward tasks to the User Container and then persist results to a Celery Backend, which is S3 in our internal deployment. The Forwarder also emits metrics useful for Endpoint monitoring and can perform pre/post-processing hooks for incoming Predict requests.
</li>
</ol>
By leveraging a separate container LLM Engine isolates system dependencies and logic from user code, enabling the same bundle to be leveraged for sync/async/streaming endpoints without any dependency conflicts or any changes in user logic. Separation also enables users to have the option of complete customization should they need to bring their own image.
<h3>Autoscaling</h3>
MLE&rsquo;s can specify how to autoscale their endpoints with the requests_per_worker field. For example, a value of 2 would mean that each worker should handle about ~2 concurrent requests. Users can also specify the min_worker and max_worker they are willing to scale in order to balance costs with reliability.
<h3 dir="ltr">Sync/Streaming</h3>
To autoscale sync/streaming endpoints, LLM Engine leverages the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter" target="_blank" rel="noopener">Prometheus Adapter</a> to create custom metrics for Horizontal Pod Autoscalers. Specifically, LLM Engine creates a request-concurrency-average metric whose target value is equal to the user-provided requests_per_worker value. To compute this custom value, LLM Engine leverages Istio&rsquo;s Request Duration Metric (e.g istio_request_duration_milliseconds) to approximate the average concurrency per endpoint.
<h3>Async</h3>
LLM Engine scales async endpoints based on the task queue length. Autoscaling based on external queues is not a standard feature in K8s, so LLM Engine contains an Autoscaler component. The Autoscaler polls queues for their lengths and then adjusts the workers for corresponding endpoints. The autoscaler simply updates the replicas field for the corresponding k8s deployments objects. The desired replicas value is calculated as the &ldquo;max queue length over the last 10 minutes&rdquo; divided by the user provided requests_per_worker value.
<h3>Scale From Zero</h3>
Some endpoints benefit from scaling down to 0 workers for particular use cases. An example would be experimental endpoints only used for demos or model zoo endpoints that receive very little traffic. To support scaling from zero, LLM Engine makes use of <a href="https://keda.sh/" target="_blank" rel="noopener">KEDA</a>. A KEDA scaler is configured to scale on the length of a Redis list. Inside the Gateway, LLM Engine pushes items to the Redis list, but caps the length of the list at one, in addition to setting an expiration time on the list. This has the effect of enabling scaling from zero pods to one pod. The request is saved inside the Gateway so that LLM Engine can wait for the pod to scale up to serve the request.
<h3>Cacher</h3>
For the request volume LLM Engine makes to the K8s API, the K8s API server is slow and easily overwhelmed. Simple list pod operations can take over a few seconds, especially as the number of pods grows in the cluster. LLM Engine has a cacher service that periodically queries for the Kubernetes API for all relevant cluster states and saves it to Redis cache. This takes load off the core K8s API server and significantly improves latency for simple GET requests sent by users.
The same service is responsible for updating Endpoint image caches in the nodes. This enables different endpoints to autoscale much faster, enabling users to efficiently provision GPU resources for their workloads. Without fast autoscaling, users would often bump up their min_workers to a value much higher than required and consequently rack up a higher compute bill. For details about fast autoscaling, we defer to a <a href="https://keda.sh/" target="_blank" rel="noopener">previous blog post</a> dedicated to this topic.
<h3>Self-Hosting</h3>
LLM Engine's self-hosting feature is compatible with AWS and Azure environments. It utilizes a Helm chart for configuring and managing the LLM Server, ensuring seamless integration within a cloud environment. Key components of this setup (visualized in our system diagram above) include dependencies like PostgreSQL for database needs and Redis for caching and queuing.
Overall, this self-hosting setup offers a robust, scalable solution for deploying LLM Engine, aligning with contemporary cloud infrastructure needs and providing a flexible, powerful environment for LLM applications.
Get started by downloading the <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">open-source repo from Github</a>. If you need a full-stack platform to customize, build, test, and deploy enterprise-ready Generative AI models and production applications, you can learn more about <a href="https://scale.com/genai-platform">Scale GenAI Platform</a>.
<h3 dir="ltr">Acknowledgments</h3>
Special thanks to the Scale team that made this work possible:
Sean Shi, Yi Xu, Will Song, Yunfeng Bai, Katie Wu, Sai Atmakuri, Ian Macleod

Learn how Scale optimizes Open-Source model hosting with LLM Engine, available as an Open-Source package and as part of Scale GenAI Platform.

Optimizing Open-Source Model Hosting with LLM Engine

At Scale, we host a diverse set of deep learning models for both internal and external users. The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time, we were able to reduce cost while maintaining a stable latency SLA. We&rsquo;d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with&nbsp;<a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener">LLM Engine</a>.
<h2>Why Cold Start Time Is Important</h2>
Without a fast enough cold start, users will often permanently provision GPUs for peak traffic. GPU hosting costs are significantly more expensive than the average CPU-based microservice. To formalize this idea in equations,
<img style="display: block; margin-left: auto; margin-right: auto;" title="n_warm_pods.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/153d7eae-b671-4ea1-a33d-fab7dc827ae0" alt="" width="650" height="75">
If we can cold start pods and make predictions within the latency SLA, we won&rsquo;t need any warm pods; otherwise we need to keep some warm pods based on the max amount of traffic we want to keep within SLA and the throughput per node.
<img style="display: block; margin-left: auto; margin-right: auto;" title="throughput_per_node.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/4c30cc1d-9231-4298-9b0f-410d73bc3911" alt="" width="650" height="128">
Then we could calculate compute seconds per wall clock second by dividing the total amount of requests and per pod throughput and plus number of warm pods. Lastly we could get the cost for serving all requests by multiplying compute seconds with the cost per second and duration.
As a concrete example, the chart below illustrates the hosting costs of one of our products as a function of user traffic. The X-axis shows the number of requests and the Y-axis is cost. The blue line is the cost curve for a configuration that has 3x the cold start time than the red line while all other configuration values are the same. When cold start time is high (blue line), we need to keep more warm pods to maintain latency SLA when there are more requests; however if cold start time is short enough (red line), we could spin up pods right when requests come in and only keep a small amount of warm pods for safety. This difference made a huge impact on cost.
<img style="display: block; margin-left: auto; margin-right: auto;" title="line_chart.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f92e2f99-dcf7-4516-afe6-a1707c2a11db" alt="" width="500" height="326">
<h2>Measurements</h2>
To understand where time is spent during cold start, we measured LLM endpoint cold start time for a Llama 2 7B model by looking at Kubernetes events and duration for each step that runs in the container. Here&rsquo;s the time breakdown:
<img style="display: block; margin-left: auto; margin-right: auto;" title="Cold_start_time_breakdown.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7f1f16d9-483d-4219-b02b-04b72176513d" alt="" width="700" height="395">
As shown in the chart, the actual model loading time is quite small and most of the time is spent on pulling docker images and downloading model weights.
<h2>Faster Pod Initialization</h2>
Since pulling images from repositories takes the majority of the time, we focused our efforts here first. From our prior experiences working on optimizing inference for various deep learning models like image generation models, we&rsquo;ve already encountered similar problems, but with LLMs the docker images are bigger and the problem is more prominent. We tried a few ideas and eventually optimized away this portion of time by caching images onto the nodes using Kubernetes daemonsets. We utilized the same technique for LLM images. The following chart describes the process:
<img style="display: block; margin-left: auto; margin-right: auto;" title="llm_cold_start_flow.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1086257a-01a0-4bbd-ad9e-04f15fcb5879" alt="" width="700" height="519">
A <a href="https://github.com/scaleapi/llm-engine/blob/main/model-engine/model_engine_server/infra/services/image_cache_service.py" target="_blank" rel="noopener">cacher</a> periodically scans through the database to get all &ldquo;<a href="https://github.com/scaleapi/llm-engine/blob/main/clients/python/llmengine/model.py#L51" target="_blank" rel="noopener">high priority</a>&rdquo; models, together with a full set of (GPU, docker image) pairs. We then construct and create/replace one daemonset for each type of GPU, and run each image with sleep commands like <code>/bin/sh -ec 'while : ; do sleep 30 ; done'</code>. This way we could dynamically maintain the set of images and preload them onto the nodes, and effectively eliminate docker image pulling time.
In addition to image caching, we reduced time to provision new nodes by creating&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/charts/model-engine/templates/balloon_a10_deployment.yaml" target="_blank" rel="noopener">balloon deployments</a> per GPU type to prewarm nodes. These pods have low priority and will be preempted when actual workloads are created.
<h2 dir="ltr">Faster Model Weights Loading</h2>
We utilize <a href="https://github.com/peak/s5cmd" target="_blank" rel="noopener">s5cmd</a> to download model weights which is <a href="https://joshua-robinson.medium.com/s5cmd-for-high-performance-object-storage-7071352cc09d" target="_blank" rel="noopener">much faster</a> than aws-cli. We used to put all files into a tarball for simplicity, but found that it&rsquo;s bad for concurrent download. We instead stored model files with 2GB chunks and had s5cmd download all the files in parallel. We also did some quick benchmarks for s5cmd parameters and chose 512 for --numworkers and 10 for --concurrency. With those changes we pushed the download time of the Llama 2 7B model (12.6GB) from 85s to 7s, achieving 14.4Gbps, which is close to the EBS volume bandwidth limit (16Gbps) for our host. Here is an example how we invoke s5cmd:
<pre class="language-python"><code>s5cmd --numworkers 512 cp --concurrency 10 {source_folder} {destination_folder}</code></pre>
We heavily utilize <a href="https://github.com/huggingface/text-generation-inference" target="_blank" rel="noopener">text-generation-inference</a> (TGI) to serve models. TGI prefers <a href="https://github.com/huggingface/safetensors" target="_blank" rel="noopener">safetensors</a>, a file format for storing tensors, and it loads <a href="https://huggingface.co/docs/safetensors/speed" target="_blank" rel="noopener">2 times faster</a> than raw PyTorch bin files. With safetensor files it could load the model in less than 20 seconds. Thus by default now we store only safetensors (with <code>model.save_pretrained(output_dir, safe_serialization=True</code>)) to avoid TGI having to convert PyTorch bin files on the fly.
<h2 dir="ltr">Summary</h2>
With all the optimizations applied we successfully reduced cold start time from more than 6 minutes to less than 40 seconds. This greatly improved our GPU utilization and our ability to deal with spiky traffic. Given the current 30-40s lag, it is now possible to save cost by having zero workers for sporadic workloads.
<h2 dir="ltr">Future Work</h2>
<ul>
<li dir="ltr" aria-level="1">
Though this approach conceptually works, we still need to implement (and open source) auto scaling from zero since Kubernetes&rsquo; Horizontal Pod Autoscaler does not scale from zero.
</li>
<li dir="ltr" aria-level="1">
We fine tune models with PEFT (Parameter-Efficient Fine-Tuning). The code is currently private and we will open source in the near future.
</li>
<li dir="ltr" aria-level="1">
We currently merge back PEFT trained weights onto the base model. This is cost inefficient since we need to maintain one deployment per fine-tuned model, and the cold start optimizations presented here only partially mitigate this problem. Internally, we have implemented dynamic loading / unloading for <a href="https://arxiv.org/abs/2106.09685" target="_blank" rel="noopener">LoRA</a> and <a href="https://arxiv.org/abs/2205.05638" target="_blank" rel="noopener">IA3</a> adapters with DeepSpeed-sharded models. This way only one deployment is needed per base model, and each process would dynamically swap out adapters when serving traffic from different fine-tunes. The future work is to open-source this framework and get adapters working with continuous batching.
</li>
<li dir="ltr" aria-level="1">
To better generalize our image cache beyond just &ldquo;high-priority&rdquo; endpoints, we are investigating lazy-loading of images with projects like&nbsp;<a href="https://github.com/containerd/stargz-snapshotter/tree/main" target="_blank" rel="noopener">stargz</a>. 
</li>
</ul>
&nbsp;
<h2 dir="ltr">Next Steps</h2>
Looking to customize and serve LLMs? Try&nbsp;<a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> today!&nbsp;
Learn more about how Scale can help you <a href="https://scale.com/custom-llms">customize LLMs for your unique use case</a>.

The recent boom of large language models (LLMs) brought a set of new technical challenges and pod cold start is one of the most important. By reducing cold start time we were able to reduce cost, due to its significant impact on the number of warm pods if a stable latency SLA needs to be maintained. We’d like to share our learnings on why cold start is an important problem and how we reduced the cold start time of LLMs with LLM Engine.

Yunfeng Bai

Optimizing Open-Source Model Hosting with LLM Engine

How To Reduce Cold Start Times For LLM Inference