Jihan Yin

At Scale, we have always believed that building custom LLMs through fine-tuning is key to unlocking greater performance for any given organization&rsquo;s specific use case. We work with enterprise customers to implement cutting-edge <a href="https://scale.com/generative-ai-platform">enterprise Generative AI</a> solutions, combining the best large language models with the latest research techniques and balancing our solutions for both effectiveness with efficiency to optimize model performance.
Recently, Parameter-Efficient Fine-Tuning (PEFT) and Mixture-of-Experts (MoE) techniques have risen in popularity &mdash; each with its unique focus. While PEFT prioritizes efficiency, MoE pushes the boundaries of model performance. This blog post will briefly explore the core concepts of PEFT and MoE before diving into a new approach that synergistically combines these methods, offering an efficient and effective way to fine-tune large language models.
<h2 dir="ltr">Background</h2>
<h3 dir="ltr">Parameter-efficient Fine-tuning (PEFT)</h3>
Traditional fine-tuning, where each task requires a distinct set of weights, becomes untenable with models scaling to hundreds of billions of parameters. Not only does hosting different weights for each model become inefficient and cost-prohibitive, but reloading weights for various tasks also proves too slow. PEFT techniques address this by modifying only a small portion of the weights relative to the full model size, keeping the bulk of the model unchanged.
PEFT methods typically require a considerably smaller memory footprint (e.g. &lt; 1% of total parameters) while closely approximating the performance of full fine-tuning. These methods can be broadly categorized:
<ul>
<li dir="ltr" aria-level="1">
Adapters: Techniques that fine-tune a part of the model or insert small, trainable modules between layers, enabling efficient fine-tuning with minimal additional parameters. Examples include BitFit, (IA)&sup3;, and LoRA (and its variants).
</li>
<li dir="ltr" aria-level="1">
Prompt Tuning: This involves fine-tuning a set of input &ldquo;prompts&rdquo; that guide the model&rsquo;s responses, adapting output with minimal changes to existing parameters. Methods can be either hand-crafted or learned, with examples like Prefix Tuning and P-Tuning.
</li>
</ul>
Given the multitude of fine-tuning options to choose from, we performed a comprehensive benchmark across these techniques, detailed in our paper, <a href="https://arxiv.org/pdf/2304.14999.pdf">&ldquo;Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs&rdquo;</a>. Our findings include a detailed decision framework to choose the best technique given the task type (e.g. classification, generation) along with the data volume. For example, there are several dimensions to consider when deciding between memory, performance, and time constraints. In addition, we found that LoRA/(IA)&sup3; could be further optimized by selectively choosing which parts of the model to train, such as only PEFT&rsquo;ing the last few layers of an LLM, while maintaining performance.
<h3>Mixture of Experts (MoE)</h3>
Mixture of Experts (MoE) for language models is a modification over the transformer architecture, where the model consists of various &lsquo;expert&rsquo; sub-networks. These sub-networks each specialize in a different aspect or types of data. In an MoE model, there is also an additional gating/routing mechanism that dynamically determines which expert or combination of experts is best suited for the given input during inference. This approach enables the model to handle a wider array of tasks and understand the minor nuances of different domains better than a monolithic model. By distributing learning across different specialized experts, MoE models can achieve higher performance and scalability. Recent papers such as <a href="https://arxiv.org/pdf/2112.06905.pdf">&ldquo;GLaM: Efficient Scaling of Language Models with Mixture-of-Experts&rdquo;</a> and <a href="https://arxiv.org/pdf/2202.09368.pdf">&ldquo;Mixture-of-Experts with Expert Choice Routing&rdquo;</a> offer further insights into this approach. In addition, these methods allow us to sacrifice memory consumption for more efficient floating point operations during training and inference.
<h2>Parameter-efficient Mixture of Experts</h2>
The paper <a href="https://arxiv.org/pdf/2309.05444.pdf">&ldquo;Pushing Mixture of Experts to the Limit&rdquo;</a> came to our attention as it heralds a blend of PEFT and MoE to facilitate both efficient and effective fine-tuning. Intrigued by its potential, we implemented and benchmarked the method ourselves to assess its effectiveness. The work has proposed MoE variations of two popular adapter PEFT approaches: LoRA and (IA)&sup3;, which are named MoLORA and MoV respectively. However, this method was only evaluated on the FLAN T-5 models, which is an encoder-decoder model.
We will provide an overview of the MoV approach and delve into the implementation of one of the proposed methods. MoLoRA can be implemented similarly with the main differences being: only modifying the key/value matrices, applying matrix multiplication instead of element-wise, and adding a dimension during computations for LoRA rank.
<h3>What is MoV?</h3>
Mixture of Vectors (MoV) builds upon the foundational concept of (IA)&sup3;, where a pretrained model remains largely unchanged except for three learned vectors per attention block. These vectors interact element-wise with the key, value, and feed-forward layers within the transformer&rsquo;s self-attention block. The image below from the original paper provides a clear depiction.
<img src="https://img.plasmic.app/img-optimizer/v1/img/0cb7e308238e6679c177c7e66a96d0a5.png?f=webp&amp;q=75" alt="" width="1731" height="1227">
Overview of (IA)&sup3;, taken from the <a href="https://arxiv.org/pdf/2205.05638.pdf">paper</a>.
In Mixture of Vectors (MoV), the overall concept remains the same, but instead of learning one vector for each of the three tensors, we learn \(n\) (no. of experts) of them and combine them through a routing mechanism. The diagram below from the paper, gives an overview of this.
<img src="https://img.plasmic.app/img-optimizer/v1/img/c9ec04f99c82ad1fdd15cd845942d01c.png?f=webp&amp;q=75" alt="" width="1731" height="1521">
<h3>Implementation</h3>
Next, we will guide you through the implementation of the MoV and Router layers essential for this methodology. Additionally, we&rsquo;ll discuss adapting these to fine-tune the LLaMA-2 model.
<h4>Router</h4>
A router is a linear layer that selects which experts to send the input towards. In MoV, the router is combined with the output from our experts, which are (IA)&sup3; modules, and allows for conditional computation instead of using all of our parameters.
<pre class="language-python"><code>class Router(nn.Module):
 def __init__(self, input_dim, num_experts):
 super().__init__()
 self.ff = nn.Linear(input_dim, num_experts)
 def forward(self, x):
 logits = self.ff(x)
 probs = F.softmax(logits, dim=-1)
 return logits, probs</code></pre>
First, we define a linear layer that takes in the original input dimension and outputs a tensor with the corresponding number of experts. In the forward call, we first compute the logits with our dense layer and also return our probabilities from a softmax.
<h4>MoV Layer</h4>
The MoV layer combines the probabilities of the router network along with the outputs of each expert, which is an (IA)&sup3; vector. We are computing the following equation, where \(s_i\) is the routing probability of the current expert, x is the token representation, and \(E_i\) is the current (IA)&sup3; vector&rsquo;s output:
<img src="https://img.plasmic.app/img-optimizer/v1/img/4b9c91955fe00119f3e10fe17eb127fc.png?f=webp&amp;q=75" alt="" width="1731" height="357">
Although we can also implement Top-k routing, where we zero out the non-selected experts, the authors found that soft merging, which is a &ldquo;weighted average of all experts computed within a specific routing block&rdquo;, performed the best.
<pre class="language-python"><code>class MoV(nn.Module):
 def __init__(self, linear_layer, num_experts):
 super(MoV, self).__init__()
 # Original linear layer
 self.original_layer = linear_layer
 self.router = Router(self.original_layer.in_features, num_experts)
 self.experts = nn.Parameter(torch.ones(num_experts, linear_layer.out_features))
 def prepare_model_gradients(self):
 self.experts.requires_grad_(True)
 self.router.ff.weight.requires_grad_(True)
 def forward(self, x):
 frozen_output = self.original_layer(x)
 _, gating_probs = self.router(x)
 # Compute the weighted sum of expert outputs
 mov_combined = torch.einsum("bse,ed-&gt;bsd", gating_probs, self.experts)
 return frozen_output * mov_combined</code></pre>
To implement the MoV layer, we first store the original linear layer along with initializing the Router layer from the previous section and the experts, which are (IA)&sup3; vectors. In the forward pass, we first compute the original output representation and then router probabilities. Afterward, we compute the weighted sum of the expert outputs along with the gating probabilities and rescale the original output. We also provide a prepare_model_gradients() method to set these tunable parameters and freeze the rest of the model in the next part.
<h4>Adding MoV to LLaMA-2</h4>
We stepped through the implementation specifics, but to get these layers integrated into an actual model, we need to iterate through the entire pretrained LLM and selectively apply theses MoV layers. For our experiments, we use the AutoModelForCausalLM on the Llama-2 decoder-only models.
<pre class="language-python"><code>def adapt_model_with_moe_peft(model, experts):
 # Only modify the key/value and linear activations
 llama_regex_match = "(.*(self_attn|LlamaAttention).(k_proj|v_proj).weight)|(.*LlamaMLP.down_proj.weight)"
 
 for n, _ in model.named_parameters():
 if re.search(regex_match, n) is None:
 continue
 # Get module that the parameter belongs to
 module_name = ".".join(n.split(".")[:-1])
 module = attrgetter(module_name)(model)
 module_parent_name = ".".join(n.split(".")[:-2])
 module_key_name = n.split(".")[-2]
 module_parent = attrgetter(module_parent_name)(model)
 setattr(module_parent, module_key_name, MoV(module, experts))

 # Freeze base model and set MoV weights as tunable
 for m in model.modules():
 m.requires_grad_(False)
 if isinstance(m, MoV):
 m.prepare_model_gradients()</code></pre>
When we print the model object, which is a Llama-2-7B model, we can see the defined embedding, 32 decoder layers, and language modeling head. Diving into the decoder, each layer consists of a self-attention layer, a multi-layer perceptron, input normalization, and a post-attention normalization. First, we define a regex to match the (IA)&sup3; implementation where the parameters of the query/key and linear activations are modified. Then, we iterate through the model&rsquo;s layers to find the valid parameters to inject our MoV layer. Finally, we need to freeze the base model and set the MoV layers as tunable, excluding the original layer. Note that using one expert is equivalent to applying (IA)&sup3;.
There are other tricks to improve our training efficiency, such as gradient checkpointing or mixed-precision training. After we add these trainable MoV layers into the decoder model, we can calculate the total number of parameters being tuned. With Llama-2 model, we get:
<div dir="ltr" align="left">
<table style="border-collapse: collapse; border-color: rgb(126, 140, 141);" border="1"><colgroup><col width="103"><col width="105"></colgroup>
<tbody>
<tr>
<td style="border-color: rgb(126, 140, 141);">
# experts
</td>
<td style="border-color: rgb(126, 140, 141);">
MoV (7B)
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
1
</td>
<td style="border-color: rgb(126, 140, 141);">
524352
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
10
</td>
<td style="border-color: rgb(126, 140, 141);">
5243520
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
20
</td>
<td style="border-color: rgb(126, 140, 141);">
10487040
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
60
</td>
<td style="border-color: rgb(126, 140, 141);">
31461120
</td>
</tr>
</tbody>
</table>
</div>
Note that using 1/10/20/60 expert(s) modifies less than 0.001%/0.01%/0.02%/0.05% of the total parameters, respectively. This is still incredibly memory efficient!
<h2>Experiments</h2>
We evaluate across 4 different datasets:
<ul>
<li dir="ltr" aria-level="1">
The ScienceQA dataset is generated from elementary and high school multiple choice questions, where we select around 6000 samples (see our blog <a href="https://scale.com/blog/fine-tune-gpt-3.5">How to Fine-Tune GPT-3.5 Turbo With OpenAI API</a> for more on fine-tuning with this dataset).
</li>
<li dir="ltr" aria-level="1">
Corpus of Linguistic Acceptability (CoLA) uses 23 linguistics publications to evaluate grammaticality with around 10000 samples.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Microsoft Research Paraphrase Corpus (MRPC) uses newswire articles and checks whether the sentence pairs are or are not paraphrases with 5000 pairs.
</li>
<li dir="ltr" aria-level="1">
Recognizing Textual Entailment (RTE) uses news and Wikipedia text for textual entailment, where we are given a premise and hypothesis and evaluate whether these texts logically follow (entail) or do not, with around 3000 samples.
</li>
</ul>
All of our listed experiments use the Llama-2-7b model with the MoV technique. For our MoV runs, we default the learning rate to 2e-4 and run for 10 epochs. For full-tuning, we use a learning rate of 3e-5 and 5 epochs. In addition, we select the checkpoint that corresponds to the lowest validation loss. For evaluation, we use exact string match accuracy across the gold label and prediction.
&nbsp;
<div dir="ltr" align="left">
<table style="border-collapse: collapse; border-color: rgb(126, 140, 141);" border="1"><colgroup><col width="114"><col width="86"><col width="91"><col width="91"><col width="92"><col width="92"></colgroup>
<tbody>
<tr>
<td style="border-color: rgb(126, 140, 141);">
Dataset
</td>
<td style="border-color: rgb(126, 140, 141);">
MoV-1
</td>
<td style="border-color: rgb(126, 140, 141);">
MoV-10
</td>
<td style="border-color: rgb(126, 140, 141);">
MoV-20
</td>
<td style="border-color: rgb(126, 140, 141);">
MoV-60
</td>
<td style="border-color: rgb(126, 140, 141);">
Full
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
Science QA
</td>
<td style="border-color: rgb(126, 140, 141);">
74.61%
</td>
<td style="border-color: rgb(126, 140, 141);">
79.52%
</td>
<td style="border-color: rgb(126, 140, 141);">
79.90%
</td>
<td style="border-color: rgb(126, 140, 141);">
80.62%
</td>
<td style="border-color: rgb(126, 140, 141);">
81.00%
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
CoLA
</td>
<td style="border-color: rgb(126, 140, 141);">
82.55%
</td>
<td style="border-color: rgb(126, 140, 141);">
84.28%
</td>
<td style="border-color: rgb(126, 140, 141);">
84.95%
</td>
<td style="border-color: rgb(126, 140, 141);">
83.89%
</td>
<td style="border-color: rgb(126, 140, 141);">
85.91%
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
MRPC
</td>
<td style="border-color: rgb(126, 140, 141);">
79.90%
</td>
<td style="border-color: rgb(126, 140, 141);">
83.09%
</td>
<td style="border-color: rgb(126, 140, 141);">
83.58%
</td>
<td style="border-color: rgb(126, 140, 141);">
85.54%
</td>
<td style="border-color: rgb(126, 140, 141);">
85.54%
</td>
</tr>
<tr>
<td style="border-color: rgb(126, 140, 141);">
RTE
</td>
<td style="border-color: rgb(126, 140, 141);">
76.17%
</td>
<td style="border-color: rgb(126, 140, 141);">
82.67%
</td>
<td style="border-color: rgb(126, 140, 141);">
81.23%
</td>
<td style="border-color: rgb(126, 140, 141);">
80.14%
</td>
<td style="border-color: rgb(126, 140, 141);">
80.14%
</td>
</tr>
</tbody>
</table>
</div>
From our results, we observe that using the MoE PEFT method consistently outperforms PEFT with an average four percent delta. Additionally, in some scenarios, such as with MRPC and RTE, MoV is equal or better than full-tuning our model. Lastly, we should note that increasing the number of experts does not always translate to better downstream performance. For example, Science QA and MRPC increase in performance as we scale from 1 to 60 experts, noting that this task can have even more experts. Both CoLA and RTE drop in performance after the 20th and 10th expert, respectively.
We recommend carefully tuning the number of experts while being mindful of the memory overhead. These trends are similarly observed by the authors. In addition, we can empirically state from our results that MoE PEFT helps close the gap between PEFT methods and full-tuning across both encoder-decoder and decoder-only models.
<h2>Conclusion</h2>
The MoE PEFT methods have great empirical benefits while being extremely memory effective. We are excited to further experiment with these methods along with providing a working implementation of MoV with Llama-2 models for anyone to try!
As we continue to test more methods, we will add what works best with LLMs to <a href="https://github.com/scaleapi/llm-engine">llm-engine</a> so stay tuned for new changes that you can experiment with on your own! We also incorporate these methods into our <a href="https://scale.com/generative-ai-platform">Enterprise Generative AI Platform (EGP)</a> and work with customers to fine-tune models for their unique use cases, implement cutting-edge <a href="https://scale.com/blog/retrieval-augmented-generation-to-enhance-llms">retrieval augmented generation</a>, and help them implement Generative AI Applications. We will continue to incorporate the latest research and techniques into our open-source packages, products, and processes as we help organizations unlock the value of AI.
&nbsp;

We explore PEFT and MoE before diving into a new approach that combines the methods, offering an efficient and effective way to fine-tune llms.

Efficient and Effective Fine-Tuning Using Mixture-of-Experts PEFT

Earlier this week, Meta announced the release of <a href="https://ai.meta.com/llama/" target="_blank" rel="noopener noreferrer">Llama 2</a>. Llama 2 includes both a base pre-trained model and a fine-tuned model for chat available in three sizes. While the performance of the pre-trained model is impressive, fine-tuning the base Llama-2 model can unlock even greater performance on most language tasks. This process can be quite complicated, from reserving the right compute resources to properly implementing the training and inference code.
Scale's new open-source <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener noreferrer">LLM Engine</a> repository makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<pre class="language-javascript"><code> from llmengine import FineTune

response = FineTune.create(
 model="llama-2-7b",
 training_file="s3://my-bucket/path/to/training-file.csv",
)

print(response.json())</code></pre>
You can also follow the code examples in this&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/examples/finetune_llama_2_on_science_qa.ipynb" target="_blank" rel="noopener noreferrer">notebook</a>.
<h3>Dataset</h3>
For this example, we will use <a href="https://scienceqa.github.io/" target="_blank" rel="noopener noreferrer">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. Each question may have textual context and image context and contains a thorough explanation and lecture supporting the solution.
<img src="https://lh3.googleusercontent.com/hB2u6hfFZMXlmtwOLgT3-wrJKdlyAdd9MssPkfvoysG2KRdnafhfxHfJ4IVRKDt1Bj71LikV-vtmXBK1P11sOqS8ea7nGnD1_1mVIKAIcQn4OIbKNKESnpSobwRFBsnTAbYT_7t_RARN83xNiwGRitk">
An example sample from ScienceQA
Currently, LLM Engine supports fine-tuning on prompt-completion pairs. Let's first convert this dataset into the supported format, a CSV with two columns: <code>prompt</code> and <code>response</code>.
Before you get started, install the required dependencies.
<pre class="language-php"><code>pip install datasets==2.13.1 smart_open[s3]==5.2.1 pandas==1.4.4</code></pre>
We can load the dataset from Hugging Face, and observe the dataset's features.
<pre class="language-javascript"><code>from datasets import load_dataset
from smart_open import smart_open
import pandas as pd
dataset = load_dataset('derek-thomas/ScienceQA')
dataset['train'].features</code></pre>
A commonly used format for feeding ScienceQA examples is
<pre class="language-javascript"><code>Context: A baby wants to know what is inside of a cabinet. Her hand applies a force to the door, and the door opens.
Question: Which type of force from the baby's hand opens the cabinet door?
Options: (A) pull (B) push
Answer: A.</code></pre>
Since the format of <code>options</code> in the Huggingface dataset is only a list of possible answers, we need to convert this list into the example format from above by adding the enumeration prefix.
<pre class="language-javascript"><code>choice_prefixes = [chr(ord('A') + i) for i in range(26)] # A-Z
def format_options(options, choice_prefixes):
 return ' '.join([f'({c}) {o}' for c, o in zip(choice_prefixes, options)])
</code></pre>
Now, let's write the formatting function to turn a single sample from this dataset into a <code>prompt</code>&nbsp;and <code>response</code> to feed into the model.
<pre class="language-javascript"><code>def format_prompt(r, choice_prefixes):
 options = format_options(r['choices'], choice_prefixes)
 return f'''Context: {r["hint"]}\nQuestion:{r["question"]}\nOptions:{options}\nAnswer:'''
def format_response(r, choice_prefixes):
 return choice_prefixes[r['answer']] </code></pre>
Finally, let's construct the dataset. Note that some samples in ScienceQA only have an image for context - we'll be skipping those in this example as Llama-2 is purely a language model, and cannot accept image inputs.
<pre class="language-javascript"><code> def convert_dataset(ds):
 prompts = [format_prompt(i, choice_prefixes) for i in ds if i['hint'] != '']
 labels = [format_response(i, choice_prefixes) for i in ds if i['hint'] != '']
 df = pd.DataFrame.from_dict({'prompt': prompts, 'response': labels})
 return df </code></pre>
LLM Engine supports training with a training and validation dataset. If you only provide a training dataset, the LLM Engine will randomly split 10% from the dataset for validation. Splitting your dataset prevents the model from overfitting on your training data, resulting in poor generalization to live data it might see during inference.
These dataset files must be stored in a publicly accessible URL so that LLM Engine can read them. For this example, we will be saving the datasets to s3. We also have the preprocessed <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_train.csv" target="_blank" rel="noopener noreferrer">train dataset</a> and <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_val.csv" target="_blank" rel="noopener noreferrer">validation dataset</a> publicly&nbsp;available in Github Gists - you can directly replace <code>train_url</code> and <code>val_url</code> with these links.
<pre class="language-javascript"><code>train_url = 's3://...'
val_url = 's3://...'
df_train = convert_dataset(dataset['train'])
with smart_open(train_url, 'wb') as f:
 df_train.to_csv(f)
df_val = convert_dataset(dataset['validation'])
with smart_open(val_url, 'wb') as f:
 df_val.to_csv(f)</code></pre>
Now, we can start fine-tuning via the LLM Engine API.
<h3>Fine-Tuning</h3>
First, we need to install LLM Engine.
<pre class="language-python"><code>pip install scale-llm-engine</code></pre>
Next, you'll need to set up your Scale API key. Follow the instructions in the&nbsp;<a href="https://github.com/scaleapi/llm-engine#-quick-start" target="_blank" rel="noopener noreferrer">README</a> to acquire your unique API key. Advanced users can also follow the guide for <a href="https://scaleapi.github.io/llm-engine/guides/self_hosting/" target="_blank" rel="noopener noreferrer">self-hosting LLM Engine</a> to avoid needing a Scale API key.
<pre class="language-javascript"><code>import os
os.environ['SCALE_API_KEY'] = 'xxx'</code></pre>
Once you have everything set up, fine-tuning your model requires only one API call. Today, we'll be working with the 7 billion parameter version of Llama-2, which proves powerful enough for most use cases.
<pre class="language-javascript"><code>from llmengine import FineTune
response = FineTune.create(
 model="llama-2-7b",
 training_file=train_url,
 validation_file=val_url,
 hyperparameters={
 'lr':2e-4,
 },
 suffix='science-qa-llama'
)
run_id = response.fine_tune_id</code></pre>
With the <code>run_id</code>, you can monitor the status of your job and get live updates on per-epoch metrics, like train and validation loss.
ScienceQA is a large dataset, so it may take an hour or two for training to complete.
<pre class="language-javascript"><code>while True:
 job_status = FineTune.get(run_id).status
 # Returns one of `PENDING`, `STARTED`, `SUCCESS`, `RUNNING`,
 # `FAILURE`, `CANCELLED`, `UNDEFINED` or `TIMEOUT`
 print(job_status)
 if job_status == 'SUCCESS':
 break
 time.sleep(60)
#Logs for completed or running jobs can be fetched with
logs = FineTune.get_events(run_id)</code></pre>
<h3 class="ql-syntax" spellcheck="false">Inference &amp; Evaluation</h3>
Once you are done fine-tuning, you can start generating responses to any input. However, before that, let's ensure the model exists and is ready to accept inputs.
<pre class="language-php"><code>ft_model = FineTune.get(run_id).fine_tuned_model</code></pre>
It may take a few minutes for your first inference call to be served. After that, inference should be relatively quick.
Now, let's evaluate the performance of our Llama-2 model fine-tuned on ScienceQA.
<pre class="language-python"><code> import pandas as pd

#Helper a function to get outputs for fine-tuned model with retries
def get_output(prompt: str, num_retry: int = 5):
 for _ in range(num_retry):
 try:
 response = Completion.create(
 model=ft_model, 
 prompt=prompt, 
 max_new_tokens=1, 
 temperature=0.01
 )
 return response.output.text.strip()
 except Exception as e:
 print(e)
 return ""
#Read the test data
test = pd.read_csv(val_url)
test["prediction"] = test["prompt"].apply(get_output)
print(f"Accuracy: {(test['response'] == test['prediction']).mean() * 100:.2f}%")
 </code></pre>
This achieves 82.15% accuracy. This is pretty good! Let's see how this compares with the original base Llama-2 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses. We also see how things compare against fine-tuning a model of similar size, Mosaic's <a href="https://www.mosaicml.com/blog/mpt-7b" target="_blank" rel="noopener noreferrer">MPT</a>.
<img src="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/blog-llama-1.png">
*Results from <a href="https://arxiv.org/abs/2302.00923" target="_blank" rel="noopener noreferrer">MM-COT paper.</a>
As we can see, fine-tuning makes a huge difference in the case of both language models. The performance gain from fine-tuning Llama-2 on ScienceQA was a&nbsp;26.59% absolute difference! This is in addition to the fact that inference with a fine-tuned model is cheaper than few-shot example prompts, due to shorter prompt length. This fine-tuned Llama-2-7B model also outperforms GPT-3.5, a 175 billion parameter model!
We can also see that the new Llama-2 model outperforms MPT in both the fine-tuned and few-shot prompting settings, showcasing its strength as both a base and fine-tunable model.
As a side experiment, we also used LLM Engine to fine-tune and evaluate LLAMA-2's performance on several tasks from <a href="https://gluebenchmark.com/" target="_blank" rel="noopener noreferrer">GLUE</a>, a set of commonly used NLP benchmark datasets.
<img src="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/blog-llama-2.png">
<h3>Conclusion</h3>
Fine-tuning makes a huge difference in unlocking greater performance from base pre-trained models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. Happy fine-tuning! 🚀

Deploy and fine-tune models using our open-sourced LLM Engine Repo. Learn how in our latest blog. 

Jihan Yin

Efficient and Effective Fine-Tuning Using Mixture-of-Experts PEFT

How to Fine-Tune Llama 2 With LLM Engine