Jihan Yin

Earlier this week, Meta announced the release of <a href="https://ai.meta.com/llama/" target="_blank" rel="noopener noreferrer">Llama 2</a>. Llama 2 includes both a base pre-trained model and a fine-tuned model for chat available in three sizes. While the performance of the pre-trained model is impressive, fine-tuning the base Llama-2 model can unlock even greater performance on most language tasks. This process can be quite complicated, from reserving the right compute resources to properly implementing the training and inference code.
Scale's new open-source <a href="https://github.com/scaleapi/llm-engine" target="_blank" rel="noopener noreferrer">LLM Engine</a> repository makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<pre class="language-javascript"><code> from llmengine import FineTune

response = FineTune.create(
 model="llama-2-7b",
 training_file="s3://my-bucket/path/to/training-file.csv",
)

print(response.json())</code></pre>
You can also follow the code examples in this&nbsp;<a href="https://github.com/scaleapi/llm-engine/blob/main/examples/finetune_llama_2_on_science_qa.ipynb" target="_blank" rel="noopener noreferrer">notebook</a>.
<h3>Dataset</h3>
For this example, we will use <a href="https://scienceqa.github.io/" target="_blank" rel="noopener noreferrer">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. Each question may have textual context and image context and contains a thorough explanation and lecture supporting the solution.
<img src="https://lh3.googleusercontent.com/hB2u6hfFZMXlmtwOLgT3-wrJKdlyAdd9MssPkfvoysG2KRdnafhfxHfJ4IVRKDt1Bj71LikV-vtmXBK1P11sOqS8ea7nGnD1_1mVIKAIcQn4OIbKNKESnpSobwRFBsnTAbYT_7t_RARN83xNiwGRitk">
An example sample from ScienceQA
Currently, LLM Engine supports fine-tuning on prompt-completion pairs. Let's first convert this dataset into the supported format, a CSV with two columns: <code>prompt</code> and <code>response</code>.
Before you get started, install the required dependencies.
<pre class="language-php"><code>pip install datasets==2.13.1 smart_open[s3]==5.2.1 pandas==1.4.4</code></pre>
We can load the dataset from Hugging Face, and observe the dataset's features.
<pre class="language-javascript"><code>from datasets import load_dataset
from smart_open import smart_open
import pandas as pd
dataset = load_dataset('derek-thomas/ScienceQA')
dataset['train'].features</code></pre>
A commonly used format for feeding ScienceQA examples is
<pre class="language-javascript"><code>Context: A baby wants to know what is inside of a cabinet. Her hand applies a force to the door, and the door opens.
Question: Which type of force from the baby's hand opens the cabinet door?
Options: (A) pull (B) push
Answer: A.</code></pre>
Since the format of <code>options</code> in the Huggingface dataset is only a list of possible answers, we need to convert this list into the example format from above by adding the enumeration prefix.
<pre class="language-javascript"><code>choice_prefixes = [chr(ord('A') + i) for i in range(26)] # A-Z
def format_options(options, choice_prefixes):
 return ' '.join([f'({c}) {o}' for c, o in zip(choice_prefixes, options)])
</code></pre>
Now, let's write the formatting function to turn a single sample from this dataset into a <code>prompt</code>&nbsp;and <code>response</code> to feed into the model.
<pre class="language-javascript"><code>def format_prompt(r, choice_prefixes):
 options = format_options(r['choices'], choice_prefixes)
 return f'''Context: {r["hint"]}\nQuestion:{r["question"]}\nOptions:{options}\nAnswer:'''
def format_response(r, choice_prefixes):
 return choice_prefixes[r['answer']] </code></pre>
Finally, let's construct the dataset. Note that some samples in ScienceQA only have an image for context - we'll be skipping those in this example as Llama-2 is purely a language model, and cannot accept image inputs.
<pre class="language-javascript"><code> def convert_dataset(ds):
 prompts = [format_prompt(i, choice_prefixes) for i in ds if i['hint'] != '']
 labels = [format_response(i, choice_prefixes) for i in ds if i['hint'] != '']
 df = pd.DataFrame.from_dict({'prompt': prompts, 'response': labels})
 return df </code></pre>
LLM Engine supports training with a training and validation dataset. If you only provide a training dataset, the LLM Engine will randomly split 10% from the dataset for validation. Splitting your dataset prevents the model from overfitting on your training data, resulting in poor generalization to live data it might see during inference.
These dataset files must be stored in a publicly accessible URL so that LLM Engine can read them. For this example, we will be saving the datasets to s3. We also have the preprocessed <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_train.csv" target="_blank" rel="noopener noreferrer">train dataset</a> and <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_val.csv" target="_blank" rel="noopener noreferrer">validation dataset</a> publicly&nbsp;available in Github Gists - you can directly replace <code>train_url</code> and <code>val_url</code> with these links.
<pre class="language-javascript"><code>train_url = 's3://...'
val_url = 's3://...'
df_train = convert_dataset(dataset['train'])
with smart_open(train_url, 'wb') as f:
 df_train.to_csv(f)
df_val = convert_dataset(dataset['validation'])
with smart_open(val_url, 'wb') as f:
 df_val.to_csv(f)</code></pre>
Now, we can start fine-tuning via the LLM Engine API.
<h3>Fine-Tuning</h3>
First, we need to install LLM Engine.
<pre class="language-python"><code>pip install scale-llm-engine</code></pre>
Next, you'll need to set up your Scale API key. Follow the instructions in the&nbsp;<a href="https://github.com/scaleapi/llm-engine#-quick-start" target="_blank" rel="noopener noreferrer">README</a> to acquire your unique API key. Advanced users can also follow the guide for <a href="https://scaleapi.github.io/llm-engine/guides/self_hosting/" target="_blank" rel="noopener noreferrer">self-hosting LLM Engine</a> to avoid needing a Scale API key.
<pre class="language-javascript"><code>import os
os.environ['SCALE_API_KEY'] = 'xxx'</code></pre>
Once you have everything set up, fine-tuning your model requires only one API call. Today, we'll be working with the 7 billion parameter version of Llama-2, which proves powerful enough for most use cases.
<pre class="language-javascript"><code>from llmengine import FineTune
response = FineTune.create(
 model="llama-2-7b",
 training_file=train_url,
 validation_file=val_url,
 hyperparameters={
 'lr':2e-4,
 },
 suffix='science-qa-llama'
)
run_id = response.fine_tune_id</code></pre>
With the <code>run_id</code>, you can monitor the status of your job and get live updates on per-epoch metrics, like train and validation loss.
ScienceQA is a large dataset, so it may take an hour or two for training to complete.
<pre class="language-javascript"><code>while True:
 job_status = FineTune.get(run_id).status
 # Returns one of `PENDING`, `STARTED`, `SUCCESS`, `RUNNING`,
 # `FAILURE`, `CANCELLED`, `UNDEFINED` or `TIMEOUT`
 print(job_status)
 if job_status == 'SUCCESS':
 break
 time.sleep(60)
#Logs for completed or running jobs can be fetched with
logs = FineTune.get_events(run_id)</code></pre>
<h3 class="ql-syntax" spellcheck="false">Inference &amp; Evaluation</h3>
Once you are done fine-tuning, you can start generating responses to any input. However, before that, let's ensure the model exists and is ready to accept inputs.
<pre class="language-php"><code>ft_model = FineTune.get(run_id).fine_tuned_model</code></pre>
It may take a few minutes for your first inference call to be served. After that, inference should be relatively quick.
Now, let's evaluate the performance of our Llama-2 model fine-tuned on ScienceQA.
<pre class="language-python"><code> import pandas as pd

#Helper a function to get outputs for fine-tuned model with retries
def get_output(prompt: str, num_retry: int = 5):
 for _ in range(num_retry):
 try:
 response = Completion.create(
 model=ft_model, 
 prompt=prompt, 
 max_new_tokens=1, 
 temperature=0.01
 )
 return response.output.text.strip()
 except Exception as e:
 print(e)
 return ""
#Read the test data
test = pd.read_csv(val_url)
test["prediction"] = test["prompt"].apply(get_output)
print(f"Accuracy: {(test['response'] == test['prediction']).mean() * 100:.2f}%")
 </code></pre>
This achieves 82.15% accuracy. This is pretty good! Let's see how this compares with the original base Llama-2 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses. We also see how things compare against fine-tuning a model of similar size, Mosaic's <a href="https://www.mosaicml.com/blog/mpt-7b" target="_blank" rel="noopener noreferrer">MPT</a>.
<img src="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/blog-llama-1.png">
*Results from <a href="https://arxiv.org/abs/2302.00923" target="_blank" rel="noopener noreferrer">MM-COT paper.</a>
As we can see, fine-tuning makes a huge difference in the case of both language models. The performance gain from fine-tuning Llama-2 on ScienceQA was a&nbsp;26.59% absolute difference! This is in addition to the fact that inference with a fine-tuned model is cheaper than few-shot example prompts, due to shorter prompt length. This fine-tuned Llama-2-7B model also outperforms GPT-3.5, a 175 billion parameter model!
We can also see that the new Llama-2 model outperforms MPT in both the fine-tuned and few-shot prompting settings, showcasing its strength as both a base and fine-tunable model.
As a side experiment, we also used LLM Engine to fine-tune and evaluate LLAMA-2's performance on several tasks from <a href="https://gluebenchmark.com/" target="_blank" rel="noopener noreferrer">GLUE</a>, a set of commonly used NLP benchmark datasets.
<img src="https://static.scale.com/uploads/6019a18f03a4ae003acb1113/blog-llama-2.png">
<h3>Conclusion</h3>
Fine-tuning makes a huge difference in unlocking greater performance from base pre-trained models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. Happy fine-tuning! 🚀

Deploy and fine-tune models using our open-sourced LLM Engine Repo. Learn how in our latest blog. 

Jihan Yin

How to Fine-Tune Llama 2 With LLM Engine