Introduction
Welcome to the Noob series, where we simplify complex concepts that people often search for but find difficult to grasp due to intricate mathematics and technical jargon. If you’ve come across the term ‘fine-tuning’ in the context of Large Language Models (LLMs), you’re in the right place. Though fine-tuning has long been a part of traditional machine learning, its prominence has surged with the advent of LLMs. These pre-trained models provide a versatile foundation that you can adapt to suit your specific tasks, needs, and style. This adaptation process is known as fine-tuning. However, to fully comprehend fine-tuning, it’s essential to first understand “pre-training.” Fine-tuning is essentially the process of “tuning” an existing pre-trained model. Let’s delve into these concepts so that you’ll be well-equipped to discuss them in the future.
What is pre-training?
Imagine starting with a newly created model, equipped with millions or billions of randomly assigned parameters. If you attempt to teach it a specific task, such as classifying movies into categories, it would need to learn the entire English language from scratch using a limited dataset. This is akin to teaching biology to a toddler who hasn’t yet grasped language or basic scientific concepts.
Pre-training addresses this challenge by allowing the model to learn complex and general concepts from vast amounts of data. This stage demands substantial computational power and data. Once pre-trained, the model possesses an understanding of language. During pre-training, the model learns a simple skill: predicting the next word. By presenting a piece of text with the next word hidden, the model guesses what comes next. Accurate predictions result in minimal loss, while inaccurate ones cause significant loss, prompting the model to adjust.
For instance, in the diagram above, if given the sentence “The cat sat on the ____”, the model learns that “carpet” is more likely than “car”. Repeating this training across billions of sentences, books, and articles enables the model to become proficient at predicting the next word, absorbing grammar, facts, reasoning patterns, and more. After pre-training, the model has a foundational understanding of language, serving as a platform for subsequent tasks. This is why they are referred to as foundation models.
In practice, you rarely perform pre-training yourself. Instead, you download pre-trained models like Llama, Mistral, or Qwen and build upon them. This leads us to our primary focus: fine-tuning.
What is fine tuning?
A common misconception among beginners is that once a model is trained, its weights are permanently fixed. In reality, a pre-trained model’s weights are set to “good values” that encode intelligence and perform well on general tasks. With a pre-trained model, you can customize this intelligence to meet your specific needs using task-specific data—this process is known as “fine-tuning.” The data requirements for fine-tuning are significantly lower than for pre-training, as you only need examples for the task at hand.
Fine-tuning is akin to chefs receiving general culinary training at a school, and then acquiring restaurant-specific skills when they join a restaurant. Since you’re not starting from scratch, it requires less effort—similar to how training someone already familiar with basic cooking is easier than training someone new to the kitchen. The diagram below illustrates the difference between pre-training and fine-tuning.
How does the fine tuning work?
We’ve already explored next token prediction and the pre-training process. Now, let’s examine the fine adjustment loop.
The process involves presenting the model with task-specific data, such as a movie, and asking it to categorize the movie. The model’s response is compared to the ideal response, and adjustments are made to the weights, repeating the process until the model improves at the downstream task. Two key differences in fine-tuning compared to pre-training are:
- Data → Small, high-quality, task-specific data instead of the entire Internet.
- Learning rate → A low learning rate and few passes, to enable the model to adapt without losing its general skills.
Two common types of fine tuning
Fine-tuning can be categorized into two types, depending on the number of model parameters you want to adjust or adapt:
- Complete focus: In this approach, every parameter in the model can change. By running the loop, the billions of parameters move slightly toward your task. The main challenge with this method is memory: you need enough to maintain and update the entire model, which requires substantial hardware for large LLMs. There’s also a risk of catastrophic forgetting, where the model becomes proficient in a specific task but loses its general abilities.
- Efficient Parameter Fine Tuning (PEFT): Instead of updating each weight in the network, PEFT techniques freeze the base model and introduce a small set of new trainable parameters, training only those. Various techniques like LoRA, QLoRA, and prompt tuning exist, though their details are beyond this article’s scope. PEFT requires less memory and training time, with a reduced risk of losing acquired knowledge. For most LLM fine-tuning, this is the default choice.
Is fine tuning always the solution?
While fine-tuning is effective for teaching models new skills, styles, or tasks, it’s not always the best or first choice. Sometimes, crafting a better prompt can address your problem without additional training. Similarly, when real-time information retrieval is necessary, Retrieval Augmented Generation (RAG) is a better solution, especially for large or frequently changing facts. These approaches aren’t competitors; in practice, most systems integrate them. It’s crucial to consider these options before committing to a fine-tuning phase.
Additional Resources
If you’re interested in fine-tuning with LoRA, consider these resources:
- Cuddly face PEFT: The leading open-source library for LoRA, QLoRA, and prompt tuning. Start with the documentation and examples.
- TRL Cuddly Face: Integrates with PEFT and provides an SFTTrainer for the supervised fine-tuning loop.
- Carefree: A beginner-friendly path to LoRA/QLoRA, with free Colab and Kaggle notebooks, approximately 2x faster training, and lower VRAM usage.
- Axolotl: Once you’re comfortable, this popular configuration-based (YAML) tool allows you to run fine-tuning pipelines without extensive coding.
- The original LoRA paper: “LoRA: Low-rank adaptation of large language models.”
- The QLoRA document: “QLoRA: effective development of LLMs.”
For a practical project, take a small instructional model like an 8B Llama, Qwen, or Gemma, open an Unsloth QLoRA notebook, and fine-tune it with a few hundred clear examples of your task. Observe the training loss decline. Once you’ve done this, the concepts discussed in this article will become more tangible.
Kanwal Mehreen is a machine learning engineer and technical writer with a passion for data science and the intersection of AI and medicine. She co-authored the ebook “Maximizing Productivity with ChatGPT.” As a 2022 Google Generation Scholar for APAC, she champions diversity and academic excellence. Kanwal is also recognized as a Teradata Diversity in Tech Fellow, Mitacs Globalink Research Fellow, and Harvard WeCode Fellow. She is a strong advocate for change, having founded FEMCodes to empower women in STEM fields.
For more information, visit the original article Here.
“`
