LLM FineTuning

Pre-trained large language models (LLM) boast impressive capabilities like text generation, summarization, and coding, they aren't universally apt for every task. There will be instances where an LLM might fall short in fulfilling a specific requirement.

For such challenges, fine-tuning the LLM is a viable recourse. Essentially, fine-tuning involves retraining the base model using new data. While it's a potent method, it's also intricate and might not be the immediate remedy to consider due to its complexity and potential costs. However, for organizations leveraging LLMs, it remains an essential strategy to be aware of.

Understanding the intricacies of fine-tuning, even if you're not directly implementing it, can guide informed decisions when working with large language models.

LLM fine-tuning techniques

Fine-tuning LLMs isn't a one-size-fits-all process; different methods serve distinct purposes. Sometimes, the goal is to adapt a model for a new function. For instance, if you have an LLM adept at generating text and you aim to harness it for tasks like sentiment analysis or topic categorization, you'll need to modify its architecture slightly before the fine-tuning process.

In such scenarios, you'll primarily utilize the embeddings generated by the transformer component of the model. These embeddings are numerical representations that encapsulate the various attributes of the input. While certain language models directly produce embeddings, others, like those in the GPT series, employ these embeddings to generate textual tokens.

To adapt the model, you'll link its embedding layer to a classification module, such as a series of fully connected layers, to transform the embeddings into class likelihoods. Here, the training focuses on the classifier, using the embeddings produced by the model. The attention layers of the LLM remain static and don't require updates, leading to substantial computational savings. Yet, the classifier's training necessitates a supervised dataset comprising text samples and their respective classes. The dataset's magnitude is contingent upon the task's intricacy and the classification component.

However, there are situations where modifying the transformer model's parameter weights becomes imperative. This involves activating the attention layers and executing a comprehensive fine-tuning across the model, a process that can be both resource-intensive and complex, especially with larger models. Occasionally, to mitigate fine-tuning expenses, parts of the model remain static. There exist multiple strategies to economize the fine-tuning of LLMs, which will be elaborated on subsequently.

However, in some cases, updating the knowledge of the model is not enough and you want to modify the behavior of the LLM. In these situations, you will need a supervised fine-tuning (SFT) dataset, which is a collection of prompts and their corresponding responses. SFT datasets can be manually curated by users or generated by other LLMs. Supervised fine-tuning is especially important for LLMs such as ChatGPT, which have been designed to follow user instructions and stay on a specific task across long stretches of text. This specific type of fine-tuning is also referred to as instruction fine-tuning.

Unsupervised vs. Supervised Fine-Tuning (SFT)

At times, the primary goal is to enhance an LLM's existing knowledge. For instance, adapting the model to understand specialized medical texts or to grasp a new language is often desired. In such instances, leveraging unstructured datasets, like articles and research papers from medical periodicals, proves beneficial. The objective is to immerse the model in sufficient tokens that encapsulate the essence of the new field or the anticipated input for its designated function.

One of the notable benefits of using unstructured data is its scalability. Models can undergo training using unsupervised or self-supervised techniques. A vast majority of foundational models are nurtured on these unstructured datasets, often teeming with hundreds of billions of tokens. Procuring this kind of data to adapt the model to a novel area can be fairly straightforward, more so when organizations have access to internal databases and documentation.

Reinforcement Learning from Human Feedback (RLHF)

Elevating SFT or directive fine-tuning, certain enterprises employ reinforcement learning derived from human feedback (RLHF). This approach, though potent, is intricate and costly. It entails enlisting human evaluators and establishing auxiliary models for the fine-tuning process. As a result, currently, RLHF is predominantly within the reach of organizations or AI research centers with substantial technical and monetary prowess.

While multiple methods exist to implement RLHF, the core principle remains consistent: Training an LLM on a vast number of tokens produces text sequences that are contextually coherent. However, this generated text might not align with the user's needs or the application's demand. This is where RLHF plays a pivotal role, integrating human judgment to guide the LLM. Human evaluators assess the model's output based on certain prompts. These assessments then serve as cues, refining the model to produce highly-rated results.

A prominent instance of RLHF in action is ChatGPT. OpenAI adapted this model, drawing inspiration from their InstructGPT methodology. The initial step involved adapting a GPT-3.5 model using SFT, anchored on a collection of crafted prompts and responses. Subsequently, human evaluators were engaged to assess the model's output for diverse prompts. This human-generated feedback was utilized to devise a reward model, aiming to replicate human preferences. The final stage witnessed the language model undergoing an intensive reinforcement learning cycle. Here, the LLM formulates outputs, the reward model evaluates them, and the LLM adjusts its parameters to optimize its rewards.

Parameter-efficient fine-tuning (PEFT)

A captivating segment of research in LLM fine-tuning focuses on minimizing the expenses associated with altering the model parameters. This objective is captured by parameter-efficient fine-tuning (PEFT), an array of strategies aiming to decrease the parameters requiring modifications.

Several PEFT approaches exist, with low-rank adaptation (LoRA) emerging as a favored method, particularly in the open-source language model community. The premise of LoRA is rooted in the understanding that adapting a foundational model for a specific task doesn't necessitate the alteration of all its parameters. Instead, a compact matrix can aptly mirror the dimension of the specific task with exceptional precision.

Employing LoRA for fine-tuning emphasizes training this concise matrix rather than tweaking the primary LLM's parameters. The parameter weights derived from the LoRA model are subsequently merged with the core LLM or appended during the deduction process. Impressively, LoRA can slash fine-tuning expenditures by a whopping 98%. Additionally, it provides the convenience of housing several refined mini-models, which can be effortlessly integrated into the LLM as needed.