Fine-tuning (machine learning)

In machine learning, fine-tuning is an approach to transfer learning in which the weights of a pre-trained model are trained on new data.[1] Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen" (not updated during the backpropagation step).[2]

For some architectures, such as convolutional neural networks, it is common to keep the earlier layers (those closest to the input layer) frozen because they capture lower-level features, while later layers often discern high-level features that can be more related to the task that the model is trained on.[2][3]

Fine-tuning is common in natural language processing (NLP), especially in the domain of language modeling. Large language models like OpenAI's GPT-2 can be fine-tuned on downstream NLP tasks to produce better results than the pre-trained model can normally achieve.[4] Models that are pre-trained on large and general corpora are usually fine-tuned by reusing the model's parameters as a starting point and adding a task-specific layer trained from scratch.[5] Fine-tuning the full model is common as well and often yields better results, but it is more computationally expensive.[4] Full fine-tuning is also more prone to overfitting and may cause the model to perform worse on data outside of the distribution of training data used during finetuning.[6]

Fine-tuning is typically accomplished with supervised learning, but there are also techniques to fine-tune a model using weak supervision.[7] Reinforcement learning is also used to fine-tune language models like ChatGPT (a fine-tuned version of GPT-3) and Sparrow by means of reinforcement learning from human feedback.[8][9]

Low-Rank Adaptation (LoRA)[10] trains low-rank matrices ("update matrices") that adds to existing weights. The basic idea is as follows: Suppose we have a ${\displaystyle N\times N}$ matrix ${\displaystyle A}$ in the model, where ${\displaystyle N}$ is large, then we can either modify ${\displaystyle A}$ itself into some ${\displaystyle A'}$, or define ${\displaystyle A':=A+VW^{T}}$, and train ${\displaystyle V,W}$. Here ${\displaystyle V,W}$ are of size ${\displaystyle N\times r}$, and ${\displaystyle r\ll N}$ is the "low-rank" of the update matrix ${\displaystyle VW^{T}}$.

LoRA is often used for language models, but it can also be used for image models.[11]