In-context learning (natural language processing)

In natural language processing, in-context learning, few-shot learning or few-shot prompting is a prompting technique that allows a model to process examples before attempting a task.[1][2] The method was popularized after the advent of GPT-3[3] and is considered to be an emergent property of large language models.[4]

A few-shot prompt normally includes n examples of (problem, solution) pairs known as "shots", with the overall usage of such a prompt being known as n-shot prompting.[5][6] For instance, the following is a one-shot prompt for review sentiment classification: Review: This movie sucks. Sentiment: negative. Review: I love this movie. Sentiment: If the model outputs "positive", then it has correctly solved the task.[4]

The term zero-shot prompting is often used to signify that no examples are provided.[7][8][9] An example of a zero-shot prompt for a question-answering task would be "Who wrote the book On the Origin of Species?".

In-context learning was initially proposed as an alternative to fine-tuning a pre-trained language model on a task-specific dataset.[3] The term is misleading as in fact no parameters are changed in the model and as a result no actual learning takes place. Instead the prompt primes the model for subsequent inference. The main advantages of in-context learning over fine-tuning are a reduction in the amount of task-specific data needed and a reduced potential of overfitting by learning an overly narrow distribution from a large but narrow fine-tuning dataset.[3] Few-shot performance of large language models has been shown to achieve competitive results on NLP tasks, sometimes surpassing prior state-of-the-art fine-tuning approaches.[3][10] Examples of such NLP tasks are translation, question answering, cloze tasks, unscrambling words, and using a novel word in a sentence. The creation and optimization of such few-shot prompts is part of the now active field of study of prompt engineering.[11][12]

While few-shot prompting has performed competitively when compared to fine-tuned models, it has its own drawbacks. For example, it has been shown that the order in which the shots are listed can make the difference between state-of-the-art and random guess performance. A set of few-shot examples that works well in some specific order with one model may not work at all when used with a different model.[13] Despite these shortcomings, the commonly used Transformer model can encode principled learning algorithms based on gradient descent inside their weights and enable mesa-optimization[14] i.e. learn-to-learn small models based on the data given in-context when making predictions.[15][16][17][18]

A common example of in-context learning is chain-of-thought prompting, where few-shot examples are given to teach the model to output a string of reasoning before attempting to answer a question.[19] This technique has been shown to improve performance of models in tasks that require logical thinking and reasoning.[20]