Language model

Maximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is

$${\displaystyle P(w_{m}\mid w_{1},\ldots ,w_{m-1})={\frac {1}{Z(w_{1},\ldots ,w_{m-1})}}\exp(a^{T}f(w_{1},\ldots ,w_{m}))}$$

where ${\displaystyle Z(w_{1},\ldots ,w_{m-1})}$ is the partition function, ${\displaystyle a}$ is the parameter vector, and ${\displaystyle f(w_{1},\ldots ,w_{m})}$ is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on ${\displaystyle a}$ or some form of regularization.

The log-bilinear model is another example of an exponential language model.

Neural language models (or continuous space language models) use continuous representations or embeddings of words to make their predictions.[10] These models make use of neural networks.

Continuous space embeddings help to alleviate the curse of dimensionality in language modeling: as language models are trained on larger and larger texts, the number of unique words (the vocabulary) increases.[lower-alpha 1] The number of possible sequences of words increases exponentially with the size of the vocabulary, causing a data sparsity problem because of the exponentially many sequences. Thus, statistics are needed to properly estimate probabilities. Neural networks avoid this problem by representing words in a distributed way, as non-linear combinations of weights in a neural net.[11] An alternate description is that a neural net approximates the language function. The neural net architecture might be feed-forward or recurrent, and while the former is simpler the latter is more common.

Typically, neural net language models are constructed and trained as probabilistic classifiers that learn to predict a probability distribution

$${\displaystyle P(w_{t}\mid \mathrm {context} )\,\forall t\in V.}$$

That is, the network is trained to predict a probability distribution over the vocabulary, given some linguistic context. This is done using standard neural net training algorithms such as stochastic gradient descent with backpropagation.[11] The context might be a fixed-size window of previous words, so that the network predicts

$${\displaystyle P(w_{t}\mid w_{t-k},\dots ,w_{t-1})}$$

from a feature vector representing the previous k words.[11] Another option is to use "future" words as well as "past" words as features,[12] so that the estimated probability is

$${\displaystyle P(w_{t}\mid w_{t-k},\dots ,w_{t-1},w_{t+1},\dots ,w_{t+k}).}$$

This is called a bag-of-words model. When the feature vectors for the words in the context are combined by a continuous operation, this model is referred to as the continuous bag-of-words architecture (CBOW).[13]

A third option that trains slower than the CBOW but performs slightly better is to invert the previous problem and make a neural network learn the context, given a word.[13] More formally, given a sequence of training words ${\displaystyle w_{1},w_{2},w_{3},\dots ,w_{T}}$, one maximizes the average log-probability

$${\displaystyle {\frac {1}{T}}\sum _{t=1}^{T}\sum _{-k\leq j\leq k,j\neq 0}\log P(w_{t+j}\mid w_{t})}$$

where k, the size of the training context, can be a function of the center word ${\displaystyle w_{t}}$. This is called a skip-gram language model.[14] Bag-of-words and skip-gram models are the basis of the word2vec program.[15]

Instead of using neural net language models to produce actual probabilities, it is common to instead use the distributed representation encoded in the networks' "hidden" layers as representations of words; each word is then mapped onto an n-dimensional real vector called the word embedding, where n is the size of the layer just before the output layer. The representations in skip-gram models have the distinct characteristic that they model semantic relations between words as linear combinations, capturing a form of compositionality. For example, in some such models, if v is the function that maps a word w to its n-d vector representation, then

$${\displaystyle v(\mathrm {king} )-v(\mathrm {male} )+v(\mathrm {female} )\approx v(\mathrm {queen} )}$$

where ≈ is made precise by stipulating that its right-hand side must be the nearest neighbor of the value of the left-hand side.[13][14]

A positional language model[16] assesses the probability of given words occurring close to one another in a text, not necessarily immediately adjacent. Similarly, bag-of-concepts models[17] leverage the semantics associated with multi-word expressions such as buy_christmas_present, even when they are used in information-rich sentences like "today I bought a lot of very nice Christmas presents".

Despite the limited successes in using neural networks,[18] authors acknowledge the need for other techniques when modelling sign languages.