Continuous Bag of Words (CBoW)

Predicts a middle word giving surrounding context words, i.e. predict word with in the middle with index $k$ of a sequence of $2k+1$ words.

Use a sliding window with size $k$ to generate pairs of $(x,y)$ values, where $x=2k$ words and $y=$ middle word

Input: Start with one-hot vectors $e_i\in {\mathbb{R}}^{|V|}$ for $2k$ context words, with $k$ as the span of the model

Embedding Layer:

• Add a shared linear transformation $W\in {\mathbb{R}}^{D\times |V|}$ of size $D\times |V|$ applied to each input one-hot vector $e_i$.
• We get a dense vector $v_i$ of size $D$, the size of the new word embedding, often chosen as $D\ll |V|$
• $\mathrm{\forall }i,v_i=W{e}_i$
• This layer learns the word embeddings.

Context Aggregation Layer:

• Sums all vectors $u=\sum _k{v}_k$, which combines the contextual meaning vectors together
• Non-trainable layer, which produces $u\in {\mathbb{R}}^D$
• Roughly encapsulates the meaning of the entire incomplete sentence of $2k$ input words

Prediction Layer:

• Final linear layer (matrix $R$, size $|V|\times D$) followed by a softmax function
• Produces an output $o=softmax(R\times u)$ with $R\in {\mathbb{R}}^{|V|\times D}$ so that $o\in {\mathbb{R}}^{|V|}$
• Predicts probability distribution over the entire vocabulary $V$ for the missing word

Training:

• Use a sliding window over a large text corpus to generate (context words, middle word) pairs
• Classification loss is preferably a negative logarithm: $$L_t = -\log(o_t = e_t|e_{-t})$$ where $e_{-t}=(e_{t-k},...e_{t-1},e_{t+1},...,e_{t+k})$

This and SkipGram are referred to as Word2Vec approaches.