Gated Recurrent Units

A simpler alternative to Long Short-Term Memory, which combines their forget and input gates to an Update Gate, and merges cell state and hidden state

Hidden State $h_{t}$ - the only state vector

Gates (sigmoid layers):

Update Gate $z_{t} = σ (W_{z} x_{t} + U_{z} h_{t - 1} + b_{z})$
Reset Gate $r_{t} = σ (W_{r} x_{t} + U_{r} h_{t - 1} + b_{r})$

Candidate Hidden State - could replace $h_{t - 1}$ and become the next memory vector $h_{t}$

{\tilde{h}}_{t} = \tanh (W_{h} x_{t} + U_{h} (r_{t} ⊙ h_{t - 1} + b_{h})

where $r_{t} = 0$ means the previous hidden state will be entirely discarded, and 1 means entirely kept

New Hidden State - will replace $h_{t - 1}$ by combining proportion of $h_{t - 1}$ and $\tilde{h_{t}}$

h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ \tilde{h_{t}}

where $⊙$ is the element wise multiplication, $z_{t} = 1$ means new hidden state is entirely based on candidate hidden vector $\tilde{h_{t}}$ , and 0 means entirely based on previous hidden state $h_{t - 1}$

Predicted output - defined as simple linear operation with a weight matrix $V$

y_{t} = V h_{t}