Highway Layer

Variant of the fully connected Linear layer, with an additional gated residual connection

It produces a standard mapping with a non-linear activation function $y = g (W_{1} x + b_{1})$

\begin{aligned} z & = t ⊙ g (W_{1} x + b_{1}) + (1 - t) ⊙ x \\ t & = σ (W_{2} x + b_{2}) \end{aligned}

Examples:

# Blocks to be used for the highway connection
self.highway = nn.Linear(128, 128)
self.transform = nn.Linear(128, 128)

# Some Highway Layers
h = self.highway(x)
t_gate = torch.sigmoid(self.transform(x))
c_gate = 1 - t_gate
x_ = h * t_gate + x * c_gate