Activation functions

They are an additional function $A=s(Z)$ that occurs after a linear operation $Z=WX+B$ to introduce non-linearity to models.

Examples

• Sigmoid function
• tanh function

These two activation functions are often used in binary classification models, are reminiscent of Logistic regression and are simply differentiable

However, as the outputs are squeezed between $[0,1]$ or $[-1,1]$, training deep networks with them would cause Vanishing gradients

• ReLU function
• Leaky ReLU function

To allow for multi-class classification:

• Softmax function

Out-of-syllabus

• ELU function
• Swish function
• Mish function
• Gated Linear Unit