Artificial Neural Network Activation Functions in Exact Analytical Form (Heaviside, ReLU, PReLU, ELU, SELU, ELiSH)

Activation functions are fundamental elements in artificial neural networks. The mathematical formulation of some activation functions (e.g. Heaviside function and Rectified Linear Unit function) are not expressed in an explicit closed form. This made them numerically unstable and computationally complex during estimation. This paper introduces a novel explicit analytic form for those activation functions. The proposed mathematical equations match exactly the original definition of the studied activation function. The proposed equations can be adapted better in optimization, forward and backward propagation algorithm employed in an artificial neural network.