techRxiv_2021.pdf (694.07 kB)
Download fileArtificial Neural Network Activation Functions in Exact Analytical Form (Heaviside, ReLU, PReLU, ELU, SELU, ELiSH)
Activation functions are fundamental elements in artificial neural
networks. The mathematical formulation of some activation functions (e.g.
Heaviside function and Rectified Linear Unit function) are not expressed in an
explicit closed form. This made them numerically unstable and computationally
complex during estimation. This paper introduces a novel explicit analytic form
for those activation functions. The proposed mathematical equations match
exactly the original definition of the studied activation function. The
proposed equations can be adapted better in optimization, forward and backward
propagation algorithm employed in an artificial neural network.
History
Email Address of Submitting Author
khatibrh@hotmail.comSubmitting Author's Institution
Rafik Hariri UniversitySubmitting Author's Country
- Lebanon