TechRxiv
techRxiv_2021.pdf (694.07 kB)
Download file

Artificial Neural Network Activation Functions in Exact Analytical Form (Heaviside, ReLU, PReLU, ELU, SELU, ELiSH)

Download (694.07 kB)
preprint
posted on 2021-08-06, 02:30 authored by Rami AlkhatibRami Alkhatib
Activation functions are fundamental elements in artificial neural networks. The mathematical formulation of some activation functions (e.g. Heaviside function and Rectified Linear Unit function) are not expressed in an explicit closed form. This made them numerically unstable and computationally complex during estimation. This paper introduces a novel explicit analytic form for those activation functions. The proposed mathematical equations match exactly the original definition of the studied activation function. The proposed equations can be adapted better in optimization, forward and backward propagation algorithm employed in an artificial neural network.

History

Email Address of Submitting Author

khatibrh@hotmail.com

Submitting Author's Institution

Rafik Hariri University

Submitting Author's Country

  • Lebanon

Usage metrics

    Licence

    Exports