POLYSIGMOID: UNA FUNCIÓN DE ACTIVACIÓN POLINOMIAL PARA REDES NEURONALES (POLYSIGMOID: A POLYNOMIAL ACTIVATION FUNCTION FOR NEURAL NETWORKS)
Resumen
En este artículo se propone una nueva función de activación que aproxima la función de activación Sigmoid, la cual es ampliamente usada en redes neuronales artificiales. El interés de aproximar a la función Sigmoid es para reducir la evaluación de un gran número de términos polinomiales (teóricamente infinito) lo cual genera un alto costo computacional. El enfoque es aproximarla mediante un polinomio a trozos de orden finito llamado PolySigmoid. La función PolySigmoid contempla dos parábolas para aproximar la parte no lineal de la función Sigmoid que es continua, creciente y acotada. Adicionalmente, dada la relación que hay entre las funciones Sigmoid y Tanh (tangente hiperbólica) también es posible proponer una función PolyTanh que aproxima la función Tanh, y esto da pauta a aproximar otras funciones de activación. Experimentalmente, se valora la aproximación de ambas funciones PolySigmoid y PolyTanh mediante resultados de clasificación con la base de datos MNIST.
Palabras Clave: Función de activación, Redes neuronales artificiales, Clasificación.
Abstract
In this paper, a novel activation function is proposed as an approximation to the Sigmoid activation function. The Sigmoid function is widely used in neural networks, and its approximation is to reduce the evaluation of a large number (theoretically infinite) of polynomial terms which generates a high computational cost. The Sigmoid activation function is approximated with a piecewise polynomial called PolySigmoid. The PolySigmoid function considers two parabolas to approximate the non-linear sections since it is a continuous, monotone, and non-decreasing function. Additionally, there is a relationship between the Sigmoid and Tanh functions, then it is possible to propose another function PolyTanh to approximate the Tanh function, which opens the possibility to approximate other activation functions. Experimentally, the approximations of PolySigmoid and PolyTanh are valuated through the classification results of a neural network trained with the MNIST dataset.
Keywords: Activation function, Artificial neural networks, Classification.
Texto completo:
PDFReferencias
Agostinelli, F. Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830, 2014.
Bengio, Y. Deep Learning. MIT Press, 2016.
Catalbas, B., Morgül, Ö. Deep learning with ExtendeD Exponential Linear Unit (DELU). Neural Computing and Applications, No.35(30), 22705-22724, 2023.
Cybenko, G. Approximation by superpositions of a sigmoidal function. Math-ematics of Control, Signals, and Systems, No. 2(4), 303-314, 1989.
Davies, B. Integral Transforms and their Applications. Springer, No. 28, 2002.
Haykin, S. Neural networks and learning machines. Pearson Prentice Hall, 2008.
Hendrycks, D., Gimpel, K. Gaussian error linear units (GELUs). Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Es-paña, 5-10, 2016.
Herrera-Alcántara, O., Arellano-Balderas, S. Adaptive Morphing Activation Function for Neural Networks. Fractal Fract., No. 8(8), 444, 2024.
Kingma, D., Ba, J. Adam: A Method for Stochastic Optimization. 3rd Interna-tional Conference for Learning Representations, San Diego, CA, 2015.
Kumar, R. APTx: better activation function than MISH, SWISH, and ReLU's variants used in deep learning. arXiv preprint arXiv:2209.06119, 2022.
Lippmann, R. An Introduction to Computing with Neural Nets. ASSP Maga-zine, IEEE, 1987.
Misra, D. Mish: A Self Regularized Non-Monotonic Neural Activation Func-tion. arXiv:1908.08681, 2019.
Narayan, S. The generalized sigmoid activation function: Competitive super-vised learning. Information Sciences, No. 99(1–2), 69-82, 1997.
Qiumei, Z., Dan, T., Fenghua, W. Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function. IEEE Access No. 7, 151359-151367, 2019.
Ramachandran, P., Zoph, B., Le, Q.V. Searching for Activation Functions. arXiv:1710.05941v2, 2017.
Rosenblatt, F. Principles of Neurodynamics. Spartan, Washington, DC, 1961.
Zhou, Y., Li, D., Huo, S., Kung, S. Shape autotuning activation function. Ex-pert Syst. Appl. No. 171, 114534, 2020.
URL de la licencia: https://creativecommons.org/licenses/by/3.0/deed.es
Pistas Educativas está bajo la Licencia Creative Commons Atribución 3.0 No portada. 
TECNOLÓGICO NACIONAL DE MÉXICO / INSTITUTO TECNOLÓGICO DE CELAYA
Antonio García Cubas Pte #600 esq. Av. Tecnológico, Celaya, Gto. México
Tel. 461 61 17575 Ext 5450 y 5146
pistaseducativas@itcelaya.edu.mx