Fractal properties of neural networks
Abstract
The work is devoted to the research of neural networks’ properties, which have been extremely intensively used in various applied directions recently. The study of their general and fundamental properties is becoming more and more actual due to their wide application.
The key goal of the work is to investigate the reaction field of an artificial neural network in the space of all possible input signals of a certain length. Based on the example of a simple perceptron, zones where the reaction field of the neural network has a structurally complex nature are studied.
Research methods: To research an output signal field, software was developed, which allowed modeling and visualization of the output signal field over the space of all input signals. The software also allowed changing of activation functions, weights, and thresholds of each neuron, which made it possible to research the influence of all these factors on the structural complexity of the output signal field.
As a result, the study established that, in general, within the space of input signals, there are shadow zones where the response field of the neural network has a self-similar fractal structure. Conditions for the appearance of symmetry of such structures were determined, the influence of activation functions, weights and thresholds of network neurons on the properties of fractal structures was investigated. It was revealed that the input layer of neurons predominantly influences these properties. Dependences of the fractal dimension of the structures on the neuron weights were obtained. Changes occurring with the increase in the dimensionality of the input signal space were discussed.
The presence of shadow zones with a fractal output signal field is important for understanding the functioning of artificial neural networks. Such shadow zones define regions within the input signal space where the neural network’s response is extremely sensitive even to minute changes in input signals. This sensitivity leads to a fundamental change in output signals with a slight change in input signals.
Downloads
References
/References
W. S. McCulloch and W. Pitts, "A logical calculus of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5, no. 4, pp. 115–133, 1943. https://doi.org/10.1007/BF02478259
F. Rosenblatt, "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain," Psychological Review, vol. 65, no. 6, pp. 386–408, 1958. https://doi.org/10.1037/h0042519
F. Rosenblatt, Principles of Neurodynamics. Washington, DC: Spartan Books, 1962. https://doi.org/10.2307/1419730
Olazaran, Mikel, “A Sociological Study of the Official History of the Perceptrons Controversy,” Social Studies of Science, vol. 26, no. 3, p.611-659, 1996. https://doi.org/10.1177/030631296026003005
M. L. Minsky and S. A. Papert, Perceptrons. Cambridge, MA: MIT Press, 1969. https://doi.org/10.1016/s0361-9230(99)00182-3
J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proceedings of National Academy of Sciences, vol. 79, no. 8, pp. 2554–2558, April 1982. https://doi.org/10.1073%2Fpnas.79.8.2554
J. J. Hopfield, "Neural with graded response have collective computational properties like those of two-state neurons," Proceedings of the National Academy of Sciences, vol. 81, no. 10, pp. 3088-3092, May 1984. https://doi.org/10.1073/pnas.81.10.3088
J. J. Hopfield, "Learning algorithms and probability distributions in feed-forward and feed-back networks," Proceedings of the National Academy of Sciences, vol. 84, no. 23, pp. 8429-8433, December 1, 1987. https://doi.org/10.1073/pnas.84.23.8429
Freund, Y.; Schapire, R. E., "Large margin classification using the perceptron algorithm," Machine Learning, vol. 37, no. 3, pp. 277-296, 1999. https://doi.org/10.1023/A:1007662407062
Mehryar Mohri1, Afshin Rostamizadeh, “Stability Bound for Stationary Phi-mixing and Beta-mixing Processes,” Journal of Machine Learning Research (JMLR), vol. 11, pp. 798-814, 2010. https://doi.org/10.48550/arXiv.0811.1629
Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N. et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012. http://dx.doi.org/10.1109/MSP.2012.2205597
Jurafsky D., Martin J.H., Speech and language processing, 2nd edition. NJ: Prentice Hall, 2008.
Shinde, B. S., Dani, A. R., "The origins of digital image processing and application areas in digital image processing medical images," IOSR Journal of Engineering, vol. 1, no. 1, pp. 066-071, 2012. http://dx.doi.org/10.9790/3021-0116671
He, K., Zhang, X., Ren, S., Sun, J., "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. http://dx.doi.org/10.1109/CVPR.2016.90
Schmidhuber, J., "Deep learning in neural networks: An overview," Neural networks, vol. 61, no. 3, pp. 85-117, 2015. http://dx.doi.org/10.1016/j.neunet.2014.09.003
Benoit B. Mandelbrot, The Fractal Geometry of Nature. San Francisco: W.H.Freeman and Co, 1982.
Richard M. Crownover, Introduction to Fractals and Chaos. University of Missouri-Columbia: Jones and Bartlett Publishers, 1995.
Takens, F., “Detecting strange attractors in turbulence,” Dynamical Systems and Turbulence, vol. 898, pp. 366-381, 1981. https://doi.org/10.1007/BFb0091924
W. S. McCulloch and W. Pitts, "A logical calculus of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5, no. 4, pp. 115–133, 1943. https://doi.org/10.1007/BF02478259
F. Rosenblatt, "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain," Psychological Review, vol. 65, no. 6, pp. 386–408, 1958. https://doi.org/10.1037/h0042519
F. Rosenblatt, Principles of Neurodynamics. Washington, DC: Spartan Books, 1962. https://doi.org/10.2307/1419730
Olazaran, Mikel, “A Sociological Study of the Official History of the Perceptrons Controversy,” Social Studies of Science, vol. 26, no. 3, p.611-659, 1996. https://doi.org/10.1177/030631296026003005
M. L. Minsky and S. A. Papert, Perceptrons. Cambridge, MA: MIT Press, 1969. https://doi.org/10.1016/s0361-9230(99)00182-3
J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proceedings of National Academy of Sciences, vol. 79, no. 8, pp. 2554–2558, April 1982. https://doi.org/10.1073%2Fpnas.79.8.2554
J. J. Hopfield, "Neural with graded response have collective computational properties like those of two-state neurons," Proceedings of the National Academy of Sciences, vol. 81, no. 10, pp. 3088-3092, May 1984. https://doi.org/10.1073/pnas.81.10.3088
J. J. Hopfield, "Learning algorithms and probability distributions in feed-forward and feed-back networks," Proceedings of the National Academy of Sciences, vol. 84, no. 23, pp. 8429-8433, December 1, 1987. https://doi.org/10.1073/pnas.84.23.8429
Freund, Y.; Schapire, R. E., "Large margin classification using the perceptron algorithm," Machine Learning, vol. 37, no. 3, pp. 277-296, 1999. https://doi.org/10.1023/A:1007662407062
Mehryar Mohri1, Afshin Rostamizadeh, “Stability Bound for Stationary Phi-mixing and Beta-mixing Processes,” Journal of Machine Learning Research (JMLR), vol. 11, pp. 798-814, 2010. https://doi.org/10.48550/arXiv.0811.1629
Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N. et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012. http://dx.doi.org/10.1109/MSP.2012.2205597
Jurafsky D., Martin J.H., Speech and language processing, 2nd edition. NJ: Prentice Hall, 2008.
Shinde, B. S., Dani, A. R., "The origins of digital image processing and application areas in digital image processing medical images," IOSR Journal of Engineering, vol. 1, no. 1, pp. 066-071, 2012. http://dx.doi.org/10.9790/3021-0116671
He, K., Zhang, X., Ren, S., Sun, J., "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. http://dx.doi.org/10.1109/CVPR.2016.90
Schmidhuber, J., "Deep learning in neural networks: An overview," Neural networks, vol. 61, no. 3, pp. 85-117, 2015. http://dx.doi.org/10.1016/j.neunet.2014.09.003
Benoit B. Mandelbrot, The Fractal Geometry of Nature. San Francisco: W.H.Freeman and Co, 1982.
Richard M. Crownover, Introduction to Fractals and Chaos. University of Missouri-Columbia: Jones and Bartlett Publishers, 1995.
Takens, F., “Detecting strange attractors in turbulence,” Dynamical Systems and Turbulence, vol. 898, pp. 366-381, 1981. https://doi.org/10.1007/BFb0091924
