Prospects of using deep learning models for semantic image segmentation on autonomous devices
Abstract
Relevance. The implementation of deep learning models for semantic segmentation on autonomous devices is a promising direction for the development of intelligent systems capable of analyzing visual information without constant connection to external resources. This enables the creation of more autonomous and efficient systems that can operate in real-time and under resource constraints. Such an approach is highly significant for various industries, including robotics, autonomous vehicles, medical diagnostics, and other fields where high accuracy and speed of image processing are required.
Goal. The goal of this work is to explore the possibilities and challenges of using deep learning models for semantic segmentation on autonomous devices. This includes analyzing the efficiency of the models, their adaptation to the limited resources of the devices, and developing methods to ensure the security of access to the trained models.
Research methods. The research methods include theoretical analysis, systematization, and generalization of the use of deep learning models in autonomous devices. Special attention is given to the parameters affecting the memory footprint of the models and the specifics of implementing trained models into proprietary software products. Additionally, modern approaches to encrypting models to ensure their security have been considered.
Results. A comparative analysis of traditional models and deep learning models for semantic segmentation of images has been conducted. Significant potential of deep learning technology for creating autonomous intelligent systems is identified. Various deep learning models currently used for semantic segmentation of images have been reviewed. The impact of key parameters on the efficiency of models on devices with limited resources has been determined, and the role of model size has been considered. Recommendations for implementing trained models into software products are presented, including optimizing models to reduce the size and increase the speed. Special attention has been paid to the analysis of encrypting trained models. It is shown that ensuring the security of access to trained deep learning models for semantic segmentation of images on autonomous devices requires a comprehensive approach that combines hardware and software solutions.
Conclusions. Further developments in the field of deep learning for semantic segmentation on autonomous devices will contribute to the development of more efficient and autonomous systems for a wide range of applications, including computer vision, robotics, and more. Ensuring the security of access to trained deep learning models for semantic segmentation of images on autonomous devices requires a comprehensive approach that combines hardware and software solutions. This not only protects the intellectual property of developers but also ensures the integrity and confidentiality of the data processed by autonomous devices when performing semantic segmentation tasks.
Downloads
References
/References
R. Szeliski, Computer vision: algorithms and applications. Springer Nature, 2022. 925 p. https://link.springer.com/book/10.1007/978-3-030-34372-9.
Hao S., Zhou Y., Guo Y. A brief survey on semantic segmentation with deep learning. Neurocomputing. 2020. V. 406. P. 302-321. https://www.sciencedirect.com/science/article/abs/pii/S0925231220305476.
Guo Y., Liu Y., Georgiou T., Lew M. A review of semantic segmentation using deep neural networks. Int. J. Multimed. Info Retr. 2018. V. 7. P. 87-93. https://link.springer.com/article/10.1007/s13735-017-0141-z.
Yu H., Yang Z., Tan L., Wang Y., Sun W., Sun M., Tang Y. Methods and datasets on semantic segmentation: a review. Neurocomputing. 2018. V. 304. P. 82-103. https://www.sciencedirect.com/science/article/abs/pii/S0925231218304077.
Kutsik A. Ya. Analysis of satellite imagery based on semantic segmentation, bachelor thesis. Igor Sikorsky KPI, 2020. https://ela.kpi.ua/items/5c1bdf70-a6a6-45ab-8a63-312296420f6c. [in Ukrainian]
Ryabko A.V. Analysis and evaluation of satellite image segmentation methods, Ukrainian Scientific and Technical Conference 'Sustainable Development of Communication, Navigation, Surveillance Systems, and Air Traffic Organization CNS/ATM - 2023», November 29-30. P. 4. https://it-visnyk.kpi.ua/?page_id=2165. [in Ukrainian]
Glyboka Yu.O. Investigation of the quality of human image segmentation methods under the influence of additive noise, master thesis, Kharkiv National University of Radio Electronics, 2022. https://openarchive.nure.ua/entities/publication/6bad8449-2d47-4c70-87dc-927980da23e7. [in Ukrainian]
Hua Y., Marcos D., Mou L., Zhu X., Tuia D. Semantic segmentation of remote sensing images with sparse annotations. IEEE Geoscience and Remote Sensing Letters. 2022. V. 19. P. 1-5. https://arxiv.org/abs/2101.03492.
Zhang Y., Chi M. Mask-R-FCN: a deep fusion network for semantic segmentation. IEEE Access. 2020. V. 8. P. 155753-155765. https://ieeexplore.ieee.org/document/9151932.
O’Mahony N., Campbell S., Carvalho A., et al. Deep learning vs. traditional computer vision, Advances in Computer Vision: Proceedings of the 2019 Computer Vision Conference (CVC), Vol. 11, Springer International Publishing, 2020. P. 128-144. https://arxiv.org/abs/1910.13796.
Panella F., Lipani A., Boehm J. Semantic segmentation of cracks: data challenges and architecture. Automation in Construction. 2022. V. 135. P. 104110. https://www.sciencedirect.com/science/article/abs/pii/S0926580521005616.
Mairittha N., Mairittha T., Inoue S. On-device deep learning inference for efficient activity data collection. Sensors (Basel). 2019. V. 19. P. 3434. https://www.mdpi.com/1424-8220/19/15/3434.
Cui T. Review of deep learning and mobile edge computing in autonomous driving. Вісник Львівської політехніки. 2022. V. 12. P. 208-218. https://science.lpnu.ua/sites/default/files/journal-paper/2023/jan/29757/221029maket-210-220.pdf.
Grigorescu S., Trasnea B., Cocias T., Macesanu G. A survey of deep learning techniques for autonomous driving. J. Field Robotics. 2020. V. 37. P. 362-386. https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.21918.
Zhang Z., Li J. A review of artificial intelligence in embedded systems. Micromachines. 2023. V. 14. P. 897. https://www.mdpi.com/2072-666X/14/5/897.
Merone M., Graziosi A., Lapadula V., Petrosino L., d'Angelis O., Vollero L. A practical approach to the analysis and optimization of neural networks on embedded systems. Sensors. 2022. V. 22. P. 7807. https://www.mdpi.com/1424-8220/22/20/7807.
Helms D., Amende K., Bukhari S., et al., Optimizing neural networks for embedded hardware. SMACD/PRIME 2021, International Conference on SMACD and 16th Conference on PRIME, online. 2021. P. 1-6. https://ieeexplore.ieee.org/document/9547911.
Song W. Hardware accelerator systems for embedded systems. Advances in Computers, vol. 122. Elsevier, 2021. P. 23-49. https://www.sciencedirect.com/science/article/abs/pii/S0065245820300917.
Yesuf M., Assefa B. Model compression techniques in deep neural networks. Pan African Conference on Artificial Intelligence. Cham: Springer Nature Switzerland. 2022. P. 169-190. https://link.springer.com/chapter/10.1007/978-3-031-31327-1_10.
Lohn A., Scaling AI. Technical report, Center for Security and Emerging Technology, 2023. https://cset.georgetown.edu/publication/scaling-ai/.
Acun B., Murphy M., Wang X., et al. Understanding training efficiency of deep learning recommendation models at scale. 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE. 2021. https://arxiv.org/abs/2011.05497.
Dong K., Zhou C., Rian Y., Li Y. MobileNetV2 model for image classification. 2nd International Conference on Information Technology and Computer Application (ITCA). IEEE, 2020. P. 476-480. https://ieeexplore.ieee.org/document/9422058.
Berthelier A., Chateau T., Duffner S., et al. Deep model compression and architecture optimization for embedded systems: a survey. J. Signal Processing Systems. 2021. V. 93. P. 863-878. https://link.springer.com/article/10.1007/s11265-020-01596-1.
Hadidi R., Cao J., Xie Y., et al. Characterizing the deployment of deep neural networks on commercial edge devices. IEEE International Symposium on Workload Characterization (IISWC), IEEE. 2019. P. 35-48. https://ieeexplore.ieee.org/document/9041955.
Mavrovouniotis S. Ganley M. Hardware security modules. Secure Smart Embedded Devices, Platforms and Applications. New York, NY: Springer New York, 2013. P. 383-405.
Vembu S. K., Chattopadhyay A., Saha S. Authenticating edge neural network through hardware security modules and quantum-safe key management. 2024 37th International Conference on VLSI Design and 2024 23rd International Conference on Embedded Systems (VLSID), Kolkata, India. 2024. P. 318-323. https://ieeexplore.ieee.org/document/10483401.
Ezirim K., Khoo W., Koumantaris G., et al. Trusted platform module – a survey. The Graduate Center of The City University of New York, 11. 2012. https://www.researchgate.net/profile/Kenneth-Ezirim/publication/287984174_Trusted_Platform_Module_-_A_Survey/links/567af54608ae197583812a7c/Trusted-Platform-Module-A-Survey.pdf.
Köylü T.Ç., Wedig Reinbrecht C.R., Gebregiorgis A., et al. A survey on machine learning in hardware security. ACM Journal on Emerging Technologies in Computing Systems. 2023. V. 19. P. 1-37. https://dl.acm.org/doi/10.1145/3589506.
R. Szeliski, Computer vision: algorithms and applications. Springer Nature, 2022. 925 p. https://link.springer.com/book/10.1007/978-3-030-34372-9.
Hao S., Zhou Y., Guo Y. A brief survey on semantic segmentation with deep learning. Neurocomputing. 2020. V. 406. P. 302-321. https://www.sciencedirect.com/science/article/abs/pii/S0925231220305476.
Guo Y., Liu Y., Georgiou T., Lew M. A review of semantic segmentation using deep neural networks. Int. J. Multimed. Info Retr. 2018. V. 7. P. 87-93. https://link.springer.com/article/10.1007/s13735-017-0141-z.
Yu H., Yang Z., Tan L., Wang Y., Sun W., Sun M., Tang Y. Methods and datasets on semantic segmentation: a review. Neurocomputing. 2018. V. 304. P. 82-103. https://www.sciencedirect.com/science/article/abs/pii/S0925231218304077.
Куцик А.Я. Аналіз супутникових знімків на основі семантичної сегментації, кваліфікаційна робота, КПІ ім. І. Сикорського, 2020. https://ela.kpi.ua/items/5c1bdf70-a6a6-45ab-8a63-312296420f6c.
Рябко А.В. Аналіз та оцінка методів сегментації супутникових зображень, Всеукр. Науково-технічна конференція «Сталий розвиток систем зв’яязку, навігації, спостереження та організації повітряного руху CNS/ATM - 2023», 29-30 листопада 2023 р., с. 4. https://it-visnyk.kpi.ua/?page_id=2165.
Глубока Ю.О. Дослідження якості методів сегментації зображення людини в умовах дії адитивних завад, кваліфікаційна робота, Харківський національний університет радіоелектроніки, 2022. https://openarchive.nure.ua/entities/publication/6bad8449-2d47-4c70-87dc-927980da23e7.
Hua Y., Marcos D., Mou L., Zhu X., Tuia D. Semantic segmentation of remote sensing images with sparse annotations. IEEE Geoscience and Remote Sensing Letters. 2022. V. 19. P. 1-5. https://arxiv.org/abs/2101.03492.
Zhang Y., Chi M. Mask-R-FCN: a deep fusion network for semantic segmentation. IEEE Access. 2020. V. 8. P. 155753-155765. https://ieeexplore.ieee.org/document/9151932.
O’Mahony N., Campbell S., Carvalho A., et al. Deep learning vs. traditional computer vision, Advances in Computer Vision: Proceedings of the 2019 Computer Vision Conference (CVC), Vol. 11, Springer International Publishing, 2020. P. 128-144. https://arxiv.org/abs/1910.13796.
Panella F., Lipani A., Boehm J. Semantic segmentation of cracks: data challenges and architecture. Automation in Construction. 2022. V. 135. P. 104110. https://www.sciencedirect.com/science/article/abs/pii/S0926580521005616.
Mairittha N., Mairittha T., Inoue S. On-device deep learning inference for efficient activity data collection. Sensors (Basel). 2019. V. 19. P. 3434. https://www.mdpi.com/1424-8220/19/15/3434.
Cui T. Review of deep learning and mobile edge computing in autonomous driving. Вісник Львівської політехніки. 2022. V. 12. P. 208-218. https://science.lpnu.ua/sites/default/files/journal-paper/2023/jan/29757/221029maket-210-220.pdf.
Grigorescu S., Trasnea B., Cocias T., Macesanu G. A survey of deep learning techniques for autonomous driving. J. Field Robotics. 2020. V. 37. P. 362-386. https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.21918.
Zhang Z., Li J. A review of artificial intelligence in embedded systems. Micromachines. 2023. V. 14. P. 897. https://www.mdpi.com/2072-666X/14/5/897.
Merone M., Graziosi A., Lapadula V., Petrosino L., d'Angelis O., Vollero L. A practical approach to the analysis and optimization of neural networks on embedded systems. Sensors. 2022. V. 22. P. 7807. https://www.mdpi.com/1424-8220/22/20/7807.
Helms D., Amende K., Bukhari S., et al., Optimizing neural networks for embedded hardware. SMACD/PRIME 2021, International Conference on SMACD and 16th Conference on PRIME, online. 2021. P. 1-6. https://ieeexplore.ieee.org/document/9547911.
Song W. Hardware accelerator systems for embedded systems. Advances in Computers, vol. 122. Elsevier, 2021. P. 23-49. https://www.sciencedirect.com/science/article/abs/pii/S0065245820300917.
Yesuf M., Assefa B. Model compression techniques in deep neural networks. Pan African Conference on Artificial Intelligence. Cham: Springer Nature Switzerland. 2022. P. 169-190. https://link.springer.com/chapter/10.1007/978-3-031-31327-1_10.
Lohn A., Scaling AI. Technical report, Center for Security and Emerging Technology, 2023. https://cset.georgetown.edu/publication/scaling-ai/.
Acun B., Murphy M., Wang X., et al. Understanding training efficiency of deep learning recommendation models at scale. 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE. 2021. https://arxiv.org/abs/2011.05497.
Dong K., Zhou C., Rian Y., Li Y. MobileNetV2 model for image classification. 2nd International Conference on Information Technology and Computer Application (ITCA). IEEE, 2020. P. 476-480. https://ieeexplore.ieee.org/document/9422058.
Berthelier A., Chateau T., Duffner S., et al. Deep model compression and architecture optimization for embedded systems: a survey. J. Signal Processing Systems. 2021. V. 93. P. 863-878. https://link.springer.com/article/10.1007/s11265-020-01596-1.
Hadidi R., Cao J., Xie Y., et al. Characterizing the deployment of deep neural networks on commercial edge devices. IEEE International Symposium on Workload Characterization (IISWC), IEEE. 2019. P. 35-48. https://ieeexplore.ieee.org/document/9041955.
Mavrovouniotis S. Ganley M. Hardware security modules. Secure Smart Embedded Devices, Platforms and Applications. New York, NY: Springer New York, 2013. P. 383-405.
Vembu S. K., Chattopadhyay A., Saha S. Authenticating edge neural network through hardware security modules and quantum-safe key management. 2024 37th International Conference on VLSI Design and 2024 23rd International Conference on Embedded Systems (VLSID), Kolkata, India. 2024. P. 318-323. https://ieeexplore.ieee.org/document/10483401.
Ezirim K., Khoo W., Koumantaris G., et al. Trusted platform module – a survey. The Graduate Center of The City University of New York, 11. 2012. https://www.researchgate.net/profile/Kenneth-Ezirim/publication/287984174_Trusted_Platform_Module_-_A_Survey/links/567af54608ae197583812a7c/Trusted-Platform-Module-A-Survey.pdf.
Köylü T.Ç., Wedig Reinbrecht C.R., Gebregiorgis A., et al. A survey on machine learning in hardware security. ACM Journal on Emerging Technologies in Computing Systems. 2023. V. 19. P. 1-37. https://dl.acm.org/doi/10.1145/3589506.