Machine learning methods for solving semantics and context problems in processing textual data

doi:10.26565/2304-6201-2022-56-03

Ihor Malyha Kharkiv National University of V.N. Karazin,4 Svobody Square, Kharkiv, Ukraine, 61077 https://orcid.org/0000-0002-5708-7739
Serhiy Shmatkov Kharkiv National University of V.N. Karazin,4 Svobody Square, Kharkiv, Ukraine, 61077 https://orcid.org/0000-0002-0298-7174

DOI: https://doi.org/10.26565/2304-6201-2022-56-03

Keywords: Machine learning, natural language processing, semantics, context, textual data, neural networks, transformers, BERT, GPT-3, data analysis, sentiment analysis, semantic analysis

Abstract

Topicality. As machine learning capabilities expand and impact many aspects of modern life, such as natural language processing, understanding semantics and context in textual data is becoming increasingly important. Semantics and context play a significant role in the ability of machines to understand human language. They are central elements in various applications such as machine translation, sentiment analysis, spam detection, voice recognition, and others. However, these aspects are often neglected or underestimated when processing textual data. Despite significant progress in this area, the problem of semantics and context remains unresolved, which reduces the efficiency and accuracy of many machine learning systems.

Goal: The main goal of this article is to investigate the problem of understanding semantics and context in machine learning in the textual data processing. The article aims to identify the main challenges associated with understanding semantics and context, and how they affect various aspects of text processing. Additionally, current techniques and approaches used in the field of machine learning for solving those problems have been analyzed and their limitations identified.

Research methods. Analysis, explanation, classification.

The results. It has been found that despite significant advances in machine learning technologies, problems of semantics and context in processing textual data are still existing. They affect the quality and accuracy of decisions made by machine learning based systems, which can lead to incorrect analysis and distortion of data. It has been found that even modern transformer-based models can face challenges in understanding semantics and context, especially in complex and multi-valued scenarios.

Conclusions. On the basis of the conducted research, it has been concluded that the problem of semantics and context in the processing of textual data is significant and requires further study. The existing methods and technologies show high results in some cases, but may be insufficient in others, especially complex ones. It is proposed to continue research in this area, to develop new methods and approaches that will be able to effectively solve these problems. It is also important to study how different contextual factors affect the semantics of textual data and how these effects can be taken into account when designing and using machine learning systems.

Downloads

Download data is not yet available.

Author Biographies

Ihor Malyha, Kharkiv National University of V.N. Karazin,4 Svobody Square, Kharkiv, Ukraine, 61077

PhD student

Serhiy Shmatkov, Kharkiv National University of V.N. Karazin,4 Svobody Square, Kharkiv, Ukraine, 61077

Doctor of science, professor; Head of the Department of Theoretical Theoretical and Applied System Engineering

References

/

References

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Available at: https://arxiv.org/abs/1301.3781.

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. Available at: https://arxiv.org/abs/1409.0473.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Available at: https://www.aclweb.org/anthology/N19-1423/.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. Available at: https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.

Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global Vectors for Word Representation. Available at: https://www.aclweb.org/anthology/D14-1162/.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. Available at: https://papers.nips.cc/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html.

Davidov, D., Tsur, O., & Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. Available at: https://aclanthology.org/W10-2914/.

Blodgett, S. L., Green, L., & O'Connor, B. (2018). Demographic Dialectal Variation in Social Media: A Case Study of African-American English. Available at: https://aclanthology.org/D16-1120/.

Howard, J., & Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. Available at: https://www.aclweb.org/anthology/P18-1031/.

Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. Available at: https://www.aclweb.org/anthology/N18-1202/.

Huang, P. S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013). Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. Available at: https://posenhuang.github.io/papers/cikm2013_DSSM_fullversion.pdf.

Xu C., McAuley J., (2018). The Importance of Generation Order in Language Modeling. Available at: https://www.aclweb.org/anthology/D18-1324/.

Suzuki M., Matsuo Y., (2020). A survey of multimodal deep generative models. Available at: https://arxiv.org/abs/2207.02127.

Felbo, B., Mislove, A., Søgaard, A., Rahwan, I., & Lehmann, S. (2017). Using Millions of Emoji Occurrences to Learn Any-domain Representations for Detecting Sentiment, Emotion and Sarcasm. Available at: https://www.aclweb.org/anthology/D17-1169/.

Reyes A., Rosso P., (2016). Mining Subjective Knowledge from Customer Reviews: A Specific Case of Irony Detection. Available at: https://aclanthology.org/W11-1715.pdf.

Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016). Hierarchical Attention Networks for Document Classification. Available at: https://www.aclweb.org/anthology/N16-1174/.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. (2019). Better language models and their implications. Available at: https://openai.com/blog/better-language-models/.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., ... Amodei, D. (2020). Language Models are Few-Shot Learners Available at: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bf5478631ec67e564d04505b-Paper.pdf.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Available at: https://openreview.net/pdf?id=rJ4km2R5t7.

Lu, X., Xiong, C., Parikh, A. P., & Socher, R. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. Available at: https://arxiv.org/abs/1908.02265.