Multimodal and cognitive approaches to academic discourse in AI-supported learning
Abstract
This article examines how academic discourse is reshaped in higher education through the integration of artificial intelligence (AI) and multimodal design, understood here in the sense of multimodal discourse theory (not multimodal AI models). Grounded in cognitive linguistics, sociocognitive discourse theory and multimodal semiotics, the study analyzes how academic concepts are structured and communicated in AI-enhanced learning environments. It focuses on two Micromodules developed at the University of Osnabrück – Welcome to the AI Jungle and Expedition AI. Micromodules are short multimedia units suitable for Blended Learning that integrate text, visuals, interactivity, and AI-generated feedback within the Stud.IP Learning Management System (LMS). Using a combination of cognitive discourse analysis and multimodal content analysis, the study explores how learners engage with the concepts of learning, argumentation, and autonomy in AI-mediated contexts.
Our findings show that learners navigate content using conceptual metaphors like LEARNING IS A JOURNEY, reinforced by modular layout and AI feedback mechanisms. Argumentation is shaped through additive elaboration rather than critical opposition, while autonomy is bounded by interface cues and AI prompts. The study also analyzes how AI systems – specifically tailored to and embedded within the LMS – can participate as semiotic agents, influencing meaning-making through tone, visual presence, and structured interaction. These patterns suggest a shift toward dialogic, hybrid academic discourse in which agency is distributed across human and non-human actors. The article argues that in AI-supported, multimodal learning environments, academic literacy should be reimagined as something created through collaboration between students, educators, and digital tools. It also highlights that developing critical digital literacy is essential for designing curricula that meet the demands of the future.
Downloads
References
Chevalier, M., Giang, C., El-Hamamsy, L., Bonnet, E., Papaspyros, V., Pellet, J.-P., Audrin, C., Romero, M., Baumberger, B., & Mondada, F. (2022). The role of feedback and guidance as intervention methods to foster computational thinking in educational robotics learning activities for primary school. Computers & Education, 180, 104431. https://doi.org/10.1016/j.compedu.2022.104431
Dalton‑Puffer, C. (2013). A construct of cognitive discourse functions for conceptualizing content–language integration in CLIL and multilingual education. European Journal of Applied Linguistics, 1(2), 216–253
DigiFLEd Project. (2024–2026). Modernisation of University Education Programmes in Foreign Languages by Integrating Information Technologies (101128713 — DigiFLEd— ERASMUS-EDU-2023-CBHE). Retrieved from https://digifled.eu/
Evans, K. (2007). Concepts of bounded agency in education, work, and the personal lives of young adults. International Journal of Psychology, 42(2), 85–93.
Fauconnier, G., & Turner, M. (2002). The way we think: Conceptual blending and the mind's hidden complexities. New York, NY: Basic Books.
Fleming, L. (2013). Expanding learning opportunities with transmedia practices: Inanimate Alice as an exemplar. Journal of Media Literacy Education, 5(2). https://doi.org/10.23860/jmle-5-2-3
Hadwin, A. F., Järvelä, S., & Miller, M. (2018). Self-regulation, co-regulation, and socially shared regulation in collaborative learning environments. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 83–106). New York, NY: Routledge.
Jones, M. V., Smith, N., Burns, D., Braithwaite, E., Turner, M., McCann, A., Walker, L., Emmerson, P., Webster, L., & Jones, M. (2022). A systematic review of resilient performance in defence and security settings. PLoS ONE, 17(10), e0273015. https://doi.org/10.1371/journal.pone.0273015
Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2024). Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task [Preprint]. arXiv. https://arxiv.org/pdf/2506.08872
Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. London, New York: Routledge.
Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London, New York: Routledge.
Krysanova, T., & Shevchenko, I. (2021). Conceptual blending in multimodal construction of negative emotions in film. In A. Pawelec, A. Shaw, & G. Szpila (Eds.), Text–Image–Music: Crossing the borders (pp. 357–371). Switzerland: Peter Lang. https://doi.org/10.3726/b18012
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago, IL: University of Chicago Press.
Langacker, R. W. (2008). Cognitive grammar: A basic introduction. England: Oxford University Press.
Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23(2), 157–172.
Little, D. (1991). Learner autonomy: Definitions, issues and problems. Ireland: Authentik Language Learning Resources Ltd.
Mercer, S. (2011). Understanding learner agency as a complex dynamic system. System, 39(4), 427–436. https://doi.org/10.1016/j.system.2011.08.001
Mousavi, A., Mares, C., & Stonham, T. J. (2015). Continuous feedback loop for adaptive teaching and learning process using student surveys. International Journal of Mechanical Engineering Education, 43(4), 247–264. https://doi.org/10.1177/0306419015606618
Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal
of Social Issues, 56(1), 81–103. https://doi.org/10.1111/0022-4537.00153
Piwowar, A., & Dovhaniuk, E. (2025). Mikromodule in Stud.IP erstellen: Lessons learned aus dem virtUOS [Presentation]. Germany: University of Osnabrück.
Shevchenko, I., & Shastalo, V. (2021). The conceptual metaphor of modesty in English and Ukrainian. Cognitive Studies, (21), 1–11. https://doi.org/10.11649/cs.2462
Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Researcher, 27(2), 4–13. https://doi.org/10.2307/1176193
Sundar, S. S., & Kim, J. (2019). Machine heuristic: When we trust computers more than humans with our personal information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–9). New York, NY: ACM Press. https://doi.org/10.1145/3290605.3300768
Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. New Haven, Conn: Yale University Press.
Toulmin, S. E. (2003). The uses of argument (2nd ed.). England: Cambridge University Press. https://doi.org/10.1017/CBO9780511840005
van Dijk, T. A. (2008). Discourse and context: A sociocognitive approach. England: Cambridge University Press.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – Where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0
Sources for illustrations
University of Osnabrück. (2024a). Expedition KI – Wie textgenerative KI tickt [Micromodule]. Zentrum für digitale Lehre, Campus-Management und Hochschuldidaktik (virtUOS), Universität Osnabrück. Retrieved May 20, 2025, from https://digitale-lehre.uni-osnabrueck.de/ideenpool-mikromodule/
University of Osnabrück. (2024b). Mikromodule – Übersicht. Zentrum für digitale Lehre, Campus-Management und Hochschuldidaktik (virtUOS). Retrieved March 19, 2025, from https://digitale-lehre.virtuos.uni-osnabrueck.de/mikromodule-uebersicht/
University of Osnabrück. (2024c). Willkommen im KI-Dschungel [Micromodule]. Zentrum für digitale Lehre, Campus-Management und Hochschuldidaktik (virtUOS), Universität Osnabrück. Retrieved March 19, 2025, from https://digitale-lehre.uni-osnabrueck.de/ideenpool-mikromodule/
Copyright (c) 2025 Ella Dovhaniuk, Alexander Piwowar, Natalya Oliynyk
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors, who publish with this journal, accept the following conditions:
The authors reserve the copyright of their work and transfer to the journal the right of the first publication of this work under the terms of the Creative Commons Attribution License (CC BY), which allows other persons to freely distribute a published work with mandatory reference to the authors of the original work and the first publication of the work in this journal.
Authors have the right to enter into separate additional agreements for the non-exclusive dissemination of the work in the form in which it was published by this journal (for example, to post the work in the electronic institutions' repository or to publish as part of a monograph), provided that the link to the first publication of the work in this journal is given.
The journal policy allows and encourages the authors to place the manuscripts on the Internet (for example, in the institutions' repositories or on personal websites), both before the presentation of this manuscript to the editorial board and during review procedure, as it contributes to the creation of productive scientific discussion and positively affects the efficiency and dynamics of citing the published work (see The Effect of Open Access).
