Multimodal text-image synergy in representing interpersonal relations in picture books
Abstract
Picture books as multimodal narratives contain rich data for analysis of multimodal meaning-making. Verbal and visual modes offer various choices for representing different interpersonal relations. In this paper, I focus on the following research question: how visual and verbal resources are used to construct interactive meanings between the reader and characters and between the characters of picture books.
In my analysis, I follow Halliday’s social semiotic approach to language within systemic-functional linguistics, according to which every semiotic system operates at three levels corresponding to ideational, interpersonal and textual metafunctions. I focus on the interpersonal level, using methodology and typologies from Kress and van Leeuwen’s (2006) visual grammar, further developed by Painter, Martin, and Unsworth (2012) for the analysis of picture books. Visual resources used for representing interpersonal relations are connected with the categories of contact, social distance, attitude, pathos, affect, ambience, and graduation. Verbal means of realizing interpersonal relations include, among others, mood structures and attitudinal lexis. Martin and White’s (2005) appraisal theory is applied to classify appraisal resources according to three systems: attitude, engagement and graduation.
I have analysed interpersonal text-image relationship in two picture books: The Paper Bag Princess by Munsch (2005) and Snow White in New York by French (1989). The analysis of visual resources at the interpersonal level has demonstrated a predominance of long shots, direct eye contact with a reader is rare with the exception of several close-ups. Front-on images situate characters in an equal position with readers, creating a sense of involvement. Overall, the interactions between the visual and verbal components create a synergetic effect in representing interpersonal relations in picture books.
Downloads
References
Halliday, M.A.K. (1985). An introduction to functional grammar. London: Edward Arnold.
Halliday, M.A.K. (1978). Language as social semiotic. London: Edward Arnold.
Joosen, V. (2011). Critical and creative perspectives on fairy tales: An intertextual dialogue between fairy-tale scholarship and postmodern retellings. Detroit, MI: Wayne State University Press.
Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design. London: Routledge.
Kress, G., & van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual communication, 1(3), 343–369.
Ledin, P., & Machin, D. (2020). Introduction to multimodal analysis. London: Bloomsbury.
Lewis, D. (2001). Reading contemporary picture books: Picturing text. London: Routledge.
Martin, J.R., & White, P.R.R. (2005). The language of evaluation: Appraisal in English. Basingstoke & New York, NY: Palgrave MacMillan.
Martinec, R., & Salway, A. (2005). A system for image-text relations in new (and old) media. Visual communication, 4(3), 337–371.
Nikolajeva, M., & Scott, C. (2001). How picture books work. New York & London: Garland Publishing.
O’Halloran, K.L., & Lim, F.V. (2014). Systemic functional multimodal discourse analysis. In S.Norris & C.D.Maier (Eds.), Interactions, images and texts: A reader in multimodality (pp. 137–154). Berlin: Mouton de Gruyter. https://doi.org/10.1515/9781614511175.137
Otto, C., Springstein, M., Anand, A., & Ewerth, R. (2020). Characterization and classification of semantic image-text relations. International Journal of Multimedia Information Retrieval, 9, 31–45. https://doi.org/10.1007/s13735-019-00187-6
Painter, C. (2017). Multimodal analysis of picture books. In B. Kümmerling-Meibauer (Ed.), The Routledge companion to picture books (pp. 420–428). London: Routledge. https://doi.org/10.4324/9781315722986
Painter, C., Martin, J.R., & Unsworth, L. (2012). Reading visual narratives: Image analysis in children’s picture books. Sheffield: Equinox.
Santamaría-García, C. (2022). A semiotic and multimodal analysis of interactive relations in picture books that challenge female gender stereotypes. In A. J. Moya-Guijarro & E. Ventola (Eds.), A multimodal approach to challenging gender stereotypes in children’s picture books (pp. 144–163). New York & London: Routledge. https://doi.org/10.4324/9781003145875
Tseng, C. (2013). Cohesion in film: tracking film elements. Basingstoke: Palgrave Macmillan.
Van Leeuwen, T. (2020). Multimodality and multimodal research. In L. Pauwels & D.Mannay (Eds.), The SAGE handbook of visual research methods (pp. 465–508). Thousand Oaks, CA: SAGE publications, Inc. https://doi.org/10.4135/9781526417015
White, P.R.R. (2015). Appraisal theory. In K. Tracy, C. Ilie, & T. Sandel (Eds), The international encyclopedia of language and social interaction. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781118611463.wbielsi041
Yefymenko, V. (2022). Multimodality and cross-modal cohesion in manga.
Cognition, communication, discourse, 24, 103–114. https://doi.org/10.26565/2218-2926-2022-24-08
Copyright (c) 2024 Victoria Yefymenko
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors, who publish with this journal, accept the following conditions:
The authors reserve the copyright of their work and transfer to the journal the right of the first publication of this work under the terms of the Creative Commons Attribution Non-Commercial Non-Derivs License (CC BY-NC-ND), which allows other persons to freely distribute a published work with mandatory reference to the authors of the original work and the first publication of the work in this journal.
Authors have the right to enter into separate additional agreements for the non-exclusive dissemination of the work in the form in which it was published by this journal (for example, to post the work in the electronic institutions' repository or to publish as part of a monograph), provided that the link to the first publication of the work in this journal is given.
The journal policy allows and encourages the authors to place the manuscripts on the Internet (for example, in the institutions' repositories or on personal websites), both before the presentation of this manuscript to the editorial board and during review procedure, as it contributes to the creation of productive scientific discussion and positively affects the efficiency and dynamics of citing the published work (see The Effect of Open Access).
