SciELO - Scientific Electronic Library Online

 
vol.22 issue1The Relationship between Bilingualism and Identity in Expressing Emotions and ThoughtsIncidence of general didactics in foreign language teaching-learning process in Cuba author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


Íkala, Revista de Lenguaje y Cultura

Print version ISSN 0123-3432

Íkala vol.22 no.1 Medellín Jan./Apr. 2017

https://doi.org/10.17533/udea.ikala.v22n01a04 

Theoretical Articles

Reading with Eyes Wide Open: Reflections on the Impact of Multimodal Texts on Second Language Reading1

Leyendo con ojos bien abiertos: reflexiones sobre el impacto de los textos multimodales en la lectura en segunda lengua

Lire avec les yeux grands ouverts: réflexions sur l'impact de textes multimodaux dans la lecture en langue seconde

Abraham Paul1 

Farías Miguel2 

1BA, Romance Languages and Education, Boston College, Ed.M., TESL, Boston University, Ed. D. Reading and Language, Harvard University. Chair of the TESOL Master's Program, Simmons College Mailing address: Simmons 300 The Fenway Boston, MA 02115 E-mail: paul.abraham@simmons.edu

2Universidad de Santiago de Chile Teacher of English, Universidad de Chile, M.A. Applied Linguistics, Ohio University, Ph.D. Literature and Linguistics, The Catholic University of America, Washington, D.C. Full Professor of Applied Linguistics at the Faculty of Humanities, Universidad de Santiago de Chile. Mailing address: Alameda 3363, Estación Central, Santiago, Chile. E-mail: miguelfariasf@gmail.com


Abstract:

In the context of what has been called a new sociosemiotic landscape in communications, this article explores how reading multimodal texts demands a change in literacy practices and reflects on the impact of these changes on second language reading. We review the literature that has attempted to incorporate multimodal texts in the discussion of second language reading and evaluate it in light of the practices second language readers engage in when comprehending multimodal texts. Implications of this transition from understanding reading as an individual, text-based, and univocal activity to approaching it as literacy practice are included. We finish with reflections and conclusions derived from such explorations.

Keywords: sociosemiotic landscape; multimodal texts; second language reading; literacy.

Resumen:

En el contexto de lo que se ha denominado como un nuevo paisaje sociosemiótico en las comunicaciones, este artículo explora la forma como la lectura de textos multimodales requiere de cambios en las prácticas de literacidad y reflexiona sobre el impacto de estos cambios en la lectura en segunda lengua. Reseñamos la literatura que ha intentado incorporar los textos multimodales en las discusiones de la lectura en segunda lengua y la evaluamos a la luz de las prácticas que los lectores de una segunda lengua utilizan en los procesos de comprensión lectora. Se discuten las implicaciones de esta transición entre concebir la lectura como actividad individual, unívoca y basada en el texto a una concepción plural como práctica de literacidad. Finalizamos con reflexiones y conclusiones derivadas de esta evaluación crítica.

Palabras clave: paisaje sociosemiótico; textos multimodales; lectura en segunda lengua; literacidad

Résumé:

Dans le contexte de ce que l'on a appelé un nouveau paysage socio-sémiotique dans les communications, cet article explore pourquoi la lecture de textes multimodaux demande un changement dans les pratiques de littératie et réfléchit sur l'impact de ces changements sur la lecture en langue étrangère. Nous présentons différents auteurs qui ont essayé d'incorporer des textes multimodaux dans la discussion sur la lecture en langue étrangère et nous confrontons leurs positions en tenant compte des pratiques que les lecteurs en langue étrangère engagent pour comprendre les textes. Nous discutons ensuite les implications de la transition d`une conception de lecture en tant qu`activité individuelle et univoque à une conception plus complexe de la lecture en tant que pratique de littératie. Finalement, nous proposons nos réflexions et conclusions tirées de cette discussion.

Mots-clés : univers socio-sémiotique; textes multimodaux; lecture de deuxième langue; alphabétisation

Introduction

That reading is a pivotal human activity is worth underscoring as a starting point in this attempt to expand current notions of reading to incorporate multimodal texts. Although reading has always been multimodal because of the various modes involved -from font types and letter size to hyperlinks and accompanying audio- in recent years both the how and the what of reading have changed with the advent of mass information and communication technologies that have produced new formats to present language and provided access to design software that allows for quick and personal creation of texts. That the International Reading Association (IRA) has recently changed its name to International Literacy Association (ILA) is not a random resemantization, but rather points in the direction we want this exploration and reflection on reading and multimodality to go: the adaptation and transformation of traditional reading models to include the semiotic dimension needed to fully account for the understanding of reading multimodal texts as a literacy practice.2

In order to open up the explanatory dimension of reading to include multimodal texts, we need to make use of conceptual frames that lie beyond the traditional cognitive models that regard reading as an individual and often passive activity. As any new applied disciplinary area attempting to establish a core of basic assumptions to frame research and teaching practices, second language multimodal reading is agglutinating ideas from such diverse fields as learning psychology and sociosemiotics. The introduction of a semiotic approach to texts that multimodality brings to reading has revolutionized our understanding of the new meaning-making processes in which citizens in today's world engage. In her thought provoking article, Siegel (2012) sets the context for multimodality with references to youth and their new literacy capabilities: "It is tempting to suggest that this is the time of multimodality: A time when the privileged status of language is being challenged by the ease with which youth can access semiotic resources of all varieties -visual, aural, gestural, and spatial- to assemble meanings" (p. 671).

The drawing in Figure 1 done by a Chilean university student in an English education program provides a good illustration of the multiple modes of communication that this learner has at his disposal. This drawing was a response to a course task that asked students to graphically represent their conception of how they learn English as a second language. What do you, as readers, see in Figure 1? Are you surprised? Is anything missing? Would you add anything else?

Figure 1 Mental map by a student in a Chilean EFL teacher education program. 

The array of language inputs the student represents in his metaphorical drawing ranges from a printed novel and a traditional classroom instructor to television programs, videos, radio music, and the Internet. His path to the world of English language learning, signaled by an arrow, includes different media such as books, television, radio, computer, and whiteboard, which make possible various modes, including writing and images on the page and screens, sounds, gaze, gestures, and speech. Therefore, we wonder how to make use of these more non-traditional formats within the school setting and, particularly, in the context of second language reading education. We ask ourselves what multimodality can bring to the processes of teaching and learning second language reading. What makes the inclusion of multimodal texts different from the typical use of accompanying visuals that support a printed message?

Because of the availability of technology, communication has morphed into a multiplicity of formats (for example, emails, tweets, videos, DVD's, text messaging, WhatsApp) that are not being fully exploited in schools. Young learners are particularly adept at making use of these novel communication systems and have created a new reading and writing culture based on digital social networks. In this respect, Gee (2004) has mentioned that "young people today are often exposed outside of school to processes of learning that are deeper and richer than the forms of learning to which they are exposed in schools" (p. 107). Gee (2004) calls these users "millennials" while Prensky (2001) refers to them as "digital natives;" others have called them "screenagers." Whatever they are called, these young people constitute a reading community that engages daily in semiotic and interpreting processes recently being explored by authors like Cassany (2006). For educators, the question seems to be how to make use of the knowledge that these learners bring to school in order to enhance their critical appraisal of the texts they encounter in their daily lives. More specifically, we are interested in the impact of these changes for second language educators who are dealing or will deal with emerging genres associated with these new technologies. More than responding intellectually to the challenge posed by this new sociosemiotic scenario that might serve as a model of second language multimodal reading, we want to encourage language educators to incorporate these changes into their practice with a critical lens. Further, we encourage them to recognize and discuss in their classrooms that reading is a social practice where the knowledge, skills, attitudes, and values constructed from texts encompass the historical, individual, and social use of the written code.

In what follows, we explore multimodality from the broad perspectives assumed by work in learning psychology and second/foreign language multimodal learning and reading to arrive at some implications at the juncture of such concepts dealing with the critical reading of multimodal texts in second language learning processes. We finish with reflections and conclusions derived from these explorations.

Multimodality in Second Language Teaching and Learning

Research on multimodal texts has been conducted around the concept of multimodal/multimedia learning by Mayer and his team (Mayer, 2001, 2005; Moreno and Mayer, 2007; Mayer and Sims, 1994) and much of the related research has involved testing a number of principles postulated by Mayer to account for cognitively sound presentations of instructional materials that can lead to meaningful learning in areas such as biology, physics, and mechanics. Presentations are understood as verbal and pictorial representations such as on-screen text or narration (voice) and static graphics or animation (video) used in most computer-based educational multimedia, of which PowerPoint is the best known. At least three principles are relevant to second language learning -multimedia, coherence, and redundancy. The multimedia principle claims that students learn better from words and pictures than from words alone. The coherence principle states that students learn better when extraneous material is excluded rather than included, and the redundancy principle postulates that students learn better from animation and narration than from animation, narration, and text (Mayer, 2001). Redundant presentations duplicate information using the same channel, such as an image and the written word defining it, which are both processed visually.

Schnotz ( 2002, 2005) took a different approach. Following a similar line of research concerning the dual channel theory postulated by Paivio, he proposed an integrated theory of learning from visual and textual information to deal with what he calls domain learning, i.e., discipline or content learning as in the school curriculum: biology, history, physics, etc. Furthermore, Schnotz and Baadte (2008) emphasized the distinction between domain learning and language learning claiming that "in language learning, things are different because the primary goal of learning is not to learn about a specific domain, but to master a new language" (p. 22).

This distinction was tested in Farias et al. (2014) where three groups of English as a foreign language (EFL) learners were exposed to three different types of presentations. The first included text and narration (GTN), the second still image, text, and narration (GITN), and the third, moving image (video), text, and narration (GVTN). The study tested the acquisition of concrete vocabulary as measured by retention and transfer tests. The results indicated that the consistently higher results by the GITN over GTN confirm the basic premise of the multimodal learning principle regarding the advantage of presentations including text and image by means of which learners actively integrate textual and pictorial information into a coherent mental model. In turn, higher results of both GITN and GVTN over GTN may indicate that presentations that included images offered more options for different learning style preferences. This finding may indicate that redundancy does not seem to be as much of a problem for second language learning as it was for domain learning, at least for vocabulary learning in adult learners with beginner and pre-intermediate proficiency levels. Redundancy, involving the duplication of information via the same visual channel, text and image, in this case helped learners to retain and transfer the lexical items being presented. Farias et al. (2014) speculated that

this seems to be the case primarily in beginning and pre-advanced second language learners that require the text, the voice and the image to construct their lexicons in the second language as they have not yet attained automaticity in matching sound and text (p. 34).

Coming from the psychology of learning, Mayer's and Schnotz' models provide robust explanations for how multimodal texts are cognitively processed, but reading, in the context of second/foreign language learning, is also a literacy practice that involves the reader's purposes and active participation in understanding the text's sociosemiotic contexts.

Ground-breaking in establishing the field of multimodal discourse and its ensuing approach of sociosemiotics are Kress (2010) and Kress and van Leeuwen (1996, 2001). Based on the Hallidayian tradition of Systemic Functional Linguistics (SFL), Kress and van Leeuwen have described multimodal texts as those that combine two or more semiotic systems. These researchers postulated a grammar of visual design that could account for the semiotic processes by which multimodal texts are produced and comprehended in today's societies. Their ideas were also incorporated in the discussions by the New London Group on the new literacies (multiliteracies) that citizens need to be familiar with for effective communication (Cope and Kalantzis, 2000).

Specifically addressing language learning, Kress (2000) reiterates the need for language teachers to look at the context in which language is used and the accompanying resources, such as images, that are co-present in any text and contribute to its meaning. He introduces the concepts of modes and affordances to discuss the interplay between image, text and other modes of making meaning and the options users have at their disposal to create their intended meanings.

Affordances can be defined (Kress, 2010) as the (social and cultural) potentialities and constraints offered by different modes, i.e. what is possible to express and represent or communicate easily and what is less straightforward or even impossible with the resources of a mode. Kress (2000) stressed the notion of the functionality of modes and their affordances by mentioning that images are based on the logic of display in space whereas writing is based on the logic of succession in time. He reminds us of the perspective of TESOL teachers by noting that when asked to look at other features in the text that convey meaning, they often reply that their business is language, as the only mode that fully represents meaning.

What is interesting about the notions of affordances and modes for this reflection on multimodal reading is that, traditionally, reading has been conceived of as using the logic of sequentiality in time afforded by the written printed page whereas the introduction of visuals has brought with it the logic of simultaneity in space, as images are represented concurrently. Hyperlinks, in turn, introduce depth to texts as they may lead to endless embeddings.

As for the visual dimension of multimodal texts in second language acquisition, Royce (2002) explored the concept of multimodality in the TESOL classroom through the analysis of a multimodal text taken from a science textbook. His aim was to call the attention of TESOL teachers to the richness of the intersemiotic relations established between text and image. In the tradition of SFL, he claimed that "the visual and the verbal modes complement each other to realize an inter-semiotically coherent multimodal text" (p. 192). His intention was to encourage TESOL professionals to help learners develop multimodal communicative competence and "not suggest that students carry out semiotic interrelatedness analyses in the classroom" (p. 198). In the context of second language learning, what Royce indicates is the need to explore and exploit the potential of the other modes of making meaning that accompany language, which may be in line with developing visual literacy skills in second language learners as they are increasingly exposed to multimodal texts. Royce (2007) further explores the term multimodal communicative competence to call attention to the visual dimension that makes up multimodal texts, not as a competence dealing with modes in isolation but having to do with "how students can become competent in interpreting and constructing appropriate meanings multimodally" (p. 374).

Second-language reading comprehension

Before the recent attention to texts as sociosemiotic and multimodal conveyors of meaning, reading was historically seen from three different cognitive perspectives, and these models were readily adopted to describe second language reading: "bottom-up," "top-down," and "interactive." Bottom-up theorists, including Gough (1972) and LaBerge and Samuels (1974), emphasized the ability to decode or put into sounds what is seen in a text. In this model, the power rests in the text and it is the job of the reader to "decode" it much like a puzzle. Alternatively, top-down reading models focused on what readers bring to the process (Goodman, 1967; Smith, 1971). In this model, readers were seen to sample the text and contrast it with their world knowledge to make sense of what was written; therefore power rests with the reader.

For those reading theorists who recognized the importance of both the text and the reader in the reading process, an amalgamation of the two emerged: the interactive approach. The interactive model (Rumelhart, 1977; Stanovich, 1989) stressed both what is written on the page and what the reader brings to it using both top-down and bottom-up skills, an interaction between what is on the page and what a reader knows. Early second language reading research embraced a top-down model, but has more recently recognized how crucial bottom-up skills are in the reading process (see Birch, 2007 and Grabe, 2008).

Ironically, second language reading has long included the multimodal components of texts to support deficits in print knowledge or conventions. So, how does this emphasis on multimodal texts today differ from what has always been part of reading instruction in the second language classroom? A major difference is that in multimodal texts, a reader's attention to visuals is used for their meaning potential in understanding the text as a totality and not merely for pointing to their ornamental, aesthetic, or ancillary functions.

In light of the literature here reviewed, we suggest the following ideas that may help in expanding reading models to encompass multimodal texts:

a) The bottom-up view that centers on the linguistic code at the lexical and syntactical levels needs to expand to include the visual dimension in its relation to the textual references and, thus, identify the role that images and other visual components, such as layout and color, are playing in this relation. Are they ancillary, ornamental, complementary, concurrent, or a combination of all of these? What are the meanings that visuals, other modes, and language bring to the text? How are they organized and mentally represented? Research by Unsworth (2006, 2008), Unsworth and Chan (2009), and Unsworth, Thomas, and Bush (2004) has defined the metalanguage necessary to describe image/text relations in educational materials for school literacy and provided guidelines to include multimodal texts as resources for assessment in the Australian curriculum.

b) The top-down view that engages background and previous knowledge has to broaden to incorporate an evaluation of the culturally motivated affordances that the textual and visual modes bring to the construction of the text's architecture. The activation of verbal and nonverbal textual representations provides fertile ground for inferences and text integration. For example, an emoticon introduces emotional states that are conventionally designed according to fixed patterns of facial expressions, whereas language is more polysemic in that words convey multiple connotations. (All of this is despite the recent creation of catalogs of emoticons.) Such an awareness of modes is mentioned by Siegel (2012): "If, for example, students do not consider the ways modes are historically and culturally produced, they may simply reproduce designs that reflect and reinforce the status quo" (p. 675). The critical dimension of reading may be introduced here by paying attention to the cultural "weights" and functions of different modes. Why is an image used instead of a linguistic description? What similar texts combine image and language in the same manner (awareness of genres)? Such an awareness of modes, which can be part of a critical multimodal literacy curriculum, may also be conducive to critical media literacy (Buckingham, 2008) where the (adult) second language learner may comparatively evaluate the semiotic design of different genres of multimodal texts.

c) The interactive dimension of reading should not be limited to the interplay between bottom-up and top-down processes but expand to the relations between images and text, i.e., the various intersemiotic senses generated by the visual and verbal modes. The understanding of these intersemiotic relations, in the case of second language reading, are influenced by at least two factors: learning preferences and proficiency level. Some authors have postulated that visual learners or visualizers (Vincent, 2005) benefit more from having images in the text; others have hypothesized that low proficiency second language learners would resort more to images in search for meaning (see, for example, Liu, 2004). This modal intersemiotic interactivity may open the language learner avenues to explore cultural artifacts such as graphic novels, graffiti, memes, and other emerging genres. The adult learner interacts with a multimodal text at various levels, from intersemiotically assembling the meanings furnished by the different modes to transmediating (see Yi, 2014) in the act of interculturally interpreting the idiosyncrasies in the text.

Regarding these three views to reading, Serafini (2011) has provided guidelines to expand students' interpretive repertoires by attending to the new theories, perspectives, and practices used to comprehend multimodal texts. His recommendations include, among others, using an NCI chart to approach three levels of meaning, the first of which involves the bottom-up-like process of describing the elements we notice (N). The second calls for top-down connections (C) to background knowledge and considerations of what they might mean, and, the third to interpreting their implications (I) in the sociocultural context outside the text.

Multimodality and (second) language reading

This section reviews selected studies that have incorporated the multimodal dimension either in first or second language reading, highlighting the implications of these studies for second/foreign language multimodal reading processes.

In her 2014 paper, Yi mentions that there is a small but growing body of research in L1 on digital and multiliteracy practices, but little on multilingual readers and writers in second or foreign language contexts. Yi focused on adolescent English language learners (ELL). She informs teachers about implementing multimodal literacy in the L2 language classroom by addressing the constructs in multimodal literacy research and their possibilities and challenges in L2 teaching and learning. She distinguishes multimodality from multiliteracies, the latter as proposed by the New London Group (1996) to address the multiplicity of "communication channels and media and the increasing saliency of cultural and linguistic diversity (p. 63)." For multimodality, she quotes Jewitt (2009), who describes it as "concerned with signs and starts from the position that like speech and writing, all modes consist of sets of semiotic resources that people draw on and configure in specific moments" (p. 159).

Additionally, she addressed "transmediation," the act of translating meaning from one sign system to another, as especially important when we teach ELLs. She cautions that there is a false dichotomy between print-based literacy and multimodal literacy, as the ability to work in both "media" is essential. This coexistence of modes is at the heart of our position, as well.

Yi suggests that multimodal literacies provide adolescents with a means to explore and express their identities, to improve their academic language, to develop critical literacy and perspectives, and to connect in and out of school experiences. She also sees challenges, such as access to digital technologies, varied skill-levels among learners, learners' avoiding extended discourse in English, and the prescribed curriculum and high stakes testing as not always allowing for multimodal literacy instruction. More importantly, the value placed on printed text prevents both teachers and students from considering multimodal literacies to be legitimate. As we move forward, we concur with Yi on the need to examine how students learn the knowledge and skills to consume, produce, and evaluate multimodal texts. Teachers also need to determine how they can support their students in developing multimodal competence.

A number of researchers address some of these concerns about multimodal second language reading empirically and, in their research, offer techniques that can be applied pedagogically. For example, Harris (2011) investigated how effective print-based comprehension models for reading and assessing multimodal texts were. He discussed the impact that multimodal texts have had on students and how teachers, not being millennials but with some multimodal literacy (Lotherington & Jenson, 2001; Yi, 2014), can take advantage of traditional models of reading comprehension to approach multimodal texts. His study included a group of Australian elementary students in grade 5 who answered assessment task sheets designed for each episode in a series of online adventures. A key component was the inclusion of the here-hidden-head model used in answering the questions: "here" for the literal answers that are found in the text itself, often linguistically similar to the question posed and for which readers are merely asked to recognize information in the text; "hidden" for answers that are scattered along the text and that may paraphrase the text; and "head" for inferential answers that require making connections to previous knowledge. The results were processed according to four foci: students' interest levels, rating descriptors, task completion, and comprehension, all of which favored the inclusion of multimodal texts as an assessment tool for specific outcomes in the curriculum. The reader may have noticed that behind the three Hs model, top-down and bottom-up strategies are integrated.

With the premise that multimodal learning materials are frequently used in education to enhance learning outcomes, Brante et al. (2013) examined whether contrasts in such materials are likely to support reading comprehension for all readers. These researchers studied the use of text only or text plus picture material with 46 young adults (n=46), nineteen of whom had low phonological awareness (PA) and 27 had high PA. Comprehension was checked using open interview questions. Learning materials were designed to focus readers on aspects critical to understanding the content through the use of contrasts. While well-known pictures aided information recall, contrasts described in the text were most effective for learning. The notion of contrasts comes from variation theory. One finding of the study was that contrasts allow a learner to focus on a specific feature that differentiates one object or phenomenon from another and seem to be essential for learning to occur. Another outcome in this study is that, "if pictures are to contribute to reading comprehension, readers must be invited to pay attention to them" (p. 34). Surprisingly, these researchers found that those subjects with lower levels of phonological awareness in fact paid less attention to the visuals, as well. So it seems that those readers who were less able decoders, also were less able to "decode" the accompanying visuals. The researchers conclude that "the construction of texts matters; using clear and sharp contrasts makes it possible for learners to discern the values of critical aspects" (p. 34). In this respect, we hypothesize that it may be the case that for the intersemiotic relation between image and text to trigger some contrast for intermediate and advanced language learners, the image or the text needs to provide a surplus of meaning, be it as expansion, extension, or elaboration (Martinec and Salway, 2005), as is the case of graphs and tables. The beginner, particularly in vocabulary learning, would need the image and text to be iconic, having a concurrent intersemiotic relation

In a study designed to teach children how to engage in close reading of text, Dalton (2013) used a technology-based strategy, multimodal hypertext commentary, and remix. She describes this process "in its broadest sense, as a focused re-reading of the text in which the reader goes beyond the basic understanding of the text" (p. 643). The research draws on familiar text analysis practices while offering opportunities for engagement and learning. Dalton used the classic Aesop's fable, "The Lion and the Mouse," to illustrate how students use hyperlinks to deal with written, visual, and audio commentary. The use of hypertext with commentary and illustration offers students multiple pathways to think, critique, or express their understanding of written and illustrated text. Reading text and viewing illustrations is integrated as an authentic literacy practice that can engage readers.

In a similar vein, in second language reading, Plass et al. (1998) carried out studies on multimodal learning investigating English-speaking learners of German in a multimedia learning environment. These learners read a text in German that was presented to them through software that offered annotated translations of a key lexicon, a video clip illustrating the lexical item, or both. The results showed that learners did better when they used verbal and visual annotations (which confirms Mayer's multimedia principle) and that they comprehended the story better when they used their preferred mode of annotation, i.e. verbal or visual.

In another study, Margolin et al. (2013) examined the impact of technology on reading comprehension by comparing paper, computers, and e-readers and found that these three different presentation modes do not differentially affect comprehension of narrative or expository text. Results from their research suggest that reading can happen effectively in a variety of presentation formats and that, at least in educational settings, page and screen coexist.

Finally, a study by Lee (2013) examined the use of multimodal reading response to text with a group of Taiwanese ESL university students. After reading Dr. Jekyll and Mr. Hyde, students were given great latitude in how they responded to it to demonstrate comprehension and create response. Multimodal reading responses included role playing, reader's theater, movie clips, music, setting maps, the creation of a book jacket, comic strips, introductions to the author, summaries of important scenes, crossword puzzles, and even commercials. Using such multimodal reading responses "lifted" the limits that may have confined these readers to respond in one mode, traditionally writing, sustained students' reading experience with reading, and improved their comprehension. Most importantly, Lee moves beyond the timid conclusions reached by Harris (2011) to claim that multimodal responses motivated the students and, as the author suggests, promised "a possibility for them to evolve into autonomous and proficient English Readers" (p. 192).

Discussion and conclusions

When introducing their model of second language multimodal learning, Plass and Jones (2005) posed the key question: "In what way can multimedia support second-language acquisition by providing comprehensible input, facilitating meaningful interaction, and eliciting comprehensible output?" (p. 471). From the literature reviewed here and our experience with multimodal texts, we suggest that multimodal texts more effectively support second language reading by providing input that caters to different learning styles and that they are familiar, authentic, and contextualized to the learners' lives. Moreover, these texts facilitate learners' meaningful interaction not only intratextually, by exploring the text/image semiosis, but also intertextually, by allowing readers to become literate in the different genres that are constructed multimodally.

Based on this brief review and our experience as second language teaching educators, we recommend that second language reading teachers reinvent their instructional focus to engage learners in critical multimodal reading practices that can tap the meaning potential of multimodal texts. Teachers need to set aside time for an intensive focus on the visual material of a lesson. Graphs, charts, photographs, and illustrations need to be exploited fully in the second language classroom. Accompanying visuals enhance the text in many ways, from providing easy access to comprehensible input by identifying abstract concepts or precise concrete nouns (a platypus, the flicker of a flame, or a traffic jam) to enhancing understanding of the "printed" text, by providing contrasts to it, and allowing students to engage in the classroom academic conversations related to the material at hand.

While visuals and audio have been traditionally used in pre-reading activities to activate pre-existing schemata (Rumelhart, 1980) and formulate guesses about the possible macrostructure of the text, the idea now is to integrate these modes by interrogating them with students with regards to their function and the meaning they contribute to understanding the multimodal text. Teachers need to create opportunities for students to reflect not only on the specific message of accompanying visuals but also to analyze the role of these visuals to determine if they expand, enhance, or describe the related print. This kind of analysis allows readers to make full use of all incoming input, to evaluate similarities and differences, and to interpret a breadth of meaning that emerges from this two-pronged attention, thus exploring the intersemiotic relations between visuals and text in the construction of meaning. As Dalton (2013) suggests, remix and hyperlinks can be used to engage readers in close reading. Even using a simple conceptual map, an image or a metacognitive activity (like the one shown in Fig. 1) for after-reading activities can highlight text organization and macrostructure.

Preparation in critically reading the intersemiotic relations in multimodal texts in the pre-reading, reading, and post-reading stages may, in turn, usher learners into the appraisal of moving images in videos or movies, i.e. media literacy (Buckingham, 2008). Balzalgette and Buckingham (2013) have expressed their concern in this respect when they mention that multimodality has come to reinforce the traditional distinction between print and non-print texts but that, in schools, this results in a "continuing neglect of the moving image media, which are central to the life experience of young children" (p. 95). At stake here, perhaps, is the battle between text and image; a discussion worth having in teacher education where a critical appraisal of genres should include the conflicting rendition of the same message that results from choosing different semiotic resources, as, for example, in advertising and scientific texts.

Some of these ideas have been concretely translated by a number of researchers. Explicitly incorporating multimodal texts, Serafini (2012) builds on the work of Freebody and Luke, who described the roles of "code breaker," "text participant," "text user," and "text analyst" by characterizing the four roles of the reader as "navigator," "interpreter," "designer," and "interrogator." Though his observations are not strictly directed toward second language readers, Serafini (2012) reconceptualized reading as a social practice involving the construction of meaning in a socially mediated context. Taking the notion of design from Kress, in our opinion, the role of "designer" is one of the most important in that it assigns agency to the reader in the navigation and interpretation of the text. With gestures to Barthes, we adhere to Serafini's claim that "the text to be read does not come to the reader ready-made; the text comes as semiotic potential, where the text to be interpreted is designed during the act of reading" (2012, p. 158).

These ideas bring new perspectives to second language educators in order to open their eyes widely and incorporate in reading education the full array of modes present in today's multimodal texts. The metaphor used in the title of this paper is meant to shift the attention from reading as linguistic "saccades" to zooming out to all modes included in the text and critically analyzing the contexts of production and reception. In line with some of the principles posited by Freire and Macedo (1987) about the social and political implications of reading the word and reading the world, attention to multimodality in reading provides learners with the added value of understanding not only the linguistic code but also all the codes used by discourse communities to construct meaning. In this respect, Barragán and Gómez (2012) and Villa (2008) have suggested interesting approaches to foster critical visual literacy in the language classroom, which may well be applied to second language reading education.

Nevertheless, this invitation to read with eyes wide open the multimodal texts encountered in the process of learning English as a second/foreign/international language calls for the participation of different agents in decision-making positions regarding the school curriculum. Siegel (2012) has mentioned that policies are slow to include multimodal transformations of school literacy and adds that "standards, including the Common Core, offer little coverage for teachers seeking to justify their attention to multimodality" and, citing Johnson & Kress, 2003, she adds "to dislodge the 'pedagogies of conformity'" (p. 675). However, the new emphasis on informational texts with increased numbers of photographs, graphs, and charts in second language reading education demands that teachers engage readers in visual reading. After all, these are the multimodal texts that learners encounter in their daily lives.

The research on multimodality and reading reviewed here provides insights into how crucial it is to change our approach to reading so that it truly moves beyond the linguistic mode to include visual and other modes of constructing meaning. These are the reflections of the two authors of this paper, coming from different educational and cultural milieus, Chile and the United States, yet with a common interest in second language education. The theoretical and practical tensions underpinning the inclusion of multimodal texts in second language reading education emerge from various influences, among which stands, at the macro-level, the coverage and adoption of models derived from both systemic functional linguistics and the proposal on multiliteracies by the New London Group. At the meso-level, there are national curriculum guidelines, teacher education policies, and standardized testing, which are based on different conceptions of reading, not all of them including multimodal texts. The teacher of English as a second/foreign/international language is at the micro-level faced with millennial learners who are more and more exposed to multimodal texts and urgently require guidance in reading critically, with eyes wide open, the world that comes in those texts.

References

Balzalgette, Cary, and Buckingham, David. (2013). Literacy, media and multimodality: a critical response. Literacy, 47(2), 95-102. [ Links ]

Barone, Diane. (2015).The fluid nature of literacy. Literacy Today, 33(1), 6-8. [ Links ]

Barragán, Rafael, and Gómez, Wilson. (2012). El lenguaje de la imagen y el desarrollo de la actitud crítica en el aula: propuesta didáctica para la lectura de signos visuales. Íkala, Revista de Lenguaje y Cultura, 17(1), 81-93. [ Links ]

Birch, Barbara. (2007). English L2 Reading: Getting to the Bottom (2nd Ed. ). Mahwah, NJ: Lawrence Erlbaum Associates. [ Links ]

Brante, Eva Wenas; Holmqvist Olander, Mona, and Nyström, Marcus. (2013). Exploring the impact of contrasting cases in text and picture processing. Journal of Visual Literacy , 32(2), 15-38. [ Links ]

Buckingham, David. (2008). Media Education. Cambridge: Polity Press. [ Links ]

Cassany, Daniel. (2006). Literacidad crítica: leer y escribir la ideología. In II Congreso Internacional y IV Foro Nacional de Enseñanza de la Lengua y la Literatura. October 31, Hermosillo, Sonora, Mexico. [ Links ]

Cope, Bill, and Kalantzis, Mary. (Eds.). (2000). Multiliteracies. Literacy Learning and the Design of Social Futures. London: Routledge. [ Links ]

Dalton, Bridget. (2013). Engaging children in close reading. Multimodal commentaries and illustration remix. The Reading Teacher, 66(8), 642-649. International Reading Association. [ Links ]

Farias, Miguel; Gregersen, Tammy; Obilinovic, Katica, and Orrego, Roxana. (2014). Evaluating types and combinations of multimodal presentations in the retention and transfer of concrete vocabulary in EFL learning. Revista Signos, 47(84), 21-39. DOI: 10.4067/S0718-09342014000100002 [ Links ]

Ferreiro, Emilia. (2001). Pasado y futuro del verbo leer, en pasado y presente de los verbos leer y escribir (pp. 29-44). Mexico: SEP/FCE. [ Links ]

Freire, Paulo, and Macedo, Donaldo. (1987). Literacy : reading the word and the world. South Hadley: Bergin and Garvey. [ Links ]

Gee, James Paul. (2004). Situated Language and Learning: A Critique of Traditional Schooling. New York: Routledge. [ Links ]

Goodman, Kenneth S. (1967). Reading: A psycholinguistic guessing game. Journal of the Reading Specialist, 6, 126-135. [ Links ]

Gough, Philip B. (1972). One second of reading. In James F. Kavanaugh and Ignatius G. Mattingly (Eds.). Language by Ear and by Eye (pp. 331-358). Cambridge, MA: MIT Press. [ Links ]

Grabe, Wiliam. (2008). Reading in a Second Language: Moving from Theory to Practice. Cambridge: Cambridge University Press. [ Links ]

Harris, Andrew. (2011). How effective are print-based comprehension models for reading and assessing multimodal texts? . Literacy Learning: The Middle Years, 19(3), 19-32. Retrieved from: http://search.informit.com.au/documentSummary;dn=449254865430346;res=IELHSSLinks ]

Jewitt, Carey. (2009). Introduction. In C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis (pp. 1-7). London: Routledge . [ Links ]

Kress, Gunther, and Van Leeuwen, Theo. (1996). Reading Images: The Grammar of Visual Design. London: Routledge . [ Links ]

Kress, Gunther, and Van Leeuwen, Theo. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Edward Arnold. [ Links ]

Kress, Gunther. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. New York: Routledge. [ Links ]

Kress, Gunther. (2000). Multimodality: Challenges to thinking about language. TESOL Quarterly, 34(2), 337-340. [ Links ]

LaBerge, David, and Samuels, S. Jay. (1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6, 293-323. [ Links ]

Liu, Jun. (2004). Effects of comic strips on L2 learners' reading comprehension. TESOL Quarterly , 38(2), 225-243. [ Links ]

Lotherington, Heather, and Jenson, Jennifer. (2001). Teaching multimodal and digital literacy in L2 settings: new literacies, new basics, new pedagogies. Annual Review of Applied Linguistics, 31, 226-246. [ Links ]

Lee, Hsiao-Chien (2013). An examination of ESL Taiwanese university students' multimodal reading responses. Literacy Research and Instruction, 52, 192-203. [ Links ]

Margolin, Sara J.; Driscoll, Casey; Toland, Michael J., and Kegler, Jennifer Little. (2013). E-readers, computer screens, or paper: Does reading comprehension change across media platforms? Applied Cognitive Psychology, 27, 512-519. DOI: 10.1002/acp.2930 [ Links ]

Martinec, Radan, and Salway, Andrew. (2005). A system for image-text relations in new (and old) media. Visual Communication, 4(3), 337-371. [ Links ]

Mayer, Richard E., and Sims, Valerie K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86, 389-401. [ Links ]

Mayer, Richard E. (2001). Multimedia Learning. Cambridge, UK/New York: Cambridge University Press. [ Links ]

Mayer, Richard. (Ed.). (2005). The Cambridge Handbook of Multimedia Learning. New York: CUP. [ Links ]

Moreno, Roxana, and Mayer, Richard. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309-326. [ Links ]

Plass, Jan L.; Chun, Dorothy M.; Mayer, Richard E., and Leutner, Detlev. (1998). Supporting visual and verbal learning preferences in a second language multimedia learning environment. Journal of Educational Psychology , 90, 25-36. [ Links ]

Plass, Jan, and Jones, Linda. (2005). Multimedia learning in second language acquisition. In Richard Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 467-488). New York: CUP . [ Links ]

Prensky, Marc. (2001). Digital natives, digital immigrants. Part 1. On the Horizon, 9(5), 1-6. http://dx.doi.org/10.1108/10748120110424816Links ]

Royce, Terry. (2002). Multimodality in the TESOL classroom: Exploring visual-verbal synergy. TESOL Quarterly , 36(2), 191-205. [ Links ]

Royce, Terry. (2007). Multimodal communicative competence in second language contexts. In Terry Royce and Wendy Bowcher (Eds.). New Directions in the Analysis of Multimodal Discourse (pp. 361-390). Mahwah, NJ: Lawrence Erlbaum. [ Links ]

Rumelhart, David. (1977).Toward an interactive model of reading. In Stan Dornic (Ed.), Attention and Performance VI (pp. 573-603). Hillsdale, NJ: Lawrence Erlbaum. [ Links ]

Rumelhart, David. (1980). Schemata: The Building Blocks of Cognition. In Spiro, Rand; Bertram Bruce and Wiliam Brewer (Eds.). Theoretical Issues in Reading Comprehension (pp. 33-58). Hillsdale, NJ: Lawrence Erlbaum . [ Links ]

Schnotz, Wolfgang. (2002). Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14(1), 101-120. [ Links ]

Schnotz, Wolfgang. (2005). An integrated model of text and picture comprehension. In Mayer, Richard. (Ed.). The Cambridge Handbook of Multimedia Learning (pp. 49-69). New York: CUP . [ Links ]

Schnotz, Wolfgan, and Baadte, Christiane. (2008). Domain learning versus language learning with multimedia. In Miguel Farías and Katica Obilinovic (Eds.), Aprendizaje multimodal/Multimodal learning (pp. 21-49). Santiago de Chile: Publifahu USACH. [ Links ]

Smith, Frank. (1971). Understanding Reading. New York: Holt, Rinehart & Winston. [ Links ]

Serafini, Frank. (2011). Expanding perspectives for comprehending visual images in multimodal texts. Journal of Adolescent & Adult Literacy , 54(5), 342-350. [ Links ]

Serafini, Frank. (2012). Expanding the four resources model: reading visual and multi-modal texts. Pedagogies: An International Journal, 7(2):150-164. DOI: 10.1080/1554480X.2012.656347 [ Links ]

Siegel, Marjorie. (2012). New times for multimodality? Confronting the accountability culture. Journal of Adolescent & Adult Literacy , 55(8), 671-680. DOI:10.1002/JAAL.00082 [ Links ]

Stanovich, Keith E. (1989).Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16, 32-71. [ Links ]

The New London Group. (1996). A Pedagogy of Multiliteracies: Designing Social Futures. Harvard Educational Review, 66(1), 60-92. [ Links ]

Unsworth, Len; Thomas, Angela, and Bush, Robyn. (2004). The role of images and image-text relations in group 'Basic Skills Tests' of literacy for children in the primary school years. Australian Journal of Language and Literacy , 27(1), 46-65. [ Links ]

Unsworth, Len. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning-making resources of language-image interaction. English Teaching: Practice and Critique, 5(1), 55-76. [ Links ]

Unsworth, Len. (2008). Multiliteracies and metalanguage: describing image/text relations as a resource for negotiating multimodal texts. In Coiro, J. et al. (Eds.), Handbook of Research on New Literacies (pp. 377-405). New York: Lawrence Earlbaum. [ Links ]

Unsworth, Len, and Chan, Eeveline. (2009). Bridging multimodal literacies and national assessment programs in literacy. Australian Journal of Language and, 32(3), 245-257. [ Links ]

Vincent, John. (2005). Multimodal Literacies in the School Curriculum. Melbourne: University of Melbourne. [ Links ]

Villa, Nora. (2008). Propuesta de alfabetización visual para estudiantes de educación básica apoyada en recursos hipermediales. Un aporte a la comprensión lectora. Revista Interamericana de Bibliotecología, 31(1), 207-225. [ Links ]

Yi, Youngjoo. (2014). Possibilities and challenges of multimodal literacy practices in teaching and learning English as an additional language. Language and Linguistics Compass 8(4), 158-169. DOI: 10.1111/lnc3.12076 [ Links ]

1Support to write this paper was granted by DICYT-USACH (Dirección de Investigación Científica y Tecnológica at Universidad de Santiago de Chile).

2For a discussion of the nature of literacy, see Barone (2015), current ILA President, and Ferreiro (2001).

Received: August 27, 2015; Accepted: May 02, 2016

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License