Menu
header photo

The Journal of Multimodal Rhetorics

ISSN: 2472-7318

Accessibility

To download a free dyslexia-friendly font, please visit OpenDyslexia (not associated with JOMR).

To download a free ADD/ADHD-friendly font, please visit BeeLine Reader (not associated with JOMR).

Reframing Literacy: A Framework for Uncovering Meaning in Children’s Multimodal Expression

Francesca Ciotoli, St. Thomas Aquinas College

PDF


 Purpose

This case study was conducted to examine the potential applications of multimodal analysis as a pedagogical strategy for the teaching of students who may experience marginalization due to teachers’ conceptions of literacy and disability. Specifically, this project aimed to utilize Kress and van Leeuwen’s (2006) Grammar of Visual Design (GVD) framework as a tool for analyzing the creation of one child’s images for the purposes of:

1. Understanding how visuals may be used as a set of culturally bounded resources to express and understand meaning in classrooms.

2. Exploring the potential of a multimodal approach to and analytic method for visual literacy in teaching and learning.

3. Resisting the dominant discourse of disability that positions students for whom oral and written language is not readily available as non-literate.

 

Theoretical Framework

Literacy is traditionally understood as an individual’s cognitive skills that allow them to read and write with proficiency (Serafini, 2014). The predilection for oral and written language reifies “the normal curve and normal people” (Gallagher, 2010, p. 29) and privileges those who are deemed “normal” from those who are not. Traditional conceptions of literacy have denied rich literacy opportunities for children with significant developmental disabilities because they are seen as too impaired (Kliewer & Biklen, 2001) and the “literate construction of meaning” (Kliewer, 2008) is reserved for those with a fluency in the basic-skills phonics model. Literate citizenship in classrooms then, is inextricably tied to teachers’ ableist notions of acceptable forms of expression.

Viewed from a sociocultural perspective, literacy is a highly contextual social practice that people engage in rather than an individualistic acquisition (Gee, 1996; Serafini, 2014). Multiliteracies theorists posit the importance of linguistic diversity and multimodal forms of expression (Cope & Kalantzis, 2009, 2015). Writ broadly, multiliteracies theory and pedagogy focus on a broadened definition of being literate, in which competencies are practiced in accordance with particular settings, identities, and social practices. In education, “literacies” have come to mean more than that which is linguistic or conventional. “New” literacies serve as an “umbrella term” (Lankshear, Knobel, & Curran, 2013, p.1) to describe digitized material and media and that which is “multicultural and democratic” (p. 2), such as civic literacy and critical media literacy. In sum, the concept of a single educational literacy is now understood as a collective of literacies that help us navigate life and more fully engage in a democratic society.

At the heart of multiliteracies theory is the acknowledgement that communication is more than verbal and written language. Multimodality is a term used to describe socially recognizable forms (artifacts) through which meaning can be communicated. Each mode used in a multimodal ensemble contributes to the artifact’s overall meaning; thus, the ways in which modes are used impact how the artifact is understood (Kress, 2009). In particular, visuals have gained recognition as a critical component of multiliteracies theory (Serafini, 2014). Visual literacy refers to the ability to make meaning of information that is presented in the form of an image. Visual literacy theorists assert that images, like printed and written texts, can be “read” and thus require separate and distinct competencies (Debes, 1968; Fransecky & Debes, 1972; Chauvin, 2003; Avgerinou, 2009). Being visually literate requires the ability, skills, and competencies to encode and decode visual communication.

Visual literacy theory has the potential to provide a “metalanguage” for which to teachers and students can engage in meaning-making that is contextual, subjective, and dynamic. In particular, Kress and van Leeuwen’s (2006) “Grammar of Visual Design” (GVD) is an analytic method drawn from social semiotics that allows visual images to be read as “text.” GVD is structured around three organizing principles: how objects and ideas are represented (the ideational metafunction); how the relationship between the maker and viewer is represented (the interpersonal metafunction); and how images are positioned or composed to create meaning (the textual metafunction) (Halliday, 1985). GVD provides a theoretical framework for making sense of the meanings in visual images:

1. Sign-making is a culturally, socially, and psychologically situated process.

2. Sign-makers use forms that are appropriate to their meaning-making in the mediums available to them.

3. Signs have meaning for the sign-maker and are not arbitrary.

4. The meanings of signs are often unknowable to their readers, but the metalanguage of visual images can help readers better interpret sign-makers’ meaning.

Taken together, these principles are the foundation of a multimodal approach and an analytic method that has the potential to help teachers reframe literacy and redefine with it means to be literate in their classrooms. Teachers who see beyond special education labels and can envision an “individual’s citizenship or right to full community participation…and craft responsive contexts to which one’s active citizenship might be fostered and realized” (Kliewer, 2008) demonstrate what Kliewer and Biklen (2007) call “local understanding.” A teacher’s lens of local understanding allows them to view every student, including those with disabilities, as a full citizen capable and of learning and participating as a literate citizen (Kliewer, 2008).

 

Methods of Inquiry

This case study consisted of examination of artwork created by my son, James (pseudonym) in 2014-2015, a then nine-year old boy with a diagnosis of autism spectrum disorder and obsessive-compulsive disorder who experiences challenges with both written and oral language. I squarely position myself at the nexus of scholar, educator, and mother and acknowledge the affordances and limitations such positionality incurs. I chose artwork from the aforementioned time period because James 1.) showed marked enjoyment and motivation in creating artwork at home; and 2.)  he was denied access to artmaking in his special education classroom because it was believed to increase his tendencies to “perseverate, resulting in non-appropriate behaviors” and thus placed “on extinction” (Individual Education Plan, 2015) as part of his behavior plan.

James’ oral expression was described by his Speech Therapist as:

James is more available to participate in a reciprocal conversation in regards in his recent activities, and or past experiences. Prompts are typically required to get him to engage in communication, however, he has begun a conversation by initiating the topic such as "Anna, Easter eggs". That is his way of letting his communicative partner know he wants to talk about something. He then follows verbal cues to continue the conversation by answering questions. James needs prompts to participate in a small group conversation. He is learning how to turn to the other person, call their name, wait for a response, then proceed to ask a question. James has begun to use language to describe pictures. Some prompts may require at times however, he has learned how to follow a model and is elaborating on his descriptive sentences more independently. James is sequencing pictures and events more readily with fewer prompts. He is mostly successful with who, what, where and when. He continues to have difficulty with "why" and "how" due to the abstract nature of the questions. James has a good understanding of basic and familiar synonyms and antonyms (IEP, 2014-2015).

James’ written language skills were described by his Special Education teacher as:

Writing still proves to be a challenge for him. Writing contributes to his compulsions. We switched from handwriting to using a laptop, donated by his paraprofessional. He started with writing 1 sentence and has increased to writing 3 sentences consistently. After which, his OCD tendencies contribute to his starting to cry because he wants to write something unrelated to the activity. For journal, James is given a subject by the regular education teacher. He then comes up with key ideas with his paraprofessional. James is then prompted to use the key ideas to come up with sentences (Individual Education Plan, 2014-2015).

Data collection was conducted naturalistically, at home over a two-week period, whenever James spontaneously chose to create artwork. The artwork is characterized as a multimodal ensemble (Serafini, 2014) because they combine visual image, text, and design elements. I will use the term “artifact” henceforth to refer to James’ artwork. The James’ artifacts, video-recordings of the artifact-making process, and anecdotals were used as the data for this study.

In preparing to analyze the visual images, I found it difficult to apply Kress and van Leeuwen’s methodology as presented in their book, “Reading Images: A Visual Grammar” (2006). A brief search of the literature yielded no practical application tools for using GVD. I therefore chose to create an organizational template that incorporated all elements of GVD that I could use to analyze James’ artwork (See Table 1). I organized GVD into three sections corresponding to the three metafunctions of visual literacy: ideational, interpersonal, and textual. Within each domain, I parsed out the distinct processes that may be used by makers. For example, within the ideational domain, makers may choose one of two processes of representation: a narrative process that presents unfolding actions and events, processes of change, or transitory spatial arrangements or a conceptual process that presents participants in generalized categories of class, structure, or meaning. I further delineated the distinct forms of presentation each process may take. For example, a narrative image may be represented through action or reaction depending on the vector that is formed emanating from a person or object. In my organizational template, each metafunctional domain retains its unique sets of “tools” that can be used to analyze an image and the template remains true to Kress and van Leeuwen’s original articulation. (See Table 2 for an example of how GVD was used to analyze one of James’ multimodal ensembles).

All artifacts were created at home with no help or motivation from myself or others. James had easy access to several communicative tools including white and colored paper, pencils, erasers, crayons, colored pencils, and markers. The first and third artifacts, titled, “Ann put six chickens in a chicken coop” (Image 1 in the following section) and “Kitchen!” (Image 3 in the following section) were created with paper and pencil. The second artifact, “Lollipops,” was created with white paper, pencil, and crayons (Image 2 in the following section). I developed all titles for identification purposes. James completed each artifact in one sitting and the sittings ranged in time from approximately five minutes (Image 1) to 20 minutes (Image 3). James displayed pride and joy during and at the completion of his artwork by jumping up and down and smiling. When asked to describe his creations, James usually responded with single words or short phrases such as “hangman” and “color the lollipop red.” I did not include any contextual information in my initial analyses, but strictly adhered to the GVD template to see what meaning could be gleaned from the artifacts themselves. The importance of contextual information in analysis will be discussed later.

The sample of images in the next section are analyzed using a social semiotic approach (Kress, Jewitt, Ogborn, & Tsatsarelis, 2001). Qualitative analysis of these images focused on James’ selection of available semiotic resources, modality choices and arrangement of design elements. These dimensions were examined particularly in relation to the three metafunctions (ideational, interpersonal, and textual) of visual literacy as defined by Kress and van Leeuwen (2006). Video and anecdotal analysis was then compared to the image analysis; consideration of James’ creation process and contributing contextual factors allowed for a more comprehensive understanding of his meaning-making.

 

Data Analysis

In this section I will demonstrate my use of the GVD template by describing my analysis of James’ artifacts. 

 

A chils'd drawing of a number 1, a sentence with each letter underlined like a fill-in-the-blank exercise, and a chicken.

Figure 1: “Ann put six chickens in a chicken coop.”
 

James uses a narrative process to represent an event. James’ narrative “tools” include: unidirectional transaction (a vector emanates from the chicken’s eye that is looking right at the viewer; text without balloons that is related to the image (text refers to chickens and there is a drawing of a chicken); geometrical symbolism (line directly underneath each letter of the text); and secondary participants that are related to the main participants in ways other than vectors (hangman noose and text are related to each other as parts of the game but peripherally related to the chicken drawing). It appears that the ideational metafunction of this image is to represent the experience of playing hangman.

James uses many tools, often as part of “systems” that make the relationship between himself, and the viewer understood. Contact is immediately established between the maker and viewer through the chicken’s direct gaze which “demands” attention. The chicken is viewed from one side and has a large, single open eye that take up most of its head space, giving the distinct impression that the chicken is staring at the viewer in an up-close and personal way. Further, the close proximity of the hangman’s scaffold and text (directly above the chicken) communicates that they are meaningful to viewer. A modality system helps a viewer to determine the credibility of a representation, further establishing a relationship between his/herself and the maker. Modality markers serve as cues for determining the “truth value” of a representation. James’ use of black pencil on white paper gives no color saturation, differentiation, modulation, or background to the image. The drawing has little detail, aside from those features which distinguish it as a chicken (feet, beak, and feathers on top of its head and tail) and little depth aside from some parts which have multiple drawing lines that create a slight shading effect. Together, these modality markers indicate that the interpersonal metafunction is technical or factual in nature – James provides a straightforward, accurate depiction of his experience playing hangman.

The position of images and text can also be analyzed to understand James’ meaning making. The placement of the hangman’s scaffold and text directly above the picture of the chicken can be interpreted as an ideal-real relationship. In other words, James is relating the idea of a chicken in the form of a hangman’s game and then realizing the subject in his drawing of a chicken. Salience is conveyed by the elements that draw the most attention through visual cues such as size, sharpness of focus, placement in visual field, and cultural symbols is integral to the maker’s and viewer’s meaning-making. In addition, the contrast between text and visuals in this art helps “frame” the message that is relayed. The textual metafunction of “Ann puts six chickens in a chicken coop” can be understood as symbolizing parts of a whole game of hangman: the hangman’s noose, the phrase to be guessed, and a picture hint. (See Table 2 for the completed Analysis Template).     

The second artifact represents the human world through conceptual rather than narrative means (ideational metafunction). The subjects (lollipops) can be interpreted as pertaining to a generalized category of lollipop candy: each participant (lollipop) is of equal size and the image is repeated five times. Each lollipop is equidistant from the next and the lollipops are laid out in a horizontal sequence with an order indicated by color (red, orange, yellow, green, purple) and label (Color name, “Color the Lollipop [color name]”). The colors appear to be directly linked to the declarative statements written alongside the lollipop stems which indicate to “color the lollipop” the indicated color.

A series of drawings of different color lollipops and descriptions.

 

A close-up of two of the lollipops, the red and orange.

Figure 2: "Lollipops."

 

In this artifact, the maker-viewer relationship (interpersonal metafunction) is established through participants “gazing out” at the viewer. The lollipops are viewed from the front, establishing immediate contact. This effect is heightened by the size of the lollipops. Each lollipop fills the entire vertical space of the page and remaining space is used for accompanying text. Participant demand, close social distance, and a frontal perspective combine to directly address the viewer. In addition, the speech act to “Color the Lollipop” demands the viewer attention. A declarative statement conveys that something must be done and requires immediate attention. The fact that the lollipops are already colored implies that the act was performed by the maker and is being shared with the viewer. When considering the expectations of James’ classroom, this could be interpreted as a retelling of daily academic drills, but using a subject drawn from his own experience and interest.

Modality choices also inform the maker-viewer relationship. This visual series has high modality in that there is a high degree of correspondence between what we know a lollipop to be and what is depicted. Although it cannot be described as photorealistic, it has “higher reality value’ (Kress & van Leeuwen, 2006, p. 159) when considering the importance of color in the subject matter that is accentuated by full color saturation and the absence of color differentiation and modulation. Pencil lines that depict the outline of the lollipop candy and stick and the swirls within the candy are visible but not prominent and seem to serve the single function of distinguishing the items as lollipops. The primacy of color is shown against a vibrant white background and through the written texts that identify the color used by name and demand that the lollipop be colored the named color. The initial use of the color appears to be bolded through a technique of writing over the letters several times. The viewer can be reasonably certain that color is an essential vehicle for meaning and therefore the lack of other elements of modality do not decrease the artifact’s validity.

The degree to which the images relate to each other (textual metafunction) is especially important in gleaning meaning from this artifact. The fact that it is a series and was produced and constructed according to a specific pattern tells the viewer of the maker’s intentionality. Interpretation of that intentionality can be understood by examining the placement, salience, and framing of the images. Because the colors correspond to those in the rainbow (with the exception of the missing blue) the viewer can read images from left to right. Several separate sheets of paper were used to create a whole picture: the maker placed in the order of red, orange, yellow, green, and purple. The lollipops fill the entire page, colors are sharp and vibrant, and text is placed in the same place to the left and right of each lollipop stick. Although, there are no border lines used to frame each individual lollipop, it appears that the images were meant to be viewed as a series, given the importance of color and the use of different pieces of paper placed horizontally alongside each other. The papers actually overlap each other further supporting the idea that the series is intended to be viewed as one image rather than as five separate ones.

The third artifact can be viewed as conceptual in that participants are categorically related to each other (ideational metafunction). James depicts five items which can be understood using an analytical part-whole structure where the “whole” is the kitchen that is made up of “parts,” in this case, items found in a kitchen. The set is inclusive in that what is shown is not exhaustive: there are many more items to be found in a kitchen.

 

A child's drawings of a fridge, a cup, a bowl, a stool, and a pan.

Figure 3: "Kitchen!"
 

A familiar relationship between the maker and viewer is established through the direct gaze and close social contact of participants (interpersonal relationship). Collectively, the items can be interpreted as having an objective orientation. The maker’s representation is not dependent upon the relationship of the maker and viewer: a cup is a cup. In this way, the image takes on a scientific or technological role, that is further enhanced by the directly frontal (cup) and top-down angles (refrigerator, bowl, stool, and table) used. These angles do not indicate the position of the viewer, but rather “neutralize” (Kress & van Leeuwen, 2006, p. 144) perspective and attitude, further enhancing the image’s technical orientation. The primary use of the top-down angle provides “maximum power,” (Kress & van Leeuwen, 2006, p. 145) in establishing “theoretical, objective knowledge” of the subject. The use of black pencil on white paper also contributes to image’s technical orientation. Details are minimal – those incorporated seem to be crucial to the viewer’s identification of the object. For example, the refrigerator is identified partially through the drawing of the handles that might otherwise be identified as cabinets. The cup is distinguished from the bowl through the inclusion of the handle and might explain its rendering from a frontal perspective instead of top-down one (like that of the bowl).

The composition of Kitchen! differs from the previous two artifacts in that the placement of represented items do not appear to have meaning other than as part of a set; that is to say, they are not placed in a particular order and a reading from left to right or top to bottom does not interfere with the image’s overall meaning (textual metafunction). The text “Kitchen!” is placed at the top, on the left-hand side which may indicate intended it to serve as a label for the set.  Further, the image has no central element as items are of equal size, color, focus, and perspective (excepting the cup). Certain elements of each item are made more prominent through the darkening of drawn lines. The handles of the cup and the refrigerator and the bowl lid stand out, perhaps a technique the maker uses to call attention to what he perceives are the items distinguishing characteristics. Of note, are the bottom two items which can be interpreted as a subset rather than as two separate items (stool and table). Interestingly, the side of the table and the stool leg nearest to the table are darkened which lends credibility to an interpretation that they were intended to be “set within a set.” Finally, the fact that the text includes an exclamation mark is curious, although I surmise it is related to James’ excitement and interest in the topic.

 

Discussion

As with any mode and method, there were distinct affordance and limitations to using GVD as a framework for making meaning of multimodal ensembles. I will discuss these benefits and limitations below and make recommendations for further research. Limitations described are specific to my use of GVD as a tool for analyzing one child’s multimodal ensemble as the sole means for interpreting meaning.

Benefits of the GVD Framework

Focus on meaning-making rather than competency. The discrepancy between what James can communicate through verbal and written language and what he expresses through visual and physical means (e.g., drawing, patterning of objects, and repetitive actions) has been a long-standing issue in his education. James has always shown a marked preference for visual communication and at home uses a variety of multiple media for expression. He regularly uses Legos, paper and pencil, colored blocks, paint, and various household objects to represent his thinking, recreate his experiences, and express himself. He often creates multimodal ensembles that incorporate visual and written language. Generally, James’ teachers and therapists do not embrace his primary modes of communication; all schooling efforts have been to help him better express himself through oral and written language. Teacher after teacher insisted that because he was unable to verbally retell, summarize, and answer inferential questions, they could not move him to the next level of text complexity. Writing posed a similar problem: James could write answers to “who,” “what,” “where,” and “when” questions and his grammar and syntax was almost always spot on, but again teachers pointed to his needing writing prompts as indications of his lower-level thinking.

The influence of normalized paradigms of literacy and disability contribute to James’ teachers identifying him as less literate than his peers and the inability to consistently express himself in ways that are sanctioned (speaking and writing) make his inclusion in general education classrooms difficult. Collins (2011) calls the reading of classroom actions and interactions through the lens of deficiency “ability profiling” (p. xiii). From a sociocultural perspective, “identities” are inscribed over time as repeated instances of deficit “positioning” (Collins, 2011, p. 14) and result in the disabling of students who deviate from the ideal or norm. Teacher perceptions, grounded in institutionalized paradigms and policies concerning the “disabled student type” may influence instructional practices, particularly when those students are placed in general education classrooms.

In contrast to the established norms of language in classrooms, GVD is founded on a social semiotic theory of language which focuses on language “in-use” as a social practice (Gee, 1998). The emphasis on meaning-making, rather than on the proficiency of identifiable skills allows for a broader interpretation of literacy and what it means to be literate beyond that which is oral and written. Viewed holistically, the three multimodal ensembles demonstrate that James is highly literate. He can combine texts and visual images in coordinated ways to express his ideas about the world. For example, in “Ann put six chickens in a chicken coop,” James’ combined use of text, design elements, and drawing portrayed his experience of playing the game hangman. I know this because I understand the context: James and I play hangman regularly when waiting at a doctor’s office or at a restaurant and his favorite game on the iPad is “Hay Day” where there are indeed six chickens in every coop. The classroom teacher would like James to tell a story in a more conventional way, but his preferred modality makes his storytelling no less valid.

GVD analysis makes it difficult to view James as deficient or limited. The focus on meaning provides a strength-based, rather than deficit-based approach to James’ meaning making potentials. Underlying the GVD theoretical framework is the concept of choice: the freedom a maker has to choose what they will represent and how they will express it. GVD, thus, highlights the important relationship between the sign maker and the sign on determinations of meaning. (Kress & van Leeuwen, 2006). After an analysis of James’ three artifacts that includes consideration of context, I concluded that James makes distinct choices about the use of color. James always as access to different writing tools, including crayons, colored pencils, and lead pencils. His intentional use of color, or lack of it, is therefore meaningful. For example, he used color in cases where color was integral to the communication of his ideas as with the lollipops (Figure 2). Because a distinguishing feature of lollipops is their vibrant colors, James used bright colors in his rendering. However, in his drawing of kitchen items (Figure 3), the use of color did not interfere with meaning and in fact may have distracted from their being recognized. GVD’s focus on an interpretation of meaninallowed me to view James’ use of color as strategic and therefore as a strength to be celebrated and nurtured.

Systematic reading of multimodal texts. An examination of James’ artifacts reveals that he has a lot to “say,” despite his difficulty using oral and written language. Multimodal expression, or combinations of the modes of visuals and texts, afforded him more meaning potential, but also requires corresponding viewer multimodal literacy. The GVD framework allowed me to analyze multimodal ensembles in a systematic manner that increased my understanding of James’ meaning-making. The process of looking for and identifying elements of ideational, interpersonal, and textual metafunctions shed light on what might have been overlooked without its use. For example, I might have paid little attention to James’ use of black and white in the first artifact had I not used the GVD framework as an analytic tool. GVD focused my attention on the absence of color as a possible strategic choice. As hangman is a game that is typically played with a pencil and paper, color may in fact have interferes with meaning if the function of the image is to represent an experience of the game. Rather than viewing text and visual images as separate and tangentially related, I was able to analyze it as a multimodal ensemble in which narrative representation and realization were meaningfully orchestrated.

The systematic process of bringing together discretely identified elements of the text, visual images, and design elements in a multimodal ensemble is critically important for interpreting meaning. The use of any single tool in the GVD framework (i.e., modality markers) is most valuable when applied as part of a larger system of making meaning. My interpretation that “Ann put six chickens in a chicken coop” is a game of hangman, is supported by viewing evidence drawn from analysis of all three metafunctions: the chicken is darker than the text, each letter of text is underlined, and the scaffold is located above text at left margin edge and both are located above the chicken (ideational metafunction); the close proximity of the scaffold and text that are located directly above the chicken (interpersonal); and the top to bottom reading of the game on top and subject of the game on the bottom (textual). I understand these elements to be James’ personal experience playing hangman, highlighting that the meanings of “signs” are intimately connected to their “sign-makers” (Kress & van Leeuwen, 2006).

Open-ended and universal application. The GVD framework, when applied using the analytical tool, has the potential for use for a variety of purposes and in a wide range of contexts. GVD allowed me to choose from among many of James’ artifacts to find those I felt would be representative of a range of his capabilities and interests. The GVD framework also allowed me to approach each artifact as a separate entity and later look holistically across them at some of the emerging patterns in James’ meaning-making choices. I concluded that James’ expression is deeply situated in his own – from game play (hangman, coloring pages) to the importance of food (candy and kitchen). He conveys these familiar experiences through the strategic use of color, spatial placement, and by combining textual and visual elements.

Beyond the GVD Framework

The ways in which people with autism communicate has long been the subject of scrutiny. Less “normalized” modes of communication are even viewed as embodiments of an underlying pathology. One of the earliest examples of pathologizing nonverbal autistic expression comes from Bettelheim’s interpretation of patients’ drawings. For example, Bettelheim wrote about “Joey, the Mechanical Boy” for Scientific America, using the child’s drawing of a robot as evidence of a “rejection of human feelings” (Bettelheim, 1959, p. 116) and the desire to be a machine. Bettelheim ascribes pathological meaning to the drawings because Joey is autistic. Bettelheim’s interpretation of Joey’s drawings reflects the then-current “empty fortress” theory in which autism is the natural by-product of motherly indifference and cruelty. Joey is thus “storied” into autism, in what Yergeau (2017) calls “the “clericalization of rhetoric” (p. 14). She argues that because autistics lack the traditional constructs of meaning-making, both their rhetorical modes and their stories are invalidated as deficient, lacking, and even perverse. Yergeau writes, “Autistics don’t tell us what we want to hear, nor do they tell us in the manner in which we wish to hear it” (p.22). For Bettelheim, Joey’s drawings were rhetorical “products” and proof of his pathology.

While GVD provides a strong framework to support interpretation of James’ meaning-making, it needs further development to incorporate a consideration of context. The creations, viewed in isolation as “products” present an incomplete “picture” of James’ storytelling. For example, as James began to work on the fifth lollipop image in “Lollipops,” I asked him why he chose purple instead of blue. I surmised that James’ color choices corresponded with the colors of the rainbow. He may have had another reason behind his choice. He said, “Original Skittles,” for which I understood that there are no blue candies in original brand, Skittles® candies. Without knowledge of this context, I might have concluded that James “missed the blue” or did not know all the colors of the rainbow. It might also be relevant that James had gone to the store earlier in the day with his father and purchased two bags of original brand, Skittles® candies. Another example of the importance of context in interpretation is that immediately prior to the creation of “Ann put six chickens in a chicken coop,” James was playing a game on his iPad called “Hay Day,” an online farming simulator where participants can plant crops and trees, raise animals, and sell products. Taken together, James’ recent learning of the game hangman in combination with his love of chickens and the game Hay Day, provides salient and meaningful information for interpretation. The importance of context cannot be underestimated in both in James’ meaning-making but also in my ability to see James as highly literate.

Important information can be gathered by considering the specific context in which an artifact was created. A close examination of the process can yield rich data for analysis. For example, before James made “Kitchen!” I observed him using the iPad to look up various pictures of kitchen items. He began his search using Google Images by typing the key word “kitchen” into the search bar. Subsequently, James narrowed his search to “kitchen things” and eventually to “kitchen worksheets.” He scrolled through many pages of images until he found a preschool worksheet with approximately ten outline drawings of items found in a kitchen. James clicked the picture, enlarged it, and then clicked the “go to website” button where he found an enlarged PDF. James then selected among the ten images and chose three to draw. He later found two of the images on another web page after a similar search. These actions are more than just anecdote; they provide a rich description of James capabilities and interests that move beyond what is actually represented on paper. Contextualizing James’ process provides crucial information that supports a more comprehensive understanding of James’ meaning-making. It may in fact be beneficial to take anecdotal evidence and video of the making-process as a way of incorporating process and context into the GVD framework.

 

Implications for Research and Practice

Broadening Conceptualizations of Literacy

The results of this study indicate that meaning is communicated using a variety of modes. Understanding how meaning is made through different modes is crucial if multimodality is to be used effectively. Multimodality requires an acceptance of multiliteracies as a way of addressing the diversity of expression and communication in human beings. If teachers are to meet the needs of learners who have a wide range of communicative and expressive abilities and preferences, they will need to embrace that literacy is more than the reading, writing, and speaking of written language. GVD is a potential tool for helping teachers see the meaning potential in multimodal ensembles and the capabilities of all students in their classrooms that should be paired with a consideration of each child’s individual context. This requires a critical rethinking of the ways that disability, and autism in particular, has been viewed in schools. We must move beyond the pathologizing paradigm that harkens back to Bettelheim which strips children’s expression of their contextual details of interaction and communication (Dindar, Lindblom & Kärnä, 2017).

Developing Multiliterate Teachers

The exclusive use of particular modalities of expression and communication in classrooms can have significant consequences for all students. We need to prepare students with the skills and competencies for making sense of various modalities, including multimodal ensembles. In addition, students have a range of capabilities and interest in different modalities. Teachers too must have a wide range of knowledge and competency in multiliteracies in order to acknowledge, embrace, and foster multimodality in teaching and learning. The GVD framework can serve as sharpening lens for interpreting the meaning students are making with multimodal ensembles in the advancement of multiliteracies used in curriculum, pedagogy, and preservice education.

Instructional Applications

The GVD framework allows for the systematic analysis of multimodal ensembles. Questions remain about what, if anything, teachers can do with the understandings they have gained from students’ multimodal ensembles. Is it possible to translate information about meaning making modes into appropriate learning goals and experiences? Can GVD be applied in useful ways for teachers while also keeping true to its focus on meaning-making? To answer these questions, more research is needed. I am not suggesting GVD be taken up as intervention is traditional special education assessment in which it might be used to further pathologize interpretations of students meaning making. Instead, it could be used to counter those traditional conceptions of literacy, particularly by focusing on the contextual aspects of children’s expression. One possibility is the development of a protocol for using the GVD to help teachers connect the analysis of multimodal ensembles to building rich, contextual, strengths portraits of individual students. Teachers might use the GVD framework as one way to assess students’ knowledge, skills, and motivations in support of appropriate and meaningful learning goals, curriculum, and experiences that build on children’s competencies and interests. The development of a protocol that includes contextual information for using GVD may provide a viable link from theory to application.

 

Conclusion

The GVD framework is built on strong theoretical grounding in social semiotics theory that supports inquiry into the ways multiliteracies may be used to inform and enhances inclusive teaching practices that recognize and celebrate the myriad ways children express themselves and communicate with others. The greatest strength of GVD is its emphasis on meaning-making which provides a different perspective of what forms of expression are deemed valid and in classrooms. A careful analysis of how meaning is made through various processes and tools and a consideration of the individual context of the meaning-making act itself shifts the focus on what the child is “saying” and “doing” rather than on perceived deficits and disabilities. A strengths-based approach is a necessary orientation for inclusive teaching and the framework provides ample opportunities for children to participate in visual meaning-making modes when learning. 

 

Appendix 

(Click the link for Table 1: Grammar of Visual Design (GVD) Framework, and Table 2: Visual Literary Analysis Form – Completed Template)

 

References 

Avgerinou, M. D. (2009). Re-viewing visual literacy in the "bain d'images" era. TechTrends, 53(2), 28. doi.org/10.1007/s11528-009-0264-z

Bettelheim, B. (1959). Joey. Scientific American200(3), 116-130.

Chauvin, B. A. (2003). Visual or media literacy? Journal of Visual Literacy, 23(2). doi.org/10.1080/23796529.2003.11674596

Collins, K. M. (2013). Ability profiling and school failure: One child’s struggle to be seen as competent. Routledge.

Cope, B. & Kalantzis, M. (Eds). (2015) A pedagogy of multiliteracies: Learning by design. Palgrave Macmillan.

Cope, B., & Kalantzis, M. (2009). “Multiliteracies”: New literacies, new learning. Pedagogies: An international journal, 4(3), 164-195.

Debes, J. (1968). Some foundations for visual literacy. Audiovisual Instruction, 13(9), 961-964.

Dindar, K., Lindblom, A. & Kärnä, E. (2017). The construction of communicative (in)competence in autism: a focus on methodological decisions, Disability & Society, 32:(6), 868-891, doi:10.1080/09687599.2017.1329709

Fransecky, R. B., & Debes, J. L. (1972). Visual Literacy: A Way to Teach-A Way to Learn. Washington, DC: Association for Educational Communications and Technology.

Gallagher, D. (2010). Educational researchers and the making of normal people. In Dudley-Marling, C., & Gurn, A. (Eds.). The myth of the normal curve, 25 - 38. Peter Lang.

Gee, J. P. (2014). An introduction to discourse analysis: Theory and method. Routledge.

Halliday, M. A. (1985). Context of situation. Language, context, and text: Aspects of language in a social-semiotic perspective, 3-14.

Kliewer, C. & Biklen, D. (2007). Enacting literacy: Local understanding, significant disability, and a new frame for educational opportunity. Teachers College Record, 109(12), 2579–2600. Retrieved from http://www.tcrecord.org/Content.asp?ContentId=14490

Kliewer, C., & Biklen, D. (2001). “School's not really a place for reading”: A research synthesis of the literate lives of students with severe disabilities. Journal of the Association for Persons with Severe Handicaps, 26(1), 1-12. doi.org/10.2511/rpsd.26.1.1

Kliewer, C. (2008). Seeing all kids as readers: A new vision for literacy in the inclusive early childhood classroom. Paul H. Brookes.

Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. Routledge.

Kress, G. R., & Van Leeuwen, T. (2006). Reading images: The grammar of visual design. Psychology Press.

Jewitt, C., Kress, G., Ogborn, J., & Tsatsarelis, C. (2001). Exploring learning through visual, actional and linguistic communication: The multimodal environment of a science classroom. Educational Review53(1), 5-18.

Lankshear, C., Knobel, M., & Curran, C. (2013). Conceptualizing and researching “new literacies.” The encyclopedia of applied linguistics2. doi:10.1002/9781405198431.wbeal0182

Linton, S. (1998). Claiming disability: Knowledge and identity. NYU Press.

Mills, K. A. (2009). Multiliteracies: Interrogating competing discourses. Language and Education, 23(2), 103-116. doi.org/10.1080/09500780802152762

Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard educational review, 66(1), 60-93.

Yergeau, M. R. (2018). Authoring autism: On rhetoric and neurological queerness. Duke University Press.