Reading Multimedia Texts
UD digital literacy expert explains differences between reading in print and online
Rachel Karchmer-Klein, associate professor in the College of Education and Human Development at the University of Delaware, co-designed a research study where a group of high-achieving eighth graders were asked to engage with a digital narrative text. The story incorporated written words, sound, static images and video animations. The students quickly figured out they had to interact with it similarly to playing a video game in order to keep the story moving along.
When students were asked questions about the narrative, Karchmer-Klein and her colleague found that the students were focused so much on the written words that they missed a lot of the details.
“The details of the story and the fun and really cool stuff that came out in the narrative text were told through the sounds and the pictures,” said Karchmer-Klein, who teaches courses in literacy and educational technology at the undergraduate, graduate and doctoral levels in CEHD’s School of Education. “They engaged with it, but because they were focusing on the words that were presented in the digital texts, they missed a lot of details.”
It comes as no surprise that the students were so focused on the written words, Karchmer-Klein said, since that was what they’ve been taught to do since kindergarten.
“Overall, readers privilege the written word,” she said. “One reason is because K-12 schooling spends most of its instructional time teaching kids to read and write written language. We assume that because kids today are growing up with digital devices that they can figure out how to communicate effectively with them. We don’t spend time teaching other modalities like how to make meaning from audio, sound and moving images. Yet, there is a robust line of research indicating the need for this instruction. So just like we teach phonemic awareness, phonics and fluency, we need to teach these reading skills beginning in kindergarten.”
Karchmer-Klein said there are four unique characteristics of digital texts — the medium we use to communicate over the internet — we should teach students. The first is that digital texts are nonlinear. With a printed book (in the English language), the reader reads left to right, top to bottom. But when engaging with a digital text, reading tends to be nonlinear and many times non-sequential — the reader does not have to go through the text in a sequential manner to make sense of the text.
A second characteristic is that while a physical book is static — once printed, it can’t be changed — but a digital text is malleable. Karchmer-Klein informs students that when doing research online, a source they find can be gone or altered within a day, or even within minutes.
Technology also allows students to connect with anyone in the world.
“As soon as I, as the teacher, am allowing my students to get onto the internet, for example, I am opening a door to resources that I can’t necessarily constrain,” Karchmer-Klein said. “Whereas when I give a traditionally printed textbook to my class, I know what is in it and it can’t be modified.”
The final difference is multimodality — the use of multiple modes. Modes are signs or symbols, and their meaning comes from social and cultural uses and interpretations. Digital text consists of multiple modes pieced together. Examples are live or recorded speech, still or moving images, music, ambient sounds and tactility through touch screens. When these modes are combined with written language into one digital text, they require a high level of cognitive flexibility, meaning readers must navigate among and between modes to understand the content and progress forward in the text.
Texts, whether they be narrative or informational, typically have written language. Early literacy education tends to focus on the written word, the alphabet and phonemic awareness.
“What we don’t do is spend a lot of time talking about all of these other modalities that can actually come into play and hold the cognitive load of meaning,” Karchmer-Klein said. “So sure, we use picture books and we teach kids that pictures can hold meaning, especially when kids are really young, but that kind of all goes away as kids get older and we really focus on the words, which are important. But if you look at digital texts, there are a lot of other modalities that become not just supplemental, not just as important as the written word, but sometimes even more important.”
Some studies have shown that when people read on-screen, they don’t understand what they’ve read as well as when they read in print. But Karchmer-Klein said there are many variables that can affect one’s comprehension when reading online — such as the age of a digital device and the reader’s comfort level using it — and that reading strategies are much more important than the tool itself.
“Say I have a book and I also have a website, and I want you to use the strategy of close reading. That strategy is an evidence-based strategy. It’s been systematically studied, and it’s been proven to be a useful strategy with kids,” she said. “So the tool — whether it be I’m using this passage that is traditionally printed or I’m using it on technology — doesn’t matter. What matters is that I’m using an evidence-based strategy. The bottom line is it’s the strategy. It’s not the tool.”
The technology isn’t going anywhere, and students need to learn how to use it responsibly, Karchmer-Klein said. Much like many educators in the ’90s and early 2000s were hesitant to bring the internet into classrooms, the use of generative artificial intelligence (AI) in educational settings is now a hot topic.
“With the internet, the argument was, ‘This isn’t going away. We’ve got to figure out how to use it,’” Karchmer-Klein said. “The same thing is happening now with AI, which is why we have to figure this out, because these kids are going to have to know how to use it for the workplace. The best thing that we can do is not shy away from the things that we have today and, instead, get grounded in what we know.”
The biggest issue with technology, Karchmer-Klein said, is that people don’t know how to critically analyze information — a problem that is only getting worse with the expansion of generative AI.
Most high school and college students get their news from social media sites like Instagram and TikTok, but these students don’t understand the algorithms that these sites and search engines use — and they’re often unable to determine what’s misinformation.
“Nobody teaches you how to use social media. We just assume that people know how to use it. There’s a lot of ugly stuff that happens there, and that’s why we have so many people who are misinformed,” she said. “Teaching critical thinking skills, beginning in kindergarten and first grade, is so important, because all these skills continue to build up more and more and more and more to the higher grades.”
When schools and workplaces went virtual at the start of the pandemic, it became especially clear that most people didn’t know how to use technology, Karchmer-Klein said, also noting that a good thing that came from the pandemic was that access to technology became less of an issue.
Karchmer-Klein said “there’s no deep reading on the internet,” but a main reason for that is because students aren’t taught how to read online.
“We’ve all had instructions on how to read a book, but we haven’t had instructions on how to read online,” she said. “So it’s hard to compare these things when you’ve been instructed how to do one thing, but not on the other.”
Students need to learn that meaning can come from pictures, graphics, sounds — things other than words. They need to know that reading online is not always left to right, top to bottom — like it is in a traditionally printed text. Students need to learn that they might need to click on a link for more information and then come back to finish reading the original text.
That said, parents and educators certainly shouldn’t toss all the hard copy books.
“There needs to be a balance,” Karchmer-Klein said. “I don’t think they should be on a computer all day long. I think that’s horrible, too. But I do think it should be enough that they’re exposed to technology, just like they should be exposed to books. I love books. I would hate it if my kid didn’t have a book in front of them. I think there needs to be a balance. I think that’s the smart thing to do.”
Literacy and language
Literacy faculty at UD engage in innovative research designed to help children, families, teachers and their communities address challenges in language development, reading and writing. Supported by grants and collaborative educational partnerships, UD faculty study the neurocognitive basis of language development, leverage technology to provide high-quality instruction, and develop evidence-based interventions for early childhood through college students, including English Language Learners, historically disadvantaged students, and those with learning difficulties. The School of Education’s teacher preparation programs prepare graduates to use these evidence-based instructional approaches in their future classrooms. Learn more about literacy and language research at UD.
Read this story on UDaily.
Illustration by Jeffrey C. Chase.