Research Spotlight: Joshua Wilson

When ChatGPT burst onto the scene in November 2022, many educators and parents worried that new writing tools powered by artificial intelligence (AI) would help their students bypass important learning opportunities. Instead, as University of Delaware Associate Professor Joshua Wilson has shown, AI-powered writing and evaluation tools have actually helped students develop their writing skills and have supported teachers in providing meaningful feedback.

Now, in a recent study published in Learning and Instruction, Wilson and his co-authors turn their attention to elementary English learners (EL), investigating how this growing population of students interacts with and benefits from automated writing evaluation (AWE) software. They found that AWE technologies were equally beneficial for ELs as they are for non-ELs. Study participants accessed writing feedback to a similar extent, achieved equal gains in writing quality, focused on consistent dimensions of writing when revising and endorsed the AWE system to similar degrees, regardless of their language status.

“As AI-based feedback applications become increasingly prevalent, it’s critical that researchers examine the consequences of implementing those tools in authentic educational settings, with a particular focus on equity,” said Wilson, who specializes in literacy in UD’s College of Education and Human Development (CEHD). “This study represents a novel step forward in the field of AWE by focusing on students’ multifaceted engagement and by ensuring that there were not systematic differences in engagement that might disadvantage vulnerable subgroups. This approach sets a precedent for other investigations into AI-based feedback applications, ensuring that these technologies support equitable learning outcomes for all students.”

Automated essay evaluation software

AWE refers to a class of educational technology tools that use natural language processing and AI to provide students with automated formative feedback that supports improvements in writing quality. Wilson’s study focuses on MI Write, an AWE system designed to improve the teaching and learning of writing by providing students with automated feedback and writing scores.

When a student drafts an essay and submits it within MI Write, their essay is instantly analyzed by its scoring and feedback algorithms, which then deliver immediate automated feedback directly to the student. In addition to providing the student with a holistic writing score, it also provides a specific writing score and feedback on idea development, organization, style, sentence fluency, word choice and conventions. MI Write also includes peer review capabilities, offers multimedia lessons for skill-building and allows teachers to communicate with students through commenting features.

“[The students] seem more determined and [MI Write is] so catered to them,” said a fourth grade teacher in Wilson’s study. “It’s almost like there’s a person, like somebody [teaching] them, conferencing with them, telling them how they can improve, and it’s all of them at once. Whereas before, I wouldn’t be physically able to accomplish that quickly.”

Benefits for English learners

To assess how ELs interacted with and benefited from AWE technology, Wilson and his co-authors collected data from nearly 3,500 students in grades 3–5 within a Mid-Atlantic district that implemented MI Write in all 14 of its elementary schools during the 2017-18 school year. They collected data from ELs—students whose home language is not English and who qualify for English language services—and from non-ELs.

To investigate interaction with the AWE software, Wilson and his co-authors looked at three dimensions of engagement: 1) behavioral, or the actions students take in response to feedback; 2) cognitive, or the thinking and revision strategies that students use in response to feedback; and 3) affective, or how students feel about and perceive feedback.

Across all three dimensions, Wilson and his co-authors found similar levels of engagement across all students. For example, some students chose not to access the feedback that AWE provided, while others accessed it several times. But these differences in behavioral engagement were not associated with language status. Similarly, ELs made productive revisions to their texts to the same extent as non-ELs and they often focused on the same set of textual features when revising.

In addition, student perceptions of the AWE software were generally positive. And, interestingly, EL survey responses indicated an even higher positive perception of the automated feedback.

“Elementary-aged ELs may be more motivated to master the mechanics of writing, given that their proficiency in writing is both a barrier and a gateway to their academic success across various subjects,” Wilson said. “But, the automated nature of AWE also offers a unique and invaluable advantage for ELs: it enables them to request unlimited feedback without the fear of burdening their teacher or exposing their mistakes. AWE offers ELs autonomy in seeking feedback, which may feel empowering, allowing ELs to refine their writing skills at their own pace.”

Finally, Wilson and his co-authors found that the overall improvements in student writing over the course of the school year was not related to language status. Using MI Write’s essay-scoring system, they found that the average gain scores for ELs and non-ELs were similar after accounting for the number of drafts completed, initial draft quality, demographics and baseline literacy skills.

But, even in light of these positive findings, Wilson emphasizes that it’s important to view AWE as a teaching tool rather than as a replacement for classroom teaching.

“Our study underscores the importance of integrating AWE and AI-based systems within the context of effective teacher-led instruction,” Wilson said. “While there was no evidence that AWE uniquely disadvantaged elementary ELs, it also did not uniquely advantage them or close existing performance gaps. While AI and AWE can significantly enhance personalization and provide valuable support, they should be seen as tools, not solutions. Ensuring equitable outcomes requires a holistic approach that combines technology with skilled teaching.”

This research represents one aspect of Wilson’s larger research program in automated writing evaluation and related technologies. With Mark D. Shermis, Wilson has released a new edited collection, the Routledge International Handbook of Automated Essay Evaluation, which includes articles on large-scale and classroom-based applications of this technology to support teaching and learning.

To learn more about the reliability of automated writing evaluation, read CEHD’s spotlight on Wilson’s work. To learn more about CEHD research in language and literacy, visit its research page.

Article by Jessica Henderson. Photo by Evan Krape.

Header image caption: A teacher at CEHD’s The College School works with a student on a writing assignment using a computer.

About Joshua Wilson

Joshua Wilson is an associate professor in CEHD’s School of Education. His research focuses on ways to improve the teaching and learning of writing, especially through automated writing evaluation systems. His research has been supported by grants from federal, foundation and industry sponsors and has been published in Computers & Education, the Journal of Educational Computing Research, the Journal of Educational Psychology and the Journal of School Psychology, among others. Wilson also facilitates UD’s AI for Teaching and Learning Working Group with three colleagues.

Literacy and Language Faculty at CEHD

Wilson’s research complements the work of CEHD faculty studying literacy and language, including Steve Amendum, Christina M. Budde, Martha Buell, David Coker, Stephanie Del Tufo, Ralph Ferretti, Roberta M. Golinkoff, Myae Han, Rachel Karchmer-Klein, William Lewis, Charles A. MacArthur, Adrian Pasquarella, Kristen D. Ritchey, Carol Vukelich and Sharon Walpole.