Assessing effectiveness of PEG Writing software

Imagine an elementary classroom where students could write an essay on their computer and receive instant feedback on its organization, word choice and accuracy. They would then be able to correct errors, make revisions and resubmit their essays, allowing their teacher to focus on developing their critical thinking and higher-level writing skills.

The PEG Writing system, developed by Measurement Incorporated (MI), utilizes an automated essay scoring (AES) that immediately flags lower-level writing errors like spelling, punctuation and formatting, and provides suggestions for improving organization, sentence structure and word choice.

This automated essay evaluation (AEE) software system is being used by nearly three-quarters of a million students in the United States and several other countries.

While researchers have investigated the reliability of scoring models, Joshua Wilson, assistant professsor in University of Delaware’s School of Education, is taking a different approach. His research focuses on evaluating the effectiveness of AEE on teaching and learning.

He is looking to determine if it help students — particularly those who are at risk of failing to achieve grade-level literacy standards — improve their writing skills.

Writing instruction is gaining importance

Under the federal No Child Left Behind policy, writing instruction took a backseat to reading and math. With the introduction of Common Core standards and associated literacy assessments (e.g., Smarter Balanced) writing has once again been brought to the forefront.

“The majority of U.S. students lack sufficient writing skills,” said Wilson. “Roughly two-thirds of students in grades four, eight, and 12 fail to achieve grade-level writing proficiency, as measured by the National Assessment of Educational Progress (NAEP).”

Why does this matter? “Students with weak writing skills are at a greater risk of dropping out of school or being referred to special education programs — potentially failing to secure stable and gainful employment,” said Wilson.

Yet teachers face many barriers with respect to teaching writing.

“A major impediment is the time it takes to evaluate student writing,” said Wilson. “Subsequently, many students do not get adequate practice or feedback to improve as writers.”

Wilson has found that PEG utilizes natural language processing to yield essay ratings that are highly predictive of those assigned by human raters. Consequently, students receive more thorough feedback.

Stacy Poplos Connor, a master teacher in The College School, started working with PEG after reading about it in UDaily. She has found it to be an incredible teaching and learning resource tool.

“The students receive immediate feedback on each of the main traits of writing,” she said. “If they are struggling with a particular genre and/or editing tool, they can watch videos embedded into the site or send me messages while they write.”

Findings of the research

For the past three years, Wilson has partnered with the Red Clay Consolidated School District to pilot and research PEG Writing in grades three-five, and select classrooms in the Red Clay and Colonial school districts’ middle schools. Collaborating with a number of UD graduate and undergraduate students, Wilson has published three studies focusing on the impact PEG has on both learning and instruction.

In the classroom, he has found:

• Students’ quality of writing improves in response to PEG’s automated feedback, and it appears to be equally effective for students with different reading and writing skills.
• On average, students with disabilities participating in the general education curriculum are able to use automated feedback to produce writing of equal quality to their non-disabled peers, even when their first drafts were significantly weaker.
• PEG can identify struggling writers early in the school year so they may be referred for intervention to remediate any skill deficits.

Wilson also compared results of teachers using GoogleDocs versus PEG to evaluate its impact on writing instruction. The analysis revealed that:

• Students in the PEG Writing group demonstrated an increase in writing motivation. There were no changes in writing motivation for students in the GoogleDocs group.
• Teachers using GoogleDocs provided proportionately more feedback on lower-level writing skills, such as capitalization, punctuation, and formatting.
• While teachers using PEG Writing still commented on lower-level writing skills, they were able to provide more feedback on higher-level writing skills, including idea development.

“PEG allows me to pick a topic based on several genres while also incorporating my own teacher-created prompts that fit with my instruction,” said Connor. “It has definitely saved me time in grading and editing. I would highly recommend PEG to any teacher.”

Future research

Wilson is now piloting and researching PEG at George Read Middle School, Gunning Bedford Middle School and Castle Hills Elementary School in the Colonial School District. He is expanding his analysis to determine  if the additional practice and feedback through PEG writing can help English language learners (ELL) in elementary and middle school pass the ACCESS for ELLs test.