Evaluating Ed.D.’s ELP Capstone

In education leadership programs across the country, doctor of education (Ed.D.) candidates are tasked with identifying a significant educational problem or challenge facing the organization in which they work. They then propose and test strategies to address the issue as part of their capstone requirement.

Although the literature is rich with examples, there are few empirical studies that examine this capstone approach. Researchers at the University of Delaware set out to determine their value — are they accomplishing the anticipated objectives?

Analysis of ELPs

School of Education faculty members Joan Buttram, Liz Farley Ripple and Doug Archbald developed an evaluation framework to take stock of UD’s capstone process. Implemented four years ago, candidates in UD’s Ed.D. program now produce an Education Leadership Portfolio (ELP).

A content analysis was undertaken to answer two research questions:

• What are the critical problems students are addressing in their ELPs?

• What are the different ways they go about addressing these problems?

In order to answer the first research question, the final summary and reflection documents from the 21 portfolios were examined. The problem statement and evidence offered to support the significance of the problem were studied carefully.

For the second question, the research team examined the 205 “artifacts” produced by candidates as part of their ELPs. Twelve distinct types of artifacts were identified: reviews of research, professional development, instructional resources, program evaluation, empirical study, internal communication, external communication, tools, policies/procedures, implementation or improvement plans, problem analysis, and design models.

An initial coding framework was collaboratively developed and organized around artifact purpose and artifact type, with three purposes identified: to understand the problem, address the problem, and assess efforts to address the problem.

Coding the data

The team focused on three descriptors to compare and evaluate the problems: level, focus, and motivation.

• Level refers to the level of the system in which candidates located the problem – organization, classroom and student.

• Focus classifies whether the problem statements primary focus was on practice, culture or structure.

• Motivation refers to how candidates’ anchored their problems. Three codes emerged in this category: accountability for student outcomes, new policy mandates, and vision- or mission-driven.

To more fully explore the significance of the problems identified, a set of codes was developed which documented the sources and types of evidence used to substantiate the problem. Data was further broken down into achievement, behavior/conduct, survey and other.

What are the ELPs accomplishing?

Analysis of the students’ portfolios revealed notable patterns in the nature of the problems and the students’ strategies for addressing those problems. Almost three-fourths of the candidates focused their ELPs on “problems of practice” at the organizational level, while six of the 21 were focused at the classroom level and one at the student level.

Problem statements focused on practice are defined as those necessitating changes in the work of educators within the organization, classroom, or for students. As an example, improving literacy instruction is a problem of practice.

Over 60 percent of the ELPs were directed at efforts to improve student outcomes. Both Annual Yearly Progress and Race to the Top funds also likely influenced candidates’ choice to focus on reshaping the practices of educators within their organizations (i.e., focus on practice).

This was unsurprising, given that the majority of Ed.D. students are in, or aspire to, administrative roles, and these topics would garner the greatest return on their effort.

Ed.D candidates relied on multiple strategies to address problems, including the design and delivery of professional development, the adoption of instructional resources and pedagogy, and internal and external communications to set expectations and inform practice. In fact, slightly more than half of the artifacts were categorized as addressing the problem. This suggests that the intent for the ELPs to be action-oriented is being realized.

In addition, over half of the candidates’ artifacts involved empirical studies or program evaluations. This finding aligns with the program’s emphasis on evidence-based decision making.

Jay Owens, director of special services in the Indian River School District, has found the ELP process to be extremely valuable, saying it “has helped to drive my personal and professional growth. Guidance provided by the UD team challenged me to explore areas of my work that inevitably added tremendous depth to my products. The program has instilled in me the importance of thinking analytically about concepts during the decision making process.”

Limitations of ELPs

Few of the candidates’ problem statements addressed cultural or structural issues.

• Problem statements focused on culture are defined as those that require shifts in norms and beliefs within the organizational community. Developing a culture of learning in a new school, for example, represents a problem focused on culture.

• Problem statements focused on structure are defined as those entailing changes in structural features of the organization such as departmental structures or master schedules.

ELPs likely focused on “problems of practice” because the results are more immediate and visible. “Problems of culture” generally require significant time to demonstrate results and are not as easily observed or measured.

Another potential explanation is that problems of culture or structure may not be as well developed as problems of practice in the Ed.D courses.

The team also found that candidates proposed fewer strategies to assess the ELP’s impact. This may be a result of the ELP process. Candidates are required to prepare and defend an ELP proposal that documents a problem exists. Once candidates have successfully defended their ELP proposals, it may be less important to further investigate the problem.

While this evaluation was limited to products of a single doctoral program, the findings are beneficial for UD’s program as well as others engaging doctoral students in addressing critical problems of practice.

UD’s education faculty will continue to reflect on the design and identify ways to strengthen its value for leadership preparation and for the improvement of practice.