AERA 2022

The 2022 Annual Meeting is a dual-component conference with sessions offered on-site in San Diego, CA, and other sessions offered on a virtual platform. The 2022 theme is “Cultivating Equitable Education Systems for the 21st century.

Thursday, April 21

Developing Automated Feedback to Improve Teachers’ Formative Assessment of Student Work in Technology-Enhanced Science Curricula

Trudi Lord, Hee-Sun Lee, Chris Lore, Amy Pallant

8:00 – 9:30 AM PDT (11:00 – 12:30 PM EDT), Marriott Marquis San Diego Marina, Floor: North Tower, Ground Level, Pacific Ballroom 18

The current call for teaching disciplinary core ideas through science practices (NRC, 2012) presents a challenge for teachers who strive to provide ongoing and timely support for their students (Ruiz-Primo & Furtak, 2006). Teachers need to be aware of “how judgments about the quality of student responses (performances, pieces, or works) can be used to shape and improve the student’s competence by short-circuiting the randomness and inefficiency of trial-and-error learning” (Sadler, 1989, p.120). However, teachers rarely have either the time or the pedagogical, domain, and assessment knowledge to provide students with individualized feedback in real-time or before the next class.

Developing automated teaching feedback to assist teachers’ formative assessment is to be carried out in phases. In the first phase, this study is carried out in the context where student assessment information from an online student module is fed into a teacher dashboard that is connected to a teacher edition of the student module. See Figure 3. We hypothesize that Machine Learning (ML) algorithms can identify how teachers’ interactions with the dashboard relate to changes in student learning behaviors individually and collectively. Identifying these patterns is important to develop data-driven automated teaching feedback that would be directly integrated into the teacher dashboard in the following phase.

We recruited 40 middle school teachers across the U.S. to implement an online science curriculum module on wildfire risks and impacts. The Wildfire Module has three components: the student version of the module, the teacher edition with educative curriculum materials layered on top of the student version, and the real-time teacher dashboard. See Figure 3. The data will be collected in the fall of the 2021-2022 school year.

The three components of the Wildfire Moule will log all user interactions and artifacts. Our analysis will focus on eight scientific argumentation tasks embedded in the module where students use real-world data and simulation results to make and support claims or predictions about risks and impacts caused by wildfires. In particular, we will examine to what extent the teachers use the three main features of the dashboard:

1. Main view to track student progress throughout the module: How often do teachers click on student responses?
2. Question view to read student responses: When do teachers read full responses–during class or between classes?
3. Feedback view to give students feedback: How often do teachers provide feedback? Did feedback affect the student’s actions? Did scores increase after teacher feedback?

ML has been applied to finding patterns in unstructured data or to developing detection algorithms based on experts’ judgments (Zhai et al. 2020). The most well-known ML applications are automated scoring of student responses made available on a teacher dashboard. This study goes one step further by employing ML to identify data-driven pathways to systematically develop automated teaching feedback to enhance teachers’ formative assessment practice.

Saturday, April 23

Connecting the Dots Between Teacher Use of Educative Curriculum Materials and Student Curricular Learning Outcomes

Hee-Sun Lee, Trudi Lord, Amy Pallant

2:30 – 4:00 PM PDT (5:30 – 7:00 PM EDT), Marriott Marquis San Diego Marina, Floor: North Tower, Ground Level, Pacific Ballroom 19

Objectives. Enacting curricula equipped with innovative pedagogical ideas is challenging. To support teacher learning of a technology-enhanced plate tectonics unit on their own and at their own pace, we developed online educative curriculum materials (ECMs). These ECM features focused on how to help students formulate causal mechanistic explanations related to plate tectonics. Our hypothesis was that the more ECM features the teachers accessed, the greater learning gains the students would achieve.

Theoretical Framework. In developing ECMs, Ball and Cohen (1996) identified several factors to consider including students’ prior knowledge, teacher understanding of the content, the materials available, the environment of the classroom, and the support of the broader community. Researchers have used these factors to craft design heuristics (Davis & Krajcik, 2005). To actualize student gains, teachers need not only well-designed curriculum materials but also training that helps them implement these materials (Penuel & Gallagher, 2009).

Methods and data sources. Under a mixed methods research design, we analyzed automatically generated log data to identify how teachers interacted with ECMs. To measure student learning gain in each teacher’s classrooms, we calculated Effect Size (ES) based on Cohen’s d, i.e. mean difference between pretest and posttest divided by the pooled standard deviation. The test consisted of 16 multiple-choice items with a maximum score of 26. The test reliability was 0.74 in Cronbach’s alpha. We studied 15 middle school teachers and 11 high school teachers. Among them, 88% taught in public schools; 15% of the schools were in urban settings, 50% in suburban settings, and 35% in rural settings. An average of teaching years was 17.14 ranging from 5 to 35. The average grade level of students (n = 1,098) was 8.40; 52.9% were female; 12.5% spoke English as a second language.

Results. The students across teachers made significant gains, ranging from ES= 0.36 SD to 2.59 SD. We found a non-significant, but positive, correlation between the total amount of teacher interactions with ECMs and student learning gains, r = 0.20, p = 0.32. To explain the non-significant correlation, we further analyzed teacher interviews and surveys. We found that each classroom environment was unique in terms of student, teacher, and school characteristics, which confounded the relationship between teacher use of ECMs and student learning gain. Middle school teachers accessed significantly more ECM features than high school teachers, t(24) = 3.15, p < 0.01. The student learning gain was significantly higher when ECMs were used during class with students when not, t(24) = 2.42, p < 0.05. Scholarly significance. The path between teacher access to ECMs and student learning gain is not straightforward. Focusing on simple significant relationships in studies like ours as a sole evaluation criterion can be misleading and often deny an important opportunity to examine diverse ways in which educational interventions work. Regardless, if online ECMs are freely available, unconstrained by the limitations of travel, logistics, and cost that accompany the person-to-person modes traditionally used for in-service teacher training, a much greater number of teachers and their students can benefit.

Sunday, April 24

Water Careers: Impact of a Universal Design for Learning–Enhanced Middle School Watershed Curriculum on Students’ Career Interest

Nanette Marcum-Dietrich, Cindy Stunkard, Carolyn Staudt, Steve Kerlin

2:30 – 4:00 PM PDT (5:30 to 7:00pm EDT), San Diego Convention Center, Exhibit Hall B

This study investigates a curriculum designed to increase awareness/engagement with water concepts and career pathways using Universal Design for Learning (UDL) principles. The study investigates changes in students’ confidence and efficacy in STEM subjects, 21st century learning skills, and interest in STEM careers using the Student Attitudes toward STEM Surveys as a pre/post-assessment with data from nine teachers and 256 middle-school students in 3 states in the spring of 2021 during COVID-19 pandemic. UDL enhancements were added to the curriculum to increase the accessibility of the content with an emphasis on reaching marginalized students. The data show that those with the lowest scores on the pre-test benefited most as statistically significant positive changes were observed in all areas.

More Conferences