Turn on the news and you’ll see horrific images of flames enveloping hillsides, engulfing homes, and destroying entire communities. This year’s wildfire season is on track, once again, to be one of the worst in history. The number of acres burned in California in August 2024 is already more than double the number of acres burned in the entire 2023 fire season. With more wildfires, more families are affected either by being personally displaced by fires or by experiencing related hazards such as poor air quality. Teaching students about wildfires is not only relevant and timely but also necessary. Future scientists must understand the scientific factors that influence wildfires to be able to help mitigate wildfire risks and impacts in the decades to come.

Our National Science Foundation-funded GeoHazard project developed and tested an online module focused on wildfire risks and impacts. In the spring of 2022, as schools were regaining their footing after closures from the COVID pandemic, 18 teachers and 1,009 students participated in a field test of the Wildfire Risks & Impacts module that was focused on assessing student learning in relation to using the Wildfire Explorer simulation. The teachers and students came from a diverse set of urban, rural, and suburban middle and high schools across the United States. While past research projects relied on some in-person classroom visits, post-COVID pandemic health and travel concerns meant we needed to pivot the project’s research questions and data collection strategy.

A new paper focusing on the insights gained by analyzing how students interacted with the Wildfire Explorer simulation embedded in the curriculum module was published in the Journal of Science Education and Technology. This blog post summarizes the methods and key results of that study.

The Wildfire Explorer shows three zones, each with different terrain, vegetation, and drought levels. A graph shows the number of acres burned in each zone over time. Students can dig a fire line to impede the progress of the fire.

Data collection using logs

Log data is a digital record of all the events that happen in an application. In the Wildfire Risks & Impacts module, each student action, such as answering a question or using the Wildfire Explorer is saved and timestamped in a file that can be downloaded by researchers. Within the Wildfire Explorer simulation, actions such as setting up vegetation types, adjusting drought levels, or adding a spark to start a fire, were recorded.

For the students in our field test, this detailed record of mouse clicks generated a rich dataset that served as a record of their experience. We analyzed the log data to see what we could learn about how students used the Wildfire Explorer and to assess the effectiveness of their interactions with the simulation in contributing to their learning outcomes as measured through a pre-test and post-test comparison.

For this study, we focused on assessing the quality of students’ simulation experiences. By analyzing student logs, we identified different patterns of interaction with the Wildfire Explorer simulation. We were able to determine when students were engaging in systematic inquiry (e.g., conducting controlled experiments) or simply adjusting variables and running the simulation without meaningful intention.

For each of the 10 simulation-based tasks included in the module, we developed a detailed rubric to score students based on the systematicity of their actions in the simulation as a measure of the quality of their interactions. Tasks asked students to experiment with different variables, for example, to determine how vegetation type or drought level contribute to wildfire spread. Higher scores were awarded to students who conducted controlled experiments by varying only the target variable (e.g., drought level) while keeping other factors constant, which allowed them to observe the emergent phenomenon effectively.

In order to automatically score students’ simulation use for each task, we wrote scripts contextualized to each task. To ensure the accuracy of this automated scoring, we compared the scores generated by the computer-generated script with scores derived from analyzing the logs by hand. Although time consuming, this step was crucial in verifying that the automated system correctly interpreted the students’ actions and assigned appropriate scores.

While the log data analysis was the primary focus of this research study, we also analyzed student responses to questions within the module that asked about their prior real-world experience with wildfires, as well as data about each school and classroom. Finally, we took into consideration teachers’ self-reports on their pedagogical approaches. Our analysis explored which factors predicted positive learning outcomes.

Key findings

We found three significant predictors of student learning.

1. Simulation experience. Students’ simulation experience score was a significant predictor of their learning, with an extra large effect size, indicating that the quality of students’ simulation experiences was strongly linked to their conceptual understanding of wildfire hazards and risks. Students who engaged more systematically by running controlled experiments with the simulation showed greater learning gains.
2. Teacher effects. While teachers’ own educational background was not significant in student learning (very few teachers had specific Earth or environmental science certification), variation in their teaching strategies significantly affected student learning. For example, students of teachers who frequently facilitated classroom discussions and encouraged connections between the simulation and students’ personal experiences had the largest gains.
3. Real-world experience. Those students who had prior real-world experience with wildfires tended to perform better on the post-test. However, even students without such experiences saw a significant increase in learning from the module.

Using simulations to teach natural hazards

This study revealed the importance of well-designed simulation-based learning activities and active teacher facilitation to improve students’ understanding of complex natural hazards such as wildfires. The Concord Consortium has also developed curriculum and interactive simulations for boreal forest fires, hurricanes, inland flooding, and earthquakes. These materials are all freely available on our STEM Resource Finder. This blog post is the first in a series that highlights these materials. Stay tuned!