Automating Detection of Engineering Design Practices in Energy3D: Integrating Human and Machine Intelligence

With the expanded adoption of project-based learning and the Next Generation Science Standards (NGSS), there is a growing emphasis on students engaging in authentic science and engineering practices that mirror professional practices. Incorporating both disciplinary content knowledge and strategies for using that content knowledge, practices reflect dynamic, flexible, and potentially highly transferable skills. Helping students engage in practices is important because they extend what students know to what students can do, enable them to wrestle with complex, open-ended challenges, and are associated with gains in both confidence and students’ sense of self as a scientist or engineer.

In the case of engineering design, practices are often studied at a finer grain resolution than NGSS, though they can be hard both to define and to measure. Further, for many advanced practices, there isn’t one canonical form they should take. As research on engineering designers has demonstrated, practices are often polymorphous and may be best characterized as falling on a spectrum from underdeveloped to highly developed.¹ Looking at how professionals apply practices can illuminate why they are so fluid. First, the kinds of open-ended challenges professionals face rarely have a single clear-cut trajectory from problem to solution. Second, professionals bring their own unique backgrounds, experiences, and preferred strategies to a given problem. Thus, coupling an open-ended challenge with these individual differences leads to a variety of ways in which practices are fruitfully enacted. This is also true for students.

As any educator who has attempted a project-based learning activity in their class knows, students simultaneously engage in a wealth of practices, at their own pace, throughout the activity. While teachers routinely keep track of their students, including which students are struggling and how the class is moving through an activity, observing all the practices in which students are engaging is daunting. This, in addition to the heterogeneous nature of many practices, can make it prohibitive to monitor and support all students on their progress in developing practices. New advances in technology and research are enhancing our ability to capture the diversity of students’ practices and their idiosyncratic structure, helping both students and teachers of engineering design.

Advances in educational technology and research

Great progress has been made in developing logging systems for educational technologies as well as in techniques for extracting students’ actions in a given platform. Open-ended simulation platforms that allow students to explore similar problems engineering professionals face offer one promising area for study. Critically, these platforms need to simulate a large part of the problem context and the types of actions professionals take in it in order to enable students to also engage in a wide variety of practices. In conjunction with a logging system, these platforms may be able to capture the diversity of students’ practices in open-ended challenges. This raises another difficulty, however, in that logged actions in a wide interactive environment often result in messy data.

This is where advances in research come into play. While machine analysis may struggle over the messiness of the data, domain experts are able to analyze student log files, learn the forms their practices take in a particular context, and distinguish patterns they embody. Analysis from domain experts can then be programmed back into the machine analysis to automate this process. To see what this looks like, let’s examine an example of different students’ polymorphous practices from Energy3D, a computer-assisted design platform that enables students to design energy-efficient homes, solar panel arrays, and concentrated solar power systems.

Engineering design with Energy3D

Our Solarize Your Home project tasks students with building a model of their home and designing a solar array for their rooftop in Energy3D. After asking their parents or guardians for information about the annual energy use for their home, students need to design a solar array to meet their energy need while staying within a certain budget. Energy3D offers a physics engine that enables estimation of the energy production of solar panels; it also includes tools for estimating the cost of associated materials. A central goal for this project is to give students an authentic challenge for learning about engineering design. Two critical engineering practices involve analysis and design iteration.

Figure 1 displays two simplified practices originally reported in Schimpf and Xie.² Each bubble represents an action or set of actions. These chains of actions form a complex practice encompassing both analysis and iteration. In the first example, the student places a solar panel on the rooftop of a virtual home (Figure 2A) and conducts an annual analysis to estimate the panel’s energy production, and then moves the panel (Figure 2B) and conducts the analysis again (Figure 3).

In the second example, the student starts by turning on the heliodon, a simulated dome that shows the path of the sun for a given location. The student then “animates” the sun, which shows the path of the sun for a given day, beginning by showing the path in June (Figure 2C), and then repeating this for March and December (Figure 2D).

While at first glance these two examples?moving solar panels and using the heliodon?might seem like entirely different practices, our examination of when and how students were using these actions revealed in both cases that they were used early in their design process for the same goal: to gain a better understanding of the solar potential of their rooftop. In other words, they achieved the same ends through two different methods, one focused on a quantitative analysis of rooftop locations estimated through a single panel and the other using a qualitative visual analysis of how the sun faced the rooftop over different seasons. Across students, this single practice was polymorphous.

The pedagogical value of polymorphous practices

For design educators, capturing and displaying students’ practices could assist in several pedagogical goals. First, a key lesson in design education is that there are many ways to approach a challenge and develop a solution. While teachers can discuss this in the abstract, showing how students engaged in different practices in their current project is a powerful way to demonstrate this principle about design and help students learn from each other. Second, another difficulty students face in learning design is the desire to find the “right” way. By comparing similar final designs, a teacher may also be able to show polymorphism in their practices, providing another demonstration that there may be multiple paths toward designing an optimal solution.

Although work remains on automating extraction of student practices identified by a domain expert, there are promising paths forward. One research strand ripe for investigation is the use of regular expressions in programming, which can search through strings of characters or actions and identify those that meet a range of requirements or patterns. We are also looking at machine learning approaches that are trained on patterns identified by domain experts and then classifying new chains of actions.

^1. Crismond, D. P., & Adams, R. S. (2012). The informed design teaching and learning matrix. Journal of Engineering Education, 101(4), 738?797.
^2. Schimpf, C., & Xie, C. (2017). Characterizing Students’ Micro-Iterations Strategies through Data-logged Design Actions. 124th ASEE Annual Conference & Exposition. Presented at the American Society for Engineering Education Annual Conference, Columbus, OH.

Corey Schimpf (cschimpf@concord.org) is a learning analytics scientist.

This material is based upon work supported by the National Science Foundation under grant DRL-1503196. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.