Research on NUIs is currently one of the most active areas in computer science and engineering. It is one of the most important directions of Microsoft Research. In line with this future, our NSF-funded Mixed-Reality Labs (MRL) project has proposed a novel concept called the Natural Learning Interfaces (NLIs), which represents our latest ambition to realize the educational promise of cutting-edge technology. In the context of science education, an NLI provides a natural user interface to interact with a scientific simulation on the computer. It maps a natural user action to the change of a variable in the simulation. For example, the user uses a hot or cold source to control a temperature variable in a thermal simulation. The user exerts a force to control the pressure of a gas simulation. NLIs use sensors to acquire real-time data that are then used to drive the simulation in real time. In most cases, it involves a combination of multiple sensors (or multiple types of sensors) to feed more comprehensive data to a simulation and to enrich the user interface.
I have recently invented a technology called the Frame, which may provide a rough idea of what NLIs may look like as an emerging learning technology for science education. The Frame technology is based on the fact that the frame of a computer screen is the natural boundary between the virtual world and the physical world and is, therefore, an intuitive user interface for certain human-computer interactions. Compared with other interfaces such as touch screens or motion trackers, the Frame allows users to interact with the computer from the edges of the screen.
Collaborating with Jennie Chiu’s group at the University of Virginia (UVA), we have been working on a few Frame prototypes that will be field tested with several hundred Virginia students in the fall of 2012. These Frame prototypes will be manufactured using UVA’s 3D printers. One of the prototypes shown in this blog post is a mixed-reality gas lab, which was designed for eighth graders to learn the particulate nature of temperature and pressure of a gas. With this prototype, students can push or pull a spring to exert a force on a virtual piston, or use a cup of hot water or ice water to adjust the temperature of the virtual molecules. The responsive simulation will immediately show the effect of those natural actions on the state of the virtual system. Besides the conventional gas law behavior, students may discover something interesting. For example, when they exert a large force, the gas molecules can be liquified, simulating gas liquifying under high pressure. When they apply a force rapidly, a high-density layer will be created, simulating the initiation of a sound wave. I can imagine that science centers and museums may be very interested in using this Frame lab as a kiosk for visitors to explore gas molecules in a quick and fun way.
 |
| A mixed-reality gas lab (a Frame prototype) |
As these actions can happen concurrently, two students can control the simulation using two different mechanisms: changing temperature or changing pressure. This makes it possible for us to design a student competition in which two students use these two different mechanisms to push the piston into each other’s side as far as possible. To the best of our knowledge, this is the first collaborative learning of this kind mediated by a scientific simulation.
NLIs are not just the results of some programming fun. NLIs are deeply rooted in cognitive science. Constructivism views learning as a process in which the learner actively constructs or builds new ideas or concepts based upon current and past knowledge or experience. In other words, learning involves constructing one’s own knowledge from one’s own experiences. NLIs are learning systems built on what learners already know or what they feel natural. The key of a NLI is that it engineers natural interactions that connect prior experiences to what students are supposed to learn, thus building a bridge for stronger mental association and deeper conceptual understanding.