They’re the in thing, especially for teaching science. Everyone, it seems, is fascinated by the potential of educational games. They’re interactive and “multimedia,” they can adapt to individual students, they promote “authentic learning.” In short, they match the outsize expectations of a digital world. They’re definitely cool, but do they teach, and if so, what do they teach?

Full disclosure: I am an enthusiastic proponent of educational games. I created one called “ThinkerTools” so long ago that it ran on a Commodore 64 computer and had to be programmed in machine language to make it run fast enough. And, yes, I have no doubt that kids learn from such games. But do they learn what we think they’re learning? And how would we know if they were? Is it sufficient that they get better at the game? Surely not, else chess masters would be good at logic, and athletes would be physicists.

It is tempting to imagine that we can design educational games so cleverly that it would be impossible for a student to get good at the game without acquiring a deep understanding of whatever it is the game is trying to teach. Unfortunately, it doesn’t always work that way, as I learned from my experience with another educational game called GenScope.

GenScope was a multi-level genetics game. It linked processes at all different levels, from molecules to ecosystems, and we used it to create a bunch of engaging challenges for students. Our species of choice was dragons. We would show a dragon’s chromosomes, for instance, and ask students to figure out how to change its genes to make the dragon breathe fire. Later on, we would challenge them to breed a strain of blue dragons, or try to find two parent dragons that could only have two-legged offspring (hint: neither parent can have two legs).

We used the GenScope games in several high schools. We compared students who had used the games to others who had learned genetics by conventional means. To do this we designed a clever test that assessed precisely the reasoning skills we were trying to teach—and that we naively assumed were necessary to succeed at the games. Each time we did this, we found that the GenScope classes did no better on the test than the control group. Sometimes they did worse!

In the jargon of the trade the Holy Grail is “transfer,” and we weren’t getting much. Knowledge gained in one context is often difficult for the novice to apply to another one, even though to an expert the two situations appear very much alike.

To us, the researchers, the genetic principles behind the GenScope games were obvious, and their relevance to the questions on the test equally so. Clearly, that was not the case for the students, who became expert GenScope players but failed to apply what they learned to genetics.

There are ways around this impasse, of course, and I will describe a few in a future blog post. For the moment, though, let’s just keep in mind: there are lots of ways of getting good at an educational game. Only one of them involves learning what the game is supposed to teach.