Under the Hood: Using Brain Signals in a Block Programming Environment

It’s a hot late August day and your neighbor’s kids have set up a lemonade stand. After paying your quarter, you reach for a cup. As you lift it, you sense the paper cup is thin, so you tip it gingerly and take a sip—delicious! But how did the idea of picking something up become motion in your arms and fingers? And how did you know to grasp the cup just enough so it wouldn’t be crushed or slip out of your hand?

Electricity is the answer. Your brain’s motor cortex sends electrical signals through motor neurons to contract muscles in your arm and hand. At the same time, nerves in your fingers send electrical signals back to your brain with information about how the object feels in your hand.

In collaboration with the University of Connecticut, we’re designing activities for high school biology students that engage them in computational thinking as they learn how to use electrical signals from their brains to control virtual and real mechanical devices. We were inspired by Backyard Brains whose Arduino-based hardware kits allow students to attach an electrode to their arm, then tense their muscles and use the electricity created to make a robotic-like Grabber open and close (Figure 1). We wanted to provide students with components to program this process on their own, using Backyard Brains hardware in combination with an accessible, drag-and-drop programming environment.

We started by importing our Dataflow visual programming software into a new tool tile in CLUE, our Collaborative Learning User Environment. Using Dataflow, students can write programs that consume, process, and output data by connecting blocks, similar to Legos. It’s the perfect environment for simulating how electrical signals traverse a system (Figure 2).

We updated Dataflow’s existing Sensor block to take a measurement from an EMG sensor or surface pressure sensor connected to the Backyard Brains Muscle Spiker and bring it into Dataflow. We also added three new blocks. The Demo Output block takes a number created in Dataflow and animates a virtual version of the Grabber. The Live Output block passes a number from Dataflow to the physical device, which opens or closes to the percentage specified by the number calculated from the Logic and Math blocks. The Control block adds the ability to turn the flow of data on or off.

To make the Live Output and Sensor blocks work, we had to manage communication between the Arduino (the microprocessor that handles sensor input and controls Grabber motion) and Dataflow. For example, to get sensor data into Dataflow, first the Arduino program takes sensor readings and sends them to the serial output.

	emgReading = analogRead(A0); 
	fsrReading = analogRead(A1);
	emgStringOut = String(emgId + : + emgReading);
	fsrStringOut = String(fsrId + : + fsrReading);
	Serial.println(emgStringOut);
	Serial.println(fsrStringOut);

Meanwhile, Dataflow takes the reading from the serial port and brings it into the program on a dedicated “channel” for each block.

	 if (targetChannel){
	 targetChannel.value = parseInt(numValue, 10);
	 }

Figure 2 illustrates the use of all four blocks to approximate the cup of lemonade scenario. One Sensor block takes an EMG reading and sends it through some arithmetic blocks to create a value to close or open the Grabber, while another Sensor block keeps track of the pressure on the cup. If it’s too high, the Control block prevents the close message from getting to the Output block.

Now that students can program a path of blocks to open a Grabber using their own brainwaves, we hope that Dataflow opens up additional new areas of neuroscience exploration.

This material is based upon work supported by the National Science Foundation under grant DRL-2131097. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.