Mind-Reading AI Can Reconstruct Videos Using Your Brain Waves

Technology to instantly read your mind seems like a far-out idea, best reserved for some cutting-edge sci-fi flick. But recent findings from Purdue University are demonstrating that it is now possible to decode the brain activity that occurs during the viewing of videos — in real-time — with the help of artificial intelligence and in combination with existing fMRI (functional magnetic resonance imaging) technology.

In a recent paper published in Cerebral Cortex, the researchers describe how these new findings extend upon previous work that attempts to understand how the brain encodes complex and dynamic visual information seen from the environment. Previous work had artificial intelligence (AI) being paired with fMRI technology to decipher brain activity stemming from the viewing still images, or more complex thoughts composed of semantic building “blocks,” as well as reconstructing memories of faces.

Predicting and Decoding Natural Vision

The team’s aim was to find out what kind of computations might underlie an artificial version of natural vision and to better understand the deeper workings behind how the brain “encodes” and “decodes” visually based stimuli. In particular, the team’s research centered around a type of artificial neural network known as a convolutional neural network (CNN), which consists of a deeply layered structure that feeds information forward, with minimal pre-processing. These CNNs are inspired by the way biological neurons in visual cortexes make connections and have been used in a wide range of applications, from image and video recognition to recommendation systems and natural language processing.

“That type of network has made an enormous impact in the field of computer vision in recent years,” said Zhongming Liu, an assistant professor of biomedical, electrical and computer engineering in Purdue University in a post. “Our technique uses the neural network to understand what you are seeing.”

The team’s experiment used almost a dozen hours of fMRI data from three test subjects, collected while they were watching hundreds of video clips. Many of these showed either people or animals in movement or in natural landscapes. To simulate the action of a biological brain encoding visual information, this data was then fed into a CNN model to train it, so that it could make predictions about what kind of activity might appear in the subjects’ brains while they are viewing various types of video footage.

To imitate the decoding function of the brain when processing visual stimuli, another CNN model was used to interpret new fMRI data from the subjects to reconstruct videos that the model had never seen before. It’s a fuzzy facsimile, but it’s almost as if the software “reads” the thoughts of subjects, revealing the images on another screen. These reconstructions were also categorized, and this was done almost in real-time, with the researchers scanning participants’ brains every two seconds while they watched videos, and with the model then extrapolating the decoded images almost instantly.

Besides suggesting that mind-reading technology is not so far off in the future, the team’s findings also help to clarify what happens in the brain when it’s confronted with dynamic visual images, pinpointing which areas of the brain light up when it’s viewing different things in the same scene.

“Neuroscience is trying to map which parts of the brain are responsible for specific functionality,” said doctoral student and the paper’s lead author, Haiguang Wen. “I think what we report in this paper moves us closer to achieving that goal. A scene with a car moving in front of a building is dissected into pieces of information by the brain: one location in the brain may represent the car; another location may represent the building. Using our technique, you may visualize the specific information represented by any brain location, and screen through all the locations in the brain’s visual cortex. By doing that, you can see how the brain divides a visual scene into pieces and re-assembles the pieces into a full understanding of the visual scene.”

Most importantly, the team found that the models could be applied to a larger number of people beyond the initial test subjects, meaning that such models could potentially be someday used to better understand the mechanisms behind natural vision, or help people with visual impairments.

“We think we are entering a new era of machine intelligence and neuroscience where research is focusing on the intersection of these two important fields,” said Liu. “Our mission, in general, is to advance artificial intelligence using brain-inspired concepts. In turn, we want to use artificial intelligence to help us understand the brain. So, we think this is a good strategy to help advance both fields in a way that otherwise would not be accomplished if we approached them separately.”

Images: Purdue University.