“The brain is less activated,” Kamitani explained, so those visual clues that define an image are less apparent. The resulting images are not all that clear, especially for natural images, as it can be difficult to recall exactly how the finer details of an image might look like. Next, the team attempted to decode the brainwaves from subjects who were merely imagining the same sets of images, recalling them from memory. Nevertheless, the system was able to produce images that were recognizably close to the initial shape or letter. The team also tested out their system on translating perceptual data on geometric forms and alphabetical letters, which was a challenge as the system was only initially trained on natural images. With the help of a DGN to boost the image output, the team found that a neutral human evaluator could match the decoded image with the original 99 percent of the time.
To create more realistic-looking images, the results were further enhanced with a deep generator network (DGN), an algorithm that better captures common dominant features (such as eyes, faces and textural patterns), which will then offer visual clues to what that particular object might be. To identify what image someone is seeing, the team then used a “decoder” - which is trained on fMRI data taken while subjects were viewing natural images - to repeatedly refine the decoded information further in hundreds of calculated passes (as seen in the video above), so that the image will come closer to the original, pixel by pixel. This filtered data then acts as a template of sorts. “By using a DNN we can extract information from different levels of the brain’s visual system.” “We believe that a deep neural network is good proxy for the brain’s hierarchical processing,” said Kamitani. This raw data was then filtered through a deep neural network so that the decoding process would occur in a way that more closely matched what happens in the human brain when it perceives something. The team used fMRI (functional magnetic resonance imaging) technology to gather brain activity data from subjects as they viewed real-life images animals and other objects, as well as images of geometric shapes and letters. But it’s known that our brain processes visual information hierarchically extracting different levels of features or components of different complexities. These neural networks or AI model can be used as a proxy for the hierarchical structure of the human brain.” “Our previous method was to assume that an image consists of pixels or simple shapes. “We have been studying methods to reconstruct or recreate an image a person is seeing just by looking at the person’s brain activity,” neuroscientist and paper senior author Yukiyasu Kamitani told CNBC. In their paper published on Biorxiv, the researchers explain how their approach to this kind of AI-assisted translation of perceptual content better imitates the elaborate, hierarchical neural representations that are constructed in humans’ natural visual system. Open Deep image reconstruction: Natural images on YouTube. Watch this short video to gsee some of the experimental results: The team’s algorithm also optimizes the pixels of the decoded image so that it more closely resembles the actual object, in combination with a multiple-layered deep neural network (DNN) to simulate the same processes that occur when a human brain perceives an image. A team of scientists based in Kyoto University, Japan have now developed a new method called “deep image reconstruction,” which uses a reconstruction algorithm capable of “decoding” a “hierarchy” of complex visual information from human brain activity, such as colors and shapes. But often these reconstructions have limitations: the images reproduced by these AI models might bear little resemblance to the original object, or the system might be constrained to a predetermined set of objects to identify.īut these thought-decoding technologies are improving. Recent research has demonstrated how machines might be used to decode “blocks” of complex thoughts, reconstruct memories or even videos as they are being watched in real time. Yet more than ever, that futuristic fantasy seems closer to reality, thanks to recent advances in the development of artificially intelligent, “deep” neural networks. The meme of mind-reading machines have been a recurring science-fiction staple for decades.