How VR for animals could unlock the secrets of our own minds

The age of virtual reality (VR) is upon us, with headsets such as the HTC Vive and Oculus Rift settling into living rooms and augmented reality-enabled phones in our pockets. Behind the scenes, VR is also allowing another important – but perhaps quieter – revolution to take place in scientific research. It turns out VR is not just for humans. This technology is increasingly being used in the lab, to help scientists understand animal behaviour.

As strange as it may sound, VR for animals is not a new idea. Curious examples include a flight simulator for moths; a virtual maze for mice (built from open-source software from the Quake 2 video game); a terrifying game of chase for zebrafish and a walk in the forest for cockroaches. These approaches have been incredibly useful to study animal behaviour, but they share a crucial flaw: they need to restrain or control animals in some way, which may give misleading results.

The good news is that, with recent technical developments, particularly in the gaming industry, a fully immersive virtual environment where animals can move freely is not the farfetched idea it once was. Think of it as The Matrix for animals…

For example, one of the latest approaches comes from Andrew Straw and his team from the University of Freiburg in Germany. Named FreemoVR, the system projects realistic environments onto a screen and then monitors how animals respond. “With VR for freely moving animals, we can let the animal experience its own natural sense of motion – feeling the stretch and strain of muscles as it walks forward and feeling the sensation of really turning – while still creating artificial visual worlds to do things like test spatial navigation”, explains Straw.

Despite the use of advanced technology, the setup is quite simple: it’s just an area where both the floor and the walls are flexible computer displays. Animals in this 3D environment can be tracked by multiple high-speed cameras and sensors, which means there is no need for animals to use special headgear as was the case in the past.

(Above: FreemoVR’s VR arena for flies. Credit: MP/IMBA Graphics Department)

“We use a combination of modern computer game technology (projectors, video cards and software) and camera-based animal tracking to create virtual reality for animals that are freely moving. We can immerse animals in a holographic, reactive 3D [environment] which is essentially limited only by the imagination of the experimenter”, says Straw. “If it weren’t for the computer gaming industry, our work would be impossible”.

Playing games with fish

This may sound like an excuse to play games all day, but Straw really believes the system can help researchers learn about animal behaviour in a way that wasn’t possible before. According to the researcher, the two main ways in which this technology can be used revolve around spatial recognition and collective behaviour. In other words, how animals learn about their environment and how they behave towards other animals. “Animals need to build maps of where they are and how the body senses its own movement. That’s the kind of thing that’s really enabled by this”, says Straw.

It turns out animals really think their 3D virtual surroundings are convincing and react the same way as they would in the real world. For example, when the tank of zebrafish was invaded by Space Invaders (yes, you read that correctly), the fish ‘believed’ the incoming crowd was real, and even tried to join the group.

Deceiving poor fish with old video games may sound a little weird, but this work can explain much about how animals understand the world. “Does it even matter if the other things in the water look like Space Invaders? Is it actually the visual appearance that’s important or is it maybe the pattern of motion?” asks Straw, keen to understand how fish respond to other ‘stuff’ in the water.

(Above: Real fish swimming in a swarm of “space invaders”. Credit: FreemoVR)

Ultimately, virtual reality for animals may be a way to better understand why these animals do certain things, which may eventually also help us understand human behaviour.

For researchers, the hope is that VR (combined with already existing technology to measure brain activity in freely moving animals) represents a unique door into the animal’s brain, to study their neural signals in response to a certain environment. For example, maybe knowing which part of the mouse’s brain lights up when this small furry creature is afraid will help us understand our own brain in stressful situations.  

Stereo vision

Free-moving VR systems may be more sophisticated than previous methods, but there’s still room for improvement. Some changes are simple, such as matching the images displayed with the visual sensitivities of the animals. This is particularly true for animals that have ultraviolet vision, for example. “It’s a matter of building the right piece of equipment to project an image with the right polarisation”, says Straw. This technology is already in use in 3D cinemas, which have active polarisers that very rapidly switch the polarisation of the signal to give different signals to each eye.

The second issue is stereo vision and this is significantly trickier to solve. With the current technology, it’s impossible to render separate images for each eye in a free moving VR environment. “That might be a more fundamental difficulty”, says Straw. “That’s why these head-mounted displays are so important for humans – for us, stereo vision is incredibly important”. This is not much of an issue for lab animals, such as insects or mice, but if researchers want to move on to bigger animals, this issue needs to be solved. Just projecting an image onto the wall won’t cut it for animals that rely on stereo vision such as predators, for example. Cats will know that’s not really a mouse in the corner.

In the long term, what scientists really want is to create a multisensory environment, where animals can not only ‘see’ virtual reality, but they can also ‘smell’ and even ‘touch’. For the results from brain activity analysis to be meaningful, the animals need to be 100% ‘convinced’ that their surroundings are real. In the future, VR applied to animals could become an incredible tool to understand not only how the brain works, but perhaps more importantly for us, to study what happens when something goes wrong.

This may seem impossible for now but, with the way VR and other technologies are developing, chances are it will come sooner than we think.