Seeing Is Deceiving: Your Brain and VR

September 02, 2016 by Crytek

Seeing Is Deceiving: Your Brain and VR

You brain is constantly trying to trick you.

Everything you think you see is a lie.

It might sound like science fiction, but it’s not. Your brain is biologically programmed to make constant, small corrections to what you see, and if you’re in the business of creating new realities in VR, it’s a fact that brings some interesting challenges.

A research team made up of scientists from the University of California and the Massachusetts Institute of Technology (MIT) found that in order to maintain stability and shield our senses from an overload of visual input, the brain filters out information, which means that what we think we see is often not what is there.

"What you are seeing at the present moment is not a fresh snapshot of the world but rather an average of what you've seen in the past 10 to 15 seconds," said Jason Fischer, Ph.D., a Neuroscientist at MIT and author of a study on the subject published in the journal Nature Neuroscience. David Whitney, Associate Professor of Psychology at UC Berkley and senior author of the study adds, “This is surprising because it means the visual system sacrifices accuracy for the sake of the continuous, stable perception of objects.” They call it a continuity field, and the result is referred to as “inattentional blindness.”

Thanks, brain. 

This means human perception of reality is a highly filtered experience. Your brain edits on autopilot, and you get on with your life. Our brains’ filtering processes are something we take for granted. That is, until we start trying to recreate realistic experiences in VR.

In order to trick the brain into believing that a virtual experience is real, close attention has to be paid to these editing processes. Ali Helmy, Animation Programmer at Crytek, explained that when his team first started working on VR, they experimented with the techniques they had implemented while developing first person shooters. And that it went horribly, horribly wrong.

“You work on first person shooter games, and you’re used to thinking ‘oh let’s make it more realistic, let’s add weapon sway, let’s add inertia,’” Helmy explained. “But our brain is really used to filtering out a lot of stuff. All these things actually do happen in real life, but your brain has learned to compensate for it. So when it suddenly sees it in VR, it’s awful.”

As it turns out adding weapon sway, gait sway, and camera movement indicating a nod or the jolt of a landing – standard elements used to make fps games feel more realistic – make a VR experience intolerable. And because VR is so immersive, that doesn’t just mean a frustrated sigh and a one-star rating. It means headaches and nausea.

Why are these things so unpleasant in VR? After all, in reality you have gait sway, your head moves when you nod, and items you are carrying sway as you move. But your brain filters your perception of these movements, leaving you with a stable image of the world. So when you nod your head, your eyes remain fixed in such a way as to stabilize the image. When you’re riding a bicycle down a bumpy road, your vestibular system (the inner ear and co.) works with your brain to stabilize the image so that you don’t perceive the world as if through a shaky camera. Your brain and vestibular system are constantly compensating for the movement of images around you. You couldn’t cope otherwise – you’d be nauseous all the time. So elements that appear to add realism in a 3D game viewed on a screen don’t work in a truly immersive experience like VR because you are essentially stripping your brain of its editing power and forcing unedited (ie: images you are not prepared to deal with) onto your perception.

Enter nausea, enter headaches.

While adding these elements to a 3D game can add an element of realism (because they happen, and intellectually, we know that they happen), in VR it makes the experience feel less realistic. Add these elements to a game that you are viewing on a screen, and you get the impression that it is more realistic. But immerse yourself entirely in the world – attempt to recreate reality as it were – and those realistic elements feel awful. What adds realism in a 3D game detracts from it in VR.

In order to make a VR game feel more realistic, you have to make it less realistic.

Yeah. You read that right. Less realism=more realism in VR. (Except when it doesn’t. Making realistic graphics in VR is another story, and one we’ll be telling later in this series.)

So what can we do to make VR more realistic?

Let’s ask Mr. Helmy. “If you’re looking at another character, give them sway and give them motion and give them inertia, but one of the clear easy wins for us was in VR development was, ‘yeah let’s take out weapon sway, let’s take out movement sway. Don’t have the camera bob up and down. Let’s take out inertia.’ So if the game is actually less realistic, if you take out inertia, if you take out head bob and gun sway, your brain is like yeah, this is what I expect.

“I think some of the first major solutions were just trying to relax the simulation. Don’t simulate everything. Just make it a bit arcadey. So there’s no motion, there’s no inertia, there’s no bobbing – that really gave us big gains. The stricter your simulation is, the more fragile it is.”

Tune in on September 16th to find out what causes VR sickness and what developers can do about it.