Framing a More Immersive World on the Horizon

The expanding use of virtual reality is giving researchers more opportunities to understand just how we perceive the real world through vision.

 |  Transcript [PDF]

Virtual reality (VR) is getting more accessible to more audiences, and creators hope to find ways to make VR totally immersive. That starts with a better understanding of how human vision works so that researchers can define how people perceive the real world as events unfold before their eyes.

“In vision research, we use pictures a lot in order to study the visual system,” says Niko Troje, professor of biology at the Vision: Science to Applications (VISTA) program at York University.

“We present them on screens or on larger projection surfaces, and we assume that because the stimulation of our eyes under those conditions is similar to what we are experiencing in the real world, that what we are studying can be generalized to a real world that opens in front of our open eyes.”

But lately, researchers in the vision science field have been challenging that assumption. In particular, VR makes it clear that projections on a flat screen are not a complete substitute for vision in the real world.

In these virtual worlds, the user can see a slightly different image with each of their eyes, and the perspective of the scene responds to the movement and position of their head. It attempts to simulate vision in the real world. This visual space is fundamentally different to what we are used to seeing in pictures and movies, where there is a single camera that does not respond to its audience.

With the help of VISTA research funds, Troje has developed a new tool that allows him to carry out a side-by-side comparison between what people see in an open space in front of them versus a pictorial space like a standard picture or movie. The tool is called the Alberti Frame, making reference to the renaissance architect and mathematician Leon Battista Alberti, who believed that paintings should recreate exactly the view seen through an empty frame. You can think of it like a camera that can be placed anywhere inside virtual reality to take a 2D picture or video.

Imagine now that you were looking at a mountain in VR, and you take out your camera to freeze an Alberti frame. You could go back to that exact location and hold up the image, and if you closed one eye it would be indistinguishable from the rest of the scene. That screen would be like an open window.

But open both eyes and start moving around, and the mismatch becomes clear. Two new phenomena that give us a sense of depth perception come into play: stereopsis, because each of your eyes is in a slightly different position and sees the scene from a slightly different perspective; and active motion parallax, because what you see should change as you move your head.

Normally these phenomena are entangled, but researchers can separate them in VR. For example, both eyes can be shown the same image to eliminate stereopsis, or the visual scene can be decoupled from the user’s movements. In other words, the Alberti Frame can act like a screen in terms of one of the depth cues, but like a window in terms of the other.

“Our Alberti Frame now has not just two states, but it’s got four states,” says Troje. “So two additional ones in which it either behaves as a window in terms of stereopsis, but like a picture in terms of motion parallax, or the other way around.”

Although we know that both phenomena play important roles in vision, VR presents a new opportunity to study the effect of each one separately, and understand how each contributes to the virtual experience. This work will help creators build a more convincing illusion of reality in VR.

‹ Previous post
Next post ›

Niko Troje directs the BioMotion Lab, at York University in Toronto, Canada. He received his PhD in Biology from the Albert Ludwigs University in Freiburg, Germany, in 1994. Subsequently, he moved to the Max Planck Institute for Biological Cybernetics in Tübingen.

From 1997 until 1999, he spent two years as a Visiting Professor at Queen’s University, Kingston, Ontario. In 1999, he was awarded with a Young Researchers Award from the Volkswagen Foundation and moved to Ruhr University where he founded and directed the BioMotionLab, at the Department of Psychology.

In 2003 he re-joined Queen’s University and in 2018 he moved to York University in Toronto. At York University he is affiliated with the Centre for Vision Research, the Department of Biology and the Graduate Programs in Biology, Psychology, and Electrical Engineering & Computer Science. Troje has received numerous awards and recognitions for his work, among others the Feodor-Lynen Fellowship of the Alexander von Humboldt Foundation, the E.W.R. Steacie Memorial Fellowship from NSERC, and most recently, the lifetime achievement Humboldt Research Award. His main research interest is focused on questions concerning the nature of perceptual representations. How can a stream of noisy nerve cell excitations possibly be turned into the coherent and predictable perception of a “reality”?

Research2Reality is a groundbreaking initiative that shines a spotlight on world-class scientists engaged in innovative and leading edge research in Canada. Our video series is continually updated to celebrate the success of researchers who are establishing the new frontiers of science and to share the impact of their discoveries with the public.