Video conferencing took off during the pandemic as people around the world were self-isolating and working from home. But there are important visual cues we are used to getting in person that get lost on Zoom or Skype, and according to vision scientist Niko Troje, they affect what he calls “people perception.”
Over the past decade, Troje has been studying facial recognition and the way people move. That motion contains information about emotion, intention, and personality. The human visual system is highly sensitive to it, and users can’t extract the same depth of meaning in interactions captured on camera when they interact virtually.
“There’s many problems with systems such as Zoom or Skype that we know today,” says Troje, professor of biology at the Vision: Science to Applications (VISTA) program at York University.
“The main problem is that we are losing what I call directionality; so, if someone happens to look into the camera, the person on the other end feels being looked at and there’s no way to escape that gaze.
“Or if someone is not looking into the camera, for instance because they are looking at the screen where the other person is, we feel being looked at our chin or neck or something, but we can’t catch the other person’s gaze.”
In normal in-person interactions, we catch and break eye contact all the time. Even knowing this, there are many challenges in supporting a virtual system that could simulate it. The cameras we use have a fixed position on our devices, and it wouldn’t be practical to move it around just to catch a user’s gaze.
In the first step of Troje’s approach to this problem, he is researching how people look from slightly different viewpoint angles than the standard one captured by a fixed camera.
“We have a demo system, which functions beautifully but it’s based on computer graphics,” adds Troje. “So the person I’m talking to, I see represented as an avatar and the other person sees me represented as an avatar.”
Combining facial recognition, an understanding of biological motion, and the ability to shift to a different viewpoint enables a more natural experience. In the future, he hopes to be able to integrate this proof of concept back into a more photorealistic representation of each user, instead of using simplified computer graphics.
Virtual interactions have kept people connected during an unprecedented time, and it’s likely that many will continue using this technology even as public health measures are relaxed. Being able to replicate our in-person interactions more closely will help build a richer experience no matter how far apart we are.
