Helping Surgeons Live in the Moment

Surgical plans are imperfect guides, but new technological tools could help lighten the load for surgeons doing complex, delicate work.


Every day in the operating room, surgeons make complex visual transformations in their minds.

Any pre-operative information that they have based on medical imaging is usually static; however, during surgery there are many moving parts. That means that any surgical plan up on a screen isn’t keeping pace, and it takes a lot of mental effort to align what they see on-screen with what’s on their operating table.

Computational vision scientist Richard Wildes, Associate Director of York University’s Vision: Science to Applications (VISTA) program, wants to make their lives easier by creating tools that would update the surgical plan to match the surgery as it progresses.

It all starts with teaching computers to see in a similar manner to how we see, taking on the task of manipulating medical images like x-rays or CT scans in real time, just like a surgeon does mentally.

“[Surgeons have] the surgical plan available to them, typically on a video display someplace above where they’re working, and down in front of them, there would be the patient,” explains Wildes. “And guess what? The geometry, the layout of the organs in front of them live is very different than those captured, say, in the pre-operative imaging.

“So the surgeon has to do this very difficult, live, in-the-moment transformation between what they’re seeing on front of them on the operating table, and what they see up here in the video display. Of course, surgeons can do this, they’re very talented people; but it’s a very mentally demanding task.”

The idea is to use a video camera installed above the surgical field, giving a bird’s eye view of what is happening in the operating room. Computer vision technology would then reconstruct the 3D layout in real time and apply a transformation to the pre-operative imagery, aligning it with what the surgeon sees on the operating table.

This technology could also one day help drive surgical robotics, which are currently operated by human surgeons. Routine portions of the surgery could be assisted by computer vision to provide a surgical robot with a degree of autonomous operation, decreasing the load on human surgeons.

As computer vision continues to develop, it will play an ever greater role in assisting with complex tasks in real time. In medicine, it could help surgeons stay fully in the moment, validating the mental manipulations they do daily to help them provide the best care.

‹ Previous post
Next post ›

Richard Wildes received his PhD from the Massachusetts Institute of Technology in 1989. Subsequently, he joined Sarnoff Corporation (now SRI) as a Member of the Technical Staff in the Vision Technologies Lab. In 2001, he joined the Department of Electrical Engineering and Computer Science (EECS) at York University, Toronto, as an Associate Professor. Currently he is Associate Director of York’s Vision: Science to Applications (VISTA) program, funded for $33.3 million by the Canada First Research Excellence Fund. He also previously served as Chair of EECS and Associate Director of the York Centre for Vision Research (CVR).

Wildes’s research interests are in computational vision, especially video understanding and machine vision applications, as well as artificial intelligence. While in industry, he led teams developing vision technologies for various real-world applications, including iris recognition, where he is widely considered a pioneer. Awards and honours include the IEEE D.G. Fink Prize Paper Award, Sarnoff Corporation Technical Achievement Award, twice giving invited presentations at the USA National Academy of Sciences and currently holding a York Research Chair, Tier I.

Research2Reality is a groundbreaking initiative that shines a spotlight on world-class scientists engaged in innovative and leading edge research in Canada. Our video series is continually updated to celebrate the success of researchers who are establishing the new frontiers of science and to share the impact of their discoveries with the public.