“There’s a lot of appeal in learning from visual observations, compared to traditional interfaces for learning in a static way or learning from [mimicking] trajectories, because of the huge amount of information content available in existing videos,” Tanwani told Engadget. When it comes to teaching robots, a picture, apparently, is worth a thousand words.

“YouTube gets 500 hours of new material every minute. It’s an incredible repository, dataset,” Dr. Ken Goldberg, who runs the UC Berkeley lab and advised Tanwani’s team on this study, added. “Any human can watch almost any one of those videos and make sense of it, but a robot currently cannot — they just see it as a stream of pixels. So the goal of this work is to try and make sense of those pixels. That is to look at the video, analyze it, and… be able to segment the videos into meaningful sequences.”

To do this, the team leveraged a siamese network to train its AI. Siamese networks are built to learn the distance functions from unsupervised or weakly-supervised data, Tanwani explained. “The idea here is that you want to produce the high amount of data that is in recombinant videos and compress it into a low dimensional manifold,” he said. “Siamese networks are used to learn the distance functions within this manifold.” 

Basically, these networks can rank the degree of similarity between two inputs, which is why they’re often used for image recognition tasks like matching surveillance footage of a person with their drivers license photo. In this case, however, the team is using the network to match the video input of what the manipulator arms are doing with the existing video of a human doctor making the same motions. The goal here being to raise the robot’s performance to near-human levels. 

And since the system relies on a semi-supervised learning structure, the team needed just 78 videos from the JIGSAWS database to train their AI to perform its task with 85.5 percent segmentation accuracy and an average 0.94 centimeter error in targeting accuracy.

It’s going to be years before these sorts of technologies make their way to actual operating theaters but Tanwani believes that once they do, surgical AIs will act much like Driver Assist does on today’s semi-autonomous cars. They won’t replace human surgeons so much as augment their performance by taking over low-level, repetitive tasks. The Motion2Vec system isn’t just for suturing. Given proper training data, the AI could eventually be tasked with any of a number of duties, such as debridement (picking dead flesh and debris from a wound), but don’t expect it to perform your next appendectomy.

“We’re not there yet, but what we’re moving towards is the ability for a surgeon, who would be watching the system, indicate where they want a row of sutures, convey that they want six overhand sutures,” Goldberg said. “Then the robot would essentially start doing that and the surgeon would… be able to relax a little bit so that they could then be more rested and able to focus on more complex or nuanced parts of the surgery.” 

“We believe that would help the surgeons productively focus their time in performing more complicated tasks,” Tanwani added, “and use technology to assist them in taking care of the mundane routine.”