Robot watched surgery videos then performed with skill of human doctor

A robot, trained for the first time by watching videos of seasoned surgeons, executed the same surgical procedures as skillfully as the human doctors.

The successful use of imitation learning to train surgical robots eliminates the need to program robots with each individual move required during a medical procedure and brings the field of robotic surgery closer to true autonomy, where robots could perform complex surgeries without human help.

“It’s really magical to have this model and all we do is feed it camera input and it can predict the robotic movements needed for surgery,” says senior author Axel Krieger. “We believe this marks a significant step forward toward a new frontier in medical robotics.”

The findings led by Johns Hopkins University researchers are being spotlighted this week at the Conference on Robot Learning in Munich, a top event for robotics and machine learning.

The team, which included Stanford University researchers, used imitation learning to train the da Vinci Surgical System robot to perform fundamental surgical procedures: manipulating a needle; lifting body tissue, and suturing. The model combined imitation learning with the same machine learning architecture that underpins ChatGPT. However, where ChatGPT works with words and text, this model speaks “robot” with kinematics, a language that breaks down the angles of robotic motion into math.

The researchers fed their model hundreds of videos recorded from wrist cameras placed on the arms of da Vinci robots during surgical procedures. These videos, recorded by surgeons all over the world, are used for post-operative analysis and then archived. Nearly 7,000 da Vinci robots are used worldwide, and more than 50,000 surgeons are trained on the system, creating a large archive of data for robots to “imitate.”

While the da Vinci system is widely used, researchers say it’s notoriously imprecise. But the team found a way to make the flawed input work. The key was training the model to perform relative movements rather than absolute actions, which are inaccurate.

“All we need is image input and then this AI system finds the right action,” says lead author Ji Woong “Brian” Kim. “We find that even with a few hundred demos the model is able to learn the procedure and generalize new environments it hasn’t encountered.”

The team trained the robot to perform three tasks: manipulate a needle, lift body tissue, and suture. In each case, the robot trained on the team’s model performed the same surgical procedures as skillfully as human doctors.

 “Here the model is so good learning things we haven’t taught it,” Krieger says. “Like if it drops the needle, it will automatically pick it up and continue. This isn’t something I taught it do.”

The model could be used to quickly train a robot to perform any type of surgical procedure, the researchers said. The team is now using imitation learning to train a robot to perform not just small surgical tasks but a full surgery.

Before this advancement, programming a robot to perform even a simple aspect of a surgery required hand-coding every step. Someone might spend a decade trying to model suturing, Krieger says. And that’s suturing for just one type of surgery.

“It’s very limiting,” Krieger says. “What is new here is we only have to collect imitation learning of different procedures, and we can train a robot to learn it in a couple days. It allows us to accelerate to the goal of autonomy while reducing medical errors and achieving more accurate surgery.”

Authors from Johns Hopkins include PhD student Samuel Schmidgall; Associate Research Engineer Anton Deguet; and Associate Professor of Mechanical Engineering Marin Kobilarov. Stanford University author is PhD student Tony Z. Zhao.