AI system links surgical gestures with postoperative outcomes

Advances in artificial intelligence are creating new opportunities to evaluate surgical technique and improve patient care. In the following interview, Andrew J. Hung, MD, discusses the development of the Frame-to-Outcome (F2O) system, which automatically translates surgical video into discrete gesture sequences and identifies technique patterns associated with postoperative outcomes.In a recent study, investigators demonstrated that “the F2O-derived features—gesture frequency, duration, and transitions—predicted postoperative outcomes with accuracy comparable to human annotations (0.79 vs. 0.75; overlapping 95% CI).”¹

Hung is a professor and the vice chair of academic development in the Department of Urology at Cedars-Sinai Medical Center in Los Angeles, California.

Urology Times: We are discussing your recent publication, “End to end AI system for surgical gesture sequence recognition and clinical outcome prediction.”¹ What gap in current surgical performance assessment led your team to develop the frame-to-outcome (F2O) system?

Hung: For many years, our team has looked into creating a surgical gesture classification system that breaks down otherwise very complex surgeries into the moment-to-moment maneuvers or movements that a surgeon chooses to make. Our group not only has described these gestures and done it for hundreds of cases, but we have also found the link between these sequences of gestures. [We] think of them as the alphabet of surgery; stringing these gestures together through AI models can predict outcomes. That was our prior work.

The problem is that it is a very tedious job to watch these videos and annotate or manually determine what is the maneuver that the surgeon is making every other second. What the breakthrough was for this particular study was that we were able to automate the entire process from a raw video of a surgeon performing, let's say, the nerve-spare step of a prostatectomy, one of the key vital steps to preserving sexual function for patients. The AI pipeline can take the raw video sequence and take out the individual gestures on a frame-to-frame level. Then, [it] takes the sequence and plugs it into our prior work, which is taking the sequence of these gestures to predict what the outcome may be. So, we're able to provide the full solution with minimal human effort, such that it is truly a scalable opportunity to process thousands and thousands of surgical cases and predict outcomes. [We are] hoping that we can provide this insight to our surgeons and even patients to understand the likelihood of recovering sexual function after surgery.

Urology Times: Could you expand on what a surgical gesture is in this context?

Hung: Your simplest gestures could be a simple cut of a piece of tissue, a single push of tissue, a burn if there is a bleeding vessel that needs to be cauterized, or retraction. These are some of the more common gestures that are performed by a surgeon. We do split them into blunt dissection, sharp dissection, [and] combination maneuvers that are a predictable cluster of gestures [that] fit together. These are the building blocks of what surgery is.

Urology Times: What were the key findings from your study of this system?

Hung: This was an enabling developmental project. It was meant to put together different elements that were pre-existing. Perhaps the biggest development is taking the raw video of these gestures and on a frame-by-frame level, being able to predict what the gesture is that's being performed at that time. Each gesture is roughly 2 seconds long. So, at 30 frames per second—that's the video rate that we're processing—there's about 60 frames of video that make up a gesture. The model will decipher on a frame-by-frame basis what the gesture ought to be. There's an additional innovation, which is the change points, so the switch from a preceding gesture to the next. That was a big development—detecting what is the gesture that's happening and when the surgeon completed that specific movement or gesture and moved on to the next. A lot of the development side of things was in this space, and that is the centerpiece in terms of technological development.

The practical value, of course, is, as I mentioned earlier, taking raw video from the very beginning and taking someone seamlessly through the entire process, so that at the other end there's a single determination or prediction of outcome. In fact, we know how these patients eventually did. In our validation set, we found that indeed our overall model performs very robustly with the predictions based on this full pipeline to outcomes.

Urology Times: Your work identified prolonged tissue peeling and reduced energy use as patterns associated with erectile function recovery. What is the significance of this finding?

Hung: For many years, we have taught the procedure in this fashion. We learned by dogma. Our mentors taught us how to perform this operation in peeling the neurovascular bundle off the prostate, [but] this wasn't based on any science. The recovery rate is about 35% across our multi-institutional series. Our work represents the first of many that quantifies these actions and truly boils it down to the maneuvers that we are seeing recurrently.

The idea with the specific gestures is that they're blunt maneuvers that are peeling the nerves off along natural tissue planes. This is preventing injury, and furthermore, the reduction or the minimal use of energy is indeed protective. You can physically preserve the nerves, but if you burn it, you can cause quite a bit of damage and reduce the patients to non-function. While this was always a suspicion, we haven't had evidence to support that. Our body of work supports this in a quantitative, reproducible manner, that by using blunt dissection, using the natural tissue planes of surgery, and by minimizing the use of energy—and we see this again on the gesture level—these things do support the recovery of sexual function.

Urology Times: What would it look like for a practicing urologist to receive and apply this feedback in clinical practice?

Hung: That's a space that we're actively working towards. We would love for our fellow surgeons to have a sense of what they're doing and how it may relate to their patient outcomes. The challenge has always been that from the time of surgery to the time that patients have recovery or not, it is typically a year later. As a result, surgeons don't get the positive nor negative feedback of what's working or not in their surgical practice.

Now we're able to link all of this through AI. At least for retrospective cases, we've bridged the time of surgery to outcomes, and we've isolated, as we just discussed, various maneuvers that seem protective or that negatively impact these outcomes. The ideal environment is that we can provide positive or negative reinforcement to surgeons based on their own practices and their own patients [and determine] the likelihood of recovery function. Hopefully surgeons can learn what to continue doing and perhaps things to avoid.

Our particular study was trained from data across 5 different institutions internationally. While it's a robust study and we think it's a generalizable outcome, the best test of that, of course, is external validation. That's what we anticipate doing, perhaps in the form of a clinical trial, so that we can see if the actions that are being performed and deciphered through these gesture sequences may impact how patients recover after surgery.

Urology Times: Is there anything else that you wanted to add?

Hung: I see this study as similar to the ability to automate the sequencing of the human genome. By doing so, that unlocked the ability to discover the genetic causes of many diseases. Surgery has always been seen as an art form, as a process, but it's never been seen as something that's quantifiable. The ability to take a complex operation and reduce it to these moment-to-moment maneuvers and gestures, sequence them out, and understand them in an objective manner at a scalable pace will unlock a lot of discoveries of what makes good surgery or not within urology and beyond. That's very exciting, and for those reasons, this for me is a very important piece of work that's come out of our group.

REFERENCE