Abstract: 

When developing context-aware systems, automatic surgical phase recognition and tool presence detection are two essential tasks. There exist previous attempts to develop methods for both tasks but majority of the existing methods utilize a frame-level loss function (e.g., cross-entropy) which does not fully leverage the underlying semantic structure of a surgery, leading to sub-optimal results. In this paper, we propose multi-task learning-based, LAtent Space-constrained Transformers, referred as LAST, for automatic surgical phase recognition and tool presence detection. Our design features a two-branch transformer architecture with a novel and generic way to leverage video-level semantic information during network training. This is done by learning a non-linear compact presentation of the underlying semantic structure information of surgical videos through a transformer variational autoencoder (VAE) and by encouraging models to follow the learned statistical distributions. In other words, LAST is of structure-aware and favors predictions that lie on the extracted low dimensional data manifold. Validated on two public datasets of the cholecystectomy surgery, i.e., the Cholec80 dataset and the M2cai16 dataset, our method achieves better results than other state-of-the-art methods. Specifically, on the Cholec80 dataset, our method achieves an average accuracy of 93.12±4.71%, an average precision of 89.25±5.49%, an average recall of 90.10±5.45% and an average Jaccard of 81.11 ±7.62% for phase recognition, and an average mAP of 95.15±3.87% for tool presence detection. Similar superior performance is also observed when LAST is applied to the M2cai16 dataset.

Fig. 1. A schematic overview of the proposed LAST architecture, which consists of a visual feature extraction unit, a feature enhancement unit, and a latent space constraint unit; (A) setup at the training stage; (B) setup at the testing stage; (C) legend.

Fig. 2. A schematic overview of the visual feature extraction (VFE) unit; (A) training setup; (B) inference setup. During training, a shared backbone network is attached with two prediction heads with one for the phase prediction and the other for the tool presence detection. After training, the weights of the backbone network is fixed such that it can be used as the feature extractor.

Fig. 3. Visualization of the embedded feature distributions of two videos from different variants of the proposed method using t-SNE (A) and three challenging frames corresponding to three time points of each video as well as the labels predicted by the different variants of the proposed method in comparison with the associated ground truth (GT) labels (B). In (A), for each video we show the color-coded feature distributions of seven phases as well as the corresponding features of three selected time points with challenging frames as shown in (B).

DOI: 10.1109/TMI.2023.3279838