Abstract

Surgical instrument recognition and tissue motion analysis are two critical tasks for instrument-tissue characterization (ITC) in robot-assisted minimally invasive surgery, with broad applications in surgical scene understanding and 3D reconstruction. While precise instrument segmentation and detailed motion map estimation are the foundation of these tasks, existing methods commonly address these two tasks independently, and little attention has been paid to the fusion of these two tasks for exploring the intrinsic relationships between spatial segmentation and temporal motion. To address this, we propose ITCNet, a multi-task self supervised instrument segmentation and motion estimation network for instrument-tissue characterization, which simultaneously predicts motion maps and instrument masks from consecutive images. The proposed ITCNet is built upon a shared feature encoder for extracting unified representations and two task-specific decoders for each individual task. To dive into the inherent relationships between spatial segmentation and temporal motion, several fusion constraints are designed to facilitate the multi-task training process, including a motion-based geometric segmentation consistency and a segmentation-based edge-aware motion consistency. Meanwhile, the proposed multi-task fusion scheme enables the adaptation across different surgical scenes with few-shot and zero-shot learning strategies. Extensive experiments have been conducted on three public datasets for comparison with existing methods. Quantitative and qualitative results demonstrate that our proposed method can accurately segment surgical instruments and predict tissue motions across various surgical scenarios. Furthermore, application results on our established surgical robotic system emphasize its potential for real-world surgical robotics.

Fig. 1. Illustration of the proposed multi-task self-supervised instrument segmentation and motion estimation tasks. (a) Existing methods commonly perform these two tasks individually, and no interactions are available for two separate networks. (b) Our proposed ITCNet simultaneously leverages the spatial and temporal relationships between surgical instruments and target tissues. By introducing the multi-task fusion scheme, our proposed method can improve the performance of two individual tasks and adapt well to different surgical scenes. (c) The proposed ITCNet generalizes well to our established microsurgical robotic system for surgical suturing and provides accurate visual guidance for precise operations.

Fig. 2. Overview of the proposed ITCNet and multi-task fusion scheme. (a) The network consists of a shared feature extractor and two task-specific decoders for simultaneous surgical motion estimation and instrument segmentation. With consecutive frames as input, the network can estimate tool segmentation and motion field in parallel. (b) The detailed network architectures of the two branches. For the segmentation branch, the feature maps extracted by the shared encoder are directly sent to reconstruct segmentation masks. For the motion branch, features of two frames are concatenated to predict motion maps. (c) The basic constraints for training the motion branch include a photometric term and a smoothness term. (d) The basic constraints for training the segmentation branch by the teacher model-guided distillation strategy. (e) The proposed multi-task fusion constraints consist of the segmentation consistency and motion consistency to explore the inherent relationships of the two tasks in spatial and temporal domains.

Fig. 3. Overview of surgical robotic platform for microsurgical anastomosis. (a) The robot system is composed of two 6-axis micro-motors equipped with surgical instruments to accurately control the position and movements of needles and threads. The vision system consists of white-light microscopes to capture intraoperative RGB images. (b) Our proposed multi-task scheme for surgical motion estimation and instrument segmentation can be applied to different surgical suturing scenes.

https://doi.org/10.1016/j.media.2026.104279