SWNet: Surgical Workflow Recognition with Deep Convolutional Network

Bokai Zhang, Amer Ghanem, Alexander Simes, Henry Choi, Andrew Yoo, Andrew Min
Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, PMLR 143:855-869, 2021.

Abstract

Surgical workflow recognition has been playing an essential role in computer-assisted interventional systems for modern operating rooms. In this paper, we present a computer vision-based method named SWNet that focuses on utilizing spatial information and temporal information from the surgical video to achieve surgical workflow recognition. As the first step, we utilize Interaction-Preserved Channel-Separated Convolutional Network (IP-CSN) to extract features that contain spatial information and local temporal information from the surgical video through segments. Secondly, we train a Multi-Stage Temporal Convolutional Network (MS-TCN) with those extracted features to capture global temporal information from the full surgical video. Finally, by utilizing Prior Knowledge Noise Filtering (PKNF), prediction noise from the output of MS-TCN is filtered. We evaluate SWNet for Sleeve Gastrectomy surgical workflow recognition. SWNet achieves 90% frame-level accuracy and reaches a weighted Jaccard Score of 0.8256. This demonstrates that SWNet has considerable potential to solve the surgical workflow recognition problem.

Cite this Paper

BibTeX
@InProceedings{pmlr-v143-zhang21b, title = {{SWN}et: Surgical Workflow Recognition with Deep Convolutional Network}, author = {Zhang, Bokai and Ghanem, Amer and Simes, Alexander and Choi, Henry and Yoo, Andrew and Min, Andrew}, booktitle = {Proceedings of the Fourth Conference on Medical Imaging with Deep Learning}, pages = {855--869}, year = {2021}, editor = {Heinrich, Mattias and Dou, Qi and de Bruijne, Marleen and Lellmann, Jan and Schläfer, Alexander and Ernst, Floris}, volume = {143}, series = {Proceedings of Machine Learning Research}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v143/zhang21b/zhang21b.pdf}, url = {https://proceedings.mlr.press/v143/zhang21b.html}, abstract = {Surgical workflow recognition has been playing an essential role in computer-assisted interventional systems for modern operating rooms. In this paper, we present a computer vision-based method named SWNet that focuses on utilizing spatial information and temporal information from the surgical video to achieve surgical workflow recognition. As the first step, we utilize Interaction-Preserved Channel-Separated Convolutional Network (IP-CSN) to extract features that contain spatial information and local temporal information from the surgical video through segments. Secondly, we train a Multi-Stage Temporal Convolutional Network (MS-TCN) with those extracted features to capture global temporal information from the full surgical video. Finally, by utilizing Prior Knowledge Noise Filtering (PKNF), prediction noise from the output of MS-TCN is filtered. We evaluate SWNet for Sleeve Gastrectomy surgical workflow recognition. SWNet achieves 90% frame-level accuracy and reaches a weighted Jaccard Score of 0.8256. This demonstrates that SWNet has considerable potential to solve the surgical workflow recognition problem.} }
Endnote
%0 Conference Paper %T SWNet: Surgical Workflow Recognition with Deep Convolutional Network %A Bokai Zhang %A Amer Ghanem %A Alexander Simes %A Henry Choi %A Andrew Yoo %A Andrew Min %B Proceedings of the Fourth Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2021 %E Mattias Heinrich %E Qi Dou %E Marleen de Bruijne %E Jan Lellmann %E Alexander Schläfer %E Floris Ernst %F pmlr-v143-zhang21b %I PMLR %P 855--869 %U https://proceedings.mlr.press/v143/zhang21b.html %V 143 %X Surgical workflow recognition has been playing an essential role in computer-assisted interventional systems for modern operating rooms. In this paper, we present a computer vision-based method named SWNet that focuses on utilizing spatial information and temporal information from the surgical video to achieve surgical workflow recognition. As the first step, we utilize Interaction-Preserved Channel-Separated Convolutional Network (IP-CSN) to extract features that contain spatial information and local temporal information from the surgical video through segments. Secondly, we train a Multi-Stage Temporal Convolutional Network (MS-TCN) with those extracted features to capture global temporal information from the full surgical video. Finally, by utilizing Prior Knowledge Noise Filtering (PKNF), prediction noise from the output of MS-TCN is filtered. We evaluate SWNet for Sleeve Gastrectomy surgical workflow recognition. SWNet achieves 90% frame-level accuracy and reaches a weighted Jaccard Score of 0.8256. This demonstrates that SWNet has considerable potential to solve the surgical workflow recognition problem.
APA
Zhang, B., Ghanem, A., Simes, A., Choi, H., Yoo, A. & Min, A.. (2021). SWNet: Surgical Workflow Recognition with Deep Convolutional Network. Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 143:855-869 Available from https://proceedings.mlr.press/v143/zhang21b.html.

Related Material