Fixing the Background: Segmentation-Guided Feature Refinement for Multiple Object Tracking

Muhammad Hassan Nisar, Syeda Umme Umama, Muhammad Imran Shehzad, Hazrat Ali, Shoaib Azmat
Proceedings of the Fourth UK AI Conference 2026, PMLR 348:106-115, 2026.

Abstract

Tracking-by-detection algorithms in multiple object tracking systems rely on bounding boxes for object localization and data association. Bounding boxes also contain extraneous background information, which contaminates appearance features, thereby increasing false or missed detections. This paper proposes a multiple object tracking framework that integrates segmentation-based background removal into the tracking pipeline. Mask R-CNN is used to eliminate background clutter within the bounding boxes, producing refined detections. Deep SORT uses a Kalman filter for motion modeling and, in parallel, a convolutional neural network extracts appearance features from these refined detections. Data association is performed by evaluating the similarity between predicted tracks and detections using both motion and appearance features. The final matching between tracks and detections is performed using the Hungarian matching, enabling improved object detections over time. The robustness of data association is enhanced in challenging situations, such as occlusion and clutter, by improving appearance features. The approach demonstrated improvement in the overall tracking performance on the MOT16 challenge dataset, resulting in reduction in false or missed detections, and an increase in multiple object tracking accuracy (MOTA).

Cite this Paper

BibTeX
@InProceedings{pmlr-v348-nisar26a, title = {Fixing the Background: Segmentation-Guided Feature Refinement for Multiple Object Tracking}, author = {Nisar, Muhammad Hassan and Umama, Syeda Umme and Shehzad, Muhammad Imran and Ali, Hazrat and Azmat, Shoaib}, booktitle = {Proceedings of the Fourth UK AI Conference 2026}, pages = {106--115}, year = {2026}, editor = {Benford, Alistair and Büyükateş, Baturalp and Cabrera, Christian and Kiden, Sarah and Salili-James, Arianna and Zakka, Vincent and Zhou, Feng}, volume = {348}, series = {Proceedings of Machine Learning Research}, month = {29--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v348/main/assets/nisar26a/nisar26a.pdf}, url = {https://proceedings.mlr.press/v348/nisar26a.html}, abstract = {Tracking-by-detection algorithms in multiple object tracking systems rely on bounding boxes for object localization and data association. Bounding boxes also contain extraneous background information, which contaminates appearance features, thereby increasing false or missed detections. This paper proposes a multiple object tracking framework that integrates segmentation-based background removal into the tracking pipeline. Mask R-CNN is used to eliminate background clutter within the bounding boxes, producing refined detections. Deep SORT uses a Kalman filter for motion modeling and, in parallel, a convolutional neural network extracts appearance features from these refined detections. Data association is performed by evaluating the similarity between predicted tracks and detections using both motion and appearance features. The final matching between tracks and detections is performed using the Hungarian matching, enabling improved object detections over time. The robustness of data association is enhanced in challenging situations, such as occlusion and clutter, by improving appearance features. The approach demonstrated improvement in the overall tracking performance on the MOT16 challenge dataset, resulting in reduction in false or missed detections, and an increase in multiple object tracking accuracy (MOTA).} }
Endnote
%0 Conference Paper %T Fixing the Background: Segmentation-Guided Feature Refinement for Multiple Object Tracking %A Muhammad Hassan Nisar %A Syeda Umme Umama %A Muhammad Imran Shehzad %A Hazrat Ali %A Shoaib Azmat %B Proceedings of the Fourth UK AI Conference 2026 %C Proceedings of Machine Learning Research %D 2026 %E Alistair Benford %E Baturalp Büyükateş %E Christian Cabrera %E Sarah Kiden %E Arianna Salili-James %E Vincent Zakka %E Feng Zhou %F pmlr-v348-nisar26a %I PMLR %P 106--115 %U https://proceedings.mlr.press/v348/nisar26a.html %V 348 %X Tracking-by-detection algorithms in multiple object tracking systems rely on bounding boxes for object localization and data association. Bounding boxes also contain extraneous background information, which contaminates appearance features, thereby increasing false or missed detections. This paper proposes a multiple object tracking framework that integrates segmentation-based background removal into the tracking pipeline. Mask R-CNN is used to eliminate background clutter within the bounding boxes, producing refined detections. Deep SORT uses a Kalman filter for motion modeling and, in parallel, a convolutional neural network extracts appearance features from these refined detections. Data association is performed by evaluating the similarity between predicted tracks and detections using both motion and appearance features. The final matching between tracks and detections is performed using the Hungarian matching, enabling improved object detections over time. The robustness of data association is enhanced in challenging situations, such as occlusion and clutter, by improving appearance features. The approach demonstrated improvement in the overall tracking performance on the MOT16 challenge dataset, resulting in reduction in false or missed detections, and an increase in multiple object tracking accuracy (MOTA).
APA
Nisar, M.H., Umama, S.U., Shehzad, M.I., Ali, H. & Azmat, S.. (2026). Fixing the Background: Segmentation-Guided Feature Refinement for Multiple Object Tracking. Proceedings of the Fourth UK AI Conference 2026, in Proceedings of Machine Learning Research 348:106-115 Available from https://proceedings.mlr.press/v348/nisar26a.html.

Related Material