DDSVM: A Differentiable Framework for Deep Support Vector Machines with Iterative Geometry-Aware Optimization

Yirun Ding, Zhihui Lai
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:24942-24959, 2026.

Abstract

Modern deep networks optimize features via cross-entropy without explicit margin maximization, while classical Support Vector Machines maximize the margin on a fixed feature space. Hybrid Deep-SVM approaches typically treat the deep feature extractor as a static input to an SVM head, so SVM geometry never flows back to shape representation learning. We propose the Differentiable Deep Support Vector Machine (DDSVM), an iterative framework that alternates between re-fitting a linear SVM on $L_2$-normalized features and updating the backbone to pull each feature along the SVM-defined margin-widening direction, treating the decision boundary as a dynamic teacher for representation learning. Under spherical normalization the resulting feature gradient lies in the tangent space and optimization focuses exclusively on angular separability. DDSVM reduces top-1 error by 5.6–8.3% on image classification benchmarks under data scarcity.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ding26f, title = {{DDSVM}: A Differentiable Framework for Deep Support Vector Machines with Iterative Geometry-Aware Optimization}, author = {Ding, Yirun and Lai, Zhihui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {24942--24959}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ding26f/ding26f.pdf}, url = {https://proceedings.mlr.press/v306/ding26f.html}, abstract = {Modern deep networks optimize features via cross-entropy without explicit margin maximization, while classical Support Vector Machines maximize the margin on a fixed feature space. Hybrid Deep-SVM approaches typically treat the deep feature extractor as a static input to an SVM head, so SVM geometry never flows back to shape representation learning. We propose the Differentiable Deep Support Vector Machine (DDSVM), an iterative framework that alternates between re-fitting a linear SVM on $L_2$-normalized features and updating the backbone to pull each feature along the SVM-defined margin-widening direction, treating the decision boundary as a dynamic teacher for representation learning. Under spherical normalization the resulting feature gradient lies in the tangent space and optimization focuses exclusively on angular separability. DDSVM reduces top-1 error by 5.6–8.3% on image classification benchmarks under data scarcity.} }
Endnote
%0 Conference Paper %T DDSVM: A Differentiable Framework for Deep Support Vector Machines with Iterative Geometry-Aware Optimization %A Yirun Ding %A Zhihui Lai %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ding26f %I PMLR %P 24942--24959 %U https://proceedings.mlr.press/v306/ding26f.html %V 306 %X Modern deep networks optimize features via cross-entropy without explicit margin maximization, while classical Support Vector Machines maximize the margin on a fixed feature space. Hybrid Deep-SVM approaches typically treat the deep feature extractor as a static input to an SVM head, so SVM geometry never flows back to shape representation learning. We propose the Differentiable Deep Support Vector Machine (DDSVM), an iterative framework that alternates between re-fitting a linear SVM on $L_2$-normalized features and updating the backbone to pull each feature along the SVM-defined margin-widening direction, treating the decision boundary as a dynamic teacher for representation learning. Under spherical normalization the resulting feature gradient lies in the tangent space and optimization focuses exclusively on angular separability. DDSVM reduces top-1 error by 5.6–8.3% on image classification benchmarks under data scarcity.
APA
Ding, Y. & Lai, Z.. (2026). DDSVM: A Differentiable Framework for Deep Support Vector Machines with Iterative Geometry-Aware Optimization. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:24942-24959 Available from https://proceedings.mlr.press/v306/ding26f.html.

Related Material