Robust Non-linear Normalization of Heterogeneous Feature Distributions with Adaptive Tanh-Estimators

Felip Guimerà Cuevas, Helmut Schmid
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:406-414, 2024.

Abstract

Feature normalization is a crucial step in machine learning that scales numerical values to improve model effectiveness. Noisy or impure datasets can pose a challenge for traditional normalization methods as they may contain outliers that violate statistical assumptions, leading to reduced model performance and increased unpredictability. Non-linear Tanh-Estimators (TE) have been found to provide robust feature normalization, but their fixed scaling factor may not be appropriate for all distributions of feature values. This work presents a refinement to the TE that employs the Wasserstein distance to adaptively estimate the optimal scaling factor for each feature individually against a specified target distribution. The results demonstrate that this adaptive approach can outperform the current TE method in the literature in terms of convergence speed by enabling better initial training starts, thus reducing or eliminating the need to re-adjust model weights during early training phases due to inadequately scaled features. Empirical evaluation was done on synthetic data, standard toy computer vision datasets, and a real-world numeric tabular dataset.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-guimera-cuevas24a, title = { Robust Non-linear Normalization of Heterogeneous Feature Distributions with Adaptive Tanh-Estimators }, author = {Guimer\`{a} Cuevas, Felip and Schmid, Helmut}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {406--414}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/guimera-cuevas24a/guimera-cuevas24a.pdf}, url = {https://proceedings.mlr.press/v238/guimera-cuevas24a.html}, abstract = { Feature normalization is a crucial step in machine learning that scales numerical values to improve model effectiveness. Noisy or impure datasets can pose a challenge for traditional normalization methods as they may contain outliers that violate statistical assumptions, leading to reduced model performance and increased unpredictability. Non-linear Tanh-Estimators (TE) have been found to provide robust feature normalization, but their fixed scaling factor may not be appropriate for all distributions of feature values. This work presents a refinement to the TE that employs the Wasserstein distance to adaptively estimate the optimal scaling factor for each feature individually against a specified target distribution. The results demonstrate that this adaptive approach can outperform the current TE method in the literature in terms of convergence speed by enabling better initial training starts, thus reducing or eliminating the need to re-adjust model weights during early training phases due to inadequately scaled features. Empirical evaluation was done on synthetic data, standard toy computer vision datasets, and a real-world numeric tabular dataset. } }
Endnote
%0 Conference Paper %T Robust Non-linear Normalization of Heterogeneous Feature Distributions with Adaptive Tanh-Estimators %A Felip Guimerà Cuevas %A Helmut Schmid %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-guimera-cuevas24a %I PMLR %P 406--414 %U https://proceedings.mlr.press/v238/guimera-cuevas24a.html %V 238 %X Feature normalization is a crucial step in machine learning that scales numerical values to improve model effectiveness. Noisy or impure datasets can pose a challenge for traditional normalization methods as they may contain outliers that violate statistical assumptions, leading to reduced model performance and increased unpredictability. Non-linear Tanh-Estimators (TE) have been found to provide robust feature normalization, but their fixed scaling factor may not be appropriate for all distributions of feature values. This work presents a refinement to the TE that employs the Wasserstein distance to adaptively estimate the optimal scaling factor for each feature individually against a specified target distribution. The results demonstrate that this adaptive approach can outperform the current TE method in the literature in terms of convergence speed by enabling better initial training starts, thus reducing or eliminating the need to re-adjust model weights during early training phases due to inadequately scaled features. Empirical evaluation was done on synthetic data, standard toy computer vision datasets, and a real-world numeric tabular dataset.
APA
Guimerà Cuevas, F. & Schmid, H.. (2024). Robust Non-linear Normalization of Heterogeneous Feature Distributions with Adaptive Tanh-Estimators . Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:406-414 Available from https://proceedings.mlr.press/v238/guimera-cuevas24a.html.

Related Material