Label-Efficient Hyperspectral Image Classification via Spectral FiLM Modulation of Low-Level Pretrained Diffusion Features

Yuzhen Hu, Biplab Banerjee, Saurabh Prasad
Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation, PMLR 292:141-165, 2025.

Abstract

Hyperspectral imaging (HSI) enables detailed land cover classification, but low spatial resolution and sparse annotations pose significant challenges. We present a label-efficient framework that leverages spatial features from a frozen diffusion model pretrained on natural images. Specifically, we extract low-level representations from high-resolution decoder layers at early denoising timesteps, which transfer well to the low-texture setting of HSI. To combine spectral and spatial information, we introduce a lightweight FiLM-based fusion module that adaptively integrates spectral cues into frozen spatial features, enabling effective multimodal learning under sparse supervision. Experiments on two recent hyperspectral datasets show that our method outperforms state-of-the-art approaches using only the sparse training labels provided. Ablation studies further validate the benefit of diffusion-based features and spectral-aware fusion. Our results suggest that pretrained diffusion models can support domain-agnostic, label-efficient representation learning in remote sensing and scientific imaging tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v292-hu25a, title = {Label-Efficient Hyperspectral Image Classification via Spectral Fi{LM} Modulation of Low-Level Pretrained Diffusion Features}, author = {Hu, Yuzhen and Banerjee, Biplab and Prasad, Saurabh}, booktitle = {Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation}, pages = {141--165}, year = {2025}, editor = {Audebert, Nicolas and Azizpour, Hossein and Barrière, Valentin and Castillo Navarro, Javiera and Czerkawski, Mikolaj and Fang, Heng and Francis, Alistair and Marsocci, Valerio and Nascetti, Andrea and Yadav, Ritu}, volume = {292}, series = {Proceedings of Machine Learning Research}, month = {19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v292/main/assets/hu25a/hu25a.pdf}, url = {https://proceedings.mlr.press/v292/hu25a.html}, abstract = {Hyperspectral imaging (HSI) enables detailed land cover classification, but low spatial resolution and sparse annotations pose significant challenges. We present a label-efficient framework that leverages spatial features from a frozen diffusion model pretrained on natural images. Specifically, we extract low-level representations from high-resolution decoder layers at early denoising timesteps, which transfer well to the low-texture setting of HSI. To combine spectral and spatial information, we introduce a lightweight FiLM-based fusion module that adaptively integrates spectral cues into frozen spatial features, enabling effective multimodal learning under sparse supervision. Experiments on two recent hyperspectral datasets show that our method outperforms state-of-the-art approaches using only the sparse training labels provided. Ablation studies further validate the benefit of diffusion-based features and spectral-aware fusion. Our results suggest that pretrained diffusion models can support domain-agnostic, label-efficient representation learning in remote sensing and scientific imaging tasks.} }
Endnote
%0 Conference Paper %T Label-Efficient Hyperspectral Image Classification via Spectral FiLM Modulation of Low-Level Pretrained Diffusion Features %A Yuzhen Hu %A Biplab Banerjee %A Saurabh Prasad %B Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation %C Proceedings of Machine Learning Research %D 2025 %E Nicolas Audebert %E Hossein Azizpour %E Valentin Barrière %E Javiera Castillo Navarro %E Mikolaj Czerkawski %E Heng Fang %E Alistair Francis %E Valerio Marsocci %E Andrea Nascetti %E Ritu Yadav %F pmlr-v292-hu25a %I PMLR %P 141--165 %U https://proceedings.mlr.press/v292/hu25a.html %V 292 %X Hyperspectral imaging (HSI) enables detailed land cover classification, but low spatial resolution and sparse annotations pose significant challenges. We present a label-efficient framework that leverages spatial features from a frozen diffusion model pretrained on natural images. Specifically, we extract low-level representations from high-resolution decoder layers at early denoising timesteps, which transfer well to the low-texture setting of HSI. To combine spectral and spatial information, we introduce a lightweight FiLM-based fusion module that adaptively integrates spectral cues into frozen spatial features, enabling effective multimodal learning under sparse supervision. Experiments on two recent hyperspectral datasets show that our method outperforms state-of-the-art approaches using only the sparse training labels provided. Ablation studies further validate the benefit of diffusion-based features and spectral-aware fusion. Our results suggest that pretrained diffusion models can support domain-agnostic, label-efficient representation learning in remote sensing and scientific imaging tasks.
APA
Hu, Y., Banerjee, B. & Prasad, S.. (2025). Label-Efficient Hyperspectral Image Classification via Spectral FiLM Modulation of Low-Level Pretrained Diffusion Features. Proceedings of The TerraBytes {ICML} Workshop: Towards global datasets and models for Earth Observation, in Proceedings of Machine Learning Research 292:141-165 Available from https://proceedings.mlr.press/v292/hu25a.html.

Related Material