InstEmb: Instruction-Following Embeddings through Glimpses of the Future

Tianhao Gao, Jun Fang, Xiaohui Zhang, Zhiyuan Liu, Chao Liu, Pengzhang Liu, Qixia Jiang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:33138-33150, 2026.

Abstract

Recent advances have empowered large language models (LLMs) with remarkable fine-grained instruction-following capabilities in text generation tasks. However, embedding methods typically rely solely on the hidden state of the input’s last token, limiting their ability to capture complete semantic signals distributed across the full output tokens. Moreover, existing discrete-to-continuous re-encoding approaches introduce semantic discontinuity. To address these limitations, we propose $\textbf{InstEmb}$, a novel instruction following embedding framework. InstEmb jointly optimizes two key aspects: (1) Input-Intrinsic semantic information, achieved by employing contrastive learning focused on the representation of the last input token, and (2) Output-Aware semantic information, captured through representation self-distillation leveraging learnable look-ahead tokens without introducing additional decoding latency. Additionally, we introduce $\textbf{Dual-Anchor Alignment Pooling (DAAP)}$, explicitly aligned with our dual training objectives. Extensive experiments demonstrate that InstEmb achieves state-of-the-art performance across multiple instruction following benchmarks without benchmark-specific supervised data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gao26h, title = {{I}nst{E}mb: Instruction-Following Embeddings through Glimpses of the Future}, author = {Gao, Tianhao and Fang, Jun and Zhang, Xiaohui and Liu, Zhiyuan and Liu, Chao and Liu, Pengzhang and Jiang, Qixia}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {33138--33150}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gao26h/gao26h.pdf}, url = {https://proceedings.mlr.press/v306/gao26h.html}, abstract = {Recent advances have empowered large language models (LLMs) with remarkable fine-grained instruction-following capabilities in text generation tasks. However, embedding methods typically rely solely on the hidden state of the input’s last token, limiting their ability to capture complete semantic signals distributed across the full output tokens. Moreover, existing discrete-to-continuous re-encoding approaches introduce semantic discontinuity. To address these limitations, we propose $\textbf{InstEmb}$, a novel instruction following embedding framework. InstEmb jointly optimizes two key aspects: (1) Input-Intrinsic semantic information, achieved by employing contrastive learning focused on the representation of the last input token, and (2) Output-Aware semantic information, captured through representation self-distillation leveraging learnable look-ahead tokens without introducing additional decoding latency. Additionally, we introduce $\textbf{Dual-Anchor Alignment Pooling (DAAP)}$, explicitly aligned with our dual training objectives. Extensive experiments demonstrate that InstEmb achieves state-of-the-art performance across multiple instruction following benchmarks without benchmark-specific supervised data.} }
Endnote
%0 Conference Paper %T InstEmb: Instruction-Following Embeddings through Glimpses of the Future %A Tianhao Gao %A Jun Fang %A Xiaohui Zhang %A Zhiyuan Liu %A Chao Liu %A Pengzhang Liu %A Qixia Jiang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gao26h %I PMLR %P 33138--33150 %U https://proceedings.mlr.press/v306/gao26h.html %V 306 %X Recent advances have empowered large language models (LLMs) with remarkable fine-grained instruction-following capabilities in text generation tasks. However, embedding methods typically rely solely on the hidden state of the input’s last token, limiting their ability to capture complete semantic signals distributed across the full output tokens. Moreover, existing discrete-to-continuous re-encoding approaches introduce semantic discontinuity. To address these limitations, we propose $\textbf{InstEmb}$, a novel instruction following embedding framework. InstEmb jointly optimizes two key aspects: (1) Input-Intrinsic semantic information, achieved by employing contrastive learning focused on the representation of the last input token, and (2) Output-Aware semantic information, captured through representation self-distillation leveraging learnable look-ahead tokens without introducing additional decoding latency. Additionally, we introduce $\textbf{Dual-Anchor Alignment Pooling (DAAP)}$, explicitly aligned with our dual training objectives. Extensive experiments demonstrate that InstEmb achieves state-of-the-art performance across multiple instruction following benchmarks without benchmark-specific supervised data.
APA
Gao, T., Fang, J., Zhang, X., Liu, Z., Liu, C., Liu, P. & Jiang, Q.. (2026). InstEmb: Instruction-Following Embeddings through Glimpses of the Future. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:33138-33150 Available from https://proceedings.mlr.press/v306/gao26h.html.

Related Material