Query Refinement in Dense Retrieval Using LLM-Driven Relevance Feedback

Seyedehfatemeh Karimi, Maryam Khodabakhsh, Havva Alizadeh Noughabi, Fattane Zarrinkalam
Proceedings of the The 39th Canadian Conference on Artificial Intelligence, PMLR 318:1084-1091, 2026.

Abstract

Dense retrieval methods, which encode queries and documents into a shared semantic embedding space, have achieved strong performance in information retrieval tasks. However, their effectiveness diminishes in scenarios with limited or no domain-specific training data. To mitigate this limitation, recent approaches have leveraged large language models (LLMs) for query refinement in unsupervised dense retriever systems. A promising direction within this line of research involves using LLMs to assess the relevance of initially retrieved documents, and then incorporating the resulting relevance feedback to update the query embedding. Despite promising early results, a systematic investigation of how different prompting strategies and query update mechanisms influence retrieval performance remains absent. In this study, we explore four prompting strategies—Zero-Shot, Few-Shot, Role-Playing, and Chain-of-Thought—to guide LLMs in performing relevance judgments. Furthermore, we evaluate various query update formulas that utilize embeddings of LLM-identified relevant documents to refine query representations. Our experiments, conducted on two datasets and using two open-source LLMs, demonstrate that carefully crafted prompting combined with effective query updates can substantially enhance retrieval performance. These findings provide valuable insights for optimizing LLM-guided relevance feedback in unsupervised dense retrieval. All code and datasets are available at https://github.com/ftmkm97/ReFeed-IR.git.

Cite this Paper

BibTeX
@InProceedings{pmlr-v318-karimi26a, title = {Query Refinement in Dense Retrieval Using LLM-Driven Relevance Feedback}, author = {Karimi, Seyedehfatemeh and Khodabakhsh, Maryam and Noughabi, Havva Alizadeh and Zarrinkalam, Fattane}, booktitle = {Proceedings of the The 39th Canadian Conference on Artificial Intelligence}, pages = {1084--1091}, year = {2026}, editor = {Bouzar-Benlabiod, Lydia and Leung, Carson}, volume = {318}, series = {Proceedings of Machine Learning Research}, month = {25--29 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v318/main/assets/karimi26a/karimi26a.pdf}, url = {https://proceedings.mlr.press/v318/karimi26a.html}, abstract = {Dense retrieval methods, which encode queries and documents into a shared semantic embedding space, have achieved strong performance in information retrieval tasks. However, their effectiveness diminishes in scenarios with limited or no domain-specific training data. To mitigate this limitation, recent approaches have leveraged large language models (LLMs) for query refinement in unsupervised dense retriever systems. A promising direction within this line of research involves using LLMs to assess the relevance of initially retrieved documents, and then incorporating the resulting relevance feedback to update the query embedding. Despite promising early results, a systematic investigation of how different prompting strategies and query update mechanisms influence retrieval performance remains absent. In this study, we explore four prompting strategies—Zero-Shot, Few-Shot, Role-Playing, and Chain-of-Thought—to guide LLMs in performing relevance judgments. Furthermore, we evaluate various query update formulas that utilize embeddings of LLM-identified relevant documents to refine query representations. Our experiments, conducted on two datasets and using two open-source LLMs, demonstrate that carefully crafted prompting combined with effective query updates can substantially enhance retrieval performance. These findings provide valuable insights for optimizing LLM-guided relevance feedback in unsupervised dense retrieval. All code and datasets are available at https://github.com/ftmkm97/ReFeed-IR.git.} }
Endnote
%0 Conference Paper %T Query Refinement in Dense Retrieval Using LLM-Driven Relevance Feedback %A Seyedehfatemeh Karimi %A Maryam Khodabakhsh %A Havva Alizadeh Noughabi %A Fattane Zarrinkalam %B Proceedings of the The 39th Canadian Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Lydia Bouzar-Benlabiod %E Carson Leung %F pmlr-v318-karimi26a %I PMLR %P 1084--1091 %U https://proceedings.mlr.press/v318/karimi26a.html %V 318 %X Dense retrieval methods, which encode queries and documents into a shared semantic embedding space, have achieved strong performance in information retrieval tasks. However, their effectiveness diminishes in scenarios with limited or no domain-specific training data. To mitigate this limitation, recent approaches have leveraged large language models (LLMs) for query refinement in unsupervised dense retriever systems. A promising direction within this line of research involves using LLMs to assess the relevance of initially retrieved documents, and then incorporating the resulting relevance feedback to update the query embedding. Despite promising early results, a systematic investigation of how different prompting strategies and query update mechanisms influence retrieval performance remains absent. In this study, we explore four prompting strategies—Zero-Shot, Few-Shot, Role-Playing, and Chain-of-Thought—to guide LLMs in performing relevance judgments. Furthermore, we evaluate various query update formulas that utilize embeddings of LLM-identified relevant documents to refine query representations. Our experiments, conducted on two datasets and using two open-source LLMs, demonstrate that carefully crafted prompting combined with effective query updates can substantially enhance retrieval performance. These findings provide valuable insights for optimizing LLM-guided relevance feedback in unsupervised dense retrieval. All code and datasets are available at https://github.com/ftmkm97/ReFeed-IR.git.
APA
Karimi, S., Khodabakhsh, M., Noughabi, H.A. & Zarrinkalam, F.. (2026). Query Refinement in Dense Retrieval Using LLM-Driven Relevance Feedback. Proceedings of the The 39th Canadian Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 318:1084-1091 Available from https://proceedings.mlr.press/v318/karimi26a.html.

Related Material