Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Topics: AI (Deep Learning), LLMO, Ranking
The research paper titled “Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents” explores the use of Large Language Models (LLMs), specifically ChatGPT and GPT-4, in the context of Information Retrieval (IR) tasks, particularly passage re-ranking. The paper finds that properly instructed LLMs can outperform state-of-the-art supervised methods on popular IR benchmarks. It introduces a new test set, NovelEval, to assess the model’s ability to handle unknown knowledge and proposes a permutation distillation method to create smaller, specialized models that retain the LLMs’ ranking capabilities.