RankGPT: ChatGPT as Re-Ranking Agent
This project examines the use of generative LLMs, specifically ChatGPT and GPT-4, for relevance ranking in Information Retrieval (IR). The main goal is to explore whether ChatGPT is good at search and investigate large language models as re-ranking agents. The project aims to answer the following two questions:
- Can we use LLMs such as ChatGPT to directly output the permutations of a group of passages?
- Can we use LLMs such as ChatGPT as re-ranking agents for retrieval?
To answer the first question, the project introduces an instructional permutation generation approach that instructs LLMs to directly output the permutations of a group of passages. To answer the second question, the project trains a cross-encoder using 10K ChatGPT predicted permutations on MS MARCO.
The project includes a sliding window strategy for the instructional permutation generation, which enables LLMs to rank more passages than their maximum token limit. This strategy allows ranking from back to front, using a sliding window and re-ranking only the passages within the window at a time.
The project utilizes pyserini to retrieve 100 passages for each query and re-ranks them using instructional permutation generation. The evaluation on TREC-DL19 and other benchmarks yielded promising results, showing the effectiveness of ChatGPT as a re-ranking agent in information retrieval.
