Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

​RAG empowers LLMs to leverage external knowledge sources. As LLMs gain the ability to process longer input sequences, this opens avenues for integrating more retrieved information to potentially enhance output quality.

​Assuming that a larger retrieval set contains more relevant information (higher recall), this might lead to improved performance. But empirical findings show that for many long-context LLMs, the quality of generated output initially improves but subsequently declines as the number of retrieved passages increases.

​Our guest, Bowen Jin from the University of Illinois Urbana-Champaign, investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key factor.

To address this and enhance the robustness of long-context LLM-based RAG, he proposes both training-free and training-based approaches.

He demonstrates the effectiveness of retrieval reordering as a simple yet powerful training-free optimization.

​Additionally, he explores training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, showing their potential for substantial performance gains.

​Finally, he conducts a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

Read the paper

Join the BuzzRobot Slack to connect with the community