EfficientRAG: Boost Multi-Hop QA with Iterative Decomposition

Multi-hop question answering (QA) is a challenging task that requires complex queries to be resolved through multi-step reasoning, often surpassing the information that a single round of information retrieval can provide. To tackle this challenge, researchers from Nanjing University and Microsoft have developed EfficientRAG, a novel framework designed to enhance the efficiency and accuracy of multi-hop QA.

The EfficientRAG Framework

EfficientRAG consists of two lightweight components: Labeler & Tagger and Filter. These components function as token-level classifiers, identifying and filtering information effectively.

1. Labeler & Tagger: This component annotates useful information within documents, determining whether the retrieved segments contribute to answering the question.
2. Filter: The filter utilizes the annotated tokens along with the current query to construct new queries for subsequent retrieval rounds.

Iterative Retrieval Process

The EfficientRAG framework operates within an iterative Retrieval-Augmented Generation (RAG) system. Initially, it retrieves relevant segments from a knowledge base, labeling each segment as relevant or not, and annotating them with context, such as “KGOT in the Dimond Center.” The filter then processes the original question alongside the annotated labels to generate new queries. This iterative process continues until all segments are labeled as relevant or a maximum iteration limit is reached.

Performance Insights

Experimental results indicate that EfficientRAG outperforms existing RAG methods across three open-domain multi-hop QA datasets, including HotpotQA, MuSiQue, and 2WikiMQA. Key performance highlights include:

• Competitive Recall Rates: EfficientRAG maintains a high recall rate while minimizing the number of segments retrieved, demonstrating its efficiency in information retrieval.
• High Accuracy: In end-to-end question answering evaluations, EfficientRAG achieves accuracy levels comparable to state-of-the-art models based on large language models (LLMs), showcasing its effectiveness in handling complex queries.

Key Findings

EfficientRAG’s experiments yield several important conclusions:

• Retrieval Augmentation: The use of retrieved blocks significantly enhances model accuracy, whether employing “oracle chunks” (context directly providing the correct answer) or “mixed chunks” (a combination of correct and irrelevant contexts). Both approaches outperform direct answering without retrieved context.
• Challenges with Irrelevant Blocks: Despite improvements, irrelevant blocks still pose challenges for LLM generators, highlighting the need for refined information retrieval techniques to boost performance.

Query Decomposition Benefits

In the 2WikiMQA dataset, the performance of different segment settings using GPT-3.5/GPT-4/Llama3-8B as generators illustrates that query decomposition substantially improves retrieval outcomes. The number of “oracle chunks” retrieved through query decomposition exceeds results from directly retrieving the original query, indicating that breaking down complex queries enhances retrieval effectiveness.

Iterative Decomposition

EfficientRAG also introduces an iterative decomposition method, termed EfficientRAG Decompose. This method achieves higher recall rates with fewer chunks. When retrieving about 20 chunks, the recall rate achieved by EfficientRAG Decompose is comparable to the performance of LLM Decompose when retrieving about 200 chunks, showcasing the superiority of EfficientRAG Decompose in retrieval efficiency.

Appendix: Chain-of-Thought (CoT) Prompting for HotpotQA

To illustrate the application of EfficientRAG, we provide a detailed prompt for using the Llama-3 8B model in a Chain-of-Thought (CoT) question-answering task applied to HotpotQA.

CoT Prompting for HotpotQA:

As an assistant, your task is to answer the question after <Question>. You should first think step by step about the question and give your thought, then answer the <Question>. Your answer should be after <Answer> in JSON format with the key “thought” and “answer” and their values should be strings.

Here are some examples for your reference:

: What is the name of this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and was born on December 5, 1932?
:

{"thought":"Modern Record is a big R&B label with artists including Etta James, Joe Houston, Little Richard, Ike, Tina Turner, and John Lee Hooker in the 1950s and 1960s. Little Richard is an American musician, singer, actor, and songwriter, born on December 5, 1932. So the answer is Little Richard.","answer": "Little Richard"}

: Between Chinua Achebe and Rachel Carson, who had more diverse jobs?
:

{"thought":"Chinua Achebe was a Nigerian novelist, poet, professor, and critic. Rachel Carson was an American marine biologist, author, and conservationist. Chinua Achebe has 4 jobs while Rachel Carson has 3 jobs. So the answer is Chinua Achebe.","answer": "Chinua Achebe"}

: Remember Me Ballin’ is a CD single by Indo G that features an American rapper born in what year?
:

{"thought":"Remember Me Ballin’ is the CD single by Indo G that features Gangsta Boo, who is named Lola Mitchell, an American rapper born in 1979. So the answer is 1979.","answer": "1979"}

Conclusion

EfficientRAG represents a significant advancement in multi-hop question answering, demonstrating that effective query decomposition and retrieval strategies can lead to substantial improvements in performance and accuracy. By leveraging EfficientRAG, researchers and practitioners can enhance their systems’ capabilities in managing complex multi-hop queries, paving the way for more intelligent and responsive AI applications.