In the rapidly evolving landscape of artificial intelligence, Retrieval-Augmented Generation (RAG) has emerged as a game-changing technology. RAG systems enhance the capabilities of large language models (LLMs) by retrieving relevant facts from external knowledge bases, ensuring that responses are not only coherent but also grounded in accurate, up-to-date information.
However, as we move into 2024, the limitations of traditional RAG systems have become increasingly apparent, particularly when dealing with global questions about extensive text documents. These systems, designed primarily for query-focused summarization, often struggle with full-text retrieval tasks and are constrained by the context window limitations of LLMs, leading to potential loss of crucial information.
Microsoft’s GraphRAG: A Leap Forward in AI Question-Answering
To address these challenges, Microsoft introduced GraphRAG, an innovative approach that leverages the power of graph-based text indexing. GraphRAG constructs a sophisticated knowledge structure in two key stages:
- Entity Knowledge Graph Creation: The system analyzes source documents to identify and interconnect key entities, forming a comprehensive knowledge graph.
- Community Summary Generation: For clusters of closely related entities, GraphRAG pre-generates summaries, creating a network of condensed, interconnected information.
When presented with a question, GraphRAG utilizes these community summaries to generate partial responses, which are then synthesized into a final, comprehensive answer. This method has shown remarkable improvements over traditional RAG baselines, particularly for global understanding questions on datasets of up to 1 million tokens.
Figure 1: Visual representation of the GraphRAG process, from document ingestion to answer generation.
Performance Metrics
Recent benchmarks have shown that GraphRAG outperforms standard RAG systems by:
- Improving answer comprehensiveness by 35%
- Enhancing response diversity by 28%
- Reducing factual inconsistencies by 42%
These improvements are particularly significant in fields requiring deep, multifaceted understanding of complex topics, such as scientific research, legal analysis, and strategic business intelligence.
The Cost Conundrum: Balancing Performance and Accessibility
While GraphRAG offers impressive capabilities, its computational demands present a significant barrier to widespread adoption. The process of extracting graph entities, summarizing content, and building graph indices using large language models is resource-intensive and costly.
To put this into perspective, consider the following cost breakdown for processing a 50,000-character document using GPT-4:
| Process Step | Token Consumption | Estimated Cost |
|---|---|---|
| Graph Index Construction | 270,000 tokens | $5.40 |
| Single Q&A Session | 10,000 tokens | $0.20 |
| Typical Test Run (5 queries) | 320,000 tokens | $6.40 |
These costs can quickly accumulate, making GraphRAG prohibitively expensive for many researchers, small businesses, and individual developers.
Democratizing Advanced AI: Local Models with Ollama and GraphRAG
In response to these cost challenges, a new approach has emerged, combining the power of GraphRAG with local models and Ollama. This innovative solution leverages free computational resources from the ModelScope community, making advanced AI technologies more accessible to a broader audience.
Understanding Ollama
Ollama is an open-source project that simplifies the deployment and use of large language models. It provides a user-friendly interface for running various AI models locally, offering several key advantages:
- Cost-effectiveness: Eliminates the need for expensive cloud computing resources
- Privacy: Keeps sensitive data on local systems, addressing data security concerns
- Customization: Allows for easy fine-tuning and adaptation of models to specific use cases
- Offline capability: Enables AI applications to run without constant internet connectivity
By integrating Ollama with GraphRAG, developers can now harness the power of advanced question-answering systems without incurring substantial cloud computing costs.
Technical Implementation: Bridging GraphRAG and Ollama
The integration of GraphRAG with Ollama involves several key modifications to the original GraphRAG architecture. The primary change lies in adapting the embedding call method from the OpenAI format to the Ollama format.
Code Analysis
The core of this adaptation is found in the /graphrag-local-ollama/graphrag/llm/openai/openai_embeddings_llm.py file. Here’s a breakdown of the key modifications:
class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):
# ... (previous code omitted for brevity)
async def _execute_llm(
self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]
) -> EmbeddingOutput | None:
args = {
"model": self._configuration.model,
**(kwargs.get("model_parameters") or {}),
}
embedding_list = []
for inp in input:
embedding = ollama.embeddings(model="nomic-embed-text", prompt=inp)
embedding_list.append(embedding["embedding"])
return embedding_listThis modification allows the system to leverage Ollama’s embedding capabilities, specifically using the nomic-embed-text model. The change is significant because it enables the use of locally run models, reducing dependency on cloud-based services and thereby cutting costs dramatically.
Why This Matters
The shift to Ollama’s format addresses several key issues:
- Reduced Latency: Local model inference can be faster than cloud-based API calls, especially for frequent, small queries.
- Enhanced Privacy: Sensitive data no longer needs to leave the local environment.
- Customization Potential: Users can more easily fine-tune and adapt the embedding model to their specific domains.
- Cost Control: Eliminates variable costs associated with pay-per-token cloud services.
Setting Up Your Local GraphRAG Environment
To implement this cost-effective GraphRAG solution, follow these steps:
1. Install Ollama
modelscope download --model=modelscope/ollama-linux --local_dir ./ollama-linux
cd ollama-linux
sudo chmod 777 ./ollama-modelscope-install.sh
./ollama-modelscope-install.sh2. Configure Models
For embeddings:
ollama pull nomic-embed-textFor the LLM, we’ll use ModelScope’s Mistral-7B-Instruct-v0.3:
modelscope download --model=LLM-Research/Mistral-7B-Instruct-v0.3-GGUF --local_dir . Mistral-7B-Instruct-v0.3.fp16.gguf3. Create and Configure the Model
Create a ModelFile with the necessary parameters and template, then use it to create your model:
ollama create mymistral --file ./ModelFile4. Set Up GraphRAG (Ollama Version)
Clone the repository and install dependencies:
git clone https://github.com/TheAiSingularity/graphrag-local-ollama.git
cd graphrag-local-ollama/
pip install -e .5. Prepare Your Data and Run GraphRAG
Set up your input data, initialize the environment, and run your first query:
mkdir -p ./ragtest/input
cp input/* ./ragtest/input
python -m graphrag.index --init --root ./ragtest
mv settings.yaml ./ragtest
python -m graphrag.index --root ./ragtest
python -m graphrag.query --root ./ragtest --method global "What is machine learning?"Real-World Applications and Future Prospects
The combination of GraphRAG and Ollama opens up exciting possibilities across various industries:
- Healthcare: Analyzing vast medical literature to assist in diagnosis and treatment planning.
- Legal Research: Quickly extracting relevant case law and statutes from extensive legal databases.
- Financial Analysis: Processing large volumes of market data to identify trends and investment opportunities.
- Education: Creating adaptive learning systems that can answer complex, multi-faceted questions from students.
As we look to the future, the integration of GraphRAG with local models like those offered by Ollama represents a significant step towards more accessible, powerful, and cost-effective AI systems. This approach not only democratizes access to advanced AI technologies but also paves the way for more innovative applications that can operate efficiently on local hardware.
Conclusion
The marriage of GraphRAG and Ollama marks a significant milestone in the democratization of advanced AI technologies. By leveraging local models and open-source tools, researchers, developers, and businesses can now explore the frontiers of question-answering systems and knowledge retrieval without the burden of prohibitive costs.
As we continue to push the boundaries of what’s possible with AI, solutions like this will play a crucial role in fostering innovation, enabling smaller players to compete with tech giants, and ultimately accelerating the pace of AI advancement for the benefit of all.
GitHub: https://github.com/TheAiSingularity/graphrag-local-ollama