This article provides a comprehensive overview of Retrieval-Augmented Generation (RAG), a technique that enhances AI models by allowing them to retrieve information from external sources before generating a response. It covers the core concepts, architecture, and practical considerations for implementing RAG systems.
What is RAG?
RAG addresses the limitations of standard AI models, such as:
- Knowledge Cutoff: AI models have a training cutoff date and lack awareness of recent events.
- Private Data Access: Models cannot access private or internal data.
- Retraining Costs: Retraining models is expensive and time-consuming.
RAG solves these issues by enabling the AI to retrieve relevant documents and use them as context for generating answers.
How RAG Works
The RAG process involves the following steps:
- Query: The user asks a question.
- Retrieval: The system searches for relevant information in a knowledge base.
- Augmentation: The retrieved content is combined with the original query.
- Generation: The AI generates an answer based on the augmented context.
RAG Architecture: Two Pipelines
RAG consists of two main pipelines:
1. Ingestion Pipeline
This pipeline prepares the knowledge base and runs once (or when documents are updated). The steps include:
- Loading Documents: Ingesting raw documents (PDFs, Word files, etc.).
- Chunking: Splitting documents into smaller, manageable pieces (chunks).
- Embedding: Converting chunks into numerical vectors using an embedding model.
- Storing: Storing the embedded chunks in a vector database.
2. Retrieval Pipeline
This pipeline runs every time a user asks a question. The steps include:
- Receiving Query: Receiving the user’s question.
- Embedding Query: Converting the query into a vector using the same embedding model.
- Searching: Searching the vector database for the closest matching chunks.
- Assembling Prompt: Combining the retrieved chunks with the original query into a prompt.
- Generating Answer: The LLM generates an answer based on the prompt.
Hybrid Search
Hybrid search combines vector search (semantic similarity) with BM25 (keyword search) to improve retrieval accuracy.
Chunking Strategies
Different chunking strategies are available:
- Character Split: Splits documents based on a character count.
- Recursive Character Split: Splits documents based on paragraph breaks, newlines, or sentence ends.
- Document-Specific Chunking: Splits documents based on their structure (e.g., headers in Markdown files).
- Semantic Chunking: Splits documents based on topic changes detected by an embedding model.
- Agentic Chunking: Uses an LLM to decide where to split the document.
Retrieval Strategies
Different retrieval strategies are available:
- Similarity Search: Returns the k closest matching chunks.
- Similarity Search with Score Threshold: Returns chunks with similarity scores above a minimum threshold.
- Maximum Marginal Relevance (MMR): Retrieves chunks that are relevant but not repetitive.
Improving Search Accuracy
Techniques to improve search accuracy include:
- Multi-Query Retrieval: Rewrites the user’s question into multiple variations.
- Reciprocal Rank Fusion (RRF): Merges multiple result lists and ranks chunks based on their combined score.
- Reranking: Reranks the top results using a deeper check of the query and candidate chunks.
RAG Use Cases
RAG is used in various applications, including:
- Customer support bots
- Legal teams
- Medical assistants
- Internal knowledge bases
RAG Limitations
- Latency: Extra processing steps add to response time.
- Cost: Semantic chunking, reranking models, and API calls can be expensive.
- Data Quality: RAG amplifies existing issues in the knowledge base.
Conclusion
Understanding RAG is beneficial even when working with vendors or building systems from scratch. Start with simple methods and add complexity as needed. Prioritize clean, well-organized source documents.