RAG System Design Interview Questions — How to Design a Retrieval-Augmented Generation System

Two years ago, "RAG" barely appeared in interview prep guides. Today, RAG system design is tested at virtually every AI engineering interview — ML engineer, backend engineer at an AI company, even senior SWE at companies building AI products.

The question usually sounds like: "Design a system that lets users ask questions about our 10 million internal documents." It's a RAG system design question. And if you've never thought through the pieces carefully, you'll give a hand-wavy answer that interviewers see through immediately.

This guide walks through every component of a production RAG system — chunking, embeddings, vector databases, retrieval, reranking, generation, and evaluation — and gives you the answers to the follow-up questions that actually separate candidates.

Why RAG Shows Up in Interviews Now

RAG (Retrieval-Augmented Generation) is the dominant pattern for building AI systems that need to work with private or dynamic knowledge — internal documents, product data, legal contracts, customer history. LLMs have a knowledge cutoff and a context window limit. RAG solves both: retrieve only the relevant pieces, inject them into the prompt, generate a grounded answer.

It's in interviews because it's become a standard production architecture, not a research topic. Companies like Notion, Confluence, Glean, and thousands of enterprise AI products are built on RAG. Understanding it deeply is now table stakes for any AI engineering role.

RAG Architecture — The Full Picture

A production RAG system has two pipelines that run at different times:

Ingestion pipeline (offline, runs when documents are added/updated):

Raw documents → Preprocessing → Chunking → Embedding → Vector store

Query pipeline (online, runs per user request):

User query → Query embedding → Vector search → Reranking → LLM generation → Response

Let's walk through each step. Most interview answers stop at "embed documents, search, then ask GPT." The ones that pass cover what's inside each step.

Step 1: Chunking and Ingestion

Before you can embed a document, you need to split it into chunks. This is the step most people underestimate — and it's where a lot of RAG failures originate.

Why chunking matters: Embedding models have a token limit (typically 512–8192 tokens). A 50-page PDF can't be embedded as a single unit. More importantly, you want your chunks to be semantically coherent — a chunk should contain one idea, so when a user asks about that idea, the chunk scores high in retrieval.

The main chunking strategies:

Fixed-size chunks — split every 256, 512, or 1024 tokens with some overlap (e.g., 10–20% overlap between adjacent chunks to avoid cutting sentences mid-thought). Simple to implement. Works adequately for uniform documents (transcripts, support tickets). Fails on structured documents (legal contracts, technical documentation) where topic boundaries don't align with fixed sizes.

Sentence-based chunks — use an NLP library (spaCy, NLTK) to split on sentence boundaries, then group sentences until a token limit. Better semantic coherence than fixed-size. Still misses document-level structure.

Semantic chunking — compute embeddings for each sentence, then split where the cosine similarity between adjacent sentences drops below a threshold (indicating a topic change). Expensive but produces the most coherent chunks. Used in langchain.text_splitter.SemanticChunker.

Hierarchical / parent-child chunks — chunk at two granularities: large "parent" chunks (e.g., full paragraphs or sections) and small "child" chunks (e.g., individual sentences). Retrieve at the child level for precision, but return the parent chunk to the LLM for context. This is the approach used by LlamaIndex's auto-merging retriever.

Document preprocessing before chunking:

• Remove headers, footers, page numbers from PDFs (they pollute chunks with noise)
• Preserve structural metadata: page number, section title, document ID, date. Store this as chunk metadata alongside the embedding — you'll use it for filtering and citation.
• Handle tables specially — raw table text doesn't embed well. Either convert to markdown or use a multimodal embedding that handles structured data.

Step 2: Embedding Models

An embedding model converts a chunk of text into a dense vector (typically 768–3072 dimensions). Semantically similar texts produce vectors that are close in the embedding space.

The main choices in 2026:

What interviewers probe:

"Should you use the same embedding model for chunks and queries?"

Yes, always. The embedding space is model-specific — a query embedded with model A and chunks embedded with model B will produce meaningless similarity scores. Use the same model for both ingestion and query.

"How do you handle multilingual documents?"

Use a multilingual embedding model (Cohere embed-v3-multilingual, BGE-M3). Don't run separate models per language and try to merge the results — the embedding spaces won't align.

"What happens when you update your embedding model?"

You must re-embed all existing documents with the new model. The old embeddings are incompatible. This is a real operational cost for large corpora — 10M documents at 3ms per embedding = 8+ hours of reembedding.

Step 3: Vector Databases

A vector database stores embeddings and supports approximate nearest neighbor (ANN) search — finding the most similar vectors to a query vector efficiently.

The main options:

The ANN algorithm question:

Most vector databases use HNSW (Hierarchical Navigable Small World) — a graph-based ANN algorithm that gives O(log N) search time with excellent recall. The trade-off: high memory usage (stores the graph structure in RAM) and slower build time for large indexes.

Filtering with metadata:

Real RAG systems don't just search all vectors — they filter first. "Find the most similar chunks from documents created after 2025 in the Legal department." This requires pre-filtering on metadata before ANN search. How well a vector database handles this (without degrading recall) is a major differentiator — Qdrant's payload filtering is particularly efficient.

Step 4: Retrieval Strategies

Dense retrieval (cosine similarity between embeddings) is not always the best approach. Production RAG systems often use hybrid retrieval.

Dense retrieval (semantic): Embed the query, find closest vectors by cosine similarity. Good for: semantic questions where the user doesn't know the exact terminology ("what's our refund policy?" matches "customers are entitled to a full refund within 30 days" even without keyword overlap). Bad for: exact matches, proper nouns, product codes.

Sparse retrieval (BM25): Classic keyword search — term frequency × inverse document frequency. Good for: exact matches, technical terminology, product IDs, error codes. Bad for: synonym handling, semantic questions.

Hybrid retrieval: Run both, merge results with Reciprocal Rank Fusion (RRF):

def reciprocal_rank_fusion(results_dense, results_sparse, k=60):
    scores = {}
    for rank, doc_id in enumerate(results_dense):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(results_sparse):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.keys(), key=lambda x: scores[x], reverse=True)

RRF is robust — it doesn't require tuning the relative weights of dense vs sparse scores, which are on different scales anyway.

Multi-query retrieval: Generate multiple paraphrases of the user's query, retrieve for each, union the results. Addresses the problem of query specificity — different phrasings retrieve different relevant chunks.

Step 5: Reranking

Vector search retrieves the top-K most similar chunks quickly (~10ms for millions of vectors). But ANN similarity is approximate and embedding cosine similarity doesn't always correlate perfectly with relevance.

Reranking runs a more accurate but slower model over the top-K retrieved chunks to reorder them by true relevance. You go from top-100 ANN results → reranker → top-5 most relevant chunks to feed the LLM.

Reranker models:

• Cohere Rerank 3: API-based, state of the art, $1/1M tokens
• BGE-reranker-large: Open source cross-encoder, excellent quality
• ms-marco-MiniLM-L-6-v2: Smaller, faster, decent quality for most use cases

Cross-encoder vs bi-encoder:

Your embedding model is a bi-encoder — it encodes the query and each document independently, then compares with cosine similarity. Fast (embeddings precomputed), but misses query-document interactions.

A reranker is a cross-encoder — it encodes the query and document together, allowing full cross-attention between query and document tokens. Much more accurate, but 10–100× slower. That's why you run it only over the top-K retrieval results, not the full corpus.

Step 6: Evaluation with RAGAS

"How do you measure if your RAG system is good?" is asked in every senior interview. RAGAS (Retrieval Augmented Generation Assessment) is the standard framework.

The four RAGAS metrics:

Faithfulness — does the generated answer contain only information that's present in the retrieved context? Measures hallucination. Scored by an LLM judge: decompose the answer into claims, check each claim against the context.

Answer Relevance — does the answer actually address the user's question? A faithful answer can still be off-topic. Scored by generating synthetic questions from the answer and checking if they match the original.

Context Precision — of the retrieved chunks, what fraction were actually relevant to answering the question? Measures retrieval precision. High context precision means less noise fed to the LLM.

Context Recall — were all the facts needed to answer the question present in the retrieved chunks? Measures retrieval recall. Low context recall means the system missed relevant information.

Production Concerns

Latency budget: A user expects an answer in < 2 seconds. Budget: query embedding (50ms) + ANN search (20ms) + reranker (150ms for top-20) + LLM generation (500–1500ms for 200 tokens) = ~800ms–1800ms. The bottleneck is usually LLM generation. Use streaming to show partial results while generation continues.

Caching: Cache at the query-embedding level (same query → same embedding → same retrieval results). Redis with a 1-hour TTL covers most repeated questions in enterprise deployments.

Index freshness: When source documents are updated, you need to re-embed the changed chunks and update the vector store. Track document versions. Use change data capture (CDC) on the document store to trigger incremental updates rather than full reindexing.

Chunk attribution: Always return source citations with answers. Users don't just want the answer — they want to verify it. Store document ID, page number, and section title as chunk metadata and include it in the response.

Context window management: After retrieval and reranking, your top-K chunks might still exceed the LLM's context limit. Three approaches: (1) Hard truncation — drop the lowest-ranked chunks until you're under the limit. Simple but loses information. (2) Chunk compression — pass each retrieved chunk through a smaller, cheaper LLM to extract only the sentences directly relevant to the query, then concatenate the compressed excerpts. (3) Lost in the middle — research (Liu et al., 2023) shows LLMs attend better to information at the start and end of the context, not the middle. Put your highest-ranked chunks first and last, not in the middle.

GraphRAG: Microsoft's 2024 approach builds a knowledge graph from documents (entities as nodes, relationships as edges), then retrieves subgraphs rather than chunks. Outperforms standard RAG on questions that require connecting information across multiple documents (e.g., "how is concept A related to concept B across our entire knowledge base?"). Significantly more expensive to build than a standard vector index — only worth it for corpora where relationship traversal is the primary query pattern.

8 RAG Interview Questions with Answers

Q1: When would you use BM25 over dense retrieval?

Use BM25 (or hybrid with BM25) when: users frequently search for exact terms (product codes, error codes, names), the document corpus uses specialized vocabulary that embedding models may not encode well (legal clauses, medical terminology), or you need interpretable retrieval (you can explain why a document was retrieved via keyword matching). Pure dense retrieval is better when users ask natural language questions and you want semantic matching across synonyms and paraphrases.

Q2: How do you handle very long documents?

Three approaches: (1) Hierarchical chunking — chunk into small pieces for retrieval, return the parent section to the LLM for context. (2) Summarization — pre-summarize each document and store summaries as additional retrievable chunks alongside the full text. (3) Use a long-context LLM (Gemini 1.5 Pro has 1M token context) for document-level QA when the whole document is relevant, falling back to chunked retrieval for corpus-wide search.

Q3: What is RAGAS and what does faithfulness measure?

RAGAS is an evaluation framework for RAG systems. Faithfulness measures whether each claim in the generated answer is supported by the retrieved context — it's the primary hallucination metric. A faithfulness score of 1.0 means every claim in the answer is grounded in the context. Scored by an LLM judge that decomposes the answer into atomic claims and verifies each against the context.

Q4: Design a RAG system for a 10M document legal knowledge base.

Key design decisions: use hierarchical chunking (clause-level retrieval, section-level context), Cohere multilingual embeddings if docs are in multiple languages, Qdrant for the vector store (good filtering performance for metadata like jurisdiction, document type, date), BGE-reranker for reranking top-50 → top-5, GPT-4o for generation with strict prompt instructions to cite sources. Evaluation: RAGAS faithfulness > 0.92 (legal hallucination is high-stakes), human review pipeline for low-confidence answers.

Q5: How do you update your vector store when source documents change?

Track each document with a content hash. On each ingestion run, compute the hash of the new content and compare with the stored hash. If changed: delete the old chunks (by document ID), re-chunk and re-embed the new content, insert new chunks with a fresh timestamp. If unchanged: skip. Never do full reindexing — it's expensive and causes downtime. Use a queue (Kafka, SQS) for document update events to decouple ingestion from the update trigger.

Q6: How do you handle a question that requires information from multiple chunks?

This is multi-hop retrieval. Standard single-pass RAG fails here — one retrieval step finds the first relevant chunk but misses the second. Solutions: (1) multi-query RAG — generate sub-questions from the original question, retrieve for each, combine chunks; (2) iterative retrieval — generate a partial answer from the first retrieval, use it to formulate a follow-up query for the missing information; (3) graph-based RAG — build a knowledge graph from documents where entities are nodes and relationships are edges, traverse the graph for multi-hop questions.

Q7: What is the cold start problem in RAG and how do you address it?

Cold start: the system has no documents yet, or very few. Retrieval finds nothing relevant and the LLM either hallucinates or says "I don't know." Address it: (1) Bootstrap with curated high-quality documents before launch; (2) Implement a fallback to web search when retrieval confidence is low (no chunks above a similarity threshold); (3) Build a "not found" detector — if max similarity score < 0.7, return "I couldn't find information about this in our knowledge base" rather than generating a potentially hallucinated answer.

Q8: How do you improve retrieval recall without degrading precision?

Recall and precision are often in tension — retrieving more chunks improves recall but adds noise (hurts precision). The best approach: increase K for initial retrieval (top-50 or top-100 instead of top-5) to maximize recall, then use reranking to cut it back to top-5 for the LLM. The reranker absorbs the precision problem without sacrificing recall at the retrieval stage. Also: query expansion (add synonyms or paraphrases to the query before embedding) improves recall without increasing K.

Practice on KnowEaseAI

RAG is one of those topics where reading gives you vocabulary but doesn't prepare you for the follow-up questions. The interviewers know RAG deeply — they've built it in production. They'll push beyond the basics into chunking trade-offs, reranker internals, RAGAS metrics, and production operational concerns.