RAG after the demo

There is a particular moment in every RAG project.

You upload a handful of PDFs. You connect a vector database. You write about 50 lines of code. Then you ask:

“What is our production backup retention policy?”

The model finds the right paragraph. It cites the document. Everyone looks at each other.

“We just built an enterprise AI system.”

Maybe.

Now give it 50 million documents. Add SharePoint, Confluence, Git repositories, PDFs, spreadsheets, architecture diagrams, screenshots, recorded meetings, support tickets and databases. Let documents change continuously. Let employees have different permissions.

Delete a document. Change someone’s group membership. Ask a question containing an error code that appears exactly once in a 3,000-page manual.

Then ask:

“Why did you give me that answer?”

Welcome to production RAG.

The RAG diagram everyone draws

Most introductions start here:

It’s not wrong. It’s just the equivalent of drawing a web application as:

A production RAG system looks more like this:

Notice something?

The LLM is one box.

That’s the first important lesson.

Part I: RAG isn’t an LLM feature

It’s an information retrieval system with an LLM attached

I find this mental model much more useful:

RAG is a continuously synchronized, permission-aware information retrieval system whose final consumer happens to be an LLM.

Once you think about it that way, a bunch of architectural decisions become obvious. You suddenly care about indexing, freshness, deletion, permissions, ranking, metadata, lineage, latency, evaluation and observability.

The LLM is still important. But it isn’t where most of the hard engineering happens.

Part II: the first lie is “your data”

When somebody says:

“We have 10 million documents.”

my next question is:

“What exactly is a document?”

Because a single document.pdf might actually contain three different kinds of knowledge, each needing its own parser:

And then there’s everything that was never a PDF to begin with:

The first architectural decision isn’t “which vector database should we use?”

It’s “what is the actual shape of our knowledge?”

Part III: the ingestion pipeline is half the product

The query path gets all the attention. The ingestion path quietly determines whether the query path has anything useful to retrieve.

Think of it as a factory:

Every box can introduce information loss. And information lost here is usually impossible for the LLM to recover later.

Part IV: chunking is not “split every 500 tokens”

This is probably the most abused piece of advice in RAG tutorials:

“Use 500-token chunks with 50-token overlap.”

Fine. But why?

Consider a technical document with two chapters, Authentication and Authorization. A blind splitter doesn’t know chapters exist. It counts tokens, and wherever the count runs out, that’s a boundary:

Congratulations. We’ve created a chunk containing two different concepts simply because they happened to be close together.

The document had structure. Our index destroyed it.

Strategy 1: fixed-size

The baseline:

Good for simple documents, prototypes and predictable workloads. Bad for tables, code, technical manuals, and anything with meaningful hierarchy.

Use it as your baseline. Don’t assume it is your final answer.

Strategy 2: structure-aware

Instead of asking “where are the next 500 tokens?”, ask “what is the natural unit of meaning?”

Walk the document’s own hierarchy, and let every chunk keep its ancestry:

In the index, that chunk might look like this:

{
  "document": "database-guide.pdf",
  "chapter": "Security",
  "section": "Credential Rotation",
  "content": "Rotate every 90 days...",
  "parent": "Security"
}

That tiny bit of metadata makes a surprisingly large difference. The breadcrumb can be prepended to the text before embedding, used as a filter, and shown back to the user as a citation.

Strategy 3: parent / child retrieval

This is one of the patterns I like most for long technical documents.

Embed the children. Retrieve the children. Give the model the parent.

Why? Because the best unit for finding information isn’t always the best unit for understanding it. The child gives you precision. The parent gives you context.

Strategy 4: semantic chunking

Sometimes the document has no useful explicit structure. Then you can detect changes in meaning instead: embed each paragraph, compare it to the next one, and cut where similarity drops.

More expensive. Often more meaningful. And that’s the general pattern you’ll see throughout production RAG:

Better retrieval usually costs more computation somewhere.

Tables are not paragraphs

Consider:

Region	Revenue	Growth
US	$20M	12%
Europe	$14M	8%
APAC	$11M	23%

A naïve parser can turn this into:

US $20M 12%
Europe $14M
8% APAC $11M
23%

We’ve taken structured data and made it ambiguous. Whose growth is 8%?

A table-aware pipeline keeps the header attached to every row:

TABLE: Regional Revenue
COLUMNS: Region | Revenue | Growth
ROW: APAC | $11M | 23%

And it still preserves the original table as an artifact. That matters: the model reasons over the normalized representation, while the user can still trace the answer back to the original source.

Images aren’t “text with pixels”

Take an architecture diagram and run it through OCR:

OCR got every word, and missed the most important thing: the relationships. Nothing in that output tells you that both services publish to Kafka.

A multimodal pipeline keeps several representations of the same image, so it can be found in several different ways:

Now the image can participate in retrieval through extracted text, a generated description, visual similarity and the document around it. This is where multimodal RAG starts getting much more interesting than “PDF Q&A”.

Video is just another knowledge source

Except it has a clock.

A two-hour engineering presentation can be cut into segments, each one carrying a transcript, a summary of what was on screen, and its own embedding:

{
  "start": "42:17",
  "end": "44:03",
  "speaker": "CTO",
  "transcript": "...",
  "visual_summary": "...",
  "embedding": "..."
}

Now the answer can say:

“The CTO discussed the Kafka migration at 42:17.”

That’s fundamentally different from “somewhere in the two-hour recording…”

Part V: the index has to stay alive

Here’s a subtle problem. Your source system says Policy v17. Your search index says Policy v14. Your LLM gives a beautifully grounded answer.

It’s still wrong.

RAG introduces a new kind of distributed-systems problem:

eventual consistency between the source of truth and the retrieval layer.

The synchronization loop

A production ingestion system should think in events, and treat each kind of change as its own operation:

Not “we periodically re-index everything.” That approach becomes painful surprisingly quickly: it’s slow, it’s expensive, and between runs your index is lying to you.

Versioning matters

Every indexed object should carry lineage:

{
  "document_id": "doc-847",
  "document_version": 17,
  "content_hash": "a83f...",
  "source_updated_at": "2026-09-22T14:31:00Z",
  "indexed_at": "2026-09-22T14:34:12Z"
}

Now you can answer “which version generated this answer?” That’s a much more useful question than “which vector was closest?”, and the gap between source_updated_at and indexed_at is your freshness SLO, measured per document.

And please don’t forget DELETE

This is one of those bugs that looks ridiculous when you draw it. Step through it:

1 / 3 Before: the document is indexed and cited ← →

Deletion has to propagate. And if you run any caches in front of retrieval, it has to propagate through those too.

Part VI: security changes the entire retrieval architecture

Imagine Alice asks:

“What is the executive compensation plan?”

Your search engine finds the perfect document. Except Alice isn’t allowed to see it.

There are two possible architectures:

The left one isn’t security. It’s a prompt.

The LLM should decide how to answer. It should not decide what this user is allowed to see.

Security metadata belongs in the index

For example:

{
  "document_id": "doc-77",
  "tenant_id": "acme",
  "classification": "internal",
  "allowed_users": [],
  "allowed_groups": ["platform-engineering", "sre"]
}

Now retrieval can enforce a filter before ranking ever happens:

tenant_id = current_tenant
AND (
  user IN allowed_users
  OR any(user_groups) IN allowed_groups
)

The important thing is that this is deterministic infrastructure, not an LLM instruction. It also means a group-membership change is an indexing event, which is why it had its own lane in the sync loop above.

Tenant isolation

For SaaS applications, add another boundary. You have roughly three options, trading cost for blast radius:

The right choice depends on scale, cost, regulatory requirements and threat model. But the principle is simple:

A prompt is not an authorization boundary.

Part VII: vector search isn’t enough

Semantic retrieval is fantastic for “how do I recover a failed database?”

But try “what does error PX-429 mean?”

Now exact matching matters. The corpus contains PX-429, PX-430 and PX-431. The embedding model understands the general topic. The keyword index knows that PX-429 is exactly PX-429.

This is hybrid retrieval. It combines two different ways of asking “is this relevant?”, and fuses the two ranked lists. Reciprocal rank fusion is the usual default because it only needs ranks, not comparable scores.

Part VIII: reranking, the second chance

Imagine retrieving 50 candidates. The embedding model scores them 0.84, 0.82, 0.81, 0.80…

Those numbers are useful. But they aren’t the final word on relevance. An embedding model compares a query vector with a document vector that was computed long before the query existed. A reranker (usually a cross-encoder) reads the query and the candidate together, and it often disagrees:

This lets you use cheap retrieval to build a candidate set, and a more expensive model to make the final relevance decision:

The trick is balancing recall ↔ precision ↔ latency ↔ cost.

Part IX: complex questions need complex retrieval

Consider this question:

“Compare the 2025 and 2026 disaster recovery policies, identify changes to RTO requirements, and tell me which applications are affected.”

That’s not one retrieval problem. It’s several:

The system retrieves each piece independently, then combines the evidence. This is where RAG starts crossing into agentic retrieval.

But don’t make everything an agent

This is another easy trap. Not every question needs this:

If the question is “what is our password expiration policy?”, a straightforward retrieval pipeline is enough. If the question is “which production services violate our new password policy?”, now you need policy documents, a service inventory and probably some tools that can inspect configuration.

A useful architecture therefore puts a router in front:

The router decides what kind of problem it is, and pays for an agent only when the question needs one.

Part X: context is a scarce resource

Suppose retrieval gives you 50 chunks. Do you send all 50 to the model?

Probably not. A disciplined pipeline narrows, then widens, then fits:

This matters because irrelevant context isn’t free. It costs tokens, latency, money and attention. Too much context can make an answer worse rather than better.

Part XI: the cloud versions

If I wanted an enterprise RAG system running without owning every component, I’d probably start with managed infrastructure. All three major clouds now offer the same shape under different names:

AWS. Bedrock Knowledge Bases take care of a lot of the ingestion, vector storage, metadata filtering and reranking. EventBridge and SQS are the natural way to feed changes in from S3 and SaaS sources.

Azure. Azure AI Search does hybrid (vector plus keyword) retrieval natively and adds a semantic ranker on top, with Azure OpenAI for generation. This is especially convenient when your data and identity already live in Microsoft’s ecosystem: SharePoint, Entra ID, Blob.

Google Cloud. Vertex AI RAG Engine ingests from Cloud Storage and Drive into a RAG corpus, with a ranking API and Gemini on top.

Different APIs. Same fundamental architecture. That’s worth remembering, because cloud services change faster than RAG design principles. Managed services are great when the goal is to ship the product, not build the search engine.

Part XII: when I would build the whole thing myself

Managed RAG isn’t always the right answer. I would seriously consider a custom stack if I needed:

• custom ranking models or specialized embeddings
• unusual multimodal processing
• strict on-prem or data-residency requirements
• extreme scale
• an authorization model the managed service can’t express
• deep control over retrieval

A custom platform could look like this:

The advantage? Control.

The disadvantage? Now you own all of it.

Part XIII: code is data too

A surprising number of RAG implementations treat source code like prose. Don’t.

A useful code index knows that refreshToken is a method, inside a class, inside a module, inside a repository at a specific commit:

{
  "repository": "payments",
  "branch": "main",
  "commit": "a82c...",
  "language": "java",
  "file": "TokenService.java",
  "symbol": "refreshToken"
}

Now the query “where is the refresh token validated?” can retrieve a function. Not an arbitrary 500-token window that happens to contain half of one.

For code, AST-aware chunking is a much better mental model than token-based chunking.

Part XIV: RAG needs tests

A demo asks “does it work?” A production system asks “how often does it work, and why does it fail?”

Build an evaluation dataset of questions with known answers and known sources:

Question:          What is the production backup retention period?
Expected source:   backup-policy-v17.pdf
Expected section:  Retention
Expected answer:   35 days

Then test retrieval separately from generation:

Retrieval metrics, on one example

• Recall@K: did the relevant documents appear in the top K?
• MRR (mean reciprocal rank): how high was the first relevant result?
• nDCG: how well were all the relevant results ranked?

This separation is incredibly useful. Otherwise someone says “the answer was wrong,” and everyone starts tweaking the prompt.

Maybe the prompt wasn’t the problem. Maybe the correct document never entered the context.

Part XV: observability is not optional

For every query, I’d want a record like this:

{
  "request_id": "req-9812",
  "query": "How long are backups retained?",
  "retrieval": {
    "vector_candidates": 40,
    "keyword_candidates": 40,
    "after_fusion": 52,
    "after_reranking": 8
  },
  "latency_ms": {
    "embedding": 32,
    "retrieval": 71,
    "reranking": 184,
    "generation": 912
  },
  "sources": ["backup-policy-v17"]
}

Which, drawn as a trace, looks like this:

Now when someone asks “why was this response slow?”, you can answer. “Why did it retrieve the wrong policy?” You can answer that too.

Without this telemetry, debugging RAG is basically reading tea leaves.

Part XVI: six failure modes you’ll eventually meet

Each one lives at a different point in the pipeline:

01: the answer exists, but retrieval missed it

The correct document is in the index; the retriever just never returned it. Usual suspects: poor chunking, the wrong embedding model for your domain, an exact identifier the embedding blurred, weak query formulation, or a candidate pool that’s too small.

02: retrieval found it, but ranked it badly

The correct chunk came back as candidate #37, and you only kept ten. This is where reranking earns its keep.

03: the answer is beautifully grounded in stale data

Source says v17, index says v14, and the LLM says “according to the documentation…”. Technically grounded. Operationally wrong.

04: the user sees something they shouldn’t

This is not a retrieval-quality problem. It’s a security incident. Treat it accordingly.

05: PDFs work, diagrams don’t

Your ingestion system assumes knowledge is text. It isn’t.

06: accuracy is excellent and the bill is terrifying

You retrieve 1,000, rerank 1,000, and send 100 chunks to the LLM. Congratulations: your architecture works. Your CFO would like a word.

Part XVII: the architecture I’d actually start with

If I were building a large enterprise RAG platform today, I’d start with this:

It looks complicated. That’s because the problem is complicated.

The part I think gets missed most often

People often ask “which vector database should I use?”

That’s a reasonable question. It just isn’t the first question I’d ask. I’d ask:

1. What is your source of truth?
2. How frequently does it change?
3. How do deletions propagate?
4. What does a meaningful chunk look like?
5. What does authorization look like?
6. How do you represent images and tables?
7. Which questions are semantic, and which require exact matching?
8. How will you rerank?
9. How will you evaluate retrieval independently from generation?
10. How will you know when the index is stale?

Those decisions will have a much bigger impact on your system than whether your vector store is product A or product B.

The final mental model

The first RAG demo is four boxes. A production system is thirteen, and the four you started with are still in there:

And suddenly the phrase “RAG pipeline” makes a lot more sense.

It isn’t a vector database. It isn’t a prompt. It isn’t even primarily an LLM problem.

It’s a knowledge retrieval system. The LLM is the last mile, and in most real-world systems the hardest engineering happens long before the LLM ever sees the question.

One last thought

The interesting part of RAG isn’t getting a model to answer a question about ten documents. We already know how to do that.

The interesting problem is this:

How do you build a system that can find the right piece of knowledge out of 50 million constantly changing, permission-sensitive, multimodal artifacts, and prove to you why that piece of knowledge was used?

Once you frame the problem that way, RAG stops looking like an AI demo. It starts looking like what it really is: a search engine for the age of generative AI.

And that’s a much more interesting engineering problem.

An implementation path

If you’re building this incrementally, I’d go in roughly this order:

1. Basic RAG. Chunk, embed, retrieve, generate. Build the eval set now, not later.
2. Better retrieval. Hybrid search and reranking.
3. Production data. Event-driven sync, versioning, deletion.
4. Enterprise. ACLs in the index, tenant isolation, audit.
5. Multimodal. Tables, images, video, code.
6. Agentic RAG. Decomposition, tools and routing.

Start simple. Measure everything. Add complexity only when the retrieval problem actually demands it.