The two panels are the session in one image. On the left, the architecture as it is drawn in the design review. On the right, the same architecture under load, with every joint lit up orange.
Nothing on the right is a model failure. The model is one box out of seven, and most of the fires are upstream of it.
This session is built on two papers: a Microsoft Research Asia survey on how to make LLMs use external data wisely, which gives us a way to classify what is being asked; and a Deakin University field study of RAG systems in production, which gives us a catalogue of where they break.
Audience note: the figures in this deck were generated with NotebookLM from those two papers. Treat them as well-drawn summaries, not as the papers themselves — which is itself a lesson about grounded generation.

Where this session picks up

EMBA 8160 — AI for Leaders. Prerequisite: the RAG fundamentals session.

You already know

Documents are chunked, embedded, and retrieved by similarity; a reranker reorders the shortlist; the model answers from what it was handed.

That pipeline answers “what is the mileage rate?” very well.

Today's two questions

1. What happens when the question is harder than lookup — when it needs aggregation, a rulebook, or judgement learned from experience?

2. When the system gives a bad answer, which of the seven things that can go wrong actually went wrong?

One Architecture, Every Question

The assumption that quietly breaks most deployments

Stratifying the Query

Four levels of intent — and four different architectures

Level 1 — explicit facts. The answer exists in plain text in one place. “Where will the 2024 Summer Olympics be held?”
Three retrieval strategies: sparse (BM25/TF-IDF, keyword matching — the classic search engine), dense (embeddings, semantic meaning), and hybrid, which combines them.
Hybrid is the practical default, and worth knowing as a leader for one reason: dense retrieval alone is bad at exact identifiers. Part numbers, policy codes, account numbers, surnames — these are keyword problems, and a purely semantic system will miss them. If your corpus is full of codes, ask specifically whether retrieval is hybrid.
The processing note matters for real corpora: charts and tables need table-to-text conversion or multi-modal embeddings, or they are effectively invisible to retrieval. Most enterprise documents are full of tables.

Level 2 — implicit facts. The answer exists, but scattered: it has to be assembled. “What is the majority party in the country where Canberra is located?” requires linking Canberra to Australia and then Australia to its current government.
One retrieval pass cannot do this, because the second hop depends on the result of the first.
Three engineering answers: iterative RAG (retrieve, reason, retrieve again); graph or tree RAG, which organises chunks into structures — RAPTOR builds a recursive summary tree, GraphRAG builds a graph and detects communities in it; and SQL integration, where a natural-language question is translated into a database query.
The business version of this level is the one everyone actually wants: “how many of our contracts include this clause?” That is an aggregation over documents, not a lookup in one. Standard RAG will answer it confidently from the three chunks it happened to retrieve, and the number will be wrong.
The cost note: every extra hop is another model call. Level 2 systems are multiples more expensive and slower per question than Level 1.

Level 3 — interpretable rationales. The rule exists and is written down — in a handbook, a guideline, a documented workflow — and the job is to apply it correctly. “How should a patient with acute chest pain be managed?” under a specific clinical guidance.
The figure models this as a state machine: check guidelines, apply the rule set, produce a compliant answer. That framing is the figure’s own metaphor, but it is a useful one, because it says the system’s job is procedure-following, not recall.
Techniques: prompt tuning to enforce rule adherence, chain-of-thought to mimic the guideline’s logic, and agent workflows with planning and memory.
This is where most regulated-industry use cases actually sit, and it is the level where RAG stops being a search problem and starts being a workflow problem. Note the governance consequence: at this level the system is effectively giving procedural advice, and someone has to own whether that advice is correct.

Level 4 — hidden rationales. No one wrote the rule down. It has to be inferred from patterns in what the organisation did before. Debugging a server crash from past incident logs; pricing a deal the way this firm prices deals.
The key line is the last one in the definition: logical congruence matters more than semantic similarity. The most useful past case is not the one that reads most like the current one — it is the one that had the same underlying structure. Similarity search actively misleads here.
Three approaches: offline learning (mine the corpus for principles in advance), in-context learning (retrieve past examples rather than documents), and fine-tuning, when the reasoning style is too complex to fit in a prompt.
This is the level where “we have twenty years of institutional knowledge in this drive” meets reality. The knowledge is in there, but not in a form retrieval can reach. Expect a data-labelling project, not a search project.

The four levels, in your language

Level	A question from your business	What it needs	Relative cost
L1 · Explicit facts	“What is our parental leave entitlement?”	Hybrid retrieval over a curated index	Low
L2 · Implicit facts	“How many active contracts include a change-of-control clause?”	Iterative retrieval, graph/tree structure, or SQL	Medium — multiple model calls per question
L3 · Interpretable rationales	“Does this transaction require enhanced due diligence?”	Agent workflow that follows a documented procedure	High — plus a named owner for the procedure
L4 · Hidden rationales	“How would we normally price this deal?”	Examples mined from history; possibly fine-tuning	Highest — a data project before a build project

The distribution of your users' real questions across these four rows is the scope of the project. Sorting fifty of them costs an afternoon and routinely changes the plan.

Where Pipelines Break

Seven failure points, observed in the field

The pipeline laid out end to end — data source, indexer, retriever, context window, generator, output — with the seven failure points pinned to where they occur. One at the indexer, two at the retriever, four at the generator.
The study behind this ran a substantial experiment — on the order of fifteen thousand documents and a thousand questions — and reports that automated evaluation was more pessimistic than human raters. Useful when someone waves an automated score at you: it is a signal, not a verdict, and it can be wrong in both directions.
The key insight is the sentence bottom right, and it is the one worth arguing about: robustness cannot be designed in at the start; it evolves through validation. You cannot specify your way to a reliable RAG system. You discover the failures by running it against real questions and fixing what breaks.
Which has a budget implication: if the plan has no post-launch iteration budget, the plan is wrong.

The three failures that happen before the model ever sees the prompt.
FP1 — Missing content. The answer simply is not in the index. The risk is not the miss; it is that the system hallucinates instead of saying “I don’t know.”
FP2 — Missed top rank. The right passage is in there, ranked eighth, and you retrieve the top three. Exactly the failure a reranker exists to fix, and exactly what recall@k versus precision@k measures. Note the fix can be as blunt as raising $k$ — at the cost of more noise.
FP3 — Not in context. Retrieved, then dropped during consolidation before reaching the model. Mitigation is better metadata and smarter consolidation.
All three look identical to the user: the system did not know something it should have known. They have completely different fixes, which is why a system that cannot show you its retrieved passages cannot be debugged.

The four failures that happen after the right material has been retrieved — so no amount of retrieval tuning will help.
FP4 — Not extracted. The answer was in the context and the model did not use it. Causes: noise, contradictory passages, or “lost in the middle” — the well-documented tendency of models to attend less to material buried in the centre of a long context. This is the direct argument against “just retrieve twenty chunks to be safe.”
FP5 — Wrong format. Asked for a table, got prose. Breaks anything downstream that parses the output.
FP6 — Incorrect specificity. Too vague or too precise. Often caused by the user’s question being general — worth saying out loud, because it means some failures are fixed by interface design, not model work.
FP7 — Incomplete. Asked for A, B and C; got A and B. The most dangerous of the four, because the answer looks complete and nothing signals the gap.
Ask the room which of these seven their own teams have reported without naming it. FP6 and FP7 are usually in the pile labelled “the AI isn’t very good.”

From complaint to diagnosis

What the user says	Likely failure point	Where to look first
“It made that up.”	FP1	Is the document even in the index? Does the system ever say “I don't know”?
“It missed the obvious document.”	FP2	Retrieval depth $k$, and whether a reranker exists at all
“It found it yesterday, not today.”	FP3	Consolidation and filtering between retrieval and prompt
“The answer was right there in the source.”	FP4	Too many chunks, contradictory sources, long-context dilution
“It won't give me a table.”	FP5	The prompt's format instruction, and output validation
“It's too vague to use.”	FP6	The question as much as the system — interface and query rewriting
“It only answered half.”	FP7	Multi-part questions; decomposition before answering

The Design Decisions

Chunking, and the choice between context and weights

The trade-off drawn as a balance, and now tied to specific failure points: small chunks give precise retrieval but strip context, which produces FP4 — the model has the fragment and cannot use it. Large chunks preserve meaning but add noise and cost, which produces FP3 — good material crowded out during consolidation.
Strategies: heuristic chunking (fixed size or punctuation — simple and risky) versus semantic chunking on meaning-based boundaries, which performs better.
The embedding note is a genuinely useful procurement point: open-source embedding models often perform comparably to closed commercial ones on short text. Embeddings are frequently the cheapest place to avoid vendor lock-in, and the one teams are least likely to question.
There is no correct chunk size in the abstract. It is found by testing against your own golden set — which is the validation loop from the previous section, again.

Three ways to get organisational knowledge into a model: put it in the context (RAG), train a small proxy model to guide retrieval, or fine-tune the base model so the knowledge lives in the weights.
Each maps to a level: context for L1 and L2 facts; a small proxy where a specific task needs to be fast and cheap; fine-tuning for L4, where the reasoning pattern itself is the thing to learn.
The middle path is the one most people have never heard of and is worth flagging: a small trained model that guides retrieval can cut latency and cost substantially without touching the large model.
The strategic note in orange is the single most actionable line in this deck: do not fine-tune for explicit facts. It is expensive, it goes stale, it cannot cite, and it hallucinates around the edges. Facts belong in the index.

The decision matrix, and the sentence at the bottom is the one to remember: RAG is for facts, fine-tuning is for form.
Top left: facts plus retrieval — high suitability. Bottom left: facts plus fine-tuning — avoid, because of hallucination and catastrophic forgetting. Top middle: reasoning patterns and style plus fine-tuning — recommended.
Then the box nobody expects on a slide about accuracy: fine-tuning carries a security risk. Research indicates that fine-tuning — including lightweight methods such as LoRA — can erode or reverse a model’s safety training, and that fine-tuned models can be more vulnerable to adversarial attack.
That moves the decision out of engineering. “We’ll fine-tune it on our data” is a statement with a security review attached, and it is very rarely treated that way. If your firm fine-tunes, the safety evaluation has to be re-run afterwards — the base model’s assurances no longer transfer.
Tie back: this is the same governance principle as the Responsible AI session. Every capability decision is also a risk decision.

Operating It

What robustness looks like after launch

Three operational practices.
Continuous calibration. A RAG system receives inputs nobody anticipated. Latency, cost and accuracy can only really be validated in operation — which is the argument for running a small, permanent evaluation harness in production rather than a one-off acceptance test.
Semantic caching. Pre-populate a cache with known FAQs. It cuts cost and latency, and it directly mitigates FP1: for questions you know are common, you can guarantee a correct answer rather than hoping retrieval finds it.
The routing pipeline. This is the punchline of the whole session. Real applications get a mix of levels, so route each question to the architecture that fits it: L1 and L2 to vector search, L3 to agent workflows, L4 to fine-tuned models.
That is the resolution of the one-size-fits-all trap from the opening: not one architecture, but a router in front of several. Worth stating plainly — the deliverable is a portfolio of retrieval strategies with a classifier in front, not a chatbot.

The same checklist, as governance gates

Gate 1 — Scope

Fifty real user questions, sorted into the four levels. Sponsor signs off on the distribution, not on a demo.

Fails if: nobody can produce fifty real questions.

Gate 2 — Build

Chunking tested, not assumed. Metadata and citation identifiers carried end to end. Access control enforced at retrieval.

Fails if: citations cannot be traced to a source document.

Gate 3 — Validate

Failures reported by failure point, split retrieval versus generation. Post-launch iteration budget committed.

Fails if: quality is reported as a single accuracy number.

Three gates, three refusals. Each one costs days at the start and saves quarters later.

Class Discussion

Discussion Question 1

Sorting the Queries

Your operations team asks for “a chatbot over our SOPs.” Their example questions include: “What is the escalation threshold?”, “How many incidents last quarter breached it?”, and “Should we escalate this one?”

Sort these into levels. What does the mix tell you about scope, cost, and who from the business has to be on the project?

Discussion Question 2

Diagnosing the Complaint

Three months after launch, your HR assistant gets these complaints in the same week: “it invented a policy,” “it gave me half the answer,” and “the right document was in there but it used an old one.”

Name the likely failure point for each. What evidence would you ask for to confirm the diagnosis — and which of the three worries you most?

Discussion Question 3

The Fine-Tuning Proposal

A vendor proposes fine-tuning an open model on ten years of your claims files, arguing it will “learn how we assess claims” and remove the need for a retrieval system entirely.

Which part of that proposal is defensible and which is not? What would you require before approving it — and what would you still keep in an index?

Discussion Question 4

Buying Observability

Two vendors bid. Vendor A reports 94% accuracy on your documents. Vendor B reports 89%, broken out by retrieval and generation, with the failure points named and a dashboard showing which questions failed.

Which do you pick, and how do you defend that choice to a procurement committee that sees 94 > 89?

Discussion Question 5

Synthesis

You are the executive sponsor of a knowledge assistant for a 2,000-person professional services firm. Users will ask for policy facts, cross-document counts, procedural guidance, and “how do we usually handle this?” — all four levels, on day one.

Design the phased plan. Which level do you ship first and why? What do you deliberately refuse to build in phase one, and how do you explain that refusal to the partners who asked for all four?

Takeaways

Classify the question before choosing the architectureFour levels, four designs. The mix of levels in your users' real questions is the scope of the project.

“It gave a bad answer” is seven different problemsThree happen before the model sees the prompt. Insist that failures are named, not averaged.

RAG is for facts, fine-tuning is for formAnd fine-tuning is a security decision as well as a quality one.

Robustness is discovered, not specifiedBudget for the iteration after launch, and build for observability rather than for a single accuracy number.

Sources & Figure Credits

Query stratification — Levels 1–4 Siyun Zhao et al., Microsoft Research Asia, Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make Your LLMs Use External Data More Wisely (2024).

The seven failure points Scott Barnett et al., Applied Artificial Intelligence Institute, Deakin University, Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024).

About these figures The blueprint illustrations in this deck were generated with NotebookLM from the two papers above — they are a synthesis, not an excerpt. Wording, emphasis and framing (the “leaking house,” the state machine, the balance scale) are the tool's, not the authors'. Read the papers before citing any specific figure or number.

◀ Slides