Retrieval Augmented Generation

Grounding the model in your knowledge, not just its own

EMBA 8160 — AI for Leaders
J. Mack Robinson College of Business, Georgia State University

Outline

Why retrieval — the problem RAG solves, and the alternatives
Building the index — chunking, embeddings, vector databases
How similarity works — feature spaces, distance, nearest neighbors
Making retrieval better — fine-tuned embeddings and reranking
Measuring quality — precision, recall, and what to hold your team to
From retrieval to answers — grounded generation and the prompt

Why Retrieval?

The problem RAG solves — and the two alternatives

Three ways to give a model your knowledge

1. Put it in the prompt

Paste the documents in every time. Simple, immediate, always current.

Breaks when: the corpus is bigger than the context window, or you pay for the same 200 pages on every question.

2. Fine-tune the model

Continue training on your material so the knowledge is "in the weights."

Breaks when: facts change. Fine-tuning is excellent at teaching form — tone, format, task behaviour — and unreliable at teaching facts. It cannot cite, and it cannot forget on request.

3. Retrieve it on demand

Search your documents at question time and hand the model only the relevant passages.

Why it wins: current, citable, permission-aware, and updated by re-indexing a document rather than retraining a model.

These are not mutually exclusive. Production systems routinely retrieve and fine-tune — retrieval for the facts, fine-tuning for the behaviour.

This is the whole session on one slide — three ideas stacked.
Dense retrieval (top left): the query and the documents are both turned into vectors; you return the documents whose vectors sit closest to the query’s. This is “semantic” search — it matches meaning, so “how do I take time off?” finds the PTO policy even though neither word appears in it.
Reranking (top right): first-stage retrieval is fast and approximate. A reranker takes the shortlist and reorders it with a slower, much more accurate model. Note what happened in the figure — result #3 became #1. That reordering is often the single largest quality gain in the pipeline.
RAG (bottom): retrieval plus generation. The model writes the answer and cites which retrieved passages it used.
Ask the room: which of the three is the part most vendors demo? Usually the generation. The quality lives in the first two.

Building the Index

From a pile of documents to something you can search by meaning

Two generations of “text as numbers.”
Bag-of-words (left): count the words. “My cat is cute” becomes a vector of counts over the vocabulary. It works, and keyword search still uses a refined version of it — but it has no idea that cat and kitten are related. Every word is its own unrelated dimension.
Word2Vec (right): train a small network on the task “are these two words neighbors?” and the vectors it learns place related words near each other. The table is the intuition — cats and puppy both score high on “animal,” baby scores high on “newborn” and “human.”
The caveat on the slide matters: the dimensions do not actually correspond to human concepts like “animal” or “fruit.” That is a teaching fiction. The dimensions are learned and individually uninterpretable — which is exactly why you cannot audit an embedding by reading it.

Chunking is the most consequential, least discussed decision in the entire pipeline.
Options shown: one chunk per sentence, one per paragraph, or a sliding overlapping window. On the right, the same choice at the character and token level, with and without overlap.
Why overlap: a sentence that begins “It must be approved by the CFO” is useless if the previous sentence — the one that says what it is — landed in a different chunk. Overlap buys back some of that lost context.
The failure this causes in practice: a clause is split across a boundary, so no single chunk contains the full rule, so the retriever never surfaces it and the model answers from a partial one. The answer looks fluent and is wrong.
Structure beats arbitrary length. Chunking on real document structure — sections, clauses, table rows — outperforms “every 500 tokens” in nearly every serious system.

The full query path: documents are chunked and embedded into the vector database ahead of time; at question time the query goes through the same embedding model and is compared against the stored vectors; the most similar chunks come back.
The orange callout is the part leaders should hear: vector databases also store metadata, and you can filter on it. Department, document type, effective date, and — critically — access control.
That metadata filter is your permissions boundary. If retrieval ignores who is asking, a RAG assistant becomes an extremely efficient way to leak documents to people who should not see them. “Filter before you retrieve, not after” is the design rule.
Ask the room: who owns the access-control list for the index? In most failed pilots the answer is “nobody thought about it.”

The index is a set of decisions, not a database

Chunk size & overlapToo big retrieves noise, too small loses context. Drive it from document structure.

Embedding modelFixes the geometry of your search space. Changing it means re-embedding everything.

Metadata schemaWhat you can filter on later — source, date, owner, department, sensitivity.

Refresh cadenceHow stale answers are allowed to be, and what happens when a document is withdrawn.

Access controlEnforced at retrieval time, inherited from the source system — not re-invented in the chatbot.

What stays outDrafts, superseded policy, personal data. The index is a publication decision.

How Similarity Works

Feature spaces, distance metrics, and nearest neighbors

Cosine similarity measures the angle between two vectors and ignores their length.
Why that matters for text: a two-page memo and a two-paragraph summary of the same policy point in nearly the same direction but have very different magnitudes. Under Euclidean distance the length difference dominates; under cosine they are correctly judged similar.
The figure makes the reverse case too — in Case 2, cosine gets it wrong and Euclidean gets it right. Neither metric is universally correct.
In practice cosine similarity (or the closely related dot product on normalised vectors) is the default for text embeddings, for the two reasons on the slide: small magnitude differences and very high dimensionality.
If someone tells you their similarity score is 0.87, the right follow-up is “0.87 of what, and is that good?” Cosine scores are not probabilities and they are not comparable across embedding models.

Making Retrieval Better

Fine-tuned embeddings and reranking

The standard two-stage architecture. First stage: search millions of documents fast and imprecisely, returning maybe 50 candidates. Second stage: a reranker reads the query and each candidate together and scores relevance properly.
The distinction that matters: first-stage retrieval compares two vectors that were computed independently. A reranker is a cross-encoder — it sees the query and the document at the same time, so it can judge whether this passage actually answers this question. Much more accurate, far too slow to run over the whole corpus.
This is usually the cheapest large quality win available. If a pilot is “retrieving the right document but not at the top,” reranking is the first thing to try — no re-indexing required.
The cost is latency and per-query spend. It is a per-question cost, unlike embedding, which is a per-document cost.

Does It Work?

Measuring a retrieval system honestly

Precision at k asks: of the top $k$ results, how many were actually relevant? Compute it at each position, average over the positions where a relevant document appeared, and you have average precision for one query. Average that across a test suite of queries and you have mean average precision (MAP) — one number to compare two search systems.
The middle example is the important one: the system found the relevant document, but it placed two irrelevant ones above it, and the score fell from 1.0 to 0.3. Rank is penalised, not just presence. That is correct behaviour — an answer buried at position 8 is an answer nobody reads, and a model handed three irrelevant passages first is a model primed to answer from them.
Note the prerequisite hiding in the phrase “for query in the test suite”: someone had to decide which documents are relevant for each test query. That labelled set is the real deliverable. Without it you are not measuring, you are guessing.

What to hold the team to

Question	Metric	Failure it catches
Did we find the right passage at all?	Recall@k	Chunking and indexing problems — the answer is not in the shortlist
Did we put it near the top?	MAP / MRR	Ranking problems — fixable with reranking
Is the answer supported by what we retrieved?	Groundedness	The model inventing detail the sources do not contain
Do the citations point at the right place?	Citation accuracy	Plausible-looking references to the wrong document
Does it answer what was asked?	Answer relevance	Correct, well-sourced, off-topic responses
Can we afford it at volume?	Latency & cost / query	A pilot that cannot survive its own success

The first two measure retrieval; the next three measure generation. A system can fail either half, and the fixes are completely different — so insist on both being reported separately.

From Retrieval to Answers

Grounded generation and the prompt

The complete pipeline, and it is only two steps: retrieval, then grounded generation.
Look at the prompt on the right — that is the entire mechanism. The retrieved passages are pasted in as {context}, the user’s question as {question}, and the instruction says to answer using the relevant information provided above.
“Grounded” is doing real work in that sentence. The instruction constrains the model to the supplied passages instead of its own parameters. It reduces hallucination substantially; it does not eliminate it, because nothing stops a model from blending what it read with what it already believed.
The uncomfortable implication: if retrieval returns the wrong passages, a better language model makes things worse, not better. It will write a more convincing wrong answer. Garbage in, eloquent garbage out.
This is also where “I don’t know” has to be engineered. A system that cannot decline when retrieval comes back empty will fabricate, every time.

The anatomy of the instruction wrapped around the retrieved context: persona, instruction, context, format, audience, tone, and the data itself.
In a RAG system most of these are written once by the team and reused for every query. Only the data and the question change. So the prompt is a piece of software — it should be version-controlled, reviewed, and tested like any other.
Format is the one with operational consequences: if a downstream system parses the output, an unspecified format will eventually break it.
Audience is the one with reputational consequences: the same retrieved policy needs a different answer for an employee than for a regulator.
This is also where your guardrails live — instructions to cite sources, to decline when the context is insufficient, and to stay within scope.

Where RAG actually breaks

The passage was never retrievedSplit across a chunk boundary, or phrased so differently from the query that the embedding missed it. The model answers from second-best sources and sounds fine.

The index is staleThe policy changed on Monday; the index refreshed on Friday. Confidently current, factually obsolete.

Conflicting sourcesThree versions of the same document, all retrievable. Similarity has no notion of authority — it cannot tell you which one is in force.

Permission leakageRetrieval that ignores who is asking turns a chatbot into a very efficient document exfiltration tool.

Citations that do not check outThe answer is right, the reference points somewhere else. Erodes trust faster than being wrong.

No graceful "I don't know"When retrieval returns nothing useful, an unconstrained model fills the gap from memory — the single most damaging failure mode.

Six questions for your next design review

What is in the index, who decided, and what was deliberately left out?
How does retrieval know who is asking, and where is that enforced?
How fresh is the index, and what happens when a document is withdrawn?
Show me the golden set — and retrieval and generation scored separately.
What does the system do when it cannot find a good answer?
What is the cost and latency per query at ten times this volume?

Class Discussion

Discussion Question 1

Retrieve, Fine-tune, or Both?

Your firm wants an assistant that answers client questions about your service agreements. The agreements are renegotiated constantly, and the answers must sound like your firm’s house style.

Which knowledge goes in the index and which goes in the model — and how would you justify the split to a skeptical CFO?

Discussion Question 2

The Chunking Decision

A compliance manual contains numbered clauses, many with exceptions that appear two or three paragraphs after the rule itself. Your vendor proposes chunking every 500 tokens with no overlap.

What goes wrong, how would you detect it before go-live, and what would you propose instead?

Discussion Question 3

Similarity Is Not Authority

Your shared drive holds the 2023, 2024 and 2025 travel policies. All three are indexed. A user asks what the mileage reimbursement rate is, and the system returns the 2023 rate with a confident citation.

Whose failure is this, and what are three different places in the architecture where it could be fixed?

Discussion Question 4

Proving It Works

A vendor demos a RAG assistant on your documents. Every question they ask it is answered beautifully. They quote “92% accuracy.”

What do you ask them before signing, and what would you require in a pilot to generate evidence you would actually trust?

Discussion Question 5

Synthesis

You are sponsoring a RAG assistant over HR policies, benefits documents, and the employee handbook, available to all 4,000 employees. Legal is nervous, HR is enthusiastic, and IT has a vector database licence already.

Identify the three biggest risks, name one concrete mitigation for each, and state the single metric you would put in front of the steering committee each month. What would make you stop the project?

Takeaways

RAG is a search problem wearing an AI costumeThe language model is the last and smallest step. Quality is decided in chunking, embedding, and ranking.

"Similar" is a modelling choiceMetric, chunk size, and $k$ all change which evidence reaches the model — and therefore the answer.

A better model cannot fix bad retrievalIt only makes the wrong answer more persuasive.

Without a golden set, you have no evidenceYou have demos. Insist on retrieval and generation measured separately, and re-measured on every change.

Figure Credits

Jay Alammar & Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024) Semantic search and reranking overview; document → chunk → vector-database pipeline; bag-of-words and word2vec; dense retrieval; token, word, sentence and document embeddings; chunking strategies; nearest-neighbor retrieval; embedding fine-tuning; two-stage reranking; mean average precision; the RAG pipeline; and the elements of a prompt.

John D. Kelleher, Brian Mac Namee & Aoife D'Arcy, Fundamentals of Machine Learning for Predictive Data Analytics (MIT Press) The athlete draft dataset; feature space and similarity metrics; Euclidean and Manhattan distance; the nearest-neighbor algorithm; Voronoi tessellation and decision boundaries; k-nearest-neighbors; and feature pre-processing and normalisation.

Cosine similarity worked example Sasi Kumar, “Cosine Similarity vs Euclidean Distance,” Medium — credited on the figure itself.

Figures are reproduced for classroom instruction. Titles and captions were adapted for this deck; please verify each citation against your own copy before redistributing the slides.

◀ Slides