The model was trained on general internet data and knows nothing about your company: not your knowledge base, not your docs, not fresh news after its training. Ask about an internal policy and it will either admit it doesn't know or invent a plausible answer. RAG solves this: it finds the relevant chunks of your data and mixes them into the request so the model answers from facts, not from memory.

Why you can't just "load everything"

The naive first thought is to paste the whole document corpus into the prompt. It won't work: the context window is finite, gigabytes won't fit, and excess text confuses the model and costs money. So for each question you must put in context only the relevant chunks. The task reduces to search: how do you quickly fetch, from a large corpus, exactly what's relevant to the question?

Embeddings: text as a vector of meaning

Ordinary keyword search does this poorly: the user asks "how do I return an item," while the document says "processing a purchase refund" — different words, same meaning. This is where embeddings help.

An embedding turns text into a vector (a long list of numbers) that encodes its meaning. The key property: texts close in meaning get close vectors, even if the words differ. "Return an item" and "processing a refund" end up near each other in this space of meaning, while "borscht recipe" is far away.

Once meaning becomes a vector, finding similar text becomes geometry. All document chunks are converted to embeddings in advance and stored. A question comes in — it's converted to a vector too, and you look for the nearest chunk vectors. This is semantic search: finding by closeness of meaning, not by matching words. These vectors are stored and searched in a vector database — its own article.

How RAG comes together

RAG (retrieval-augmented generation) puts it all together in two stages.

Preparation (in advance): documents are cut into chunks of a reasonable size, each chunk is converted to an embedding and stored in a vector database. This is indexing the knowledge base.

Answering (per question):

  1. the user's question is turned into an embedding;
  2. semantic search fetches the few most relevant chunks;
  3. these chunks are mixed into the prompt together with the question and the instruction "answer only from these fragments";
  4. the model formulates an answer grounded in the facts, not its memory.

The result — answers from your current data, with source references, and noticeably fewer made-up claims: the model was given facts and asked not to go beyond them.

Pitfalls to know

  • chunk quality decides everything. Cut documents badly (too coarse or too fine) and search fetches the irrelevant, ruining the answer. Chunking is not a detail.
  • RAG doesn't cancel verification. Even with facts in context the model can distort them when paraphrasing; you still verify the answer and show sources to the user.
  • index freshness. Data changes — the index must be updated, or the model answers from stale chunks.

Where it applies

RAG is the most common way to build a useful LLM feature on your own data: a chat over docs and a knowledge base, support that answers from policies, search over internal data in "plain language," a codebase assistant. Almost any "ask my system in natural language" feature is RAG under the hood.

In short

  • The model doesn't know your data; RAG finds relevant chunks and puts them into the prompt so it answers from facts.
  • An embedding turns text into a vector of meaning: texts close in meaning have close vectors, even with different words.
  • Semantic search fetches the right chunks by vector closeness, not word matches; they're stored in a vector database.
  • RAG = index the corpus in advance (chunks → embeddings), then per question find the relevant and mix it into the prompt. This sharply cuts made-up content.
  • Pitfalls: chunking quality, index updates, and the same answer verification.

Next — vector databases: where and how embeddings are stored and nearest vectors are searched at scale.