Using an AI agent to write code is one thing. Embedding AI into the product itself — chat support, smart search, autocomplete, document parsing — is another. That's no longer a "developer's helper" but a part of the application that you build. Let's break down the building blocks of such a feature and how it differs from ordinary code.

The four parts of any LLM feature

Almost any feature built on a language model is made of the same pieces:

  1. Prompt — the instruction to the model: what to do and in what form to answer.
  2. Context — the data you mix into the request: the user's message, document chunks, conversation history.
  3. Model call — sending the prompt with context and getting a response.
  4. Response handling — parsing what the model returned and acting on it: show the user, store, call a function.

All of LLM engineering is about assembling these four parts well. The model is a ready-made "engine"; your job is to feed it good input and carefully take its output.

The prompt: an instruction, not magic

A prompt usually splits into two parts: the system part (who the model is and by what rules it works — "you are a support assistant, answer briefly and only from the provided context") and the user part (the concrete request). A good prompt is specific: it sets a role, the answer format, boundaries ("don't make things up if there's no data"). A bad one is vague, and the model fills the gaps with guesses.

The key insight: a prompt is a specification, not a spell. The clearer you describe the task, the more predictable the answer. A big part of the work on an LLM feature is iterating on the prompt against real examples.

Context: the model only knows what you give it

The model doesn't know your data — not your order database, not fresh documents, not even the previous message unless you pass it. Everything it needs to answer, you put into the request context. Hence two limits:

  • the context window is finite — only a limited amount fits (see context), so you can't just "load the whole database"; you select the relevant chunks;
  • excess context hurts — junk in the request confuses the model and raises the cost (see tokens and cost).

How to pick the right chunks from a large corpus is a big topic of its own, solved by RAG and embeddings.

Structured output

If the model's answer goes straight to the user, plain text is enough. But if the feature is embedded in a program (the model classifies a ticket, extracts fields from an email, decides what to do next), you need a machine-readable answer — usually JSON to a given schema. Modern models can return structured output from a schema description; this turns the model from a "conversationalist" into a component whose result you can reliably parse and use in code. The mechanics of calling functions by schema are covered in tool calling.

How it differs from ordinary code

The main thing to get used to: the model's output is non-deterministic. The same input can give different answers, and the model sometimes errs confidently — outputs plausible falsehoods. An ordinary function always gives one output for one input; an LLM does not.

Hence a different approach to quality:

  • not "pass/fail," but evaluation. You check an LLM feature on a set of examples, measuring in what fraction of cases the answer is acceptable — rather than expecting an exact match.
  • defense on the output. Since the model can err, you check the answer: validate the structure, reject the invalid, fall back on failure.
  • grounding on data. To keep the model from inventing, you give it facts in context and ask it to answer only from them — this sharply reduces made-up content.

Same acceptance principle as for AI-written code: trust, but verify with a machine.

In short

  • An LLM feature = prompt + context + model call + response handling. The model is the engine; your job is the input and the output.
  • The prompt is a specification (role, format, boundaries), not magic; you refine it on examples.
  • Context is limited and must be relevant; selecting the right data is solved by RAG.
  • Embedded features need structured output (JSON to a schema), not just text.
  • The model's output is non-deterministic: measure quality by evaluation on examples, and defend the answer with validation and grounding on data.

Next — LangChain: how to orchestrate these parts when a feature grows beyond a single call.