RAG rests on one operation: given a question vector, find the nearest chunk vectors. With a dozen documents you can do this by brute force. With millions — you can't: you need a specialized engine that stores embeddings and searches among them quickly for similar ones. That's a vector database.

What it does

An ordinary database searches by exact match or range: "where id = 42," "price under 1000." A vector database answers a different question: "which records are closest to this vector in meaning." It stores embeddings and, given a query vector, quickly returns the k nearest. Everything else (metadata, chunk texts) usually sits alongside, so that together with the found vector you get the text and a source reference.

Finding the exact nearest vectors among millions is expensive: you'd have to compare the query with every one. So vector databases use approximate nearest-neighbor search (ANN): special indexes (the most popular is HNSW, a graph of links between similar vectors) find almost the closest, but thousands of times faster than a full scan. "Approximate" means the truly nearest vector may occasionally miss the results — in practice this barely affects answer quality, while search stays instant at any scale.

Another important capability is metadata filtering: "find the semantically similar, but only among this user's documents and only from the last year." Semantic search combined with ordinary filters covers most real tasks.

pgvector or a dedicated database

The main fork when choosing.

pgvector — a PostgreSQL extension. It adds a "vector" type and nearest-neighbor search to ordinary Postgres. A huge plus: the vectors sit next to your ordinary data, in the same database, under the same transactions and backups. No need to run and sync a separate system. For most projects, where embeddings range from thousands to a few million, this is enough — and it's a reasonable default. On PostgreSQL itself — see its section.

Dedicated vector databases (Qdrant, Weaviate, Milvus, Pinecone and others) are specialized systems built purely for vector search. You reach for them when you have tens or hundreds of millions of vectors and more, need extreme scale, fine index tuning, or vector search is the service's central load. You pay for it with a separate system to operate.

The rule is the same as with any specialized store: start with pgvector (your data is in Postgres anyway), and move to a dedicated database when scale or requirements justify it — not "for the future."

Pitfalls

  • choice of metric and embedding model. Vectors from one model can't be searched among vectors from another — they're incomparable. Change the embedding model and you have to reindex everything.
  • indexing cost. Converting millions of chunks to embeddings is millions of model calls; that's a noticeable one-off (and, on updates, recurring) expense.
  • not vectors alone. Often the best result comes from a hybrid: semantic search plus ordinary keyword search — they complement each other.

Where it applies

A vector database is a mandatory component of any RAG application: a knowledge base, document search, "find similar" recommendations, deduplication and clustering by meaning. Anywhere the question is "find the similar in meaning, not by exact match."

In short

  • A vector database stores embeddings and quickly finds the nearest in meaning — the foundation of RAG.
  • Scale is handled by approximate nearest-neighbor search (ANN, the HNSW index): almost exact, but thousands of times faster than a scan; plus metadata filtering.
  • pgvector (a Postgres extension) is the default: vectors next to your data, no separate system. Dedicated databases are for very large scale.
  • Pitfalls: incompatibility of vectors from different models (reindex on change), indexing cost, the value of a hybrid with keyword search.

Next — agent applications: when the model doesn't just answer but decides for itself which tools to call to get the task done.