Elasticsearch: how a search engine works

← Back to the section

When an online store grows to millions of products, the query "find everything containing the word 'chocolate'" starts to take seconds in a regular database: SQL scans every row looking for a match. Elasticsearch solves this task in milliseconds — thanks to a special data structure called an inverted index.

How the inverted index works

In a regular database an index looks like this: "value → row". For example, a B-tree on the id column says: "id=42 → this table row". This works great for lookups by exact values.

An inverted index is built the other way around: "word → list of documents". Elasticsearch parses each document into separate words in advance (this is called tokenization) and remembers which documents each word appeared in.

Documents:
  1: { name: "Chocolate candy" }
  2: { name: "Chocolate bars" }
  3: { name: "Cookies with chocolate" }

Inverted index (after splitting into words):
  candy      → [1]
  chocolate  → [1, 2, 3]   ← stemming merges word forms
  bar        → [2]
  cookie     → [3]

The query "find chocolate" is a single dictionary lookup: we immediately find [1, 2, 3]. No scanning through millions of rows.

The price: the index is built on every write. Elasticsearch writes slower than PostgreSQL on simple INSERTs, but reads text incomparably faster.

Elasticsearch is built on top of Apache Lucene — the same library that powers Solr and PostgreSQL full-text search (tsvector). Elasticsearch adds two layers on top of Lucene: clustering (multiple machines, shards, replication) and a REST API for working with data over JSON.

Document and index

• Document — a JSON object with a unique _id. The equivalent of a row in SQL.
• Index — a collection of documents with the same structure. The equivalent of a table.

# Save a document with a specific id
PUT /products/_doc/3
{
  "name": "Candy",
  "category_id": 1,
  "price": 150
}

# Save a document, id is generated by ES
POST /products/_doc
{ "name": "Cookies", "price": 80 }

Why a document isn't visible immediately

This is one of the main features of Elasticsearch that surprises developers at the start.

Inside each shard (more on shards below) data is stored in immutable segments — files on disk. You can't write a document "straight into a segment": the segment is closed. Instead, new documents first accumulate in memory.

Every second Elasticsearch performs a refresh: it flushes the buffer into a new segment, and that segment becomes visible for search. This is exactly why ES is called near-real-time: a document written right now will appear in search results in about 1 second.

For most applications this is fine. A user saved a product — they see it a second later. If you need immediate visibility, you can force a refresh after the write, but that loads the system.

During bulk data loading it's better to temporarily disable refresh:

# Disable automatic refresh
PUT /products/_settings
{ "index": { "refresh_interval": "-1" } }

# ... load the data ...

# Restore it and refresh manually
PUT /products/_settings
{ "index": { "refresh_interval": "1s" } }
POST /products/_refresh

This gives a 5–10x speedup during the initial load of millions of documents.

Another consequence: updating a document is not an in-place edit. ES marks the old version as deleted and creates a new segment with the new version. Old segments are periodically merged into larger ones (this is called a merge), and only then do the deleted documents physically disappear.

Cluster: how data is distributed across machines

A single server can't handle terabytes of data and thousands of requests per second. Elasticsearch is designed from the ground up to run in a cluster — several machines working as a single whole.

A cluster has several types of nodes:

• Master nodes — watch over the cluster state: which indices exist, where shards live, which nodes are alive. For reliability you need at least three master nodes (so that if one is lost, the remaining two can elect a new leader).
• Data nodes — store data and run searches. These are exactly what you need to scale when there is more data.
• Coordinating nodes — accept requests from the application, distribute them across data nodes, gather and return the result. By default any node can perform this role.

A small cluster usually combines roles: three machines act as both master and data. As load grows, the master role is moved onto dedicated machines.

Shards and replicas

An index is cut into primary shards — parts distributed across different data nodes. The number of primary shards is set when the index is created and cannot be changed afterward.

Each primary shard has replicas — exact copies that live on other nodes. There can be several replicas, and their number can be changed at any time.

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

Why this is needed:

• Read scaling: queries are distributed between the primary and its replicas. More replicas — more parallel readers.
• Fault tolerance: if the node with a primary shard goes down, one of the replicas automatically becomes the new primary.

When writing a document, Elasticsearch computes which shard it lands in: shard = hash(id) % number_of_primary_shards. The write goes to the primary, which synchronously forwards it to the replicas.

A good size for a single shard is between 10 and 50 GB. Shards that are too small create unnecessary overhead; shards that are too large slow down search and maintenance.

Mapping: the index schema

Mapping describes the document structure: what type each field has and how to index it.

Dynamic mapping works out of the box: Elasticsearch determines the type itself on the first write. The string "price": "150" becomes a field of type text, the number "price": 150 becomes type long. Convenient for experiments, but dangerous in production: if the first document set the wrong type, all subsequent ones will be coerced to it, and queries will start giving unexpected results.

In production you always define an explicit mapping:

PUT /products
{
  "mappings": {
    "properties": {
      "name":        { "type": "text", "analyzer": "russian" },
      "category_id": { "type": "long" },
      "price":       { "type": "scaled_float", "scaling_factor": 100 },
      "tags":        { "type": "keyword" },
      "created_at":  { "type": "date" }
    }
  }
}

The main field types:

• text — a string for full-text search. Split into words, stored in the inverted index. For descriptions and names.
• keyword — a string as is, without splitting. For categories, statuses, tags — anything that needs exact lookup or sorting.
• text + keyword together — a common trick: one field is available for both full-text search and exact matching.
• long, integer, float, scaled_float — numbers.
• date — a date in ISO 8601 format.
• boolean — true/false.
• geo_point — coordinates for geographic queries.
• dense_vector — a vector for semantic search (ES 8+).

An important limitation: a field's type cannot be changed after the index is created. If you made a mistake — you need to create a new index with the correct mapping and move the data into it via the _reindex API. That's why mapping is worth thinking through in advance.

Analyzers: how text turns into words

When Elasticsearch indexes a field of type text, it doesn't just split the string on spaces. The text passes through an analyzer — a chain of three steps:

1. Character filters — preprocessing: strip HTML tags, replace characters.
2. Tokenizer — split into words (tokens).
3. Token filters — process each word: lowercase it, remove stop words, reduce it to its root form (stemming).

Original text:   "Chocolate «Candy» 150g"
After tokenizer: ["Chocolate", "Candy", "150g"]
After lowercase: ["chocolate", "candy", "150g"]
After stemming:  ["chocol", "candi", "150g"]

The same steps are applied to the search query. A user types "candy" — ES applies stemming, gets "candi" and looks for exactly that in the inverted index. This is why the query "candy" will find both "candies" and "candy" — all forms of the same word.

Built-in analyzers:

• standard — splitting into words + lowercase. Good for Latin script, but without stemming.
• russian — stemming + Russian stop words ("в", "на", "с" and others). Use it for Russian-language fields.
• english — the same for English.
• keyword — does not split: the whole field = one token. Applied automatically to fields of type keyword.

For autocomplete (when the user types "cho" and the system already suggests "chocolate") you use edge_ngram: at indexing time the word is split into prefixes — c, ch, cho, choc, choco, chocol, and so on. A search for "cho" instantly finds all words with such a beginning.

For specific tasks you compose a custom analyzer:

PUT /products
{
  "settings": {
    "analysis": {
      "filter": {
        "russian_stop": { "type": "stop", "stopwords": "_russian_" },
        "russian_stemmer": { "type": "stemmer", "language": "russian" }
      },
      "analyzer": {
        "ru_text": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "russian_stop", "russian_stemmer"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "ru_text" }
    }
  }
}

In short

• Elasticsearch stores data in an inverted index: word → list of documents. This makes full-text search fast even across millions of records.
• ES operates in near-real-time mode: a new document is visible in search about 1 second after the write, not instantly.
• Updating a document means creating a new version, not an in-place edit. The old version is removed at the next segment merge.
• An index is split into primary shards (fixed at creation) and replicas (can be changed). Shards let you store more data; replicas speed up reads and provide fault tolerance.
• Mapping is the index schema. In production it is defined explicitly before the first write; a field's type cannot be changed without recreating the index.
• An analyzer splits text into tokens both at indexing and at search time — which is why queries find different forms of the same word.
• For Russian text, use the russian analyzer or a custom one with the Russian Stemmer.

What to read next

• Query DSL and relevance scoring — how to formulate queries against the index.
• Elasticsearch clients — connecting from Java and other languages.
• Operations — managing indices, snapshots, monitoring.