How Queries Work in Elasticsearch and Why Documents Are Ranked the Way They Are

← Back to the section

When you type "chocolate candy" into a search box and see a list of products, Elasticsearch does not simply look for those two words. It calculates how well each document matches the query and sorts the results by that score. Let's see how all of this works.

Two modes: "how well it matches" and "match or not"

The most important distinction: a query works in one of two modes.

Full-text search (query context) — Elasticsearch asks "how well does this document answer the query?" and computes a numeric score (_score). Documents are sorted by it, the most relevant first.

Exact filter (filter context) — the question is "does the document match or not?" Yes or no, with no score. Such filters are cached, so they run faster.

An example where both modes are used together:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "chocolate" } }
      ],
      "filter": [
        { "term":  { "in_stock": true } },
        { "range": { "price": { "gte": 50, "lte": 500 } } }
      ]
    }
  }
}

match in must is full-text search and affects the score. The filters in filter are exact conditions and are cached.

A practical rule: put everything that is not for ranking into filter — it is faster and does not distort the relevance score.

Types of queries

match — text search

The most common query for text fields:

{ "match": { "name": "chocolate candies" } }

Before searching, the text goes through the same analyzer used during indexing: the string is split into terms and lowercased. Endings are stripped only when the analyzer has a stemmer — the standard analyzer does not do that. As a result, it looks for documents that contain at least one of the resulting terms.

To require all terms:

{ "match": { "name": { "query": "chocolate candies", "operator": "and" } } }

match_phrase — exact phrase

{ "match_phrase": { "name": "chocolate candies" } }

The terms must appear in the right order and next to each other — this suits exact names. The slop parameter allows a small gap between words.

multi_match — search across several fields at once

{
  "multi_match": {
    "query": "chocolate",
    "fields": ["name^3", "description^1", "tags^2"],
    "type": "best_fields"
  }
}

name^3 means that a match in the name field is worth three times as much as one in description. Handy when the title matters more than the body.

The search modes differ in how per-field scores are combined:

• best_fields — the maximum is taken. Suitable when all the query words usually sit in one field.
• most_fields — the sum is taken: the same word may have landed in different fields.
• cross_fields — all fields count as one, like a first name and a last name split across columns.

term and terms — exact value

{ "term":  { "category_id": 1 } }
{ "terms": { "category_id": [1, 2, 3] } }

term does not analyze the text, so it fits numbers, identifiers, and keyword fields, not text ones. On a text field the result surprises: Elasticsearch looks for the literal value in an index that stores already processed terms.

range — a range

{ "range": { "price": { "gte": 50, "lte": 500 } } }
{ "range": { "created_at": { "gte": "now-7d/d", "lte": "now/d" } } }

Works with numbers and dates. Date math is supported: now-7d is seven days ago, /d rounds down to the start of the day.

exists — the field is filled in

{ "exists": { "field": "image_url" } }

Finds documents where the field is present and not null.

bool — the foundation of all complex queries

bool lets you combine queries:

{
  "bool": {
    "must":     [ ... ],   // required, affects the score
    "filter":   [ ... ],   // required, no score
    "should":   [ ... ],   // preferred, raises the score on a match
    "must_not": [ ... ]    // exclude
  }
}

must and filter are both required; the only difference is the score. must_not throws documents away. The interesting one is should: add to the first example

"should": [ { "term": { "is_featured": true } } ]

— and recommended products move up, while the rest stay in the results: with a must and a filter next to it, should is optional. Left alone in a bool, should becomes required — at least one of its conditions has to match (minimum_should_match defaults to 1).

How Elasticsearch decides who is first: the BM25 algorithm

When you search for "chocolate", why does one document rank above another? Elasticsearch uses the BM25 formula (Best Matching 25) — a standard algorithm from information retrieval theory.

Simplified, the score depends on three things:

• Term frequency — does the word "chocolate" appear in the document 5 times? That is better than once. But not 5 times better — the returns diminish.
• Term rarity — "chocolate" appears in 10% of all documents, while "and" appears in 99%. A rare word carries more information, so a match on it is worth more.
• Document length — a short title "Chocolate candy" with a match is worth more than a long description with the same word.

The mechanics show up in a small program: the index here is three products, the filter drops what is out of stock, and the rest go through the same formula.

live example

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;

public class RelevanceDemo {

    record Doc(String title, String text, boolean inStock) {}
    record Hit(String title, double score) {}

    static final double K1 = 1.2;
    static final double B = 0.75;
    static final double AVG_LEN = 12;
    static final int TOTAL_DOCS = 1000;
    static final int DOCS_WITH_TERM = 120;

    public static void main(String[] args) {
        String term = "chocolate";
        List<Doc> docs = List.of(
                new Doc("Dark chocolate 70", "chocolate cocoa bar", true),
                new Doc("Candy gift box", "candy assortment chocolate milk wafers caramel nougat cookies sweets chocolate gift box for new year", true),
                new Doc("Milk chocolate", "chocolate milk 33 cocoa", false));

        double idf = Math.log(1 + (TOTAL_DOCS - DOCS_WITH_TERM + 0.5) / (DOCS_WITH_TERM + 0.5));
        System.out.printf(Locale.ROOT, "idf(%s) = %.2f: the word is in %d documents out of %d%n",
                term, idf, DOCS_WITH_TERM, TOTAL_DOCS);

        List<Hit> hits = new ArrayList<>();
        for (Doc doc : docs) {
            if (!doc.inStock()) {
                System.out.println("filtered out " + doc.title() + ": no score computed");
                continue;
            }
            String[] tokens = doc.text().toLowerCase(Locale.ROOT).split("[^\\p{L}\\p{N}]+");
            long freq = Arrays.stream(tokens).filter(term::equals).count();
            double norm = 1 - B + B * tokens.length / AVG_LEN;
            hits.add(new Hit(doc.title(), idf * freq * (K1 + 1) / (freq + K1 * norm)));
        }

        hits.sort(Comparator.comparingDouble(Hit::score).reversed());
        for (Hit hit : hits) {
            System.out.printf(Locale.ROOT, "%.2f  %s%n", hit.score(), hit.title());
        }
    }
}

The first product wins not by the number of hits: one occurrence in a three-word name outweighs two in a fifteen-word description. The third got no score at all — the filter removed it before the formula applied.

BM25 settings rarely need tuning.

Controlling the score: promoting important documents

Sometimes documents have to rank higher for business reasons rather than for a text match: "new products matter more", "recommended products at the top".

A simple way is boost on a single clause: { "match": { "name": { "query": "chocolate", "boost": 3 } } } triples the contribution of the name, exactly like writing name^3 in multi_match.

function_score — combined ranking

For more complex logic there is function_score:

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "chocolate" } },
      "functions": [
        {
          "filter": { "term": { "is_featured": true } },
          "weight": 2.0
        },
        {
          "gauss": {
            "created_at": {
              "origin": "now",
              "scale": "30d",
              "decay": 0.5
            }
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

Here the base score from the text match is multiplied by the sum of two factors: a recommended product gets a weight of 2, and fresh products are lifted by a Gaussian decay (after 30 days the score drops by half). field_value_factor is usually added next to them: it mixes a numeric field such as popularity into the score.

Start with one or two factors — it is easy to overcomplicate things with function_score.

Aggregations: facets and analytics

Aggregations compute statistics over the found documents. They are what builds catalog facets: "X products in each category", "price from Y to Z".

A query with aggregations need not return the documents at all — set size: 0 and get the statistics only:

{
  "query": { "match": { "name": "chocolate" } },
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category_id", "size": 10 }
    }
  }
}

In the response the terms aggregation returns a list of buckets with counters — those are the ready-made facets:

"by_category": {
  "buckets": [
    { "key": 1, "doc_count": 50 },
    { "key": 2, "doc_count": 12 }
  ]
}

Aggregations nest inside one another: put another one inside by_category, "avg_price": { "avg": { "field": "price" } }, and the average price is computed per category. This is a single HTTP request instead of several queries to a database.

Putting it all together: a catalog query

A real query for a product catalog with search, filters, and facets:

{
  "size": 20,
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "chocolate candies",
            "fields": ["name^3", "description"],
            "type": "best_fields",
            "fuzziness": "AUTO"
          }
        }
      ],
      "filter": [
        { "terms": { "category_id": [1, 2] } },
        { "range": { "price": { "gte": 50, "lte": 500 } } },
        { "term":  { "in_stock": true } }
      ]
    }
  },
  "aggs": {
    "categories": { "terms": { "field": "category_id", "size": 20 } }
  }
}

fuzziness: AUTO lets you find documents even with typos: for long words, deviations of up to two characters are allowed.

Pagination: why from works poorly on deep pages

Standard pagination through from and size has a limit. The query from: 9000, size: 20 makes Elasticsearch process and sort the first 9,020 documents on each shard and then throw away 9,000 of them. Past ten thousand the door is shut entirely — index.max_result_window is 10,000, so from: 10000, size: 20 returns an error.

For page-by-page browsing (especially infinite scroll), it is better to use search_after:

{
  "size": 20,
  "query": { "match": { "name": "chocolate" } },
  "sort": [ { "_score": "desc" }, { "sku": "desc" } ],
  "search_after": [0.78, "SKU-12345"]
}

search_after takes the sort values of the last document on the previous page and continues from there. You cannot jump straight to page 50 — but there are no performance problems.

The second sort field is mandatory, and it has to be unique: relevance scores of different products collide easily, and without such a tie-breaker pages start to overlap. _id looks like the obvious choice, but in Elasticsearch 8 sorting on it is disabled by default — it costs too much memory. So a regular key field such as a SKU is used instead.

In short

• Queries work in two modes: query context calculates a relevance score, filter context only sifts documents without a score — and gets cached.
• match is for text, term/terms for exact values, range for ranges; term on a text field is almost always a miss.
• bool puts it together: must and filter are required, should promotes, must_not excludes; a lone should becomes required.
• The relevance score comes from BM25: term frequency in the document, term rarity in the index, and field length.
• You can promote the documents you need through boost or function_score — starting with one or two factors.
• Aggregations compute statistics over the found documents and give you facets; deep pagination needs search_after, not a large from (from + size hits the 10,000 limit).

What to read next

• Fundamentals — how the index is built and how documents get into search.
• Clients and integration — how to send these queries from a Java application.
• Operations — performance and index management.
• Search: PostgreSQL FTS or Elasticsearch — where tsvector is enough and where a separate engine is needed.