← Back to the section

When you type "chocolate candy" into a search box and see a list of products, Elasticsearch does not simply look for those two words. It calculates how well each document matches the query and sorts the results by that score. Let's see how all of this works.

Two modes: "how well it matches" and "match or not"

The most important distinction in Elasticsearch is that a query can work in two modes:

Full-text search (query context) — Elasticsearch asks the question "how well does this document answer the query?" and calculates a numeric score (_score). Documents are sorted by this score — the most relevant come first.

Exact filter (filter context) — the question is "does the document match or not?" The answer is yes or no, with no score. Such filters are cached, so they work faster.

An example where both modes are used together:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "chocolate" } }
      ],
      "filter": [
        { "term":  { "in_stock": true } },
        { "range": { "price": { "gte": 50, "lte": 500 } } }
      ]
    }
  }
}

match in must is full-text search and affects the score. The filters in filter are exact conditions and are cached.

A practical rule: put everything that is not for ranking into filter — it is faster and does not distort the relevance score.

Types of queries

match — text search

The most common query for text fields:

{ "match": { "name": "chocolate candies" } }

Before searching, the text goes through an analyzer (the same one used during indexing): words are split into terms and endings are removed. As a result, it looks for documents that contain at least one of the resulting terms.

To require all terms:

{ "match": { "name": { "query": "chocolate candies", "operator": "and" } } }

match_phrase — exact phrase

{ "match_phrase": { "name": "chocolate candies" } }

The terms must appear in the right order and next to each other. This suits exact names. The slop parameter allows a small gap between words.

multi_match — search across several fields at once

{
  "multi_match": {
    "query": "chocolate",
    "fields": ["name^3", "description^1", "tags^2"],
    "type": "best_fields"
  }
}

name^3 means that a match in the name field is worth three times as much as one in description. Handy when the title matters more than the body.

Search modes:

  • best_fields — the final score = the maximum of the scores across the individual fields. Suitable when the words most often appear in a single field.
  • most_fields — the sum of the scores. Suitable for cases where the same word may appear in different fields.
  • cross_fields — treats all fields as one. Handy for a first name and a last name spread across different fields.

term and terms — exact value

{ "term":  { "category_id": 1 } }
{ "terms": { "category_id": [1, 2, 3] } }

term does not analyze the text, so it is not suitable for text fields — only for numbers, identifiers, and keyword fields. On text fields the result will be unexpected: Elasticsearch will look for the literal value in the index, where already processed terms are stored.

range — a range

{ "range": { "price": { "gte": 50, "lte": 500 } } }
{ "range": { "created_at": { "gte": "now-7d/d", "lte": "now/d" } } }

Works with numbers and dates. For dates, math is supported: now-7d is seven days ago, /d rounds down to the start of the day.

exists — the field is filled in

{ "exists": { "field": "image_url" } }

Finds documents where the field is present and not equal to null.

bool — the foundation of all complex queries

bool lets you combine queries:

{
  "bool": {
    "must":     [ ... ],   // required, affects the score
    "filter":   [ ... ],   // required, no score
    "should":   [ ... ],   // preferred, raises the score on a match
    "must_not": [ ... ]    // exclude
  }
}

Example: find products with "chocolate" in the name, in categories 1 or 2, priced 50–500, and preferably in stock:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "chocolate" } }
      ],
      "filter": [
        { "terms": { "category_id": [1, 2] } },
        { "range": { "price": { "gte": 50, "lte": 500 } } }
      ],
      "should": [
        { "term": { "in_stock": true } }
      ]
    }
  }
}

Products in stock will get a higher score, but out-of-stock products will still appear in the results.

How Elasticsearch decides who is first: the BM25 algorithm

When you search for "chocolate", why does one document rank above another? Elasticsearch uses the BM25 formula (Best Matching 25) — a standard algorithm from information retrieval theory.

Simplified, the score depends on three things:

  • Term frequency — does the word "chocolate" appear in the document 5 times? That is better than once. But not 5 times better — the returns diminish.
  • Term rarity — "chocolate" appears in 10% of all documents, while "and" appears in 99%. A rare word carries more information, so a match on it is worth more.
  • Document length — a short title "Chocolate candy" with a match is worth more than a long description with the same word.

In most cases you do not need to change the BM25 settings — the algorithm works well out of the box.

Controlling the score: promoting important documents

Sometimes you need certain documents to rank higher not because of a text match, but because of business logic: "new products matter more", "recommended products at the top".

A simple way is boost for individual queries:

"must": [
  { "match": { "name":        { "query": "chocolate", "boost": 3 } } },
  { "match": { "description": { "query": "chocolate", "boost": 1 } } }
]

function_score — combined ranking

For more complex logic there is function_score:

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "chocolate" } },
      "functions": [
        {
          "filter": { "term": { "is_featured": true } },
          "weight": 2.0
        },
        {
          "field_value_factor": {
            "field": "popularity",
            "modifier": "log1p",
            "factor": 0.5
          }
        },
        {
          "gauss": {
            "created_at": {
              "origin": "now",
              "scale": "30d",
              "decay": 0.5
            }
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

Here the base score from the text match is multiplied by the sum of three factors: a recommended product gets a weight of 2, popular products are lifted through the logarithm of the number of sales, and fresh products are scored higher through a Gaussian decay (after 30 days the score drops by half).

Start with one or two factors — it is easy to overcomplicate things with function_score.

Aggregations: facets and analytics

Aggregations are the calculation of statistics over the found documents. They are exactly what build catalog facets: "X products in each category", "price from Y to Z".

A query with aggregations does not need to return the documents themselves — you can set size: 0 and get only the statistics:

{
  "query": { "match": { "name": "chocolate" } },
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category_id", "size": 10 }
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 100 },
          { "from": 100, "to": 500 },
          { "from": 500 }
        ]
      }
    }
  }
}

The result:

{
  "aggregations": {
    "by_category": {
      "buckets": [
        { "key": 1, "doc_count": 50 },
        { "key": 2, "doc_count": 12 }
      ]
    },
    "price_ranges": {
      "buckets": [
        { "key": "*-100",   "doc_count": 30 },
        { "key": "100-500", "doc_count": 25 },
        { "key": "500-*",   "doc_count": 7  }
      ]
    }
  }
}

Aggregations can be nested inside one another — for example, to compute the average price in each category:

{
  "aggs": {
    "by_category": {
      "terms": { "field": "category_id" },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}

This is a single HTTP request instead of several queries to a database.

Putting it all together: a catalog query

A real query for a product catalog with search, filters, and facets:

{
  "from": 0,
  "size": 20,
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "chocolate candies",
            "fields": ["name^3", "description"],
            "type": "best_fields",
            "fuzziness": "AUTO"
          }
        }
      ],
      "filter": [
        { "terms": { "category_id": [1, 2] } },
        { "range": { "price": { "gte": 50, "lte": 500 } } },
        { "term":  { "in_stock": true } }
      ]
    }
  },
  "sort": [
    { "_score": "desc" },
    { "created_at": "desc" }
  ],
  "aggs": {
    "categories":   { "terms":     { "field": "category_id", "size": 20 } },
    "price_ranges": { "histogram": { "field": "price", "interval": 100 } }
  }
}

fuzziness: AUTO lets you find documents even with typos: for long words, deviations of up to two characters are allowed.

Pagination: why from works poorly on deep pages

Standard pagination through from and size has a limitation. The query from: 10000, size: 20 forces Elasticsearch to process and sort the first 10020 documents on each shard, and then throw away 10000 of them. At large offsets this slows down noticeably.

For page-by-page browsing (especially infinite scroll), it is better to use search_after:

{
  "size": 20,
  "query": { "match": { "name": "chocolate" } },
  "sort": [ { "_score": "desc" }, { "_id": "desc" } ],
  "search_after": [0.78, "product-12345"]
}

search_after takes the values from the last document of the previous page and continues the list from that point. You cannot jump straight to page 50 — but there are no performance problems.

In short

  • Queries work in two modes: query context calculates a relevance score, filter context just filters without a score. Filters are cached and faster.
  • The main query types: match for text, term/terms for exact values, range for ranges.
  • bool combines queries through must, filter, should, must_not.
  • The relevance score is computed by BM25 — it accounts for the frequency of a word in the document, the rarity of the word in the index, and the document length.
  • You can promote the documents you need through boost or function_score (for complex logic).
  • Aggregations compute statistics over the found documents — they are used for catalog facets.
  • For deep pagination, use search_after instead of a large from.
  • Fundamentals — how the index is built and how documents get into search.
  • Clients and integration — how to send these queries from a Java application.
  • Operations — performance and index management.