The BM25 rank feature

The bm25 rank feature implements the Okapi BM25 ranking function used to estimate the relevance of a text document given a search query. It is a pure text ranking feature which operates over an indexed string field. The feature is cheap to compute, about 3-4 times faster than nativeRank, while still providing a good rank score quality wise. It is a good candidate to use in a first phase ranking function when ranking text documents.

Ranking function

The bm25 feature calculates a score for how good a query with terms $q_1,...,q_n$ matches an indexed string field t in a document D. The score is calculated as follows:

Where the components in the function are:

• $IDF\left(q_i\right)$: The inverse document frequency (IDF) of query term i in field t. This is calculated as:

  $$log(1+\frac{N-n\left(q_i\right)+0.5}{n\left(q_i\right)+0.5})$$

  N is the total number of documents on the content node. The local value BM25 uses in this case is exposed as the num_docs_indexed rank feature. $n\left(q_i\right)$ is the number of documents containing query term i for field t, which is calculated per index existing for that field. The max value among the indexes is used in the calculation, which typically comes from the largest disk index.

  As the IDF is calculated per content node and index, slight variations might occur. To use the same IDF across all content nodes, set it as the significance on each query term using annotations.

• $f(q_i,D)$: The number of occurrences (term frequency) of query term i in the field t of document D. For multi-value fields we use the sum of occurrences over all elements.
• $field\_len$: The field length (in number of words) of field t in document D. For multi-value fields we use the sum of field lengths over all elements.
• $avg\_field\_len$: The average field length of field t among the documents on the content node. Can be configured using rank-properties. The actual index statistic is also available as the averageFieldLength(name) rank feature.
• $k_1$: A parameter used to limit how much a single query term can affect the score for document D. With a higher value the score for a single term can continue to go up relatively more when more occurrences for that term exists. Default value is 1.2. Can be configured using rank-properties.
• $b$: A parameter used to control the effect of the field length of field t compared to the average field length. Default value is 0.75. Can be configured using rank-properties.

Example

In the following example we have an indexed string field content, and a rank profile using the bm25 rank feature. Note that the field must be enabled for usage with the bm25 feature by setting the enable-bm25 flag in the index section of the field definition.

schema example {
  document example {
    field content type string {
      indexing: index | summary
      index: enable-bm25
    }
  }
  rank-profile default {
    first-phase {
      expression {
        bm25(content)
      }
    }
  }
}

If the enable-bm25 flag is turned on after documents are already fed then proton performs a memory index flush followed by a disk index fusion to prepare the posting lists for use with bm25.

Use the custom component state API on each content node and examine pending_urgent_flush to determine if the preparation is still ongoing:

/state/v1/custom/component/documentdb/mydoctype/subdb/ready/index

Scoring labeled parts of the query

bm25(content) sums the contribution of every query term searching content. When a query is built from several parts, it is often useful to score those parts separately. This is done by attaching a label to the query items, using the label annotation in YQL, and referring to that label from the rank profile:

select * from example where
    ({label:"must"}content contains "vespa") and
    ({label:"nice"}(content contains "ranking" or content contains "relevance"))

A label set on an operator enclosing others, as on the or above, is inherited by every term inside it, so a label can be given per clause instead of per term.

The bm25(field: fieldname, label: label) rank feature scores only the terms carrying the given label:

rank-profile labeled {
  first-phase {
    expression: bm25(field: content, label: must) + 0.5 * bm25(field: content, label: nice)
  }
}

One score per label

With many labels, or when the labels are not known when the schema is written, use bm25_for_labels(field) instead. It returns a tensor<float>(label{}) with one cell per label that scores in the field, holding the sum of the BM25 scores of that label's terms:

rank-profile labels {
  inputs {
    query(label_weights) tensor<float>(label{})
  }
  first-phase {
    expression: sum(bm25_for_labels(content) * query(label_weights))
  }
  match-features {
    bm25_for_labels(content)
  }
}

For the query above, the match feature might be returned as:

"bm25_for_labels(content)": {
  "type": "tensor<float>(label{})",
  "cells": {
    "must": 0.7568,
    "nice": 0.3213
  }
}

Other typical uses are reduce(bm25_for_labels(content), max, label) to rank by the best scoring labeled part, or feeding the tensor to a machine-learned model. A label only gets a cell when it actually scores, and terms without a label are not included at all, so reduce(bm25_for_labels(content), sum, label) equals bm25(content) only when every scoring term carries exactly one label.