nativeRank Reference

The nativeRank feature produces a reasonable text ranking score which is computed at an acceptable performance, and is a good candidate for first phase ranking. The nativeRank feature is a linear combination of the normalized scores computed by the features nativeFieldMatch, nativeProximity, and nativeAttributeMatch. All these features are described in detail below. See the configuration properties section for how to configure the features.

nativeFieldMatch

The nativeFieldMatch feature captures how well query terms match searched index fields by looking at the number of times a term occurs in a field and how early in the field it occurs. The significance and weight of the terms are also taken into account such that unusual terms give a higher rank contribution than common ones.

The score for nativeFieldMatch is calculated as follows:

$$\mathit{nativeFieldMatch}=\frac{\sum _i^n{\mathit{termSignificance}}_i\times {\mathit{termWeight}}_i\sum _j^m{\mathit{fieldWeight}}_j\left({\mathit{firstOccImp}}_j\times {\mathit{firstOccBoost}}_{ij}+\left(1-{\mathit{firstOccImp}}_j\right)\times {\mathit{numOccBoost}}_{ij}\right)}{\sum _i^n{\mathit{termSignificance}}_i\times {\mathit{termWeight}}_i\sum _j^m{\mathit{fieldWeight}}_j\times {\mathit{fmMaxTable}}_j}$$

where n is the number of query terms searched in index fields, m is the number of fields searched by query term i, firstOccImp_j is the firstOccurrenceImportance for field j, and firstOccBoost_ij, numOccBoost_ij and fmMaxTable_j are given below.

$${\mathit{firstOccBoost}}_{ij}={\mathit{firstOccurrenceTable}}_j\left[\frac{{\mathit{firstOcc}}_{ij}\times {\mathit{tableSize}}_j}{\max \left(6,{\mathit{fieldLength}}_j\right)}\right]$$

where firstOccurrenceTable_j is the boost table configured for field j, typically an expdecay function (see the boost tables section below), firstOcc_ij is the first occurrence of query term i in field j, and tableSize_j is the size of the boost table.

$${\mathit{numOccBoost}}_{ij}={\mathit{occurrenceCountTable}}_j\left[\frac{{\mathit{numOccs}}_{ij}\times {\mathit{tableSize}}_j}{\max \left(6,{\mathit{fieldLength}}_j\right)}\right]$$

where occurrenceCountTable_j is the boost table configured for field j, typically a loggrowth function (see the boost tables section below), numOccs_ij is the number of occurrences of query term i in field j, and tableSize_j is the size of the boost table.

$${\mathit{fmMaxTable}}_j={\mathit{firstOccImp}}_j\times \max \left({\mathit{firstOccurrenceTable}}_j\right)+\left(1-{\mathit{firstOccImp}}_j\right)\times \max \left({\mathit{occurrenceCountTable}}_j\right)$$

where max(boostTable_j) is the max value in that table. fmMaxTable_j is 1 if table normalization is turned off (see the property nativeRank.useTableNormalization in the configuration properties section).

The default behavior for nativeFieldMatch is to consider all query terms searching in all index fields when calculating the score. The calculation can be limited to a specified set of index fields as follows:

nativeFieldMatch(f1, f2)

In this case only query terms searching in index fields f1 and f2 are considered.

nativeProximity

The nativeProximity feature captures how near the matched query terms occur in searched index fields by looking at the word distance between query terms in query term pairs. Two query terms that are close to each other should give a higher score than two terms that are far from each other.

The score for nativeProximity is calculated as follows:

$$\mathit{nativeProximity}=\frac{\sum _j^m{\mathit{fieldWeight}}_j\sum _{ab}{\mathit{termPairWeight}}_{ab}\left({\mathit{proxImp}}_j\times {\mathit{proxTable}}_j\left[{\mathit{dist}}_{ab}-1\right]+\left(1-{\mathit{proxImp}}_j\right)\times {\mathit{revProxTable}}_j\left[{\mathit{dist}}_{ba}-1\right]\right)}{\sum _j^m{\mathit{fieldWeight}}_j\sum _{ab}{\mathit{termPairWeight}}_{ab}\times {\mathit{pMaxTable}}_j}$$

where m is the number of index fields, ab is a term pair searched for in field j, proxImp_j is the proximityImportance for field j, proxTable_j is the forward proximity boost table for field j, dist_ab is the minimum distance between occurrences of query terms a and b in field j, (a occurs before b), revProxTable_j is the reverse proximity boost table for field j, dist_ba is the minimum distance between occurrences of query terms b and a in field j (b occurs before a), and termPairWeight_ab and pMaxTable_j are given below.

For each field j we consider all query terms searched in this field and generate a set of term pairs. The slidingWindowSize parameter determines how many pairs that are generated. With a sliding window of size 3 over the terms a b c d, we first consider the terms a b c, then the terms b c d, and finally the terms c d. The following pairs are generated: ab, ac, bc, bd, and cd.

$${\mathit{termPairWeight}}_{ab}={\mathit{connectedness}}_{ab}\times \left({\mathit{termSignificance}}_a\times {\mathit{termWeight}}_a+{\mathit{termSignificance}}_b\times {\mathit{termWeight}}_b\right)$$

$${\mathit{connectedness}}_{ac}=\frac{\min \left({\mathit{connectedness}}_{ab},{\mathit{connectedness}}_{bc}\right)}{dis{t}_{ac}}$$

where dist_ac is the distance between term a and c in the query.

$${\mathit{pMaxTable}}_j={\mathit{proxImp}}_j\times \max \left({\mathit{proxTable}}_j\right)+\left(1-{\mathit{proxImp}}_j\right)\times \max \left({\mathit{revProxTable}}_j\right)$$

where max(boostTable_j) is the max value in that table. pMaxTable_j is 1 if table normalization is turned off (see the property nativeRank.useTableNormalization in the configuration properties section).

The default behavior for nativeProximity is to consider all index fields and all query terms pairs searching in these fields when calculating the score. The calculation can be limited to a specified set of index fields as follows:

nativeProximity(f1, f2)

In this case only query term pairs searching in index fields f1 and f2 are considered.

For multi-value fields, setting element-gap for the field in the rank profile enables distance calculation between adjacent elements.

nativeAttributeMatch

The nativeAttributeMatch feature captures how well query terms match searched attribute fields, and is calculated as follows:

$$\mathit{nativeAttributeMatch}=\frac{\sum _i^n{\mathit{termWeight}}_i\times {\mathit{attributeWeight}}_j\times \mathrm{sign}\left(w_{ij}\right)\times {\mathit{weightTable}}_j\left[\mathrm{abs}\left(w_{ij}\right)\right]}{\sum _i^n{\mathit{termWeight}}_i\times {\mathit{attributeWeight}}_j\times \max \left({\mathit{weightTable}}_j\right)}$$

where n is the number of query terms searched in attribute fields, weightTable_j is the boost table for attribute j, max(weightTable_j) is the max value in that table (1 if table normalization is turned off), sign(w_ij) is the sign of w_ij. w_ij is dependent on the attribute type:

• Weighted set: equals the weight associated with the key (represented by query term i) in attribute j.
• Array: equals the number of occurrences of query term i in attribute j.
• Single: equals 1.

The default behavior for nativeAttributeMatch is to consider all query terms searching in all attribute fields when calculating the score. The calculation can be limited to a specified set of attribute fields as follows:

nativeAttributeMatch(a1, a2)

In this case only query terms searching in attribute fields a1 and a2 are considered.

nativeRank

The nativeRank feature is just a linear combination of the three other features, and is calculated as follows:

$$\mathit{nativeRank}=\frac{fmw\times \mathit{nativeFieldMatch}+pw\times \mathit{nativeProximity}+amw\times \mathit{nativeAttributeMatch}}{fmw+pw+amw}$$

where fmw is the fieldMatchWeight, pw is the proximityWeight, and amw is the attributeMatchWeight.

The default behavior when calculating the native rank score is to consider all query terms searching in all defined index fields and attribute fields. In many cases though only a subset of these fields are of interest in the rank score calculation. You can set up nativeRank for a subset of fields by specifying the field names in the parameter list as follows:

first-phase {
    expression: nativeRank(title,body,tags)
}

In this case we have two index fields (title and body) and one attribute field (tags), and the nativeRank feature is calculated based on the features nativeFieldMatch(title,body), nativeProximity(title,body), and nativeAttributeMatch(tags). Note that the CPU cost of calculating the native rank score is also reduced when specifying a subset of the fields.

Variables

This is a list of the common variables used in the formulas above:

Variable	Description
attributeWeightj	The weight of attribute field j. See the schema reference for how to set this weight. The default value is 100.
connectednessab	The connectedness between query terms a and b.
fieldLengthj	The length of field j in number of words.
fieldWeightj	The weight of index field j. See the schema reference for how to set this weight. The default value is 100.
termSignificancei	The significance of query term i.
termWeighti	The weight of query term i.

Configuration properties

This is a comprehensive list of all the configuration properties to all native rank features:

Feature	Parameter	Default	Description
nativeFieldMatch(fieldName)	averageFieldLength	The actual length of the field in the given document.	When set this replaces the true field length in the nativeFieldMatch formula for all documents.
nativeFieldMatch	firstOccurrenceTable	expdecay(8000,12.50)	The default table used when calculating boost for the first occurrence in a field.
nativeFieldMatch(fieldName)	firstOccurrenceTable	The value of firstOccurrenceTable	The table used when calculating boost for the first occurrence in the given field.
nativeFieldMatch	occurrenceCountTable	loggrowth(1500,4000,19)	The default table used when calculating boost for the number of occurrences in a field.
nativeFieldMatch(fieldName)	occurrenceCountTable	The value of occurrenceCountTable	The table used when calculating boost for the number of occurrences in the given field.
nativeFieldMatch	firstOccurrenceImportance	0.5	The default importance value used for weighting the boosts for first occurrence and number of occurrences in a field. This value should be in the interval [0, 1].
nativeFieldMatch(fieldName)	firstOccurrenceImportance	The value of firstOccurrenceImportance	The importance value used for the given field.
nativeProximity	proximityTable	expdecay(500,3)	The default table used when calculating forward proximity boost in a field.
nativeProximity(fieldName)	proximityTable	The value of proximityTable	The table used when calculating forward proximity boost in the given field.
nativeProximity	reverseProximityTable	expdecay(400,3)	The default table used when calculating reverse proximity boost in a field.
nativeProximity(fieldName)	reverseProximityTable	The value of reverseProximityTable	The table used when calculating reverse proximity boost in the given field.
nativeProximity	proximityImportance	0.5	The default importance value used for weighting the boosts for forward and reverse proximity in a field. This value should be in the interval [0, 1].
nativeProximity(fieldName)	proximityImportance	The value of proximityImportance	The importance value used for the given field.
nativeProximity	slidingWindowSize	4	The size of the sliding window used when generating term pairs.
Deprecated: The elementGap rank property is deprecated and will be removed in Vespa 9.
nativeProximity	elementGap.fieldName	infinity	The gap between positions in adjacent elements in multi-value fields. Use the element-gap rank setting instead.
nativeAttributeMatch	weightTable	linear(1,0)	The default table used when calculating boost for matching in an attribute field.
nativeAttributeMatch	weightTable.attributeName	The value of weightTable	The table used when calculating boost for matching in the given attribute.
nativeRank	fieldMatchWeight	100.0	How much to weight the score from nativeFieldMatch.
nativeRank	proximityWeight	25.0	How much to weight the score from nativeProximity. If table normalization is turned off the default value is 100.0.
nativeRank	attributeMatchWeight	100.0	How much to weight the score from nativeAttributeMatch.
nativeRank	useTableNormalization	true	Whether we should use table normalization for the native rank features. Set this property to false to turn off table normalization

For example, to override the occurrenceCountTable and reverseProximityTable for the index field content, add the following to the rank profile in the sd file:

rank-properties {
    nativeFieldMatch(content).occurrenceCountTable: "linear(0,0)"
    nativeProximity(content).reverseProximityTable: "linear(0,0)"
}

See the search definitions reference for more information on rank-properties.

Boost tables

The following boost tables are supported by the native rank features:

Name	Function	Description
expdecay(w,t)	w * exp(-x/t)	Represents an exponential decay function where w is the weight controlling the amplitude and t is the tune parameter controlling the slope.
loggrowth(w,t,s)	w * log(1 + (x/s)) + t	Represents a logarithmic growth function where w is the weight controlling the amplitude, t is the tune parameter controlling the offset, and s is a scale parameter controlling the sensitivity to the variable x
linear(w,t)	w * x + t	Represents a linear function where w controls the slope and t controls the offset.

The parameters w, t, and s are floating point numbers, the same as the content of the tables. The default table size is 256 with x in the interval [0,255]. You can override this default size by specifying an optional last parameter to the table name. For instance, if you use linear(1.5,0,512) you get a table with size 512 populated with the result of evaluating the function 1.5*x + 0 for all x in the interval [0,511].

Rank types

Four predefined rank types are supported by nativeRank: about (default), identity, tags, and empty. Each type is associated with a set of boost tables that are used by the native rank features. See the rank type document for detailed information on these type.

When setting up the sd file, either use one of the predefined rank types for a field, or explicitly specify the boost tables to use for that field as a set of rank-properties. If you don't specify anything you get the boost tables associated with the about type. The about boost tables for nativeFieldMatch and nativeProximity are already optimized for textual match, while the boost table for nativeAttributeMatch is data dependent and must be optimized for each use case.

nativeRank limitations

The nativeRank feature is a pure text match scoring feature. In particular, it does not take the following concepts into account for documents that match a query:

• Static rank or any other relevancy contribution that is based on a numeric value. Use the attribute feature in a ranking expression to get this concept into the final relevancy score.
• Geographical location of a match correlated to a location associated with the query. Use the distance or closeness feature in a ranking expression to take this into account.
• The age of the matching documents. Use the freshness feature in a ranking expression to take this into account.