• [+] expand all

Query Language Reference

Vespa accepts unstructured human input and structured queries for application logic separately, then combines them into a single data structure for executing. Human input is parsed heuristically, while application queries are formulated in YQL.

A query URL looks like:

http://myhost.mydomain.com:8080/search/?yql=select%20%2A%20from%20sources%20%2A%20where%20text%20contains%20%22blues%22
In other words, yql contains:
select%20%2A%20from%20sources%20%2A%20where%20text%20contains%20%22blues%22

This matches all documents where the field named text contains the word blues.

Quote " and backslash \ characters in text values must be escaped by a backslash.

select

select is the list of summary fields requested (a field with the summary index attribute). Vespa will hide other fields in the matching documents.

select%20price,isbn%20from%20sources%20%2A%20where%20title%20contains%20%22madonna%22

The above explicitly requests the fields "price" and "isbn" (from all sources). To request all fields, use an asterisk as field selection:

select%20*%20from%20sources%20%2A%20where%20title%20contains%20%22madonna%22

from sources

from sources specifies which content sources to query. Example:

select%20%2A%20from%20music%20where%20title%20contains%20%22madonna%22

queries all document types in the music content cluster or federation source. Query in:

all sources select … from sources * where …
a set of sources select … from sources source1, source2 where …
a single source select … from source1 where …

In other words, sources is used for querying some/all sources. If only a single source is queried, the sources keyword is dropped. To restrict the query to only one schema (aka document type) use the model.restrict URL parameter. Also see federation.

where

The where clause is a tree of operators:

numeric

The following numeric operators are available: = < > <= >= range(field, lower bound, upper bound).

where%20500%20%3E%3D%20price
where%20range%28fieldname%2C%200%2C%205000000000L%29

Numbers must be in the signed 32-bit range, or the string Infinity or -Infinity. Input 64-bit signed numbers using L as suffix.

Annotation Effect
bounds Range: open or closed interval.
hitLimit Used for capped range search.

The weightedset field does not support filtering on weight. Solve this using the map type and sameElement query operator - see example.

boolean

The boolean operator is: =

where%20alive%20%3D%20true
contains

The right-hand side argument of the contains operator is either a string literal, or a function, like phrase.

contains is the basic building block for text matching. The kind of matching to be done depends on the field settings in the schema.

where%20title%20contains%20%22madonna%22
Annotation Effect
stem By default, the string literal is tokenized to match the field(s) searched. Explicitly control tokenization by using stem:
where%20title%20contains%20(%7Bstem%3A%20false%7D%22madonna%22)

The matched field must be an indexed field or attribute.

Fields inside structs are referenced using dot notation - e.g mystruct.mystructfield.

and

and accepts other and statements, or statements, userQuery, logically inverted statements - and contains statements as arguments:

where%20title%20contains%20%22madonna%22%20and%20title%20contains%20%22saint%22
or

or accepts other or statements, and statements, userQuery - and contains statements as arguments:

where%20title%20contains%20%22madonna%22%20or%20title%20contains%20%22saint%22
not

Use the ! operator to match document that does not satisfy some condition:

where%20title%20contains%20%22madonna%22%20and%20%21%28title%20contains%20%22saint%22%29
phrase

Phrases are expressed as a function:

where%20text%20contains%20phrase%28%22st%22%2C%20%22louis%22%2C%20%22blues%22%29
near

near() matches if all argument terms occur close to each other in the same document.

Annotation Effect
distance Tune closeness using distance.
onear

onear() (ordered near) is like near(), but also requires the terms in the document having the same order as given in the function (i.e. it is a phrase allowing other words interleaved). With distance 1, onear() has the same semantics as phrase().

Annotation Effect
distance Tune closeness using distance.
sameElement

sameElement() is an operator that requires the terms to match within the same struct element in an array or a map field. Example:

struct person {
    field first_name    type string {}
    field last_name     type string {}
    field year_of_birth type int {}
}

field persons type array<person> {
    indexing: summary
    struct-field first_name    { indexing: attribute }
    struct-field last_name     { indexing: attribute }
    struct-field year_of_birth { indexing: attribute }
}
field identities type map<string, person> {
    indexing: summary
    struct-field key                 { indexing: attribute }
    struct-field value.first_name    { indexing: attribute }
    struct-field value.last_name     { indexing: attribute }
    struct-field value.year_of_birth { indexing: attribute }
}

With normal AND the query persons.first_name AND persons.last_name will normally not give you what you want. It will match if a document has a persons element with a matching first_name AND any element with a matching last_name. So you will get a lot of false positives since there is nothing limiting them to the same element. However, that is what sameElement ensures.

where%20persons%20contains%20sameElement%28first_name%20contains%20'Joe',%20last_name%20contains%20'Smith',%20year_of_birth%20%3C%201940%29

The above returns all documents containing Joe Smith born before 1940 in the persons array.

Searching in a map is similar to searching in an array of struct. The difference is that you have an extra synthetic struct with the field members key and value. The above example with the identities map looks like this:

where%20identities%20contains%20sameElement%28key%20contains%20'father',%20value.first_name%20contains%20'Joe',%20value.last_name%20contains%20'Smith',%20value.year_of_birth%20%3C%201940%29

The above returns all documents that have tagged Joe Smith born before 1940 as a 'father'. The importance here is using the indirection of key and value to address the keys and the values of the map.

equiv

If two terms in the same field should give exactly the same behavior when matched, the equiv() operator behaves like a special case of or.

where%20fieldName%20contains%20equiv%28%22A%22%2C%22B%22%29

In many cases, the OR operator will give the same results as an EQUIV. The matching logic is exactly the same, and an OR does not have the limitations that EQUIV does (below). The difference is in how matches are visible to ranking functions. All words that are children of an OR count for ranking. When using an EQUIV however, it looks like a single word:

  • Counts as only +1 for queryTermCount
  • Counts as 1 word for completeness measures
  • Proximity will not discriminate different words inside the EQUIV
  • Connectivity can be set between the entire EQUIV and the word before and after
  • Items inside the EQUIV are not directly visible to ranking features, so weight and connectivity on those will have no effect

Limitations on how equiv can be used in a query:

  • equiv may not appear inside a phrase
  • It may only contain TermItem and PhraseItem instances. Operators like and cannot be placed inside equiv
  • PhraseItems inside equiv will rank like as if they have size 1

Learn how to use equiv.

uri

Used to search for urls indexed using the uri field type.

where%20myUrlField%20contains%20uri(%22vespa.ai%2Ffoo%22)

Various subfields are supported to search components of the URL, see the field type definition.

Annotation Effect
startAnchor Anchor uri.hostname at start.
endAnchor Anchor uri.hostname at end.
fuzzy

Levensthein edit distance search within a string attribute.

where%20myStringAttribute%20contains%20%28%7BprefixLength%3A1%2C%20maxEditDistance%3A2%7Dfuzzy%28%22parantesis%22%29%29

Annotations below are configuring fuzzy:

Annotation Effect
maxEditDistance An inclusive upper bound of edit distance between query and string attribute (default is 2).
prefixLength Number of characters that are considered frozen, so the fuzzy match will be performed only with the suffix left. Default is 0 (i.e. fuzzy will match across whole query)
matches

Regular expressions is supported using posix extended syntax with the limitation that it is case insensitive. Replace contains with matches to run a regexp search. A few examples running regular expression search on a field named attribute_field. Only fields defined with indexing: attribute supports regexp search. This example becomes a substring search:

where%20attribute_field%20matches%20%22madonna%22

This example matches both madonna, madona and with any number of ns:

where%20attribute_field%20matches%20%22mado%5Bn%5D%2Ba%22%3B

Here you match any string starting with mad:

where%20attribute_field%20matches%20%22^mad%22
userInput

userInput() is a robust way of mixing user input and a formal query. It allows controlling whether the user input is to be stemmed, lowercased, etc., but it also allows for controlling whether it should be treated as a raw string, whether it should simply be segmented or parsed as a query.

yql=select%20%2A%20from%20sources%20%2A%20where%20userInput%28%40animal%29&animal=panda

Here, the userInput() function will access the query property "animal", and parse the property value as a weakAnd query, resulting in the following expression:

select%20%2A%20from%20sources%20%2A%20where%20weakAnd%28default%20contains%20%22panda%22%29

Find a full example in the query API guide.

Instead of parameter substitution, the userInput() function also accepts raw strings as arguments, but this would obviously not be suited for parametrizing the query from a query profile. It is mostly intended as test feature.

Annotation Effect
grammar How to parse the user input. The query parsing mechanism has currently certain limitations for propagating annotations, therefore, for any value of grammar other than raw or segment, only the following annotations will take effect:
defaultIndex Same as model.defaultIndex in the query API.
language Language setting for the linguistics treatment of this userInput() call.
allowEmpty Whether to allow empty input for query parsing and search terms.

In addition, other annotations, like stem or ranked, will take effect as normal.

userQuery

userQuery() reads from model.queryString and parses the query using simple query language. If set, model.filter is combined with model.queryString before the parsing.

The user query is first parsed, then the resulting tree is inserted into the corresponding place in the YQL query tree. Example:

query%3Dabc%20def%20-ghi%26%0Atype%3Dall%26%0Ayql%3Dselect%20%2A%20from%20sources%20%2A%20where%20vendor%20contains%20%22brick%20and%20mortar%22%20AND%20price%20%3C%2050%20AND%20userQuery%28%29

This evaluates to a query where:

  • the numeric field price must be less than 50
  • vendor must match brick and mortar
  • the default index must contain the two terms abc and def, and not contain ghi.
rank

The first, and only the first, argument of the rank() function determines whether a document is a match, but all arguments are used for calculating rank score.

where%20rank%28a%20contains%20%22A%22%2C%20b%20contains%20%22B%22%29
dotProduct

dotProduct calculates the dot product between the weighted set in the query and a weighted set field in the document as its rank score contribution:

where%20dotProduct%28description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D%29

The result is stored as a raw score.

A normal use case is a collection of weighted tokens produced by an algorithm, to match against a corpus containing weighted tokens produced by another algorithm in order to implement personalized content exploration. See example usage of dotProduct in practical performance guide .

Refer to multivalue query operators for a discussion of usage and examples.

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where dotProduct(description, @myterms).

Field type Weighted set attribute with fast-search. Note: Also supported for regular attribute or index fields, but then with much weaker performance).
Query model Weighted set with {token, weight} pairs
Matching Documents where the weighted set field contains at least one of the tokens in the query.
Ranking Dot product score between the weights of the matched query tokens and field tokens. This score is available using rawScore or itemRawScore rank features.
Java Query Item DotProductItem
weightedSet

When using weightedSet to search a field, all tokens present in the searched field will be matched against the weighted set in the query. This means that using a weighted set to search a single-value attribute field will have similar semantics to using a normal term to search a weighted set field. The low-level matching information resulting from matching a document with a weighted set in the query will contain the weights of all the matched tokens in descending order. Each matched weight will be represented as a standard occurrence on position 0 in element 0.

where%20weightedSet%28description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D%29

weightedSet has similar semantics to equiv, as it acts as a single term in the query. However, the restriction dictating that it contains a collection of weighted tokens directly enables specific back-end optimizations that improves performance for large sets of tokens compared to using the generic equiv or or operators.

Refer to multivalue query operators for a discussion of usage and examples. Also see multi-lookup set filtering.

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where weightedSet(description, @myterms).

Field type Singlevalue or multivalue attribute or index field. (Note: Most use cases operates on a single value field).
Query model Weighted set with {token, weight} pairs.
Matching Documents where the field contains at least one of the tokens in the query.
Ranking The operator will act as a single term in the back-end. The query term weight is the weight assigned to the operator itself and the match weight is the largest weight among matching tokens from the weighted set. This operator does not produce a raw score. Due to better ranking and performance we recommend using dotProduct instead.
Java Query Item WeightedSetItem
wand

wand can be used to search for documents where weighted tokens in a field matches a subset of weighted tokens in the query. At the same time, it internally calculates the dot product between token weights in the query and the field. wand is guaranteed to return the top-k hits according to its internal dot product rank score. It is an operator that scales adaptively from or to and.

Note that total hit count becomes inaccurate when using wand.

wand optimizes the performance of using multiple threads per search in the backend, and is also called Parallel Wand.

wand also allows numeric arguments, then the search argument is an array of arrays of length two. In each pair, the first number is the search term, the second its weight:

where%20wand%28description%2C%20%5B%5B11%2C1%5D%2C%20%5B37%2C2%5D%5D%29

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where wand(description, @myterms).

Annotation Effect
scoreThreshold Minimum rank score for hits to include.
targetHits Wanted number of hits exposed to the real first-phase ranking function per content node.
where%20(%7BscoreThreshold%3A%200.13%2C%20targetHits%3A%207%7Dwand(description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D))

Refer to using wand for introduction to the WAND algorithm and example usage of wand in practical performance guide .

Field type Weighted set attribute with fast-search. Note: Also supported for regular attribute or index fields, but then with much weaker performance).
Query model Weighted set with {token, weight} pairs.
Matching Documents where the weighted set field contains at least one of the tokens in the query and where the internal dot product score for this document, is larger than the worst among the current top-k best hits. This means that more than top-k documents are matched and returned for ranking. It also means that many documents are skipped, even they match several tokens in the query because the dot product score is too low. This skipping makes wand faster than dotProduct in some cases.
Ranking Dot product score between the weights of the matched query tokens and field tokens. This score is available using rawScore or itemRawScore rank features. Note that the top-k best hits are only guaranteed to be returned when using this internal score as the final ranking expression.
Java Query Item WandItem
weakAnd

weakAnd is sometimes called Vespa Wand. Unlike wand, it accepts arbitrary word matches (across arbitrary fields) as arguments. Only a limited number of documents are returned for ranking (default is 100), but it does not guarantee to return the best k hits. This function can be seen as an optimized or:

where%20weakAnd%28a%20contains%20%22A%22%2C%20b%20contains%20%22B%22%29
Annotation Effect
scoreThreshold Minimum rank score for hits to include.
targetHits Wanted number of hits exposed to the real first-phase ranking function per content node.
where%20(%7BscoreThreshold%3A%200%2C%20targetHits%3A%207%7DweakAnd(a%20contains%20%22A%22%2C%20b%20contains%20%22B%22))

Unlike wand, weakAnd can be used to search across several fields of various types, but it does NOT guarantee to return the top-k best number of hits. It can however be combined with any ranking expression. Keep in mind that this expression should correlate with its simple internal ranking score that uses query term weight and inverse document frequency for matching terms.

Refer to using wand for a usage and examples.

Field type Multiple fields of all types (both attribute and index).
Query model Arbitrary number of query items searching across different fields.
Matching Documents that matches at least one of the tokens in the query and where the internal operator score for this document is larger than the worst among the current top-k best hits. As with wand, this means that typically more than top-k documents are matched and a lot of documents are skipped.
Ranking Internal ranking score based on query term weight and inverse document frequency for matching terms to find the top-k hits. This score is currently not available to the ranking framework. Matching terms are exposed to the ranking framework (same as when using and or or), so an arbitrary ranking expression can be used in combination with this operator. Note that the ranking expression used should correlate with this internal ranking score. bm25, nativeFieldMatch and nativeDotProduct rank features are good starting points.
Java Query Item WeakAndItem
geoLocation

geoLocation matches a position inside a geographical circle, specified as latitude, longitude, and a maximum distance (radius). Example:

where%20geoLocation%28myfieldname%2C%2063.5%2C%2010.5%2C%20%22200%20km%22%29

In this example we search for documents near 63.5° north, 10.5° east, and within a 200 km radius. So a document with a "myfieldname" position in Trondheim, Norway at N63°25'47;E10°23'36 would match. The first parameter is the name of the attribute field. The second parameter is the longitude (positive for north, negative for south). The third parameter is the latitude (positive for east, negative for west). The fourth parameter must be a string specifying the radius and its units, where the supported units include "km", "m" (for meters), "miles", and "deg" for degrees (so "deg" gives radius the same units as latitude). Any negative number for radius (e.g. "-1 m") is interpreted as an "infinite" radius, letting any geographical position at all match the geoLocation operator.

The position attribute in the schema could look like:

field myfieldname type position {
    indexing: attribute | summary
}

Arrays of positions are also possible:

field myfieldname type array<position> {
    indexing: attribute
}
Annotation Effect
label Label for referring to this term during ranking.

Properties:

Field type position attribute (single-valued or array).
Query parameters Field name, longitude, latitude, radius.
Matching Returns documents inside the given geo circle.
Ranking Use closeness(myfieldname), or distance(myfieldname) in ranking calculations. See closeness and distance documentation.
Java Query Item GeoLocationItem
nearestNeighbor

nearestNeighbor matches the top-k nearest neighbors in a multi-dimensional vector space. Points in the vector space are specified as tensors with one indexed dimension, where the size of that dimension is equal to the dimensionality of the vector space. The document positions are stored in a tensor attribute, and the query position is sent with the query request. Euclidean distance is used as the default distance metric and the exact nearest neighbors are returned. If a HNSW index is specified on the tensor, the approximate nearest neighbors are returned instead. Example:

where%20(%7BtargetHits%3A%2010%7DnearestNeighbor(doc_vector%2C%20query_vector))&input.query%28query_vector%29=%5B3%2C5%2C7%5D

In this example we search for the top 10 nearest neighbors in a 3-dimensional vector space. targetHits specifies the wanted top-k nearest neighbors to find. This parameter is required. The first parameter of nearestNeighbor is the name of the tensor attribute containing the document positions (doc_vector). The second parameter is the name of the tensor sent with the query request (query_vector). Specifying query_vector as the name means the query request must set this tensor as input.query(query_vector). The document tensor attribute is defined as follows:

field doc_vector type tensor<float>(x[3]) {
    indexing: attribute | summary
}

The last part of the YQL example specifies the query tensor, see defining query feature types This must have the same type as the document tensor. See Nearest Neighbor Search, Approximate Nearest Neighbor Search using HNSW Index and Nearest Neighbor Search Guide for more detailed examples.

Annotation Effect
targetHits This annotation is required, and specifies the number of hits nearestNeighbor aims to produce. Note that more or less hits might actually be produced. targetHits is per node involved in the query.
approximate The optional approximate annotation may be set to false to not use an approximate HNSW index. This is especially useful to compare exact and approximate results in order to perform tuning of HNSW parameters. This annotation is default true when an HNSW index is specified, otherwise it is always false.
hnsw.exploreAdditionalHits Tune how many extra nodes in the graph (in addition to targetHits) that should be explored before selecting the best hits.
label Use to mark the query operator with a label that can be referred to from the ranking expression in the rank profile. See the closeness rank feature. Useful when having multiple nearest neighbor query operators in the same query, see nearest neighbor search guide for usage example.
distanceThreshold Use to filter out hits with a higher distance than the given threshold. See nearest neighbor search guide for usage example.

Properties:

Field type Tensor attribute with one indexed dimension of size N.
Query model Tensor with one indexed dimension of size N.
Matching Returns documents where the distance (according to the distance metric used) between the document tensor and the query tensor is less than the greatest distance among the current top-k best hits. This means that typically more than top-k documents are matched and returned for ranking. This is similar to the behavior of wand. When an HNSW index is used, the top-k best hits are calculated before regular matching happens, taking the rest of the query filters into account.
Ranking Calculates a closeness score that is defined as 1 / (1 + d), where d is the distance between the document tensor and query tensor. This score is available using rawScore, itemRawScore, or closeness rank features. The raw distance is available using the distance rank feature.
Java Query Item NearestNeighborItem
nonEmpty

nonEmpty takes as its only argument an arbitrary search expression. It will then perform a set of checks on that expression. If all the checks pass, the result is the same expression, otherwise the query will fail. The checks are as follows:

  1. No empty search term
  2. No empty operators, like phrases without terms
  3. No null markers (NullItem) from e.g. failed query parsing
yql=select%20%2A%20from%20sources%20%2A%20where%20bar%20contains%20%22a%22%20and%20nonEmpty%28bar%20contains%20%22bar%22%20and%20foo%20contains%20%40foo%29&foo=

Note how "foo" is empty in this case, which will force the query to fail. If "foo" contained a searchable term, the query would not have failed.

predicate

predicate() specifies a predicate query - see predicate fields. It takes three arguments: the predicate field to search, a map of attributes, and a map of range attributes:

where%20predicate(predicate_field%2C%7B%22gender%22%3A%22Female%22%7D%2C%7B%22age%22%3A20L%7D)

Due to a quirk in YQL-parsing, one cannot specify an empty map, use the number 0 instead.

where%20predicate(predicate_field%2C0%2C%7B%22age%22%3A20L%7D)
true

Matches all documents of any type. Care must be taken when using this since processing all documents as matches is expensive. At minimum, consider restricting to only one schema where you know the corpus isn't too big, see the model.restrict URL parameter.

false

Does not match any document at all. Not useful in itself, but could potentially be used as a placeholder in the query tree.

order by

Sort using order by. Add asc or desc after the name of an attribute to set sort order - ascending order is default.

where%20title%20contains%20%22madonna%22%20order%20by%20price%20asc%2C%20releasedate%20desc

Sorting function, locale and strength are defined using the annotations "function", "locale" and "strength", as in:

where%20title%20contains%20%22madonna%22%20order%20by%20%7Bfunction%3A%20%22uca%22%2C%20locale%3A%20%22en_US%22%2C%20strength%3A%20%22IDENTICAL%22%7Dother%20desc%2C%20%7Bfunction%3A%20%22lowercase%22%7Dsomething

The rank profile determines the rank score each document will get. Results are ordered by that value by default, but order by overrides that ordering. Vespa does not optimize away the rank score computation in this case, it is still executed, even if the model score is thrown away. Use the built-in rank-profile unranked for optimal performance of sorting queries.

Annotation Effect
function Sort function, default UCA.
locale Locale identifier for the UCA sort function.
strength Strength setting for the UCA sort function.

limit / offset

To specify a slice / limit the number of hits returned / do pagination, use limit and/or offset. This can also be controlled by using native execution parameters. YQL specified values takes precedence. Limited by default values maxHits (400) and maxOffset (1000), max values can be overridden in a queryProfile.:

where%20title%20contains%20%22madonna%22%20limit%2031%20offset%2029

The above will return two hits (if there are sufficiently many hits matching the query), skipping the 29 first documents.

timeout

Set query timeout in milliseconds using timeout. This can also be controlled by using the native execution parameter timeout. YQL specified values takes precedence.:

where%20title%20contains%20%22madonna%22%20timeout%2070

Only literal numbers are valid, i.e. setting another unit is not supported.

Parameter substitution

The query operators dotProduct(field, value), weightedSet(field, value) and wand(field, value) support parameter substitution for the value parameter - example of equivalent queries:

... where weightedSet(field, {"a":1, "b":2})
... where weightedSet(field, @myset)&myset={a:1,b:2}

Use this to:

  • Simplify query generation, separating the value of the set/array from the YQL string. Quotes can be skipped unless the keys contain , or :.
  • Speed up query parsing. Using parameter substitution accelerates string parsing.

The value string can be passed in one of:

  • Array form: [[key, value], ...]
  • Map form: {key: value, ...}

The query operator userInput(value) supports parameter substitution for the value parameter:

... where userInput(@userinput)&userinput=free+text

Use this to submit the user data unchanged for parsing in Vespa, without risk of corrupting the YQL query.

Annotations

Terms and phrases can be annotated to manipulate the behavior. Add an annotation using {}:

where%20text%20contains%20%28%7Bdistance%3A%205%7Dnear%28%22a%22%2C%20%22b%22%29%29%20and%20text%20contains%20%28%7Bdistance%3A2%7Dnear%28%22c%22%2C%20%22d%22%29%29

Note that the annotation is enclosed by parentheses to scope the annotation to the operator.

All annotations are supported by the string arguments to functions like and phrase() and near() and also the string argument to the "contains" operator. Some annotations are also supported by the functions which are handled like leaf nodes internally in the query tree: phrase(), near(), onear(), range(), equiv(), dotProduct(), weightedSet(), weakAnd(), wand() and nearestNeighbor().

Refer to SelectTestCase.java for sample usage.

Annotation Default Values Description
accentDrop true boolean

Remove accents from this term if it is the setting for this field. Refer to linguistics.

allowEmpty false boolean

Whether to allow empty input for query parsing and query terms in userInput. If true, a NullItem instance is inserted in the proper place in the query tree. If false, the query will fail if the user provided input can not be parsed or is empty.

andSegmenting true|false

Force phrase or AND operator if re-segmenting (e.g. in stemming) this term results in multiple terms. Default is choosing from language settings.

annotations map

Map of string: string. Custom annotations. No special semantics inside the YQL layer. Example:

annotations : {cox: "another"}
approximate boolean

Used in nearestNeighbor. The optional approximate annotation may be set to false to disallow usage of an approximate HNSW index. This is especially useful to compare exact and approximate results in order to perform tuning of other parameters. This annotation is default true when an HNSW index is specified, otherwise it is always false.

ascending boolean

Ascending hit order. Used by hitLimit.

bounds closed enum

A numeric interval is by default a closed interval. If the lower bound is exclusive, set to leftOpen. If the upper bound is exclusive, set to rightOpen. If both bounds are exclusive, set the annotation to open.

connectivity map

Map of id: int, weight: double of explicit connectivity between this item and the item with the given id - see text matching and ranking. Example:

connectivity: {id: 4, weight: 0.8}
descending boolean

Descending hit order. Used by hitLimit.

defaultIndex default Any searchable field in the schema.

Used by userInput. Same as model.defaultIndex in the query API. If grammar is set to raw or segment, this will be the field searched.

distance 2 int

The distance-annotation sets the maximum position difference to count as a match, see near / onear. The default distance is 2, meaning match if the words have up to one separating word.

where%20text%20contains%20%28%7Bdistance%3A%205%7Dnear%28%22a%22%2C%20%22b%22%29%29
distanceThreshold 2 int

Used in nearestNeighbor. The distanceThreshold annotation may be used to filter away hits with a higher distance than the given threshold from the results. Note that one will never get more hits with distanceThreshold than you would get without it - to get more hits, increase targetHits, too. The units for the threshold depends on the distance metric used.

endAnchor true boolean

The hostname subfield of uri supports anchoring to the start and/or end of the hostname, controlled by the startAnchor and endAnchor annotations. Anchoring to the end is on by default while anchoring to the start is not. Hence

where%20myUrlField.hostname%20contains%20uri(%5C%22vespa.ai%5C%22)

will match vespa.ai and docs.vespa.ai, while

where%20myUrlField.hostname%20contains%20(%7BstartAnchor%3A%20true%7Duri(%22vespa.ai%22))

will only match vespa.ai.

filter false boolean

Regard this term as a "filter" term and not a term from the end user. Terms that are annotated with "filter:true" are not bolded. See also model.filter. Bolding of terms is controlled by schema:bolding.

function

Default sort function for strings is uca. Field sort specification can be configured in the schema, values in the query overrides the schema settings.

Numeric fields are numerically sorted.

Function Description
uca

This sorting is based on the icu library that follows the Universal Collation Algorithm. The specification of locale and strength are identical to how icu specifies them.

Both locale and strength are optional, however strength requires locale.

The locale query annotation will override locale-setting in the schema. If locale is missing from both, the lowercase function will be used by default.

lowercase

This improves the sorting by first lowercasing and normalising the strings before sorting. This is slightly more correct and might be enough for the use case. It is not that much more costly than raw sort, and less expensive than uca.

raw

Raw byteorder is a simple and fast ordering based on memcmp of utf8 for strings and correct sort order compliant binary rep for other fields is done. However, that is not correct for anything except computers, looking only at the binary representation.

grammar weakAnd raw, segment and all values accepted for the model.type argument in the query API.

How to parse userInput. raw will treat the user input as a string to be matched without any processing, segment will do a first pass through the linguistic libraries, while the rest of the values will treat the string as a query to be parsed. If query parsing fails, an error message will be returned. Example.

hitLimit int

Numeric operations support hitLimit. This is used for capped range search. An alternative to using negative and positive values for hitLimit is always using a positive number of hits (as a negative number of hits does not make much sense) and combine this with either of the ascending and descending annotations (but not both). Example: {hitLimit: 38, descending: true} would be equivalent to setting it to -38, i.e. only populate with 38 hits and start from upper boundary, i.e. descending order.

Note that hitLimit will limit the number of documents that are considered. This is a powerful optimisation that must be used with care, particularly in combination with other filters. The set of documents to be considered will be limited upfront by only selecting the N best according to the range query and the hitLimit annotation, for further query evaluation.

hitLimit is not exact, but "at least". In addition, it will only kick in if the attribute has fast-search. It will look up the upper or lower bound in the range in the dictionary and scan in ascending or descending order and select entries until it has satisfied hitLimit. You will get all documents for all the dictionary entries selected.

See the practical-search-performance-guide for an example.

hnsw.exploreAdditionalHits

Used in nearestNeighbor. When using an HNSW index, the optional hnsw.exploreAdditionalHits annotation can be used to tune how many extra nodes in the graph (in addition to targetHits) should be explored before selecting the best hits. Using a greater number here gives better quality, but worse performance.

id int

Unique ID used for e.g. connectivity.

implicitTransforms true boolean

Implicit term transformations (field defaults). If implicitTransforms is true, the settings for the field in the schema will be honored in term transforms, e.g. if the field has stemming, this term will be stemmed. If implicitTransforms is false, the search backend will receive the term exactly as written in the initial YQL expression. This is in other words a top level switch to turn off all other stemming, accent removal, Unicode normalizations and so on.

label string

Used by geoLocation and nearestNeighbor. Label for referring to this term during ranking.

language RFC 3066 language code

Language setting for the linguistics handling of userInput, also see model.language in the query API reference.

locale

Used by the UCA sort function. An identifier following unicode locale identifiers, e.g. en_US.

maxEditDistance 2 int

Used in fuzzy. An inclusive upper bound of edit distance between query and string attribute.

nfkc true boolean

NFKC normalization.

normalizeCase true boolean

Normalize casing of this term if it is the setting for this field.

origin map

Map of original: string, offset: int, length: int. The (sub-)string which produced this term. Default unset. Example:

origin: {original: "abc", offset: 1, length: 2}
prefix false boolean

Do prefix matching for this term, e.g. search for "word*".

prefixLength 0 int

Used in fuzzy. Number of characters that are considered frozen, so the fuzzy match will be performed with the suffix left.

ranked true boolean

Include this term for ranking calculation. Setting ranked to false can speed up query evaluation. Read more about schema reference. Example

scoreThreshold double / integer

Both wand and weakAnd supports scoreThreshold, which is a double for wand and an integer for weakAnd. This threshold specifies the minimum score for hits to include. The wand / weakAnd operators both expose hits that were evaluated during the search to the first-phase ranking expression, and not only targetHits.

significance double

Significance value for text ranking features - see text matching and ranking.

startAnchor false boolean

See endAnchor.

stem true boolean

Stem this term if it is the setting for this field.

strength PRIMARY
  • PRIMARY
  • SECONDARY
  • TERTIARY
  • QUATERNARY
  • IDENTICAL

Used by the UCA sort function. Default is PRIMARY, which only sorts on primary differentiating characteristics; this means that letters in uppercase/lowercase or with differences in accents only are considered equal.

suffix false boolean

Do suffix matching for this term, e.g. search for "*word".

targetHits 100 int

Used by wand, weakAnd and nearestNeighbor. It sets the wanted number of hits exposed to the real first-phase ranking function per content node. If additional second phase ranking with rerank-count is used, do not set targetHits less than the configured rank-profile's rerank-count.

usePositionData true boolean

Use term position data for text ranking features such as nativeRank. This is term position, not to be confused with geo searches. Setting "usePositionData:false" can improve query performance.

weight 100 int

Term weight, used in some text ranking features - see text matching and ranking.

where%20title%20contains%20(%7Bweight%3A200%7D"heads")

Annotations of sub-expressions

Consider the following query:

select%20%2A%20from%20sources%20%2A%20where%20%28%7Bstem%3A%20false%7D%28foo%20contains%20%22a%22%20and%20bar%20contains%20%22b%22%29%29%20or%20foo%20contains%20%7Bstem%3A%20false%7D%22c%22

The "stem" annotation controls whether a given term should be stemmed if its field is configured as a stemmed field (default is "true"). The "AND" operator itself has no internal API for whether its operands should be stemmed or not, but we can still annotate as such, because when the value of a given annotation is determined, the expression tree is followed from the term in question and up through its ancestors. Traversing the tree stops when a value is found (or there is nothing more to traverse). In other words, none of the terms in this example will be stemmed.

How annotations behave may be easier to understand of expressing a boolean query in the style of an S-expression:

(AND term1 term2 (OR term3 term4) (OR term5 (AND term6 term7)))

The annotation scopes would then be as follows, i.e. annotations on which elements will be checked when determining the settings for a given term:

term1term1 itself, and the first AND
term2term2 itself, and the first AND
term3term3 itself, the first OR and the first AND
term4term4 itself, the first OR and the first AND
term5term5 itself, the second OR and the first AND
term6term6 itself, the second AND, the second OR and the first AND
term7term7 itself, the second AND, the second OR and the first AND

Query properties

Use YQL variable syntax to initialize words in phrases and as single terms. This removes the need for caring about quoting a term in YQL, as well as URL quoting. The term will be used exactly as it is in the URL. As an example, look at a query with a YQL argument, and the properties animal and syntaxExample:

yql=select%20%2A%20from%20sources%20%2A%20where%20foo%20contains%20%40animal%20and%20foo%20contains%20phrase%28%40animal%2C%20%40syntaxExample%2C%20%40animal%29&animal=panda&syntaxExample=syntactic

This YQL expression will then access the query properties animal and syntaxExample and evaluate to:

select%20%2A%20from%20sources%20%2A%20where%20%28foo%20contains%20%22panda%22%20AND%20foo%20contains%20phrase%28%22panda%22%2C%20%22syntactic%22%2C%20%22panda%22%29%29

YQL in query profiles

YQL requires quoting to be included in a URL. Since YQL is well suited to application logic, while not being intended for end users, a solution to this is storing the application's YQL queries into different query profiles. To add a default query profile, add search/query-profiles/default.xml to the application package:

<query-profile id="default">
    <field name="yql">select * from sources * where default contains "latest" or userQuery()</field>
</query-profile>

This will add latest as an OR term to all queries not having an explicit query profile parameter. The important thing to note is how it is not necessary to URL-quote anything in the query profiles files. They operate independently of the HTTP parsing as such.

Query rewriting in Searchers

Searchers which modifies the textual YQL statement (not recommended) should be annotated with @Before("ExternalYql"). Searchers modifying query tree produced from an input YQL statement should annotate with @After("ExternalYql").

Grouping

Group / aggregate results by adding a grouping expression after a | - read more.

select%20*%20from%20sources%20*%20where%20sddocname%20contains%20%27purchase%27%20%7C%20all(group(customer)%20each(output(sum(price))))