Query Language Reference

edit page

The following numeric operators are available: =, <, >, <=, >=, range(field, lower bound, upper bound)

where%20500%20%3E%3D%20price%3B

where%20range%28fieldname%2C%200%2C%205000000000L%29%3B

Numbers must be in the signed 32-bit range, or the string "Infinity"/"-Infinity". Input 64-bit signed numbers using L as suffix.

The interval is by default a closed interval. If the lower bound is exclusive, set the annotation "bounds" to "leftOpen". If the upper bound is exclusive, set the same annotation to "rightOpen". If both bounds are exclusive, set the annotation to "open".

The number operations support an extra annotation, the integer "hitLimit". This is used for capped range search. An alternative to using negative and positive values for "hitLimit" is always using a positive number of hits (as a negative number of hits do no not make much sense) and combine this with either of the boolean annotations "ascending" and "descending" (but not both). Then "[{"hitLimit": 38, "descending": true}]" would be equivalent to setting it to -38, i.e. only populate with 38 hits and start from upper boundary, i.e. descending order. Note that hitLimit will limit the number of documents that are considered. It is dangerous to use if you have other filters too. This is a very powerful optimisation that must be used with care. The set of documents to be considered will be limited upfront by only selecting the N best according to the range query and the hitLimit annotation, for further query evaluation. The hitLimit is not exact, but 'at least'. In addition the optimisation will only kick in if the attribute has fast-search. It will lookup the upper or lower bound in the range in the dictionary and scan in ascending or descending order and select entries until it has satisfied hitLimit. You will get all documents for all the dictionary entries selected.

The weightedset field does not support filtering on weight. Solve this using the map type and sameElement query operator - see example.

The boolean operator is: =

where%20alive%20%3D%20true%3B

The right hand side argument of the contains operator is either a string literal, or a function, like phrase.

contains is the basic building block for text matching. The kind of matching to be done depends on the field settings in the schema.

where%20title%20contains%20%22madonna%22%3B

Fields inside structs are referenced using dot notation - e.g mystruct.mystructfield.

By default, the string will be tokenized to match the field(s) searched. Explicitly control tokenization by using annotations:

where%20title%20contains%20%28%5B%7B%22stem%22%3A%20false%7D%5D%22madonna%22%29%3B

Note the use of parentheses to control precedence.

where%20title%20contains%20%22madonna%22%20and%20title%20contains%20%22saint%22%3B

where%20title%20contains%20%22madonna%22%20or%20title%20contains%20%22saint%22%3B

where%20title%20contains%20%22madonna%22%20and%20%21%28title%20contains%20%22saint%22%29%3B

where%20text%20contains%20phrase%28%22st%22%2C%20%22louis%22%2C%20%22blues%22%29%3B

where%20text%20contains%20%28%5B%7B%22stem%22%3A%20false%7D%5D%22blues%22%29%3B

where%20text%20contains%20%28%5B%20%7B%22distance%22%3A%205%7D%20%5Dnear%28%22a%22%2C%20%22b%22%29%29%3B

struct person {
    field first_name    type string {}
    field last_name     type string {}
    field year_of_birth type int {}
}

field persons type array<person> {
    indexing: summary
    struct-field first_name    { indexing: attribute }
    struct-field last_name     { indexing: attribute }
    struct-field year_of_birth { indexing: attribute }
}
field identities type map<string, person> {
    indexing: summary
    struct-field key                 { indexing: attribute }
    struct-field value.first_name    { indexing: attribute }
    struct-field value.last_name     { indexing: attribute }
    struct-field value.year_of_birth { indexing: attribute }
}

where%20persons%20contains%20sameElement%28first_name%20contains%20'Joe',
%20last_name%20contains%20'Smith',%20year_of_birth%20%3C%201940%29%3B

where%20identities%20contains%20sameElement%28key%20contains%20'father',
%20value.first_name%20contains%20'Joe',%20value.last_name%20contains%20'Smith',%20value.year_of_birth%20%3C%201940%29%3B

where%20fieldName%20contains%20equiv%28%22A%22%2C%22B%22%29%3B

where%20myUrlField%20contains%20uri(%22vespa.ai%2Ffoo%22)%3B

where%20myUrlField.hostname%20contains%20uri(%5C%22vespa.ai%5C%22)%3B

where%20myUrlField.hostname%20contains%20(%5B%7B%22startAnchor%22%3A%20true%20%7D%5D%20uri(%22vespa.ai%22))%3B

Regular expressions is supported using posix extended syntax with the limitation that it is case insensitive. Replace contains with matches to run a regex search. This example becomes a substring search:

where%20title%20matches%20%22madonna%22%3B

where%20title%20matches%20%22mado%5Bn%5D%2Ba%22%3B

where%20title%20matches%20%22^mad%22%3B

Note: Only attribute fields in documents that have mode="index" is supported. It is also not optimized. Having a prefix using the ^ will be faster than not having one.

userInput() is a robust way of mixing user input and a formal query. It allows controlling whether the user input is to be stemmed, lowercased, etc, but it also allows for controlling whether it should be treated as a raw string, whether it should simply be segmented or parsed as a query.

yql=select%20%2A%20from%20sources%20%2A%20where%20userInput%28%40animal%29%3B&animal=panda

select%20%2A%20from%20sources%20%2A%20where%20default%20contains%20%22panda%22%3B

yql=select%20%2A%20from%20sources%20%2A%20where%20userInput%28%40animal%29%3B&animal=panda%20smokey

select%20%2A%20from%20sources%20%2A%20where%20default%20contains%20%22panda%22%20and%20default%20contains%20%22smokey%22%3B

yql=select%20%2A%20from%20sources%20%2A%20where%20range%28year%2C%201963%2C%202014%29
%20and%20%28userInput%28%40animal%29%20or%20userInput%28%40teddy%29%29%3B&animal=panda&teddy=bear%20roosevelt

select%20%2A%20from%20sources%20%2A%20where%20range%28year%2C%201963%2C%202014%29
%20and%20%28default%20contains%20%22panda%22%20or%20%28default%20contains%20%22bear%22%20and%20default%20contains%20%22roosevelt%22%29%29%3B

yql=select%20%2A%20from%20sources%20%2A%20where%20range%28year%2C%201963%2C%202014%29
%20and%20%28userInput%28%40animal%29%20or%20%5B%7B%22grammar%22%3A%20%22segment%22%7D%5DuserInput%28%40teddy%29%29%3B&animal=panda&teddy=bear%20roosevelt

select%20%2A%20from%20sources%20%2A%20where%20range%28year%2C%201963%2C%202014%29
%20and%20%28default%20contains%20%22panda%22%20or%20default%20contains%20phrase%28%22bear%22%2C%20%22roosevelt%22%29%29%3B

userInput() control annotations:

In addition, other annotations, like stem or ranked, will take effect as normal.

The query parsing mechanism has currently certain limitations for propagating annotation, therefore, for any value of grammar other than raw or segment, only the following annotations will take effect:

• ranked
• filter
• stem
• normalizeCase
• accentDrop
• usePositionData

userQuery() reads from model.queryString and parses the query using simple query language. If set, model.filter is combined with model.queryString before the parsing.

The user query is first parsed, then the resulting tree is inserted into the corresponding place in the YQL query tree. Example: query=abc def -ghi& type=all& yql=select * from sources * where vendor contains "brick and mortar" AND price < 50 AND userQuery();

query%3Dabc%20def%20-ghi%26%0Atype%3Dall%26%0Ayql%3Dselect%20%2A%20from%20sources%20%2A%20where%20vendor%20contains%20%22brick%20and%20mortar%22%20AND%20price%20%3C%2050%20AND%20userQuery%28%29%3B

• the numeric field price must be less than 50
• vendor must match brick and mortar
• the default index must contain the two terms abc and def, and not contain ghi.

The first, and only the first, argument of the rank() function determines whether a document is a match, but all arguments are used for calculating rank score.

where%20rank%28a%20contains%20%22A%22%2C%20b%20contains%20%22B%22%29%3B

dotProduct calculates the dot product between the weighted set in the query and a weighted set field in the document as its rank score contribution:

where%20dotProduct%28description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D%29%3B

A normal use case is a collection of weighted tokens produced by an algorithm, to match against a corpus containing weighted tokens produced by another algorithm in order to implement personalized content exploration.

Refer to multivalue query operators for a discussion of usage and examples.

When using weightedSet to search a field, all tokens present in the searched field will be matched against the weighted set in the query. This means that using a weighted set to search a single-value attribute field will have similar semantics to using a normal term to search a weighted set field. The low-level matching information resulting from matching a document with a weighted set in the query will contain the weights of all the matched tokens in descending order. Each matched weight will be represented as a standard occurrence on position 0 in element 0.

where%20weightedSet%28description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D%29%3B

Refer to multivalue query operators for a discussion of usage and examples.

wand can be used to search for documents where weighted tokens in a field matches a subset of weighted tokens in the query. At the same time, it internally calculates the dot product between token weights in the query and the field. wand is guaranteed to return the top-k hits according to its internal dot product rank score. It is an operator that scales adaptively from or to and.

wand optimizes the performance of using multiple threads per search in the backend, and is also called Parallel Wand.

wand also allows numeric arguments, then the search argument is an array of arrays of length two. In each pair, the first number is the search term, the second its weight:

where%20wand%28description%2C%20%5B%5B11%2C1%5D%2C%20%5B37%2C2%5D%5D%29%3B

where%20%5B%20%7B%22scoreThreshold%22%3A%200.13%2C%20%22targetHits%22%3A%207%7D%20%5Dwand%28description%2C%20%7B%22a%22%3A1%2C%20%22b%22%3A2%7D%29%3B

weakAnd is some times called Vespa Wand. Unlike wand, it accepts arbitrary word matches (across arbitrary fields) as arguments. Only a limited number of documents are returned for ranking (default is 100), but it does not guarantee to return the best k hits. This function can be seen as an optimized or:

where%20weakAnd%28a%20contains%20%22A%22%2C%20b%20contains%20%22B%22%29%3B

where%20%5B%7B%22scoreThreshold%22%3A%200%2C%20%22targetHits%22%3A%207%7D%5DweakAnd%28a%20contains%20%22A%22%2C%20b%20contains%20%22B%22%29%3B

Refer to using wand for a usage and examples.

geoLocation matches a position inside a geographical circle, specified as latitude, longitude, and a maximum distance (radius). Example:

where%20geoLocation%28myfieldname%2C%2063.5%2C%2010.5%2C%20%22200%20km%22%29%3B

In this example we search for documents near 63.5° north, 10.5° east, and within a 200 km radius. So a document with a "myfieldname" position in Trondheim, Norway at N63°25'47;E10°23'36 would match. The first parameter is the name of the attribute field. The second parameter is the longitude (positive for north, negative for south). The third parameter is the latitude (positive for east, negative for west). The fourth parameter must be a string specifying the radius and its units, where the supported units include "km", "m" (for meters), "miles", and "deg" for degrees (so "deg" gives radius the same units as latitude). Any negative number for radius (e.g. "-1 m") is interpreted as an "infinite" radius, letting any geographical position at all match the geoLocation operator. The position attribute in the schema could look like:

field myfieldname type position {
    indexing: attribute | summary
}

field myfieldname type array {
    indexing: attribute
}

Only the "label" annotation is currently supported for geoLocation.

nearestNeighbor matches the top-k nearest neighbors in a multi-dimensional vector space. Points in the vector space are specified as tensors with one indexed dimension, where the size of that dimension is equal to the dimensionality of the vector space. The document positions are stored in a tensor attribute, and the query position is sent with the query request. Euclidean distance is used as the default distance metric and the exact nearest neighbors are returned. If a HNSW index is specified on the tensor, the approximate nearest neighbors are returned instead. Example:

where%20%5B%7B%22targetHits%22%3A%2010%7D%5D%20nearestNeighbor%28doc_vector%2C%20query_vector%29%3B&ranking.features.query%28query_vector%29=%5B3%2C5%2C7%5D

In this example we search for the top 10 nearest neighbors in a 3-dimensional vector space. targetHits specifies the wanted top-k nearest neighbors to find. This parameter is required. The first parameter of nearestNeighbor is the name of the tensor attribute containing the document positions (doc_vector). The second parameter is the name of the tensor sent with the query request (query_vector). Specifying query_vector as the name means the query request must set this tensor as ranking.features.query(query_vector). The document tensor attribute is defined as follows:

field doc_vector type tensor<float>(x[3]) {
    indexing: attribute | summary
}

The last part of the YQL example specifies the query tensor, see defining query feature types This must have the same type as the document tensor. See Nearest Neighbor Search and Approximate Nearest Neighbor Search using HNSW Index for more detailed examples.

These annotations are supported:

• The targetHits annotation is required, and specifies the number of hits the operator should aim to produce. Note that you may get both more or less hits actually produced.
• The optional approximate annotation may be set to "false" to disallow usage of an approximate HNSW index. This is especially useful to compare exact and approximate results in order to perform tuning of other parameters. This annotation is default "true" when an HNSW index is specified, otherwise it is always "false".
• When using an HNSW index, the optional hnsw.exploreAdditionalHits annotation may be used to tune how many extra nodes in the graph (in addition to targetHits) should be explored before selecting the best hits. Using a greater number here gives better quality but worse performance.
• The standard label annotation may be used to mark the query operator with a label that can be referred to from the ranking expression in the rank profile. See the documentation for the closeness rank feature.
• The distanceThreshold annotation may be used to filter away hits with a higher distance than the given threshold from the results. Note that you will never get more hits with distanceThreshold than you would get without it; if you want more hits you need to increase the targetHits value also. The units for the threshold depends on the distance metric used.

nonEmpty takes as its only argument an arbitrary search expression. It will then perform a set of checks on that expression. If all the checks pass, the result is the same expression, otherwise the query will fail. The checks are as follows:

1. No empty search term
2. No empty operators, like phrases without terms
3. No null markers (NullItem) from e.g. failed query parsing

yql=select%20%2A%20from%20sources%20%2A%20where%20bar%20contains%20%22a%22%20and%20nonEmpty%28bar%20contains%20%22bar%22%20and%20foo%20contains%20%40foo%29&foo=

predicate() specifies a predicate query - see predicate fields. It takes three arguments: the predicate field to search, a map of attributes, and a map of range attributes:

where%20predicate(predicate_field%2C%7B%22gender%22%3A%22Female%22%7D%2C%7B%22age%22%3A20L%7D)%3B

where%20predicate(predicate_field%2C0%2C%7B%22age%22%3A20L%7D)%3B

where%20title%20contains%20(%5B%7B"weight"%3A200%7D%5D"heads")%20and%20album%20contains%20"tails"%3B