YQL Query Language Reference

The following numeric operators are available: = < > <= >= range(field, lower bound, upper bound).

where 500 >= price

where range(fieldname, 0, 5000000000L)

Numbers must be in the signed 32-bit range. Input 64-bit signed numbers using L as suffix.

For the range operator, one can also use the strings Infinity or -Infinity:

where (range(year, 2000, Infinity))

The weightedset field does not support filtering on weight. Solve this using the map type and sameElement query operator - see example.

The boolean operator is: =

where alive = true

The right-hand side argument of the contains operator is either a string literal, or a function, like phrase.

contains is the basic building block for text matching. The kind of matching to be done depends on the field settings in the schema.

where title contains "madonna"

where title contains ({stem: false}"madonna")

The matched field must be an indexed field or attribute.

Fields inside structs are referenced using dot notation - e.g mystruct.mystructfield.

and accepts other and statements, or statements, userQuery, logically inverted statements - and contains statements as arguments:

where title contains "madonna" and title contains "saint"

or accepts other or statements, and statements, userQuery - and contains statements as arguments:

where title contains "madonna" or title contains "saint"

Use the ! operator to match document that does not satisfy some condition:

where title contains "madonna" and !(title contains "saint")

Phrases are expressed as a function:

where text contains phrase("st", "louis", "blues")

near() matches if all argument terms occur close to each other in the same document.

onear() (ordered near) is like near(), but also requires the terms in the document having the same order as given in the function (i.e. it is a phrase allowing other words interleaved). With distance 1, onear() has the same semantics as phrase().

sameElement() is an operator that requires the terms to match within the same struct element in an array or a map field. Example:

struct person {
    field first_name    type string {}
    field last_name     type string {}
    field year_of_birth type int {}
}

field persons type array<person> {
    indexing: summary
    struct-field first_name    { indexing: attribute }
    struct-field last_name     { indexing: attribute }
    struct-field year_of_birth { indexing: attribute }
}
field identities type map<string, person> {
    indexing: summary
    struct-field key                 { indexing: attribute }
    struct-field value.first_name    { indexing: attribute }
    struct-field value.last_name     { indexing: attribute }
    struct-field value.year_of_birth { indexing: attribute }
}

With normal AND the query persons.first_name AND persons.last_name will normally not give you what you want. It will match if a document has a persons element with a matching first_name AND any element with a matching last_name. So you will get a lot of false positives since there is nothing limiting them to the same element. However, that is what sameElement ensures. Note that sameElement uses AND to connect the operands. To use OR, use multiple sameElement operators using logical OR.

where persons contains sameElement(first_name contains 'Joe', last_name contains 'Smith', year_of_birth < 1940)

The above returns all documents containing Joe Smith born before 1940 in the persons array.

Searching in a map is similar to searching in an array of struct. The difference is that you have an extra synthetic struct with the field members key and value. The above example with the identities map looks like this:

where identities contains sameElement(key contains 'father', value.first_name contains 'Joe', value.last_name contains 'Smith', value.year_of_birth < 1940)

The above returns all documents that have tagged Joe Smith born before 1940 as a 'father'. The importance here is using the indirection of key and value to address the keys and the values of the map.

If two terms in the same field should give exactly the same behavior when matched, the equiv() operator behaves like a special case of or.

where fieldName contains equiv("A","B")

In many cases, the OR operator will give the same results as an EQUIV. The matching logic is exactly the same, and an OR does not have the limitations that EQUIV does (below). The difference is in how matches are visible to ranking functions. All words that are children of an OR count for ranking. When using an EQUIV however, it looks like a single word:

• Counts as only +1 for queryTermCount
• Counts as 1 word for completeness measures
• Proximity will not discriminate different words inside the EQUIV
• Connectivity can be set between the entire EQUIV and the word before and after
• Items inside the EQUIV are not directly visible to ranking features, so weight and connectivity on those will have no effect

Limitations on how equiv can be used in a query:

• equiv may not appear inside a phrase
• It may only contain TermItem and PhraseItem instances. Operators like and cannot be placed inside equiv
• PhraseItems inside equiv will rank like as if they have size 1

Learn how to use equiv.

Used to search for urls indexed using the uri field type.

where myUrlField contains uri("vespa.ai/foo")

Various subfields are supported to search components of the URL, see the field type definition.

Levenshtein edit distance search within a string attribute.

where myStringAttribute contains ({prefixLength:1, maxEditDistance:2}fuzzy("parantesis"))

Annotations below are configuring fuzzy:

Regular expressions is supported using posix extended syntax with the limitation that it is case insensitive. Replace contains with matches to run a regexp search. A few examples running regular expression search on a field named attribute_field. Only fields defined with indexing: attribute supports regexp search.

This example becomes a substring search:

where attribute_field matches "madonna"

This example matches both madonna, madona and with any number of ns:

where attribute_field matches "mado[n]+a"

Here you match any string starting with mad:

where attribute_field matches "^mad"

userInput() is a robust way of mixing user input and a formal query. It allows controlling whether the user input is to be stemmed, lowercased, etc., but it also allows for controlling whether it should be treated as a raw string, whether it should simply be segmented or parsed as a query.

yql=select * from sources * where userInput(@animal)&animal=panda

Here, the userInput() function will access the query property "animal", and parse the property value as a weakAnd query, resulting in the following expression:

select * from sources * where weakAnd(default contains "panda")

Find a full example in the query API guide.

Instead of parameter substitution, the userInput() function also accepts raw strings as arguments, but this would obviously not be suited for parametrizing the query from a query profile. It is mostly intended as test feature.

In addition, other annotations, like stem or ranked, will take effect as normal.

userQuery() reads from model.queryString and parses the query using simple query language. If set, model.filter is combined with model.queryString before the parsing.

The user query is first parsed, then the resulting tree is inserted into the corresponding place in the YQL query tree. Example:

query=abc def -ghi
type=all
yql=select * from sources * where vendor contains "brick and mortar" AND price < 50 AND userQuery()

This evaluates to a query where:

• the numeric field price must be less than 50
• vendor must match brick and mortar
• the default index must contain the two terms abc and def, and not contain ghi.

The first, and only the first, argument of the rank() function determines whether a document is a match, but all arguments are used for calculating rank score.

where rank(a contains "A", b contains "B")

dotProduct calculates the dot product between the weighted set in the query and a weighted set field in the document as its rank score contribution:

where dotProduct(description, {"a":1, "b":2})

The result is stored as a raw score.

A normal use case is a collection of weighted tokens produced by an algorithm, to match against a corpus containing weighted tokens produced by another algorithm in order to implement personalized content exploration. See example usage of dotProduct in practical performance guide .

Refer to multivalue query operators for a discussion of usage and examples.

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where dotProduct(description, @myterms).

When using weightedSet to search a field, all tokens present in the searched field will be matched against the weighted set in the query. This means that using a weighted set to search a single-value attribute field will have similar semantics to using a normal term to search a weighted set field. The low-level matching information resulting from matching a document with a weighted set in the query will contain the weights of all the matched tokens in descending order. Each matched weight will be represented as a standard occurrence on position 0 in element 0.

where weightedSet(description, {"a":1, "b":2})

weightedSet has similar semantics to equiv, as it acts as a single term in the query. However, the restriction dictating that it contains a collection of weighted tokens directly enables specific back-end optimizations that improves performance for large sets of tokens compared to using the generic equiv or or operators.

Refer to multivalue query operators for a discussion of usage and examples. Also see multi-lookup set filtering.

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where weightedSet(description, @myterms).

wand can be used to search for documents where weighted tokens in a field matches a subset of weighted tokens in the query. At the same time, it internally calculates the dot product between token weights in the query and the field. wand is guaranteed to return the top-k hits according to its internal dot product rank score. It is an operator that scales adaptively from or to and.

Note that total hit count becomes inaccurate when using wand.

wand optimizes the performance of using multiple threads per search in the backend, and is also called Parallel Wand.

wand also allows numeric arguments, then the search argument is an array of arrays of length two. In each pair, the first number is the search term, the second its weight:

where wand(description, [[11,1], [37,2]])

Keys must be single or double-quoted if passed inline in YQL - alternatively, use parameter substitution to submit the weighted set with a simple format for faster query parsing - example: where wand(description, @myterms).

where ({scoreThreshold: 0.13, targetHits: 7}wand(description, {"a":1, "b":2}))

Refer to using wand for introduction to the WAND algorithm and example usage of wand in practical performance guide .

weakAnd is sometimes called Vespa Wand. Unlike wand, it accepts arbitrary word matches (across arbitrary fields) as arguments. Only a limited number of documents are returned for ranking (default is 100), but it does not guarantee to return the best k hits. This function can be seen as an optimized or:

where weakAnd(a contains "A", b contains "B")

where ({scoreThreshold: 0, targetHits: 7}weakAnd(a contains "A", b contains "B"))

Unlike wand, weakAnd can be used to search across several fields of various types, but it does NOT guarantee to return the top-k best number of hits. It can however be combined with any ranking expression. Keep in mind that this expression should correlate with its simple internal ranking score that uses query term weight and inverse document frequency for matching terms.

Refer to using wand for a usage and examples.

geoLocation matches a position inside a geographical circle, specified as latitude, longitude, and a maximum distance (radius). Example:

where geoLocation(myfieldname, 63.5, 10.5, "200 km")

In this example we search for documents near 63.5° north, 10.5° east, and within a 200 km radius. So a document with a "myfieldname" position in Trondheim, Norway at N63°25'47;E10°23'36 would match. The first parameter is the name of the attribute field. The second parameter is the longitude (positive for north, negative for south). The third parameter is the latitude (positive for east, negative for west). The fourth parameter must be a string specifying the radius and its units, where the supported units include "km", "m" (for meters), "miles", and "deg" for degrees (so "deg" gives radius the same units as latitude). Any negative number for radius (e.g. "-1 m") is interpreted as an "infinite" radius, letting any geographical position at all match the geoLocation operator.

The position attribute in the schema could look like:

field myfieldname type position {
    indexing: attribute | summary
}

Arrays of positions are also possible:

field myfieldname type array<position> {
    indexing: attribute
}

Properties:

nearestNeighbor matches the top-k nearest neighbors in a multidimensional vector space. Points in the vector space are specified as tensors with one indexed dimension, where the size of that dimension is equal to the dimensionality of the vector space. The document vectors are stored in a tensor field attribute, and the query vector is sent with the query request. The following tensor field types are supported:

• Single vector per document: Tensor type with one indexed dimension. Example: tensor<float>(x[3])
• Multiple vectors per document: Tensor type with one mapped and one indexed dimension. Example: tensor<float>(m{},x[3])

Euclidean distance is used as the default distance metric and the exact nearest neighbors are returned. When storing multiple vectors per document, the vector that is closest to the query vector is used when calculating the distance between the document and the query. If a HNSW index is specified on the tensor field, the approximate nearest neighbors are returned instead. Example:

where ({targetHits: 10}nearestNeighbor(doc_vector, query_vector))&input.query(query_vector)=[3,5,7]

In this example we search for the top 10 nearest neighbors in a 3-dimensional vector space. targetHits specifies the wanted top-k nearest neighbors to find. This parameter is required. The first parameter of nearestNeighbor is the name of the tensor field attribute containing the document vectors (doc_vector). The second parameter is the name of the tensor sent with the query request (query_vector). Specifying query_vector as the name means the query request must set this tensor as input.query(query_vector). The document tensor field attribute is defined as follows:

field doc_vector type tensor<float>(x[3]) {
    indexing: attribute | summary
}

The last part of the YQL example specifies the query tensor, see defining query feature types This must have the same type as the document tensor. See Nearest Neighbor Search, Approximate Nearest Neighbor Search using HNSW Index and Nearest Neighbor Search Guide for more detailed examples.

Properties:

nonEmpty takes as its only argument an arbitrary search expression. It will then perform a set of checks on that expression. If all the checks pass, the result is the same expression, otherwise the query will fail. The checks are as follows:

1. No empty search term
2. No empty operators, like phrases without terms
3. No null markers (NullItem) from e.g. failed query parsing

yql=select * from sources * where bar contains "a" and nonEmpty(bar contains "bar" and foo contains @foo)&foo=

Note how "foo" is empty in this case, which will force the query to fail. If "foo" contained a searchable term, the query would not have failed.

predicate() specifies a predicate query - see predicate fields. It takes three arguments: the predicate field to search, a map of attributes, and a map of range attributes:

where predicate(predicate_field,{"gender":"Female"},{"age":20L})

Due to a quirk in YQL-parsing, one cannot specify an empty map, use the number 0 instead.

where predicate(predicate_field,0,{"age":20L})

Matches all documents of any type. Care must be taken when using this since processing all documents as matches is expensive. At minimum, consider restricting to only one schema where you know the corpus isn't too big, see the model.restrict URL parameter.

Does not match any document at all. Not useful in itself, but could potentially be used as a placeholder in the query tree.

Remove accents from this term if it is the setting for this field. Refer to linguistics.

Whether to allow empty input for query parsing and query terms in userInput. If true, a NullItem instance is inserted in the proper place in the query tree. If false, the query will fail if the user provided input can not be parsed or is empty.

Force phrase or AND operator if re-segmenting (e.g. in stemming) this term results in multiple terms. Default is choosing from language settings.

Map of string: string. Custom annotations. No special semantics inside the YQL layer. Example:

annotations : {cox: "another"}

Used in nearestNeighbor. The optional approximate annotation may be set to false to disallow usage of an approximate HNSW index. This is especially useful to compare exact and approximate results in order to perform tuning of other parameters. This annotation is default true when an HNSW index is specified, otherwise it is always false.

Ascending hit order. Used by hitLimit.

A numeric interval is by default a closed interval. If the lower bound is exclusive, set to leftOpen. If the upper bound is exclusive, set to rightOpen. If both bounds are exclusive, set the annotation to open. Example:

where ({bounds:"rightOpen"}range(year, 2000, 2018))

Map of id: int, weight: double of explicit connectivity between this item and the item with the given id - see text matching and ranking. Example:

connectivity: {id: 4, weight: 0.8}

Descending hit order. Used by hitLimit.

Used by userInput. Same as model.defaultIndex in the query API. If grammar is set to raw or segment, this will be the field searched.

The distance-annotation sets the maximum position difference to count as a match, see near / onear. The default distance is 2, meaning match if the words have up to one separating word.

where text contains ({distance: 5}near("a", "b"))

Used in nearestNeighbor. The distanceThreshold annotation may be used to filter away hits with a higher distance than the given threshold from the results. Note that one will never get more hits with distanceThreshold than you would get without it - to get more hits, increase targetHits, too. The units for the threshold depends on the distance metric used.

The hostname subfield of uri supports anchoring to the start and/or end of the hostname, controlled by the startAnchor and endAnchor annotations. Anchoring to the end is on by default while anchoring to the start is not. Hence

where myUrlField.hostname contains uri("vespa.ai")

will match vespa.ai and docs.vespa.ai, while

where myUrlField.hostname contains ({startAnchor: true}uri("vespa.ai"))

will only match vespa.ai.

Regard this term as a "filter" term and not a term from the end user. Terms that are annotated with "filter:true" are not bolded. See also model.filter. Bolding of terms is controlled by schema:bolding.

Default sort function for strings is uca. Field sort specification can be configured in the schema, values in the query overrides the schema settings.

Numeric fields are numerically sorted.

How to parse userInput. raw will treat the user input as a string to be matched without any processing, segment will do a first pass through the linguistic libraries, while the rest of the values will treat the string as a query to be parsed. If query parsing fails, an error message will be returned. Example.

Numeric operations support hitLimit. This is used for capped range search. An alternative to using negative and positive values for hitLimit is always using a positive number of hits (as a negative number of hits does not make much sense) and combine this with either of the ascending and descending annotations (but not both). Example: {hitLimit: 38, descending: true} would be equivalent to setting it to -38, i.e. only populate with 38 hits and start from upper boundary, i.e. descending order.

Note that hitLimit will limit the number of documents that are considered. This is a powerful optimisation that must be used with care, particularly in combination with other filters. The set of documents to be considered will be limited upfront by only selecting the N best according to the range query and the hitLimit annotation, for further query evaluation.

hitLimit is not exact, but "at least". In addition, it will only kick in if the attribute has fast-search. It will look up the upper or lower bound in the range in the dictionary and scan in ascending or descending order and select entries until it has satisfied hitLimit. You will get all documents for all the dictionary entries selected.

See the practical-search-performance-guide for an example.

Used in nearestNeighbor. When using an HNSW index, the optional hnsw.exploreAdditionalHits annotation can be used to tune how many extra nodes in the graph (in addition to targetHits) should be explored before selecting the best hits. Using a greater number here gives better quality, but worse performance.

Unique ID used for e.g. connectivity.

Implicit term transformations (field defaults). If implicitTransforms is true, the settings for the field in the schema will be honored in term transforms, e.g. if the field has stemming, this term will be stemmed. If implicitTransforms is false, the search backend will receive the term exactly as written in the initial YQL expression. This is in other words a top level switch to turn off all other stemming, accent removal, Unicode normalizations and so on.

Used by geoLocation and nearestNeighbor. Label for referring to this term during ranking.

Language setting for the linguistics handling of userInput, also see model.language in the query API reference.

Used by the UCA sort function. An identifier following unicode locale identifiers, e.g. en_US.

Used in fuzzy. An inclusive upper bound of edit distance between query and string attribute.

NFKC normalization.

Normalize casing of this term if it is the setting for this field.

Map of original: string, offset: int, length: int. The (sub-)string which produced this term. Default unset. Example:

origin: {original: "abc", offset: 1, length: 2}

Do prefix matching for this term, e.g. search for "word*".

Used in fuzzy. Number of characters that are considered frozen, so the fuzzy match will be performed with the suffix left.

Include this term for ranking calculation. Setting ranked to false can speed up query evaluation. Read more about schema reference. Example

Both wand and weakAnd supports scoreThreshold, which is a double for wand and an integer for weakAnd. This threshold specifies the minimum score for hits to include. The wand / weakAnd operators both expose hits that were evaluated during the search to the first-phase ranking expression, and not only targetHits.

Significance value for text ranking features - see text matching and ranking.

See endAnchor.

Stem this term if it is the setting for this field.

• PRIMARY
• SECONDARY
• TERTIARY
• QUATERNARY
• IDENTICAL

Used by the UCA sort function. Default is PRIMARY, which only sorts on primary differentiating characteristics; this means that letters in uppercase/lowercase or with differences in accents only are considered equal.

Do suffix matching for this term, e.g. search for "*word".

Used by wand, weakAnd and nearestNeighbor. It sets the wanted number of hits exposed to the real first-phase ranking function per content node. If additional second phase ranking with rerank-count is used, do not set targetHits less than the configured rank-profile's rerank-count.

Use term position data for text ranking features such as nativeRank. This is term position, not to be confused with geo searches. Setting "usePositionData:false" can improve query performance.

Term weight, used in some text ranking features - see text matching and ranking.

where title contains ({weight:200}"heads")