Read the Vespa grouping guide first, for examples and an introduction to grouping -
this is the Vespa grouping reference.
Also note that using a multivalued attribute
(such as an array of doubles) in a grouping expression affects performance.
Such operations can hit a memory bandwidth bottleneck,
particularly if the set of hits to be processed is large, as more data is evaluated.
Group
Group query results using a custom expression (using the group clause):
A numerical or string constant (e.g., group(1) or group("all")) which makes one bucket with everything
A function over another expression (xorbit, md5, cat,
xor, and, or, add, sub,
mul, div, mod) or any other expression
The datatype of an expression is resolved using best-effort, similarly to how common
programming languages do to resolve arithmetic of different data-typed operands
The results of any expression are either scalar or single-dimensional arrays
add(<array>) adds all elements together to produce a scalar
add(<arrayA>, <arrayB>) adds each element together producing a new
array whose size is max(|<arrayA>|, |<arrayB>|)
Groups can contain subgroups (by using each and group operations),
and may be nested to any level.
Multiple sub-groupings or outputs can be created under the same group level, using multiple parallel each
or all clauses, and each one may be labelled using as(mylabel).
When grouping results, groups that contain outputs,
group lists and hit lists are generated.
Group lists contain subgroups, and hit lists contain hits that are part of the owning group.
The identity of a group is held by its id.
Scalar identities such as long, double, and string, are directly available from the id,
whereas range identities used for bucket aggregation are separated into the sub-nodes from and to.
Refer to the result format reference.
Multivalue attributes
A multivalue attribute is a
weighted set,
array or
map.
Most grouping functions will just handle the elements of
multivalued attributes separately, as if they were all individual values in separate documents.
If you are grouping over array of struct or maps, scoping will be used to preserve structure.
Each entry in the array/map will be treated as a separate sub-document.
The following syntax can be used when grouping on map attribute fields.
The key can either be specified directly (above) or indirectly via a key source attribute.
The key is retrieved from the key source attribute for each document.
Note that the key source attribute must be single-valued and have the same data type as the key type of the map:
When grouping on multivalue attributes, it may be useful to filter the groups so
that only some specific values are collected. This can be done by adding a
filter. The filter clause expects a filter predicate:
Use a regular expression to match the input, and include only
documents that match in the grouping. The input will usually be the
same -expression as in the "group" clause.
Example:
Here only the values that start
with a "foo" prefix in my_array are collected into groups, all others are ignored.
See also this example.
Regex filtering works on the string representation of any field type. For example,
you can also filter on boolean values using regex(true) and regex(false).
The pattern can be passed as a query parameter instead of being written into the expression,
see parameter substitution.
Range filter
Use a range filter to match documents where a field value is between a lower and an upper bound. Example:
Here, the lower bound is inclusive (year ≥ 1990) and the upper bound is exclusive (year < 2012).
Use optional bools at the end to control if the lower and upper bounds are inclusive, respectively.
The first bool sets the lower bound inclusive, and the second sets the upper bound inclusive.
Here, only documents where my_bool_attribute evaluates to true are included in the grouping.
In filter
Available since Vespa 8.757.45
.
Use in to allow only the hits where the input is equal to one of the given strings.
The input expression is followed by one or more string constants. Example:
Here, only hits where category is "foo" or "bar" are included in the grouping.
If category is a multivalue field, a hit is allowed if at least one of its values is equal to one of the strings.
Unlike regex, the match is an exact comparison of the whole value,
so "foo" does not match "foobar". Only string values are supported.
Logical predicates
Use not to negate another filter expression. It takes a single
sub-filter and matches when the sub-filter does not. Example:
These logical predicates can be nested to create complex filter conditions.
Filter expressions follow conventional precedence
rules: not is evaluated before and,
and and is evaluated before or. Operators of the same
precedence are evaluated left-to-right. Use parentheses (...) to
force a different grouping when needed.
Example:
Limit the number of groups returned for each level using max,
returning only first n groups as specified by order:
order changes the ordering of groups after a merge operation for the following
aggregators: count, avg and sum
orderwill not change ordering of groups after a merge operation
when max or min is used
Default order, -max(relevance()), does not require use of
precision
Continuations
Pagination of grouping results is managed by continuations.
These are opaque objects that can be combined and resubmitted using the continuations annotation
on the grouping step of the query to move to the previous or next page in a result list.
All root groups contain a single this continuation per select.
That continuation represents the current view, and if submitted as the sole continuation,
it will reproduce the exact same result as the one that contained it.
There are zero or one prev/next continuation per group- and hit list.
Submit any number of these to retrieve the next/previous pages of the corresponding lists
Any number of continuations can be combined in a query, but the first must always be the this-continuation.
E.g., one may simultaneously move both to the next page of one list, and the previous page of another.
Note:
If more than one continuation object is provided for the same group- or hit-list,
the one given last is the one that takes effect.
This is because continuations are processed in the order given,
and they replace whatever continuations they collide with.
If working programmatically with grouping, find the
Continuation
objects within
RootGroup,
GroupList and
HitList
result objects. These can then be added back into the continuation list of the
GroupingRequest
to paginate.
Lists created using the each keyword can be assigned a label
using the construct each(...) as(mylabel).
The outputs created by each clause will be identified by this label.
all(group(year)each(output(count()))as(by_year))
Individual output aggregators can also be assigned a label by adding
as(mylabel) to the aggregator inside output(...).
This labels the aggregation result, not the group list.
The number of intermediate groups returned from each content node
during expression evaluation to give the container node more data to consider
when selecting the groups that are to be evaluated further:
each(...) precision(1000)
A higher number costs more bandwidth, but leads to higher accuracy in some cases.
Parameter substitution
Available since Vespa 8.774.9
.
As in YQL, a grouping expression can refer to a query parameter
with @name. The reference is replaced by the value of the query parameter name.
The value is used as-is and is never parsed as grouping syntax,
so it needs no escaping and cannot change the structure of the expression.
This makes it safe to pass user input to a grouping expression:
References can be used in the select parameter,
in the grouping expression after | in YQL,
and in select.grouping.
In select.grouping, the JSON label field
can also be a reference: "label": "@name".
Parameter names follow the same rules as in YQL: they start with a letter or underscore,
followed by letters, digits, underscores or hyphens.
How the value is interpreted depends on where the reference is used:
Strings: the pattern in regex, the label in as,
and constants in expressions. The value is always a string, even when it looks like a number:
with label=42, group(@label) is the same as group("42"),
not group(42).
Numbers: the bounds of range, and the arguments of accuracy,
fixedwidth and other functions that take a numeric constant.
The value must be an integer or a decimal number, and may be negative.
Integers: max, precision,
and the number of bits in md5 (1 to 128) and xorbit (1 to 64).
The value must be an integer within the range the clause accepts.
A decimal number such as 2.5 is rejected.
Lists: a reference in in(expression, @my_list).
The value uses the same format as in (@my_list) in YQL:
either value, value, ... with each value optionally quoted, or a JSON array.
A single reference can therefore supply any number of values.
If a referenced parameter is not set, the query fails with the error Input 'name' is not set.
References cannot be used for field names, function names or the bounds of bucket.
Query parameters
The following query parameters are relevant for grouping.
See the Query API Reference for description.
Important:
The grouping session cache is only useful if the grouping expression
uses default ordering.
The drawback is that when max is specified in the grouping expression,
it might cause inaccuracies in aggregated values such as count.
It is recommended to test whether this is an issue or not,
and if so, adjust the precision parameter to still get correct counts.
The session cache stores intermediate grouping results in the content nodes when using multi-level grouping expressions,
in order to speed up grouping at a potential loss of accuracy. This causes the query and grouping expression to be run only once.
When having multi-level grouping expressions, the search query is normally re-run for each level.
The drawback of this is that, with an expensive ranking function, the query will take more time than strictly necessary.
Aggregators
Each level of grouping specifies a set of aggregates to collect for all documents
that belong to that group (using the output operation):
The documents in a group, retrieved using a specified summary class
The count of documents in a group
The sum, average, min, max, xor or standard deviation of an expression
Multiple quantiles of an expressions value
The value of an expression for the document with the smallest or largest value of another expression (argmin, argmax)
When all arguments are numeric, the result type is resolved by looking at the argument types.
If all arguments are longs, the result is a long.
If at least one argument is a double, the result is a double.
When using order, aggregators can also be used in expressions in order to get increased control over group sorting.
This does not work with expressions that take attributes as an argument, unless the expression is enclosed within an aggregator.
Using sum, max on a multivalued attribute:
Doing an operation such as output(sum(myarray)) will run the sum over each element value in each document.
The result is the sum of sums of values.
Similarly, max(myarray) will yield the maximal element over all elements in all documents, and so on.
Compute quantiles by listing the desired quantile values (comma-separated) in brackets, followed by a comma and the expression (e.g., a field):
This computes the median (p50) and 90th percentile (p90) time to delivery in days per city. Note that quantiles are computed using KLL Sketch, so they are approximate.
Use argmin(key, value) and argmax(key, value) to get the value of one expression
for the document that has the smallest or largest value of another expression:
This outputs the title of the cheapest track per artist, where min(price) would only output the price.
The key must be numeric, while the value can be of any type, including multivalue.
Documents where the key is not set are skipped, and the output is null if no document in the group has a key.
If several documents share the smallest or largest key, one of them is selected.
Both can be used in order, e.g. order(-argmin(price, title)).
Available since Vespa 8.761.27
.
Multivalue fields, such as maps and arrays, can be used for grouping.
However, using aggregation functions such as sum() on such fields can give misleading results.
Assume a map from strings to integers (map<string, int>),
where the strings are some sort of key to use for grouping.
The following expression will provide the sum of the values for all keys:
and not the sum of the values within each key, as one would expect.
It is still, however, possible to run the following expression to get the sum of values within a specific key:
Counts the number of unique groups (as produced by group).
Note that count operates independently of max
and that this count is an estimate using HyperLogLog++
which is an algorithm for the count-distinct problem
None
Long
Group aggregators
Name
Description
Arguments
Result
count
Increments a long counter every time it is invoked
None
Long
sum
Sums the argument over all selected documents
Numeric
Numeric
avg
Computes the average over all selected documents
Numeric
Numeric
min
Keeps the minimum value of selected documents
Numeric
Numeric
max
Keeps the maximum value of selected documents
Numeric
Numeric
xor
XOR the values (their least significant 64 bits) of all selected documents
Any
Long
stddev
Computes the population standard deviation over all selected documents
Numeric
Double
quantiles
Computes one or multiple quantiles of the values of an expression. Quantiles must be a number between 0 and 1 inclusive.
[Numeric+], Expr
[{"quantile":Double,"value":Double}+]
argmin
Keeps the value of the second argument for the document with the smallest value of the first argument
Numeric, Any
Any
argmax
Keeps the value of the second argument for the document with the largest value of the first argument
Subtract second argument from first, third from result, etc
Numeric+
Numeric
-
Subtract right argument from left
Numeric, Numeric
Numeric
div
Divide first argument by second, result by third, etc
Numeric+
Numeric
/
Divide left argument by right
Numeric, Numeric
Numeric
mod
Modulo first argument by second, result by third, etc
Numeric+
Numeric
%
Modulo left argument by right
Numeric, Numeric
Numeric
neg
Negate argument
Numeric
Numeric
-
Negate right argument
Numeric
Numeric
Bitwise expressions
Name
Description
Arguments
Result
and
AND the arguments in order
Long+
Long
or
OR the arguments in order
Long+
Long
xor
XOR the arguments in order
Long+
Long
String expressions
Name
Description
Arguments
Result
strlen
Count the number of bytes in argument
String
Long
strcat
Concatenate arguments in order
String+
String
Type conversion expressions
Name
Description
Arguments
Result
todouble
Convert argument to double
Any
Double
tolong
Convert argument to long
Any
Long
tostring
Convert argument to string
Any
String
toraw
Convert argument to raw
Any
Raw
Raw data expressions
Name
Description
Arguments
Result
cat
Cat the binary representation of the arguments together
Any+
Raw
md5
Does an MD5 over the binary representation of the argument,
and keeps the lowest 'width' bits
Any, Numeric(width)
Raw
xorbit
Does an XOR of 'width' bits over the binary representation of the argument.
Width is rounded up to a multiple of 8
Any, Numeric(width)
Raw
Accessor expressions
Name
Description
Arguments
Result
relevance
Return the computed rank of a document
None
Double
<attribute-name>
Return the value of the named attribute
None
Any
array.at
Array element access.
The expression array.at(myarray, idx) returns one value per document
by evaluating the idx expression and using it as an index into the array.
The expression can then be used to build bigger expressions such as
output(sum(array.at(myarray, 0)))
which will sum the first element in the array of each document.
The idx expression is capped to [0, size(myarray)-1]
If > array size, the last element is returned
If < 0, the first element is returned
Array, Numeric
Any
interpolatedlookup
Counts elements in a sorted array that are less than an expression,
with linear interpolation if the expression is between element values.
The operation interpolatedlookup(myarray, expr) is intended for generic graph/function lookup.
The data in myarray should be numerical values sorted in ascending order.
The operation will then scan from the start of the array to find the position
where the element values become equal to (or greater than) the value of the expr lookup argument,
and return the index of that position.
When the lookup argument's value is between two consecutive array element values,
the returned position will be a linear interpolation between their respective indexes.
The return value is always in the range [0, size(myarray)-1] of the valid index values for an array.
Assume myarray is a sorted array of type array<double> in each document:
The expression interpolatedlookup(myarray, 4.2) is now a per-document expression
that first evaluates the lookup argument, here a constant expression 4.2,
and then looks at the contents of myarray in the document.
The scan starts at the first element and proceeds until it hits an element value greater than 4.2 in the array.
This means that:
If the first element in the array is greater than 4.2, the expression returns 0
If the first element in the array is exactly 4.2, the expression still returns 0
If the first element in the array is 1.7 while the second element value is exactly 4.2,
the expression returns 1.0 - the index of the second element
If all the elements in the array are less than 4.2,
the last valid array index size(myarray)-1 is returned
If the first 5 elements in the array have values smaller than the lookup argument,
and the lookup argument is halfway between the fifth and sixth element,
a value of 4.5 is returned - halfway between the array indexes of the fifth and sixth elements
Similarly, if the elements in the array are {0, 1, 2, 4, 8}
then passing a lookup argument of "5" would return 3.25 (linear interpolation between
indexOf(4)==3 and indexOf(8)==4)
Array, Numeric
Numeric
Bucket expressions
Name
Description
Arguments
Result
fixedwidth
Maps the value of the first argument into consecutive buckets
whose width equals the second argument
Any, Numeric
NumericBucketList
predefined
Maps the value of the first argument into the given buckets.
Standard mathematical start and end specifiers may be used to define the width of a bucket.
The ( and ) evaluates to [ and > by default.
The buckets assume the type of the start/end specifiers
(string, long, double or raw).
Values are converted to this type before being compared with these specifiers.
(e.g., double values are rounded to the nearest integer for buckets of type long).
The end specifier can be skipped. The buckets
bucket(3)/bucket[3] are the same as bucket[3,4>.
This is allowed for string expressions as well;
bucket("c") is identical to bucket["c", "c ">.
The buckets split the value range into separate groups, so each value lands in at most one bucket.
To find the bucket for each document quickly, Vespa expects the buckets in increasing order and
without overlaps, and rejects a request where they are not.
Neighboring buckets may share a boundary, as in bucket(0,1000), bucket(1000,2000).
Any, Bucket+
BucketList
Time expressions
The field must be a long,
with second resolution (unix timestamp/epoch) -
examples.
Each of the time-functions will respect the timezone query parameter.
Name
Description
Arguments
Result
time.dayofmonth
Returns the day of month (1-31) for the given timestamp
Long
Long
time.dayofweek
Returns the day of week (0-6) for the given timestamp, Monday being 0
Long
Long
time.dayofyear
Returns the day of year (0-365) for the given timestamp
Long
Long
time.hourofday
Returns the hour of day (0-23) for the given timestamp
Long
Long
time.minuteofhour
Returns the minute of hour (0-59) for the given timestamp
Long
Long
time.monthofyear
Returns the month of year (1-12) for the given timestamp
Long
Long
time.secondofminute
Returns the second of minute (0-59) for the given timestamp
Long
Long
time.year
Returns the full year (e.g. 2009) of the given timestamp
Long
Long
time.date
Returns the date (e.g. 2009-01-10) of the given timestamp
Long
Long
List expressions
Name
Description
Arguments
Result
size
Return the number of elements in the argument if it is a list. If not return 1
Any
Long
sort
Sort the elements in the argument in ascending order if the argument is a list. If not, it is a NOP
Any
Any
reverse
Reverse the elements in the argument if the argument is a list. If not, it is a NOP
Any
Any
Other expressions
Name
Description
Arguments
Result
zcurve.x
Returns the X component of the given zcurve encoded 2d point.
All fields of type "position" have an accompanying "<fieldName>_zcurve" attribute
that can be decoded using this expression, e.g. zcurve.x(foo_zcurve)
Long
Long
zcurve.y
Returns the Y component of the given zcurve encoded 2d point
Long
Long
geo_distance
Computes the great-circle distance from a position field
to a given point.
The unit suffix .km or .miles selects the output unit.
Works on both position and array<position> fields.
For arrays, the minimum distance across all positions in the document is returned.
Converts the attribute string using unicode collation algorithm.
Groups are sorted using locale-aware sorting, with the default and primary strength values, respectively: