Grouping Reference

edit page

Read the Vespa grouping guide first, for examples and an introduction to grouping - this is the Vespa grouping reference.

Refer to groupingSessionCache for grouping performance. Also note that using a multivalued attribute (such as an array of doubles) in a grouping expression affects performance. Such operations can hit a memory bandwidth bottleneck, particularly if the set of hits to be processed is large, as more data is evaluated.

group

Group query results using a custom expression (using the group clause):

• A numerical constant
• A document attribute
• A function over another expression (xorbit, md5, cat, xor, and, or, add, sub, mul, div, mod) or any other expression
• The data type of an expression is resolved using best effort, similarly to how common programming languages does to resolve arithmetics of different data typed operands
• The results of any expression are either scalar or single dimension arrays
  • add(<array>) adds all elements together to produce a scalar
  • add(<arrayA>, <arrayB>) adds each element together producing a new array whose size is max(|<arrayA>|, |<arrayB>|)

Groups can contain subgroups (by using each and group operations), and may be nested to any level.

Multiple sub-groupings or outputs can be created under the same group level, using multiple parallel each or all clauses, and each one may be labelled using as(mylabel).

When grouping results, groups that contain outputs, group lists and hit lists are generated. Group lists contain sub-groups, and hit lists contain hits that are part of the owning group.

The identity of a group is held by its id. Scalar identities such as long, double and string, are directly available from the id, whereas range identities used for bucket aggregation are separated into the sub-nodes from and to. Refer to the result format reference.

Multivalue attributes

A multivalue attribute is a weighted set, array or map. Most grouping functions will just handle the elements of multivalued attributes separately, as if they were all individual values in separate documents. The following syntax can be used when grouping on map attribute fields.

Group on map keys:

all( group(mymap.key) each(output(count())) )

all( group(my_map{"my_key"}) each(output(count())) )

all( group(my_map{"my_key"}.my_field) each(output(count())) )

all( group(my_map{attribute(my_key_source)}) each(output(count())) )

Tensors can not be used in grouping.

order

Each level of grouping may specify how to order its groups (using order):

• Ordering can be done using any of the available aggregates
• Multi-level grouping allows strict ordering where primary aggregates may be equal
• Ordering is either ascending or descending, specified per level of ordering
• Groups are sorted using locale aware sorting

Limit the number of groups returned for each level using max, returning only first n groups as specified by order:

• order changes ordering of groups after a merge operation for the following aggregators: count, avg and sum
• order will not change ordering of groups after a merge operation when max or min is used
• Default order, max(relevancy()), does not require use of precision

continuations

Pagination of grouping results are managed by continuations. These are opaque objects that can be combined and re-submitted using the continuations annotation on the grouping step of the query to move to the previous or next page in a result list.

All root groups contain a single this continuation per select. That continuation represents the current view, and if submitted as the sole continuation, it will reproduce the exact same result as the one that contained it.

There are zero or one prev/next continuation per group- and hit list. Submit any number of these to retrieve the next/previous pages of the corresponding lists

Any number of continuations can be combined in a query, but the first must always be the this continuation. E.g. one may simultaneously move both to the next page of one list, and the previous page of another.

Note: If more than one continuation object are provided for the same group- or hit-list, the one given last is the one that takes effect. This is because continuations are processed in the order given, and they replace whatever continuations they collide with.

If working programmatically with grouping, find the Continuation objects within RootGroup, GroupList and HitList result objects. These can then be added back into the continuation list of the GroupingRequest to paginate.

Refer to the grouping guide for an example.

Labels

Lists created using the each keyword can be assigned a label using the construct each(...) as(mylabel). The outputs created by that each clause will be identified by this label.

where(true)

Only results matching the query are grouped, also in streaming search - with one exception: Using where(true) in the grouping expression groups all documents matching the selection string in the search api. Example:

all( where(true) all(group(myfield) each(output(count()))) )

Aggregators

Each level of grouping specifies a set of aggregates to collect for all documents that belong to that group (using the output operation):

• The documents in a group, retrieved using a specified summary class
• The count of documents in a group
• The sum, average, min, max, xor or standard deviation of an expression

When using order, aggregators can also be used in expressions in order to get increased control over group sorting. This does not work with expressions that takes attributes as an argument, unless the expression is enclosed within an aggregator.

Using sum, max on a multivalued attribute: Doing an operation such as output(sum(myarray)) will run the sum over each element value in each document. The result is the sum of sums of values. Similarly max(myarray) will yield the maximal element over all elements in all documents, and so on.

Multivalue fields such as maps, arrays can be used for grouping. However, using aggregation functions such as sum() on such fields can give misleading results. Assume a map from strings to integers, where the strings are some sort of key to use for grouping. The following expression will provide the sum of the values for all keys:

all( group(mymap.key) each(output(sum(mymap.value))) )

all( group(mymap{"foo"}) each(output(sum(mymap.value))) )

Expressions

all( group(docidnsspecific()) each(output(count())) )

all( group(s) order(max(uca(s, "sv"))) each(output(count())) )

all( group(s) order(max(uca(s, "sv", "PRIMARY"))) each(output(count())) )

Select parameter language grammar

request    ::= group [ "where" "(" ( "true" | "$query" ) ")" ]
group      ::= ( "all" | "each") "(" operations ")" [ "as" "(" identifier ")" ]
operations ::= [ "group" "(" expression ")" ]
               ( ( "alias" "(" identifier "," expression ")" ) |
                 ( "max"   "(" number ")" ) |
                 ( "order" "(" expList | aggrList ")" ) |
                 ( "output" "(" aggrList ")" ) |
                 ( "precision" "(" number ")" ) )*
               group*
aggrList   ::= aggr ( "," aggr )*
aggr       ::= ( ( "count" "(" ")" ) |
                 ( "sum" "(" exp ")" ) |
                 ( "avg" "(" exp ")" ) |
                 ( "max" "(" exp ")" ) |
                 ( "min" "(" exp ")" ) |
                 ( "xor" "(" exp ")" ) |
                 ( "stddev" "(" exp ")" ) |
                 ( "summary" "(" [ identifier ] ")" ) )
               [ "as" "(" identifier ")" ]
expList    ::= exp ( "," exp )*
exp        ::= ( "+" | "-") ( "$" identifier [ "=" math ] ) | ( math ) | ( aggr )
math       ::= value [ ( "+" | "-" | "*" | "/" | "%" ) value ]
value      ::= ( "(" exp ")" ) |
               ( "add" "(" expList ")" ) |
               ( "and" "(" expList ")" ) |
               ( "cat" "(" expList ")" ) |
               ( "div" "(" expList ")" ) |
               ( "docidnsspecific" "(" ")" ) |
               ( "fixedwidth" "(" exp "," number ")" ) |
               ( "interpolatedlookup" "(" attributeName "," exp ")") |
               ( "math" "." (
                              (
                                "exp" | "log" | "log1p" | "log10" | "sqrt" | "cbrt" |
                                "sin" | "cos" | "tan" | "asin" | "acos" | "atan" |
                                "sinh" | "cosh" | "tanh" | "asinh" | "acosh" | "atanh"
                              ) "(" exp ")" |
                              ( "pow" | "hypot" ) "(" exp "," exp ")"
                            )) |
               ( "max" "(" expList ")" ) |
               ( "md5" "(" exp "," number "," number ")" ) |
               ( "min" "(" expList ")" ) |
               ( "mod" "(" expList ")" ) |
               ( "mul" "(" expList ")" ) |
               ( "or" "(" expList ")" ) |
               ( "predefined" "(" exp "," "(" bucket ( "," bucket )* ")" ")" ) |
               ( "reverse" "(" exp ")" ) |
               ( "relevance" "(" ")" ) |
               ( "sort" "(" exp ")" ) |
               ( "strcat" "(" expList ")" ) |
               ( "strlen" "(" exp ")" ) |
               ( "size" "(" exp")" ) |
               ( "sub" "(" expList ")" ) |
               ( "time" "." ( "date" | "year" | "monthofyear" | "dayofmonth" | "dayofyear" | "dayofweek" |
                              "hourofday" | "minuteofhour" | "secondofminute" ) "(" exp ")" ) |
               ( "todouble" "(" exp ")" ) |
               ( "tolong" "(" exp ")" ) |
               ( "tostring" "(" exp ")" ) |
               ( "toraw" "(" exp ")" ) |
               ( "uca" "(" exp "," string [ "," string ] ")" ) |
               ( "xor" "(" expList ")" ) |
               ( "xorbit" "(" exp "," number ")" ) |
               ( "zcurve" "." ( "x" | "y" ) "(" exp ")" ) |
               ( attributeName "." "at" "(" number ")") |
               ( attributeName )
bucket     ::= "bucket" ( "(" | "[" | "<" )
                        ( "-inf" | rawvalue | number | string )
                        [ "," ( "inf" | rawvalue | number | string ) ]
                        ( ")" | "]" | ">" )
rawvalue   ::= "{" ( ( string | number ) "," )* "}"