Filter intersections reference

This is the reference for FilterIntersectionsSearcher, which counts how many results a query would have with each named filter, and each combination of filters, added. Refer to the filter intersections guide for an introduction, setup and caveats.

Each combination of filters is called a cell and is counted by its own query.

Query parameters

All parameters are read by FilterIntersectionsSearcher and only have an effect when the searcher is in the search chain that handles the request.

ParameterTypeDefaultDescription
filterIntersections.filters String (JSON) none A JSON array of objects, each with a non-empty name and a non-empty YQL where expression. Names must be unique. When absent or an empty array, the searcher does nothing.
filterIntersections.dimensions Integer 2 Maximum number of filters combined in one cell. Must be at least 1. Values above the number of filters are clamped.
filterIntersections.maxCells Integer 1000 Maximum number of cells (and hence count queries) per request, inclusive. Must be at least 1. Requests exceeding it are rejected before reaching the content nodes.
filterIntersections.separator String & String joining filter names into a bucket's key, see Result format.

A filter's where accepts anything valid after the where keyword in a YQL query, including @parameter substitution. Some operators give lower-bound counts, see the caveats.

Result format

Added to the result as the field filterIntersections under root.fields when the request has filters. A rejected request also gets the field, with no buckets:

FieldTypeDescription
filterIntersections.buckets Array of objects One entry per cell that produced an exact count, ordered by number of filters, then by key.
buckets[].key String The filter names in this cell, sorted case-insensitively and joined with the separator.
buckets[].names Array of strings The filter names in this cell, sorted case-insensitively.
buckets[].totalCount Integer Number of documents matching the base query and all filters in names. Same accuracy as totalCount: exact, except a lower bound when weakAnd, wand, nearestNeighbor or a range with hitLimit is used.

How a cell query is built

A cell is a copy of the query with these changes:

  • The cell's filters are added with AND.
  • hits and offset are set to 0.
  • summary, grouping and soft timeout are turned off.

Everything else, including the rank profile, sources and any @parameter values, is kept. Filters are parsed separately and combined as query trees, never pasted into the YQL text, so a filter cannot change the meaning of the query.

In streaming mode, content nodes replace the rank profile with unranked for any query with zero hits and no grouping. Cells therefore run with unranked there, not the query's rank profile.

Number of cells

With \(n\) filters and \(d\) dimensions, the number of cells is:

\[\text{cells}(n, d) = \sum_{k=1}^{\min(d, n)} \binom{n}{k} = \sum_{k=1}^{\min(d, n)} \frac{n!}{k!\,(n-k)!}\]
Filtersdimensions=1dimensions=2dimensions=3
3367
551525
101055175
1414105469
20202101350

A request producing more than filterIntersections.maxCells cells is rejected with:

14 filters at 4 dimensions give 1470 cells, more than the default limit of 1000 (raise it with filterIntersections.maxCells)

Execution

  • All cells are submitted concurrently to the container's shared request thread pool, together with the original query. Latency is roughly that of the slowest cell while threads are free. On a saturated container, cells may run one after another and miss the deadline.
  • The original query runs in the same thread pool, so with a short timeout and many cells, it can time out too. The whole request then fails with a timeout error and no buckets.
  • Each cell may use up to the remaining time of the request's timeout.
  • Content node load grows linearly with the number of cells.
  • Since soft timeout is off, a cell either covers the full corpus or is reported as degraded. Degraded cells are omitted, never returned with a partial count. This includes cells limited by the rank profile's match-phase, whose counts are estimates.

Errors

SituationEffect
Filters JSON invalid, filter without name or where, duplicate name, invalid YQL in a where, dimensions or maxCells below 1, or cell count above maxCells Whole request fails with error code 4, INVALID_QUERY_PARAMETER, naming the problem, for example Filter 'broken': invalid YQL: .... Nothing is sent to the content nodes.
dimensions or maxCells not an integer Whole request fails with HTTP status 400, Bad request, naming the value, for example 'abc' is not a valid integer. Nothing is sent to the content nodes.
A cell's query returns an error from the content nodes That bucket is omitted. The error is added to root.errors, prefixed with Intersection cell '<key>':. Other buckets and the main result are unaffected.
A cell's query has degraded coverage (timeout, node down, match-phase) That bucket is omitted. An error with the coverage reached is added to root.errors, for example Intersection cell 'brand&instock': degraded coverage (50%), count not exact.
A filter matches nothing Not an error. Its cells get totalCount 0.
No filterIntersections.filters, or an empty array The searcher does nothing. No filterIntersections field in the result.