# Production tests

[](/en/reference/applications/testing-production.html.md "View as Markdown") 

This is the Vespa Testing reference for [production tests](../../applications/testing.html).

Production tests verify domain specific metrics for a production deployment, such as query latency or error rate. Unlike [system](testing.html) and [staging](testing-java.html) tests, production tests check a named [metric preset](#metric-presets) against expected bounds, using Grafana metrics for the application deployed to that zone.

A production test is placed after a deployment zone in [deployment.xml](deployment.html). It acts as a gate for the rollout: if it fails, the rollout stops and subsequent zones will not receive the new version. This is useful in multi-zone deployments where the first zone serves as a canary, but a production test can validate _any_ zone before the rollout proceeds to the next, as described in [production tests](../../operations/automated-deployments.html#production-tests).

## Production test setup

To use production tests, first deploy to a zone, use [\<delay\>](deployment.html#delay) for some time, and then run the production tests against that zone. This pattern can be repeated for each zone in the pipeline, to validate it before the rollout proceeds to the next. In the following example, the zone is deployed, then the delay waits 10 minutes, and lastly the production test waits for an additional 10 minutes before aggregating the last 10 minutes of metrics data:

```
```
<prod>
  <region>aws-us-east-1c</region>
  <delay minutes="10" />
  <test>aws-us-east-1c</test>
  <!-- Deploy to the rest of the zones... -->
</prod>
```
```

The declared production test:

```
```
name: cpu check
metric: cpu-utilization-container
duration: 10m
max: 85
```
```

 **Note:** Tests based on traffic metrics, such as query or feed latency, need real queries or feeding against the zone during the test window, since without traffic there are no metrics to evaluate and the test will fail.

## Test suites

Production tests are placed under `tests/production-test/` in the application package, as one or more `.yaml` or `.json` files:

```
$ ls tests/production-test/

tests/production-test/metrics-test.yaml
```

Each file describes one or more tests. A file may contain:

- a single test, as one YAML or JSON object; or
- multiple tests, as a YAML or JSON object with a `tests` field containing a list of test objects.

## Test file structure

Each test checks a named [metric preset](#metric-presets), over a `duration` window, against a `min` and/or `max` bound. A simple example:

```
```
name: cpu check
metric: cpu-utilization-container
duration: 5m
max: 85
```
```

This test fails the production test step if the average CPU utilization of the container cluster, over the last 5 minutes, exceeds 85%.

To place multiple tests in one file, wrap them in a top-level `tests` field, containing a list of test objects. This works the same way for both YAML and JSON:

```
```
tests:
  - name: container cpu check
    metric: cpu-utilization-container
    duration: 5m
    max: 35

  - name: content cpu check
    metric: cpu-utilization-content
    duration: 5m
    max: 85
```
```

### Test specification

A full list of fields, with description:

| Name | Type | Default | Description |
| --- | --- | --- | --- |
| name | string | "\<unnamed test\>" | Name used for display purposes in logs and failure messages. |
| metric | string | | Required. The name of one of the [metric presets](#metric-presets). |
| duration | number + granularity suffix | | Required. The duration of the metrics aggregation. Suffix is one of `s|m|h|d`. For example, `duration: 10m` means: first wait 10 minutes, then aggregate the last 10 minutes of data. Tests can have different durations. Since tests run concurrently, the total waiting time is determined by the highest `duration` value, not the sum. |
| min | number | | The inclusive lower bound the query result must satisfy. At least one of `min` and `max` is required. The unit depends on the [metric preset](#metric-presets) used, see its Unit column. |
| max | number | | The inclusive upper bound the query result must satisfy. At least one of `min` and `max` is required. The unit depends on the [metric preset](#metric-presets) used, see its Unit column. |

A test that is outside its `min`/`max` bounds fails the production test step immediately — production tests are not retried.

## Examples

A single test, in YAML:

```
```
name: error rate check
metric: error-rate-pct
duration: 5m
max: 1
```
```

Multiple tests in a single YAML file:

```
```
tests:
  - name: node count check
    metric: node-count-max
    duration: 5m
    max: 10

  - name: documents count check
    metric: documents-count-active
    duration: 5m
    min: 1
```
```

A single test, in JSON:

```
```
{
    "name": "query latency check",
    "metric": "query-latency-p95",
    "duration": "5m",
    "max": 200
}
```
```

Multiple tests in a single JSON file:

```
```
{
    "tests": [
        {
            "name": "container feed latency check",
            "metric": "container-feed-latency-max",
            "duration": "5m",
            "max": 100
        },
        {
            "name": "content commit latency check",
            "metric": "content-commit-latency-avg",
            "duration": "5m",
            "max": 50
        }
    ]
}
```
```

## Metric presets

Every test checks one of the following named presets, in its `metric` field. Each preset queries the same metrics that back the application's overview dashboard, reduced to a single number. Presets are grouped below by the dashboard tab they come from.

The **Console name** column is a direct match to the graph title shown in that tab in the [Vespa Console](https://console.vespa-cloud.com/)'s metrics view. Use it to find the preset for a graph you're looking at there. The **Metric** column is the exact value to use in the `metric` field, e.g. `metric: query-latency-p95`.

### Overview

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Buckets Out of Sync (%) | `buckets-out-of-sync` | percentage (0-100) | Percentage of buckets currently out of sync, fleet-wide. |
| Container: % Nodes Down | `container-nodes-down` | percentage (0-100) | Percentage of active container nodes with at least one service down, fleet-wide. |
| Container Thread Saturation — document-api only | `container-thread-saturation-document-api-only` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running document-api only (no search). |
| Container Thread Saturation — search + document-api | `container-thread-saturation-search-document-api` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running both search and document-api. |
| Container Thread Saturation — search only | `container-thread-saturation-search-only` | percentage (0-100) | Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running search only (no document-api). |
| Content Executor Saturation | `content-executor-saturation` | percentage (0-100) | Worst-case content-node executor utilization/saturation across the match, docsum, and field-writer executors, fleet-wide. |
| Content: Groups/Nodes Down | `content-groups-nodes-down` | count | Fleet-wide count of cluster groups currently reporting at least one down node. |
| Core Dumps (1h) | `core-dumps` | count | Total number of core dumps processed across content clusters during the time window. |
| Disk Utilization (max per cluster) | `disk-utilization-container` | percentage (0-100) | Maximum disk utilization across nodes in container clusters (logs, working files). |
| Disk Utilization (max per cluster) | `disk-utilization-content` | percentage (0-100) | Maximum disk utilization across nodes in content clusters. |
| Documents per Content Cluster (max) | `documents-per-content-cluster-max` | count | Maximum number of unique documents stored, aggregated across content clusters. |
| Feed Blocked | `feed-blocked` | count | Total number of nodes currently above the feed-block resource limit, across all content clusters. |
| Feed Latency (max) | `feed-latency-max` | milliseconds | Maximum feed latency observed (OK status only). |
| Headroom to Feed Block (per content cluster) | `headroom-to-feed-block-per-content-cluster` | percentage (0-100) | Smallest remaining headroom before the feed-block limit, across memory and disk, fleet-wide. |
| HTTP 2xx Responses (sum) | `http-2xx-responses-sum` | requests/second | Rate of successful (2xx) HTTP responses across the container cluster. |
| HTTP 4xx Responses (sum) | `http-4xx-responses-sum` | requests/second | Rate of client error (4xx) HTTP responses across the container cluster. |
| HTTP 5xx Responses (sum) | `http-5xx-responses-sum` | requests/second | Rate of server error (5xx) HTTP responses across the container cluster. |
| HTTP Read Latency (avg) | `http-read-latency-avg` | milliseconds | Average HTTP-level read latency, including time to receive the request and send the response. |
| HTTP Read Latency (max) | `http-read-latency-max` | milliseconds | Maximum HTTP-level read latency observed. |
| HTTP Read Latency (p95) | `http-read-latency-p95` | milliseconds | 95th percentile HTTP-level read latency. |
| HTTP Read Latency (p99) | `http-read-latency-p99` | milliseconds | 99th percentile HTTP-level read latency. |
| HTTP Write Latency (max) | `http-write-latency-max` | milliseconds | Maximum HTTP-level write latency observed. |
| JVM Heap Pressure | `jvm-heap-pressure` | percentage (0-100) | Fleet-wide average JVM heap utilization (heap used / heap capacity) across container clusters. |
| Memory Utilization (max per cluster) | `memory-utilization-node-container` | percentage (0-100) | Maximum OS-level memory utilization across nodes in container clusters. |
| Memory Utilization (max per cluster) | `memory-utilization-node-content` | percentage (0-100) | Maximum OS-level memory utilization across nodes in content clusters. |
| QoS - Read (success %) | `qos-read-success` | percentage (0-100) | Percentage of read requests that did not return a 5xx server error. |
| QoS - Write (success %) | `qos-write-success` | percentage (0-100) | Percentage of write requests that did not return a 5xx server error. |
| Query Latency (avg) | `query-latency-avg` | milliseconds | Average query latency across the container cluster (total latency divided by request count). |
| Query Latency (max) | `query-latency-max` | milliseconds | Maximum query latency observed across the container cluster. |
| Restarts (1h) | `restarts` | count | Total number of Vespa service restarts across the fleet during the time window. |

### Query

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Degraded Queries (sum) | `degraded-queries-sum` | requests/second | Rate of queries that were degraded (soft-doomed or match-limited), summed across the container cluster. |
| Docs matched per query per rank profile | `docs-matched-per-query-per-rank-profile` | count | Average documents matched per query, aggregated across all rank profiles - a proxy for query selectivity. |
| Docsum executor accepted (rate) | `docsum-executor-accepted-rate` | operations/second | Average rate of docsum tasks accepted by the per-node docsum executor across content nodes. |
| Docsum executor queue size (max) | `docsum-executor-queue-size-max` | count | Peak length of the per-node docsum thread-pool queue on any single content node. |
| Docsum latency | `docsum-latency-avg` | seconds | Average summary-rendering (docsum) latency across content nodes, in seconds. |
| Docsum latency | `docsum-latency-max` | seconds | Peak summary-rendering (docsum) latency observed on any content node, in seconds. |
| Document summaries requested (rate) | `document-summaries-requested-rate` | requests/second | Document summaries requested per second across content nodes, derived from the docsum-latency sample count. |
| Documents Coverage | `documents-coverage` | percentage (0-100) | Percentage of the document corpus actually covered by search (documents covered vs. documents total). |
| Documents Matched (rate) | `documents-matched-rate` | operations/second | Documents matched per second across the content cluster, summed across all content nodes and rank profiles. |
| Empty Results (sum) | `empty-results-sum` | requests/second | Rate of queries returning zero results, summed across the container cluster. |
| Failed Queries (sum) | `failed-queries-sum` | requests/second | Rate of queries that failed, summed across the container cluster. |
| Grouping time per rank profile (avg) | `grouping-time-per-rank-profile-avg` | milliseconds | Average post-match time per query (result construction, grouping, packing), aggregated across all rank profiles and content nodes. |
| Grouping time per rank profile (peak) | `grouping-time-per-rank-profile-peak` | milliseconds | Peak post-match (grouping/result-construction) time observed in any snapshot interval, aggregated across all rank profiles. |
| Hits per Query (avg) | `hits-per-query-avg` | count | Average number of hits returned per query, weighted across the container cluster. |
| Match executor accepted (rate) | `match-executor-accepted-rate` | operations/second | Average rate of tasks accepted into the match executor across content nodes. |
| Match executor queue size (max) | `match-executor-queue-size-max` | count | Peak match thread-pool queue depth on any single content node. |
| Match Executor Utilization | `match-executor-utilization-avg` | percentage (0-100) | Average match-thread-pool utilization across content nodes, as a percentage. |
| Match Executor Utilization | `match-executor-utilization-max` | percentage (0-100) | Peak match-thread-pool utilization on any single content node, as a percentage. |
| Matching Queries (rate) | `matching-queries-rate` | requests/second | Per-node-average rate of queries being matched on content nodes. |
| Matching Query Latency (avg) | `matching-query-latency-avg` | milliseconds | Average matching-phase query latency across all content nodes and rank profiles. |
| Matching Query Setup Time (avg) | `matching-query-setup-time-avg` | milliseconds | Average per-query setup overhead (query decode, blueprint build, rank setup) on content nodes. |
| Queries per second by content group (min) | `queries-per-second-by-content-group-min` | requests/second | Lowest per-content-group query rate observed, deliberately using min to catch a failed node emitting stale/frozen metric values. |
| Queries per second per rank profile | `queries-per-second-per-rank-profile` | requests/second | Average per-(node, rank-profile) query rate observed at the content nodes. |
| Query Container Latency (avg) | `query-container-latency-avg` | milliseconds | Time spent inside the container processing the query, excluding backend dispatch. |
| Query Error Breakdown (sum) | `query-error-breakdown-backend-comm` | requests/second | Rate of queries that failed due to backend (content-node) communication errors. |
| Query Error Breakdown (sum) | `query-error-breakdown-empty-docsums` | requests/second | Rate of queries that failed because document summaries could not be retrieved. |
| Query Error Breakdown (sum) | `query-error-breakdown-invalid-param` | requests/second | Rate of queries that failed due to an invalid query parameter. |
| Query Error Breakdown (sum) | `query-error-breakdown-timeout` | requests/second | Rate of queries that failed due to a timeout error. |
| Query Error Breakdown (sum) | `query-error-breakdown-unhandled` | requests/second | Rate of queries that failed due to an unhandled exception in the container. |
| Query Latency (p95) | `query-latency-p95` | milliseconds | Average 95th percentile query latency. |
| Query Latency (p99) | `query-latency-p99` | milliseconds | Average 99th percentile query latency. |
| Query Rate (QPS) | `query-rate-qps-avg` | requests/second | Total application-wide query rate, summed across all container instances. |
| Query Rate (QPS) | `query-rate-qps-peak` | requests/second | Highest per-host peak queries-per-second observed across the container cluster. |
| Query Timeout (p99) | `query-timeout-p99` | milliseconds | 99th percentile of the configured per-query timeout values actually applied by clients. |
| Rank Profile Query Latency (max) | `rank-profile-query-latency-max` | milliseconds | Peak per-query matching latency on content nodes, taken across all rank profiles combined. |
| Rerank time per rank profile (avg) | `rerank-time-per-rank-profile-avg` | milliseconds | Average second-phase rerank time per query, aggregated across all rank profiles and content nodes (no non-rank-profile-labeled equivalent metric exists to prefer instead). |
| Rerank time per rank profile (peak) | `rerank-time-per-rank-profile-peak` | milliseconds | Peak second-phase rerank time observed in any snapshot interval, aggregated across all rank profiles. |
| search-handler — Utilization | `search-handler-utilization-avg` | percentage (0-100) | Average utilization of the container's search-handler thread pool. |
| search-handler — Utilization | `search-handler-utilization-max` | percentage (0-100) | Peak utilization of the container's search-handler thread pool on any single host. |
| Search Protocol Query Latency (avg) | `search-protocol-query-latency-avg` | milliseconds | Average query latency measured at the search-protocol level on content nodes. |
| Search Protocol Query Latency (max) | `search-protocol-query-latency-max` | milliseconds | Maximum query latency measured at the search-protocol level on any content node. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-avg` | count | Average adaptive soft-timeout multiplier per query, aggregated across all rank profiles. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-max` | count | Highest adaptive soft-timeout multiplier observed for any rank profile. |
| Soft doom factor per rank profile | `soft-doom-factor-per-rank-profile-min` | count | Lowest adaptive soft-timeout multiplier observed for any rank profile. |
| Soft-doomed queries per rank profile (sum) | `soft-doomed-queries-per-rank-profile-sum` | requests/second | Rate of queries that hit the soft timeout, summed across all rank profiles and content nodes. |
| Rank Profile Query Latency (avg) | `rank-profile-query-latency-avg` | milliseconds | Average per-query matching latency on content nodes, taken across all rank profiles combined. |
| Query setup time per rank profile (avg) | `query-setup-time-per-rank-profile-avg` | milliseconds | Average per-query setup overhead (query decode, blueprint build, rank setup, plus post-match finalisation) per rank profile. |

### Feed

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Container Feed Latency (max) | `container-feed-latency-max` | milliseconds | Maximum feed request latency observed at the container, across all operation types and clusters. |
| Container Feed Operations (sum) | `container-feed-operations-sum` | operations/second | Total feed operation rate (puts, updates, removes) handled by container clusters, summed across operation types. |
| Content: Commit Latency (avg) | `content-commit-latency-avg` | milliseconds | Average commit latency (time to commit fed changes to the index/store) on content nodes. |
| Content: Commit Operations (sum) | `content-commit-operations-sum` | operations/second | Rate of commit operations (batched writes applied to the index/store) on content nodes. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-put` | operations/second | Rate of PUT operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-remove` | operations/second | Rate of REMOVE operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Feed Rates (sum) | `content-storage-feed-rates-update` | operations/second | Rate of UPDATE operations processed by the storage/persistence-engine layer on content nodes, counted per document replica. |
| Content: Storage Put Latency (max) | `content-storage-put-latency-max` | milliseconds | Maximum PUT operation latency observed at the storage/persistence-engine layer on content nodes. |
| Content: Storage Update Latency | `content-storage-update-latency-avg` | milliseconds | Average UPDATE operation latency at the storage/persistence-engine layer on content nodes. |
| Content: Storage Update Latency | `content-storage-update-latency-max` | milliseconds | Maximum UPDATE operation latency observed at the storage/persistence-engine layer on content nodes. |
| Distributor Latency - gets (max) | `distributor-latency-gets-max` | milliseconds | Maximum get operation latency observed at the distributor level. |
| Distributor Latency - puts (max) | `distributor-latency-puts-max` | milliseconds | Maximum put operation latency observed at the distributor level. |
| Distributor Latency - removes (max) | `distributor-latency-removes-max` | milliseconds | Maximum remove operation latency observed at the distributor level. |
| Distributor Latency - updates (max) | `distributor-latency-updates-max` | milliseconds | Maximum update operation latency observed at the distributor level. |
| Distributor Latency - visitor (max) | `distributor-latency-visitor-max` | milliseconds | Maximum visitor operation latency observed at the distributor level. |
| Distributor Operation - gets - Failures (sum) | `distributor-operation-gets-failures-sum` | operations/second | Rate of failed get operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - puts - Failures (sum) | `distributor-operation-puts-failures-sum` | operations/second | Rate of failed put operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation Rates - gets - OK (sum) | `distributor-operation-rates-gets-ok-sum` | operations/second | Rate of successfully completed (OK) get operations at the distributor level. |
| Distributor Operation Rates - puts - OK (sum) | `distributor-operation-rates-puts-ok-sum` | operations/second | Rate of successfully completed (OK) put operations at the distributor level. |
| Distributor Operation Rates - removes - OK (sum) | `distributor-operation-rates-removes-ok-sum` | operations/second | Rate of successfully completed (OK) remove operations at the distributor level. |
| Distributor Operation Rates - updates - OK (sum) | `distributor-operation-rates-updates-ok-sum` | operations/second | Rate of successfully completed (OK) update operations at the distributor level. |
| Distributor Operation Rates - visitor - OK (sum) | `distributor-operation-rates-visitor-ok-sum` | operations/second | Rate of successfully completed (OK) visitor operations at the distributor level. |
| Distributor Operation - removes - Failures (sum) | `distributor-operation-removes-failures-sum` | operations/second | Rate of failed remove operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - updates - Failures (sum) | `distributor-operation-updates-failures-sum` | operations/second | Rate of failed update operations at the distributor level; non-zero values indicate issues with writes. |
| Distributor Operation - visitor - Failures (sum) | `distributor-operation-visitor-failures-sum` | operations/second | Rate of failed visitor operations at the distributor level; non-zero values indicate issues with writes. |
| Document Processing Latency (avg) | `document-processing-latency-avg` | milliseconds | Average document processing (docproc chain) latency. |
| Document Processing Rate (sum) | `document-processing-rate-sum` | operations/second | Rate of documents processed through document processing (docproc) chains, summed across all chains and clusters. |
| Feed Blocked (nodes above resource limit) | `feed-blocked-nodes-above-resource-limit` | count | Number of content nodes currently exceeding a resource usage limit; when greater than zero, feeding is blocked for that cluster. |
| Feed Latency (avg) | `feed-latency-avg-ms` | milliseconds | Average feed latency, in milliseconds, for successful feed operations. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-put` | operations/second | Rate of PUT operations received via the Document/HTTP API. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-remove` | operations/second | Rate of REMOVE operations received via the Document/HTTP API. |
| HTTP API Feed Rates (sum) | `http-api-feed-rates-update` | operations/second | Rate of UPDATE operations received via the Document/HTTP API. |
| HTTP API Latency (avg) | `http-api-latency-avg` | milliseconds | Average latency of Document/HTTP API requests, distinct from the generic jdisc HTTP layer latency. |
| HTTP API Pending Requests (max) | `http-api-pending-requests-max` | count | Maximum number of Document/HTTP API requests pending (in-flight, not yet completed) on the container. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-failed` | operations/second | Rate of failed Document/HTTP API operations. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-parse-error` | operations/second | Rate of Document/HTTP API operations rejected due to document parse errors. |
| HTTP API Success vs Failures (sum) | `http-api-success-vs-failures-success` | operations/second | Rate of successful Document/HTTP API operations. |
| HTTP Write Latency (avg) | `http-write-latency-avg` | milliseconds | Average end-to-end HTTP write request latency at the container's HTTP layer. |
| HTTP Write Latency (p95) | `http-write-latency-p95` | milliseconds | 95th percentile HTTP write request latency at the container's HTTP layer. |
| HTTP Write Latency (p99) | `http-write-latency-p99` | milliseconds | 99th percentile HTTP write request latency at the container's HTTP layer. |
| Memory Index — Documents (per document type) | `memory-index-docs-in-memory-max` | count | Maximum number of documents held in the in-memory index on content nodes, summed per content group (to combine per-node/stripe values) and maxed across groups (since each group holds a full replica) to avoid double-counting. |
| Persistence engine input queue | `persistence-engine-input-queue-avg` | count | Average depth of the per-node persistence engine's input operation queue (operations waiting to be dispatched to Proton) over the time window. |
| Persistence engine input queue | `persistence-engine-input-queue-max` | count | Maximum depth of the per-node persistence engine's input operation queue observed during the time window. |
| Persistence engine throttle saturation (max) | `persistence-engine-throttle-saturation-active` | count | Maximum number of concurrently active (in-flight) operations in the content node's persistence engine. |
| Persistence engine throttle saturation (max) | `persistence-engine-throttle-saturation-throttle` | count | Maximum size of the adaptive throttle window that caps concurrent operations in the content node's persistence engine. |

### Nearest Neighbor Search

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| ANN Timeout Rate (sum) | `ann-timeout-rate-sum` | requests/second | Rate of queries per second that terminated an approximate NNS search early after hitting the ANN timeout. |
| Approximate NNS Distances Computed (rate) | `approximate-nns-distances-computed-rate` | operations/second | Rate of distance computations performed during approximate NNS across the content cluster. |
| Approximate NNS Nodes Visited (rate) | `approximate-nns-nodes-visited-rate` | operations/second | Rate of HNSW graph nodes visited during approximate NNS across the content cluster. |
| Query Rate of Approximate Nearest Neighbor Search Queries | `approximate-nns-query-rate` | requests/second | Rate of incoming queries per second that perform an approximate nearest-neighbor search. |
| Query Approximate Nearest Neighbor Search Time | `approximate-nns-time-avg` | milliseconds | Average time spent performing approximate NNS, measured only across queries that actually perform it. |
| Query Approximate Nearest Neighbor Search Time | `approximate-nns-time-max` | milliseconds | Maximum time observed for a single query's approximate NNS work, on any content node. |
| Approximate NNS Visit Efficiency (nodes visited / distances computed) | `approximate-nns-visit-efficiency` | ratio (dimensionless) | Ratio of HNSW graph nodes visited to distances computed during approximate NNS; values above 1.0 indicate the filter-first (Acorn-1) heuristic is active. |
| Buckets Pending Merge (sum) | `buckets-pending-merge-sum` | count | Number of buckets currently pending merge across distributors, which rises during data redistribution or node recovery. |
| Documents Active (per document type) | `documents-active-total` | count | Total number of active documents (primary copies) across all document types and content nodes in the cluster. |
| Documents Ready (per document type) | `documents-ready-total` | count | Total number of ready (searchable) documents across all document types and content nodes in the cluster. |
| Exact NNS Distance Computation Ratio (exact / total) | `exact-nns-distance-ratio-pct` | percentage (0-100) | Share of NNS distance computations done via exact (brute-force) search rather than approximate search; high values may indicate the approximate-threshold is set too high. |
| Exact NNS Distances Computed (rate) | `exact-nns-distances-computed-rate` | operations/second | Rate of distance computations performed during exact (brute-force) NNS across the content cluster. |
| Query Latency | `nns-query-latency-avg` | milliseconds | Average total time (matching plus ranking) spent per query on the content nodes, specific to the NNS/matching pipeline. |
| Query Latency | `nns-query-latency-max` | milliseconds | Maximum total time (matching plus ranking) observed for a single query on any content node, specific to the NNS/matching pipeline. |
| Query Rate of Non-Approximate Nearest Neighbor Search Queries | `non-approximate-nns-query-rate` | requests/second | Rate of incoming queries per second that do not perform approximate NNS, including queries that fall back to exact search. |
| Query Matching Time | `query-matching-time-avg` | milliseconds | Average time spent on the core query matching process, computed as query latency minus query setup time; includes exact NNS. |
| Query Matching Time | `query-matching-time-max` | milliseconds | Approximate maximum time spent on the core query matching process, computed as max query latency minus max query setup time (not a directly measured value, since the two maxima may occur on different queries). |
| Query Rate | `query-rate` | requests/second | Rate of incoming queries per second against the content cluster's document matching pipeline (equal to approximate-nns-query-rate plus non-approximate-nns-query-rate). |
| Query Setup Time | `query-setup-time-avg` | milliseconds | Average time spent setting up and tearing down queries, including filter computation and approximate NNS overhead. |
| Query Setup Time Without Approximate Nearest Neighbor Search Time | `query-setup-time-excl-ann-avg` | milliseconds | Average time spent on query setup and teardown excluding approximate NNS time, isolating filter computation and other overhead. |
| Query Setup Time | `query-setup-time-max` | milliseconds | Maximum observed time spent setting up and tearing down a single query on any content node. |
| Soft Timeout Rate (sum) | `soft-timeout-rate-sum` | requests/second | Rate of queries per second terminated early due to hitting the soft timeout, aggregated across rank profiles. |
| Thread Pool - Match Utilization | `thread-pool-match-utilization-avg` | percentage (0-100) | Average utilization of the match thread pool across content nodes. |
| Thread Pool - Match Utilization | `thread-pool-match-utilization-max` | percentage (0-100) | Peak utilization of the match thread pool observed on any content node. |
| Total Distances Computed (sum) | `total-distances-computed-sum` | operations/second | Total NNS distance computations per second, combining exact and approximate search, as an overall indicator of NNS-related CPU cost. |

### Content Node

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Attribute Resource Usage - Feeding Blocked (max) | `attribute-feeding-blocked-pct` | percentage (0-100) | Percentage of the time window during which feeding was blocked due to attribute resource usage limits on any content node. |
| Bucket Move Pending (max) | `bucket-move-pending-max` | count | Maximum number of buckets pending a move operation (e.g. during cluster resizing or redistribution) on any content node. |
| Document Store Cache Hit Rate (max) | `document-store-cache-hit-rate-max` | percentage (0-100) | Cache hit rate for the ready document store's summary-field cache, as a percentage; low values on quiescent clusters often show up as elevated CPU IOWait from disk-backed docsum reads. |
| Document Store Disk Usage (max) | `document-store-disk-usage-max` | bytes | Maximum disk space used by the ready document store on any content node. |
| Documents Active (per document type) | `documents-count-active` | count | Number of active/searchable document copies, summed across content clusters after taking the max across replica groups within each cluster to avoid double counting. |
| Documents Ready (per document type) | `documents-count-ready` | count | Number of documents available for search (ready), summed across content clusters after taking the max across replica groups within each cluster to avoid double counting. |
| Field Writer Utilization & Saturation | `field-writer-saturation-max` | percentage (0-100) | Maximum observed saturation of Proton's field writer executor thread pool; high saturation indicates a write bottleneck. |
| Field Writer Utilization & Saturation | `field-writer-utilization-avg` | percentage (0-100) | Average utilization of Proton's field writer executor thread pool across content nodes. |
| Filestor Average Queue Wait (avg) | `filestor-average-queue-wait-avg` | milliseconds | Average time operations spent waiting in the filestor queue across content nodes. |
| Filestor Queue Size (max) | `filestor-queue-size-max` | count | Maximum filestor queue size observed on any content node; a growing queue indicates storage cannot keep up with incoming operations. |
| Maintenance Job Activity (max) | `maintenance-job-activity-attr-flush` | percentage (0-100) | Activity level of the attribute-flush maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-disk-idx-fusion` | percentage (0-100) | Activity level of the disk-index-fusion maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-docstore-compact` | percentage (0-100) | Activity level of the document-store-compaction maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-lid-compact` | percentage (0-100) | Activity level of the LID (local document id) space-compaction maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Maintenance Job Activity (max) | `maintenance-job-activity-mem-idx-flush` | percentage (0-100) | Activity level of the memory-index-flush maintenance job (0% = idle, 100% = fully active), maximum across content nodes. |
| Proton Disk Usage (avg) | `proton-disk-usage-avg` | percentage (0-100) | Relative disk usage of the content node's Proton process, as a percentage of the available disk; 100% means feeding is blocked. |
| Proton Executor Utilization | `proton-executor-utilization-avg` | percentage (0-100) | Average utilization of Proton's own executor thread pool across content nodes. |
| Proton Executor Utilization | `proton-executor-utilization-max` | percentage (0-100) | Maximum observed utilization of Proton's own executor thread pool on any content node. |
| Proton Memory Usage (avg) | `proton-memory-usage-avg` | percentage (0-100) | Relative memory usage of the content node's Proton process, as a percentage of the available memory. |
| Shared Executor Queue Size (max) | `shared-executor-queue-size-max` | count | Maximum shared executor queue size observed on any content node during the time window. |
| Shared Executor Utilization | `shared-executor-utilization-avg` | percentage (0-100) | Average utilization of Proton's shared executor thread pool across content nodes. |
| Shared Executor Utilization | `shared-executor-utilization-max` | percentage (0-100) | Maximum observed utilization of Proton's shared executor thread pool on any content node. |

### Resources

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| CPU IOWait (avg/max/min) | `cpu-iowait-avg` | percentage (0-100) | Average per-host CPU I/O-wait percentage across the cluster over the time window, indicating time spent waiting on disk I/O rather than doing CPU work. |
| CPU IOWait (avg/max/min) | `cpu-iowait-max` | percentage (0-100) | Maximum per-host CPU I/O-wait percentage across the cluster over the time window; high values point to disk being the bottleneck rather than CPU. |
| CPU IOWait (avg/max/min) | `cpu-iowait-min` | percentage (0-100) | Minimum per-host CPU I/O-wait percentage across the cluster over the time window. |
| CPU Throttled Time (sum) | `cpu-throttled-time-sum` | count | Sum of per-node CPU cgroup-throttling rate across the cluster over the time window; non-zero values indicate nodes are hitting their CPU resource limits. |
| CPU Utilization (max per cluster) | `cpu-utilization-container` | percentage (0-100) | Maximum CPU utilization across the nodes of the container cluster. |
| CPU Utilization (max per cluster) | `cpu-utilization-content` | percentage (0-100) | Maximum CPU utilization across the nodes of the content cluster. |
| default-handler-common — Utilization | `default-handler-common-utilization-avg` | percentage (0-100) | Average active threads as a percentage of pool size for the default-handler-common thread pool over the time window. |
| default-handler-common — Utilization | `default-handler-common-utilization-max` | percentage (0-100) | Peak active threads as a percentage of pool size for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Size | `default-handler-common-work-queue-size-avg` | count | Average number of pending tasks queued for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Size | `default-handler-common-work-queue-size-max` | count | Peak number of pending tasks queued for the default-handler-common thread pool observed during the time window. |
| default-handler-common — Work Queue Utilization | `default-handler-common-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the default-handler-common thread pool over the time window. |
| default-handler-common — Work Queue Utilization | `default-handler-common-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the default-handler-common thread pool observed during the time window. |
| Disk Utilization (avg/max/min) | `disk-utilization-avg` | percentage (0-100) | Average OS-level disk utilization percentage across the cluster's nodes. |
| Disk Utilization (avg/max/min) | `disk-utilization-max` | percentage (0-100) | Maximum OS-level disk utilization percentage across the cluster's nodes; sustained high values risk feed/query slowdowns from disk saturation. |
| Disk Utilization (avg/max/min) | `disk-utilization-min` | percentage (0-100) | Minimum OS-level disk utilization percentage across the cluster's nodes. |
| feedapi-handler — Utilization | `feedapi-handler-utilization-avg` | percentage (0-100) | Average active threads as a percentage of pool size for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Utilization | `feedapi-handler-utilization-max` | percentage (0-100) | Peak active threads as a percentage of pool size for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Size | `feedapi-handler-work-queue-size-avg` | count | Average number of pending tasks queued for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Size | `feedapi-handler-work-queue-size-max` | count | Peak number of pending tasks queued for the feedapi-handler thread pool observed during the time window. |
| feedapi-handler — Work Queue Utilization | `feedapi-handler-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the feedapi-handler thread pool over the time window. |
| feedapi-handler — Work Queue Utilization | `feedapi-handler-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the feedapi-handler thread pool observed during the time window. |
| GPU Memory Utilization (max) | `gpu-memory-utilization-max` | percentage (0-100) | Maximum GPU memory utilization (used/total) across GPU-equipped nodes, averaged over the time window. |
| GPU Utilization (max) | `gpu-utilization-container-max` | percentage (0-100) | Maximum GPU utilization across container-cluster nodes with GPUs attached, averaged over the time window (only produces data for GPU-equipped nodes). |
| JVM Direct Memory (avg) | `jvm-direct-memory-capacity` | bytes | Average configured JVM direct (off-heap) memory capacity on container nodes. |
| JVM Direct Memory (avg) | `jvm-direct-memory-used` | bytes | Average JVM direct (off-heap) memory currently in use on container nodes. |
| JVM GC Overhead (max) | `jvm-gc-overhead-max` | percentage (0-100) | Approximate percentage of CPU time spent in JVM garbage collection on container nodes, averaged over the time window. |
| JVM GC Pause Duration | `jvm-gc-pause-duration-avg` | seconds | Average stop-the-world JVM GC pause duration on container nodes over the time window. |
| JVM GC Pause Duration | `jvm-gc-pause-duration-max` | seconds | Worst single stop-the-world JVM GC pause observed on container nodes during the time window. |
| JVM Heap Usage (avg) | `jvm-heap-usage-capacity` | bytes | Average configured JVM heap memory capacity (max heap size) on container nodes. |
| JVM Heap Usage (avg) | `jvm-heap-usage-used` | bytes | Average JVM heap memory currently in use on container nodes. |
| JVM Native Memory (avg) | `jvm-native-memory-avg` | bytes | Average native (off-JVM) memory used by C++ libraries loaded via JNI (e.g. ONNX Runtime for embedders, linguistics) on container nodes. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-avg` | percentage (0-100) | Average OS-level memory utilization across the cluster's nodes (physical RAM usage), distinct from the Vespa-computed content memory-utilization metric. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-max` | percentage (0-100) | Maximum OS-level memory utilization across the cluster's nodes (physical RAM usage), distinct from the Vespa-computed content memory-utilization metric. |
| Memory Utilization (avg/max/min) | `memory-node-utilization-min` | percentage (0-100) | Minimum OS-level memory utilization across the cluster's nodes (physical RAM usage). |
| Resource Usage: Max Memory Utilization vs Limit | `memory-utilization-content` | percentage (0-100) | Maximum memory utilization of the content cluster's cluster controller. |
| Network Throughput (bytes received/sent) | `network-throughput-bytes-received-sent-received` | bytes | Average per-interval network bytes received by container nodes, summed across the cluster over the time window. |
| Network Throughput (bytes received/sent) | `network-throughput-bytes-received-sent-sent` | bytes | Average per-interval network bytes sent by container nodes, summed across the cluster over the time window. |
| Open Server Connections (max) | `open-server-connections-max` | count | Peak number of open HTTP server connections on container nodes observed during the time window. |
| Requests per HTTP Connection | `requests-per-http-connection-avg` | count | Average number of HTTP requests served per connection over its lifetime on container nodes during the time window. |
| Requests per HTTP Connection | `requests-per-http-connection-max` | count | Peak number of HTTP requests served by a single connection over its lifetime on container nodes during the time window. |
| search-handler — Work Queue Size | `search-handler-work-queue-size-avg` | count | Average number of pending tasks queued for the search-handler thread pool over the time window. |
| search-handler — Work Queue Size | `search-handler-work-queue-size-max` | count | Peak number of pending tasks queued for the search-handler thread pool observed during the time window. |
| search-handler — Work Queue Utilization | `search-handler-work-queue-utilization-avg` | percentage (0-100) | Average queue fill percentage (size / capacity) for the search-handler thread pool over the time window. |
| search-handler — Work Queue Utilization | `search-handler-work-queue-utilization-max` | percentage (0-100) | Peak queue fill percentage (size / capacity) for the search-handler thread pool observed during the time window. |

### Health

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Cluster State Changes (sum) | `cluster-state-changes-sum` | count | Number of cluster state changes observed in the given time window. Frequent changes are most often caused by Vespa upgrades or required node restarts. |
| Core Dumps (1h) | `core-dumps-processed-max` | count | Number of core dumps processed across all nodes in the given time window; should be 0. |
| Deactivated Containers | `deactivated-containers` | count | Average number of deactivated container instances per sample, over the given time window. Brief spikes during deployments are normal; persistently high values may indicate resource leaks. |
| Estimated Time to In-Sync | `estimated-time-to-in-sync` | seconds | Rough estimate, from the recent (fixed 30-minute) convergence rate, of the time remaining until all buckets are back in sync; capped at 7 days. |
| Failed Component Graphs | `failed-component-graphs` | operations/second | Rate of failed application component-graph reconfigurations, averaged over the given time window; nonzero after a deployment may indicate a configuration or component issue. |
| HTTP Requests Prematurely Closed | `http-requests-prematurely-closed` | operations/second | Rate of HTTP requests closed by the client before the server finished responding, averaged over the given time window. |
| Merge Bucket Pending | `merge-bucket-pending` | count | Number of pending bucket merge operations across all distributors, summed per application. |
| Node count (max) | `node-count-max` | count | Total number of nodes currently in the active state for the application. |
| Node Events (sum) | `node-events-sum` | count | Number of node events reported by the cluster controller in the given time window. |
| Node state: down (max) | `node-state-down-max` | count | Number of content nodes in the 'down' state per cluster (nodes considered unavailable by the cluster controller). |
| Node state: initializing (max) | `node-state-initializing-max` | count | Number of content nodes in the 'initializing' state per cluster (nodes currently starting up and not yet serving). |
| Node state: maintenance (max) | `node-state-maintenance-max` | count | Number of content nodes in the 'maintenance' state per cluster (nodes intentionally taken down for maintenance). |
| Node state: retired (max) | `node-state-retired-max` | count | Number of content nodes in the 'retired' state per cluster (nodes being decommissioned). |
| Node state: up (max) | `node-state-up-max` | count | Number of content nodes in the 'up' state per cluster. |
| Re-indexing remaining | `re-indexing-remaining` | percentage (0-100) | Worst-case document reindexing progress remaining across document types, as a percentage (100 = not started, 0 = complete or no reindex running). |
| Resource Usage: Attribute Address Space | `resource-usage-attribute-address-space` | percentage (0-100) | Worst-case attribute address-space utilization across content nodes; feed is blocked when this approaches 100%. |
| Resource Usage: Max Disk Utilization vs Limit | `resource-usage-max-disk-utilization-vs-limit` | percentage (0-100) | Maximum disk utilization vs the configured limit; feed is blocked when utilization exceeds the limit. |
| Service Restarts (total) | `service-restarts-total` | count | Total service restarts across all nodes, as tracked by the config sentinel. Increases indicate services crashing or being restarted. |
| Rate-Limited Requests | `rate-limited-requests-sum` | operations/second | Rate of requests checked by the RateLimitingSearcher (i.e. requests with rate.id and rate.quota set); only present if the searcher is in use. |
| Requests Rejected (Over Quota) | `requests-rejected-over-quota-sum` | operations/second | Rate of requests rejected by the RateLimitingSearcher for exceeding their quota (rate.quota); in dry-run mode these are counted but not actually rejected. |

### Autoscaling: Content

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Active nodes (max) | `active-nodes-count` | count | Maximum number of active nodes in the content cluster during the test window. |
| Active nodes (max) | `active-nodes-non-active-fraction` | fraction (0-1) | Fraction of content-cluster nodes that are not yet active, indicating an in-progress resize or data redistribution. |
| Autoscaling actions (sum) | `autoscaling-actions-sum` | count | Number of autoscaler rescale actions taken for the content cluster during the test window. |
| CPU load adjustment (peak / ideal) | `cpu-load-adjustment-ratio` | ratio (peak / ideal, 1.0 = ideally scaled) | Ratio of peak to ideal CPU load for the content cluster; above 1.0 signals scale-up pressure and below 1.0 signals the cluster is over-provisioned for CPU. |
| CPU load — peak vs ideal (max) | `cpu-load-peak-vs-ideal-ideal` | percentage (0-100) | The autoscaler's ideal CPU load target for the content cluster, as a percentage (already includes growth and failover headroom, so normally below 100). |
| CPU load — peak vs ideal (max) | `cpu-load-peak-vs-ideal-peak` | percentage (0-100) | The autoscaler's peak observed CPU load for the content cluster, as a percentage, from the config server's cluster.load.peak.cpu signal. |
| Query growth (rate vs 3h average) | `query-growth-rate-vs-3h-average-content` | ratio (test-window average / trailing 3h average) | Ratio of the content cluster's query rate during the test window to its own trailing 3-hour average, a proxy for the autoscaler's internal query-growth headroom factor. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-query-content` | operations/second | Query rate for the content cluster, one half of the query/write mix that determines the autoscaler's ideal CPU target. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-write-content` | operations/second | Write rate (put + update + remove operations per second) for the content cluster, the other half of the query/write mix that determines the autoscaler's ideal CPU target. |

### Autoscaling: Container

| Console name | `metric` | Unit | Description |
| --- | --- | --- | --- |
| Active nodes (max) | `active-nodes-container` | count | Maximum number of active container nodes in the cluster during the test window. |
| Active nodes (max) | `non-active-node-fraction-container` | fraction | Peak fraction of non-active (mid-transition) nodes in the container cluster during the test window, non-zero while a resize is in progress. |
| Query growth (rate vs 3h average) | `query-growth-rate-vs-3h-average` | ratio | Ratio of the container cluster's query rate during the test window to its trailing 3-hour average, a proxy for the autoscaler's growth-headroom signal (values above 1.0 indicate rising traffic). |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-query` | operations/second | Average query rate across the container cluster during the test window; one side of the query/write mix that shapes the autoscaler's ideal CPU target. |
| Query vs write rate — CPU mix driver (sum) | `query-vs-write-rate-cpu-mix-driver-write` | operations/second | Average write (feed) operation rate across the container cluster during the test window; the other side of the query/write mix that shapes the autoscaler's ideal CPU target. |

 Copyright © 2026 - [Cookie Preferences](#)

### On this page:

- [Production tests](#page-title)
- [Production test setup](#normal-flow)
- [Test suites](#test-suites)
- [Test file structure](#test-file-structure)
- [Test specification](#test-yaml-and-json-specification)
- [Examples](#examples)
- [Metric presets](#metric-presets)
- [Overview](#metric-presets-overview)
- [Query](#metric-presets-query)
- [Feed](#metric-presets-feed)
- [Nearest Neighbor Search](#metric-presets-nearest-neighbor-search)
- [Content Node](#metric-presets-content-node)
- [Resources](#metric-presets-resources)
- [Health](#metric-presets-health)
- [Autoscaling: Content](#metric-presets-autoscaling-content)
- [Autoscaling: Container](#metric-presets-autoscaling-container)