Production tests

This is the Vespa Testing reference for production tests.

Production tests verify domain specific metrics for a production deployment, such as query latency or error rate. Unlike system and staging tests, production tests check a named metric preset against expected bounds, using Grafana metrics for the application deployed to that zone.

A production test is placed after a deployment zone in deployment.xml. It acts as a gate for the rollout: if it fails, the rollout stops and subsequent zones will not receive the new version. This is useful in multi-zone deployments where the first zone serves as a canary, but a production test can validate any zone before the rollout proceeds to the next, as described in production tests.

Production test setup

To use production tests, first deploy to a zone, use <delay> for some time, and then run the production tests against that zone. This pattern can be repeated for each zone in the pipeline, to validate it before the rollout proceeds to the next. In the following example, the zone is deployed, then the delay waits 10 minutes, and lastly the production test waits for an additional 10 minutes before aggregating the last 10 minutes of metrics data:

<prod>
  <region>aws-us-east-1c</region>
  <delay minutes="10" />
  <test>aws-us-east-1c</test>
  <!-- Deploy to the rest of the zones... -->
</prod>

The declared production test:

name: cpu check
metric: cpu-utilization-container
duration: 10m
max: 85

Test suites

Production tests are placed under tests/production-test/ in the application package, as one or more .yaml or .json files:

$ ls tests/production-test/

tests/production-test/metrics-test.yaml

Each file describes one or more tests. A file may contain:

• a single test, as one YAML or JSON object; or
• multiple tests, as a YAML or JSON object with a tests field containing a list of test objects.

Test file structure

Each test checks a named metric preset, over a duration window, against a min and/or max bound. A simple example:

name: cpu check
metric: cpu-utilization-container
duration: 5m
max: 85

This test fails the production test step if the average CPU utilization of the container cluster, over the last 5 minutes, exceeds 85%.

To place multiple tests in one file, wrap them in a top-level tests field, containing a list of test objects. This works the same way for both YAML and JSON:

tests:
  - name: container cpu check
    metric: cpu-utilization-container
    duration: 5m
    max: 35

  - name: content cpu check
    metric: cpu-utilization-content
    duration: 5m
    max: 85

Test specification

A full list of fields, with description:

A test that is outside its min/max bounds fails the production test step immediately — production tests are not retried.

Examples

A single test, in YAML:

name: error rate check
metric: error-rate-pct
duration: 5m
max: 1

Multiple tests in a single YAML file:

tests:
  - name: node count check
    metric: node-count-max
    duration: 5m
    max: 10

  - name: documents count check
    metric: documents-count-active
    duration: 5m
    min: 1

A single test, in JSON:

{
    "name": "query latency check",
    "metric": "query-latency-p95",
    "duration": "5m",
    "max": 200
}

Multiple tests in a single JSON file:

{
    "tests": [
        {
            "name": "container feed latency check",
            "metric": "container-feed-latency-max",
            "duration": "5m",
            "max": 100
        },
        {
            "name": "content commit latency check",
            "metric": "content-commit-latency-avg",
            "duration": "5m",
            "max": 50
        }
    ]
}

Metric presets

Every test checks one of the following named presets, in its metric field. Each preset queries the same metrics that back the application's overview dashboard, reduced to a single number. Presets are grouped below by the dashboard tab they come from.

The Console name column is a direct match to the graph title shown in that tab in the Vespa Console's metrics view. Use it to find the preset for a graph you're looking at there. The Metric column is the exact value to use in the metric field, e.g. metric: query-latency-p95.

Overview

Query

Feed

Nearest Neighbor Search

Content Node

Resources

Health

Autoscaling: Content

Autoscaling: Container