• [+] expand all

Metrics

Metrics can be accessed aggregated on an application-level using /metrics/v2/values or at node-level using either /metrics/v1/values or /prometheus/v1/values.

Metric-sets

Node metrics in /metrics/v1/values are listed per service, with a set of system metrics - example:

{
    "services": [
        {
            "name": "vespa.container",
            "timestamp": 1662120754,
            "status": {
                "code": "up",
                "description": "Data collected successfully"
            },
            "metrics": [
                {
                    "values": {
                        "memory_virt": 3683172352,
                        "memory_rss": 1425416192,
                        "cpu": 2.0234722784298,
                        "cpu_util": 0.202347227843
                    },
                    "dimensions": {
                        "metrictype": "system",
                        "instance": "container",
                        "clustername": "default",
                        "vespaVersion": "8.46.19"
                    }
                },
                {
                    "values": {},
                    "dimensions": {
                        "clustername": "default",
                        "instance": "container",
                        "vespaVersion": "8.46.19"
                    }
                }
            ]
        },

The default metric-set is added to the system metric-set, unless a consumer request parameter specifies a different built-in or custom metric set - see metric list.

The Vespa metric-set has a richer set of metrics, see metric list.

The consumer request parameter can also be used in /metrics/v2/values and /prometheus/v1/values.

Example minimal metric-set; system metric-set + a specific metric:

<admin version="2.0">
    <adminserver hostalias="node1" />
    <metrics>
        <consumer id="my-metric-set">
            <metric id="vds.idealstate.garbage_collection.documents_removed.count" />
        </consumer>
    </metrics>
</admin>

Example default metric-set and more; system metric-set + default metric-set + a built-in metric:

<admin version="2.0">
    <adminserver hostalias="node1" />
    <metrics>
        <consumer id="my-metric-set">
            <metric-set id="default" />
            <metric id="vds.idealstate.garbage_collection.documents_removed.count" />
        </consumer>
    </metrics>
</admin>

Metrics from custom components

Add custom metrics from components like Searchers and Document processors:

  1. Add a MetricReceiver instance to the constructor of the component - it is injected by the Container.
  2. Declare Gauge and Counter metrics using the declare-methods on the MetricReceiver. Optionally set arbitrary metric dimensions to default values at declaration time - refer to the javadoc for details.
  3. Each time there is some data to measure, invoke the sample method on gauges or the add method on counters. The gauges and counters declared are inherently thread-safe. When sampling data, any dimensions can optionally be set.
  4. Add a consumer in services.xml for the metrics to be emitted in the metric APIs, like in the previous section.

Find a full example in the album-recommendation-java sample application.

Histograms

Metric histograms is supported for Gauge metrics. Create the metric like in album-recommendation-java, adding the histogram:

public HitCountSearcher(MetricReceiver receiver) {
    this.hitCountMetric = receiver.declareGauge(EXAMPLE_METRIC_NAME, Optional.empty(),
        new MetricSettings.Builder().histogram(true).build());
}

The histograms for the last five minutes of logged data are available as CSV per dimension at /state/v1/metrics/histograms. Example output:

# start of metric hits_per_query, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
# end of metric hits_per_query, dimensions: { "chain": "metalchain" }
# start of metric example_hitcounts, dimensions: { "query_language": "en" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
# end of metric example_hitcounts, dimensions: { "query_language": "en" }
# start of metric query_latency, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
5.69,0.000000000000,1,1.00
5.69,1.000000000000,1,Infinity
# end of metric query_latency, dimensions: { "chain": "metalchain" }
# start of metric totalhits_per_query, dimensions: { "chain": "metalchain" }
"Value","Percentile","TotalCount","1/(1-Percentile)"
1.00,0.000000000000,1,1.00
1.00,1.000000000000,1,Infinity
# end of metric totalhits_per_query, dimensions: { "chain": "metalchain" }