Container metrics

Vespa reports a certain number of standard metrics. You can also set up Vespa to report your own custom metrics.

Metrics from custom components

  1. Add a MetricReceiver ( com.yahoo.metrics.simple.MetricReceiver) instance to the constructor of the component in question - it is injected by the container
  2. Declare the gauges and counters using the declare methods on the metric receiver. Optionally set arbitrary metric dimensions to default values at declaration time - refer to the javadoc for details
  3. Each time there is some data to measure, invoke the sample method on gauges or the add method on counters. When sampling data, any dimensions can optionally be set

The gauges and counters declared are inherently thread safe. Example:

package com.yahoo.example;

import java.util.Optional;
import com.yahoo.metrics.simple.Gauge;
import com.yahoo.metrics.simple.MetricSettings;
import com.yahoo.metrics.simple.MetricReceiver;
import com.yahoo.search.Query;
import com.yahoo.search.Result;
import com.yahoo.search.Searcher;
import com.yahoo.search.searchchain.Execution;

public class HitCountSearcher extends Searcher {
    private static final String LANGUAGE_DIMENSION_NAME = "query_language";
    private static final String EXAMPLE_METRIC_NAME = "example_hitcounts";
    private final Gauge hitCountMetric;

    public HitCountSearcher(MetricReceiver receiver) {
        this.hitCountMetric = receiver.declareGauge(EXAMPLE_METRIC_NAME, Optional.empty(),
                new MetricSettings.Builder().histogram(true).build());
    }

    @Override
    public Result search(Query query, Execution execution) {
        Result result = execution.search(query);
        hitCountMetric
                .sample(result.getTotalHitCount(),
                        hitCountMetric.builder()
                                .set(LANGUAGE_DIMENSION_NAME, query.getModel().getParsingLanguage().languageCode())
                                .build());
        return result;
    }
}
Then look at the Metrics API where the new event example_hitcounts is available in the list of metrics. The histograms for the last five minutes of logged data are available as CSV per dimension at http://host:port/state/v1/metrics/histograms. In the example, that would include the estimated total hit counts for queries, grouped by language. The underlying implementation of the histograms is HdrHistogram, and the CSV is simply what that library generates itself.

Http Server metrics

The metrics from the built-in HTTP server are available in JSON using the metrics API.

The Container HTTP server is based on Jetty. Some of the metrics are gathered from the Jetty StatisticsHandler or ConnectorStatistics, with names familiar to those used to working with Jetty. Other metrics are JDisc specific. Some of the metrics are emitted with two separate names, for compatibility with different metrics frameworks.

In services.xml-terminology, a server means a Jetty connector, while there is always only one Jetty server per node in a JDisc cluster.

Server-wide Metrics

These metrics are output for the server as a whole, across all connectors, as opposed to the per-connector metrics listed in the next section.

Metric nameDescription
serverStartedMillis Time since server started
mem.heap.total Total heap size
mem.heap.free Free heap size
mem.heap.used Used heap size
serverThreadPoolSize Size of the thread pool for request processing
serverActiveThreads Number of threads that are active processing requests
serverRejectedRequests Number of requests rejected by the thread pool
jdisc.http.requests.status Number of requests to the built-in status handler
http.status.1xx Number of responses with a 1xx status
http.status.2xx Number of responses with a 2xx status
http.status.3xx Number of responses with a 3xx status
http.status.4xx Number of responses with a 4xx status
http.status.5xx Number of responses with a 5xx status

Per-connector Metrics

These metrics are output for each connector in the Jetty server.

Metric nameDescription
serverNumConnections See Jetty ConnectorStatistics
serverNumOpenConnections See Jetty ConnectorStatistics
serverConnectionsOpenMax See Jetty ConnectorStatistics
serverConnectionDurationMean, -Max, -StdDev See Jetty ConnectorStatistics
serverNumRequests, jdisc.http.requests Number of requests received by the connector
serverNumSuccessfulResponses Number of successful responses sent by the connector
serverNumFailedResponses Number of error responses sent by the connector
serverNumSuccessfulResponseWrites Number of HTTP response chunks that have been successfully written to the network.
serverNumFailedResponseWrites Number of HTTP response chunks that have not been successfully written to the network, due to some kind of I/O error.
serverBytesReceived Number of bytes the connector has received
serverBytesSent Number of bytes the connector has sent
serverTimeToFirstByte Time to first byte of response body is sent
serverTotalSuccessfulResponseLatency Time to complete successful responses
serverTotalFailedResponseLatency Time to complete failed responses