• [+] expand all


This is the Container service operational guide.

Vespa Overview

Note that "container" is an overloaded concept in Vespa - in this guide it refers to service instance nodes in blue.


Container service(s) hosts the query and feed endpoints - examples:

  • album-recommendation configures _both_ query and feed in the same container cluster (i.e. service):
    <container id="default" version="1.0">
        <document-api />
        <search />
            <node hostalias="node1" />
  • multinode-HA configures query and feed in separate container clusters (i.e. services):
    <container id="feed" version="1.0">
        <document-api />
        <document-processing />
            <node hostalias="node4" />
            <node hostalias="node5" />
    <container id="query" version="1.0">
        <search />
            <node hostalias="node6" />
            <node hostalias="node7" />

Observe that <document-api> and <search> are located in separate clusters in the second example, and endpoints are therefore different.

Container Metrics

Metrics in Vespa is subject to change, and are not documented in reference documentation. Below is a set of container metrics documented in this guide for operational purposes. A few metrics are emitted with under multiple names, for compatibility with different metrics frameworks.

Generic Container Metrics

These metrics are output for the server as a whole and are not specific to HTTP.

Metric nameDescription
serverStartedMillis Time since server started
mem.heap.total Total heap size
mem.heap.free Free heap size
mem.heap.used Used heap size

Thread Pool Metrics

Metrics for the container thread pools. The jdisc.thread_pool.* metrics have a dimension threadpool with thread pool name, e.g default-pool for the container's default thread pool. See Container Tuning for details.

Metric nameDescription
jdisc.thread_pool.size Size of the thread pool
jdisc.thread_pool.active_threads Number of threads that are active
jdisc.thread_pool.max_allowed_size The maximum allowed number of threads in the pool
jdisc.thread_pool.rejected_tasks Number of tasks rejected by the thread pool
jdisc.thread_pool.unhandled_exceptions Number of exceptions thrown by tasks
jdisc.thread_pool.work_queue.capacity Capacity of the task queue
jdisc.thread_pool.work_queue.size Size of the task queue
jdisc.http.jetty.threadpool.thread.max Jetty thread pool: configured maximum number of threads
jdisc.http.jetty.threadpool.thread.min Jetty thread pool: configured minimum number of threads
jdisc.http.jetty.threadpool.thread.reserved Jetty thread pool: configured number of reserved threads or -1 for heuristic
jdisc.http.jetty.threadpool.thread.busy Jetty thread pool: number of threads executing internal and transient jobs
jdisc.http.jetty.threadpool.thread.total Jetty thread pool: current number of threads
jdisc.http.jetty.threadpool.queue.size Jetty thread pool: current size of the job queue

HTTP Specific Metrics

These are metrics specific for HTTP. Those metrics that are specific to a connector will have a dimension containing the TCP listen port.

Metric nameDescription
jdisc.http.requests.status Number of requests to the built-in status handler
http.status.1xx Number of responses with a 1xx status
http.status.2xx Number of responses with a 2xx status
http.status.3xx Number of responses with a 3xx status
http.status.4xx Number of responses with a 4xx status
http.status.5xx Number of responses with a 5xx status
serverNumConnections The total number of connections opened
serverNumOpenConnections The current number of open connections
serverConnectionsOpenMax The max number of open connections
serverConnectionDurationMean, -Max, -StdDev The mean/max/stddev of connection duration in ms
serverNumRequests, jdisc.http.requests Number of requests received by the connector
serverNumSuccessfulResponses Number of successful responses sent by the connector
serverNumFailedResponses Number of error responses sent by the connector
serverNumSuccessfulResponseWrites Number of HTTP response chunks that have been successfully written to the network.
serverNumFailedResponseWrites Number of HTTP response chunks that have not been successfully written to the network, due to some kind of I/O error.
serverBytesReceived Number of bytes the connector has received
serverBytesSent Number of bytes the connector has sent
serverTimeToFirstByte Time to first byte of response body is sent
serverTotalSuccessfulResponseLatency Time to complete successful responses
serverTotalFailedResponseLatency Time to complete failed responses

Inspecting Vespa Java Services using JConsole

Determine the state of each running Java Vespa service using JConsole. JConsole is distributed along with the Java developer kit. Start JConsole:

$ jconsole <host>:<port>

where the host and port determine which service to attach to. For security purposes the JConsole tool can not directly attach to Vespa services from external machines.

Connecting to a Vespa instance

To attach a JConsole to a Vespa service running on another host, create a tunnel from the JConsole host to the Vespa service host. This can for example be done by setting up two SSH tunnels as follows:

$ ssh -N -L<port1>:localhost:<port1> <service-host> &
$ ssh -N -L<port2>:localhost:<port2> <service-host> &

where port1 and port2 are determined by the type of service (see below). A JConsole can then be attached to the service as follows:

$ jconsole localhost:<port1>

Port numbers:

Service Port 1 Port 2
QRS 19015 19016
Docproc 19123 19124

Updated port information can be found by running:

$ vespa-model-inspect service <servicename>

where the resulting RMIREGISTRY and JMX lines determine port1 and port2, respectively.

Examining thread states

The state of each container is available in JConsole by pressing the Threads tab and selecting the thread of interest in the threads list. Threads of interest includes search, connector, closer, transport and acceptor (the latter four are used for backend communications).