This is the Container service operational guide.
Note that "container" is an overloaded concept in Vespa - in this guide it refers to service instance nodes in blue.
Container service(s) hosts the query and feed endpoints - examples:
are located in separate clusters in the second example, and endpoints are therefore different.
Metrics in Vespa is subject to change, and are not documented in reference documentation. Below is a set of container metrics documented in this guide for operational purposes. A few metrics are emitted with under multiple names, for compatibility with different metrics frameworks.
These metrics are output for the server as a whole and are not specific to HTTP.
|serverStartedMillis||Time since server started|
|mem.heap.total||Total heap size|
|mem.heap.free||Free heap size|
|mem.heap.used||Used heap size|
Metrics for the container thread pools.
jdisc.thread_pool.* metrics have a dimension
threadpool with thread pool name,
e.g default-pool for the container's default thread pool.
See Container Tuning for details.
|jdisc.thread_pool.size||Size of the thread pool|
|jdisc.thread_pool.active_threads||Number of threads that are active|
|jdisc.thread_pool.max_allowed_size||The maximum allowed number of threads in the pool|
|jdisc.thread_pool.rejected_tasks||Number of tasks rejected by the thread pool|
|jdisc.thread_pool.unhandled_exceptions||Number of exceptions thrown by tasks|
|jdisc.thread_pool.work_queue.capacity||Capacity of the task queue|
|jdisc.thread_pool.work_queue.size||Size of the task queue|
|jdisc.http.jetty.threadpool.thread.max||Jetty thread pool: configured maximum number of threads|
|jdisc.http.jetty.threadpool.thread.min||Jetty thread pool: configured minimum number of threads|
|jdisc.http.jetty.threadpool.thread.reserved||Jetty thread pool: configured number of reserved threads or -1 for heuristic|
|jdisc.http.jetty.threadpool.thread.busy||Jetty thread pool: number of threads executing internal and transient jobs|
|jdisc.http.jetty.threadpool.thread.total||Jetty thread pool: current number of threads|
|jdisc.http.jetty.threadpool.queue.size||Jetty thread pool: current size of the job queue|
These are metrics specific for HTTP. Those metrics that are specific to a connector will have a dimension containing the TCP listen port.
|jdisc.http.requests.status||Number of requests to the built-in status handler|
|http.status.1xx||Number of responses with a 1xx status|
|http.status.2xx||Number of responses with a 2xx status|
|http.status.3xx||Number of responses with a 3xx status|
|http.status.4xx||Number of responses with a 4xx status|
|http.status.5xx||Number of responses with a 5xx status|
|serverNumConnections||The total number of connections opened|
|serverNumOpenConnections||The current number of open connections|
|serverConnectionsOpenMax||The max number of open connections|
|serverConnectionDurationMean, -Max, -StdDev||The mean/max/stddev of connection duration in ms|
|serverNumRequests, jdisc.http.requests||Number of requests received by the connector|
|serverNumSuccessfulResponses||Number of successful responses sent by the connector|
|serverNumFailedResponses||Number of error responses sent by the connector|
|serverNumSuccessfulResponseWrites||Number of HTTP response chunks that have been successfully written to the network.|
|serverNumFailedResponseWrites||Number of HTTP response chunks that have not been successfully written to the network, due to some kind of I/O error.|
|serverBytesReceived||Number of bytes the connector has received|
|serverBytesSent||Number of bytes the connector has sent|
|serverTimeToFirstByte||Time to first byte of response body is sent|
|serverTotalSuccessfulResponseLatency||Time to complete successful responses|
|serverTotalFailedResponseLatency||Time to complete failed responses|
Determine the state of each running Java Vespa service using JConsole. JConsole is distributed along with the Java developer kit. Start JConsole:
$ jconsole <host>:<port>
where the host and port determine which service to attach to. For security purposes the JConsole tool can not directly attach to Vespa services from external machines.
To attach a JConsole to a Vespa service running on another host, create a tunnel from the JConsole host to the Vespa service host. This can for example be done by setting up two SSH tunnels as follows:
$ ssh -N -L<port1>:localhost:<port1> <service-host> & $ ssh -N -L<port2>:localhost:<port2> <service-host> &
where port1 and port2 are determined by the type of service (see below). A JConsole can then be attached to the service as follows:
$ jconsole localhost:<port1>
|Service||Port 1||Port 2|
Updated port information can be found by running:
$ vespa-model-inspect service <servicename>
where the resulting RMIREGISTRY and JMX lines determine port1 and port2, respectively.
The state of each container is available in JConsole by pressing the Threads tab and selecting the thread of interest in the threads list. Threads of interest includes search, connector, closer, transport and acceptor (the latter four are used for backend communications).