Container

This is the Container service operational guide.

Note that "container" is an overloaded concept in Vespa - in this guide it refers to service instance nodes in blue.

Endpoints

Container service(s) hosts the query and feed endpoints - examples:

• album-recommendation configures _both_ query and feed in the same container cluster (i.e. service):

  <container id="default" version="1.0">
      <document-api />
      <search />
      <nodes>
          <node hostalias="node1" />
      </nodes>
  </container>

• multinode-HA configures query and feed in separate container clusters (i.e. services):

  <container id="feed" version="1.0">
      <document-api />
      <document-processing />
      <nodes>
          <node hostalias="node4" />
          <node hostalias="node5" />
      </nodes>
  </container>
  
  <container id="query" version="1.0">
      <search />
      <nodes>
          <node hostalias="node6" />
          <node hostalias="node7" />
      </nodes>
  </container>

Observe that <document-api> and <search> are located in separate clusters in the second example, and endpoints are therefore different.

Container Metrics

Please see Metrics for general information on metrics in Vespa.

Metrics from the container with description and unit can be found in Container Metrics. The most commonly used metrics are mentioned below.

Generic Container Metrics

These metrics are output for the server as a whole, e.g. related to resources. Some metrics indicate memory usage, such as mem.heap.*, mem.native.*, mem.direct.*. Other metics are related to the JVM garbage collection, jdisc.gc.count and jdisc.gc.ms.

Thread Pool Metrics

Metrics for the container thread pools. The jdisc.thread_pool.* metrics have a dimension threadpool with thread pool name, e.g default-pool for the container's default thread pool. See Container Tuning for details.

HTTP Specific Metrics

These are metrics specific for HTTP. Those metrics that are specific to a connector will have a dimension containing the TCP listen port.

Refer to Container Metrics for metrics on HTTP status response codes, http.status.* or more detailed requests related to the handling of requests, jdisc.http.*. Other relevant metrics include serverNumConnections, serverNumOpenConnections, serverBytesReceived and serverBytesSent.

Query Specific Metrics

For metrics related to queries please start with the queries and query_latency, the handled.requests and handled.latency or the httpapi_* metrics for more insights.

Feed Specific Metrics

For metrics related to feeding into Vespa we recommend using the feed.operations and feed.latency metrics.

Inspecting Vespa Java Services using JConsole

Determine the state of each running Java Vespa service using JConsole. JConsole is distributed along with the Java developer kit. Start JConsole:

$ jconsole <host>:<port>

where the host and port determine which service to attach to. For security purposes the JConsole tool can not directly attach to Vespa services from external machines.

Connecting to a Vespa instance

To attach a JConsole to a Vespa service running on another host, create a tunnel from the JConsole host to the Vespa service host. This can for example be done by setting up two SSH tunnels as follows:

$ ssh -N -L<port1>:localhost:<port1> <service-host> &
$ ssh -N -L<port2>:localhost:<port2> <service-host> &

where port1 and port2 are determined by the type of service (see below). A JConsole can then be attached to the service as follows:

$ jconsole localhost:<port1>

Port numbers:

Updated port information can be found by running:

$ vespa-model-inspect service <servicename>

where the resulting RMIREGISTRY and JMX lines determine port1 and port2, respectively.

Examining thread states

The state of each container is available in JConsole by pressing the Threads tab and selecting the thread of interest in the threads list. Threads of interest includes search, connector, closer, transport and acceptor (the latter four are used for backend communications).