Config sentinel

The config sentinel starts and stops services - and restart failed services unless are manually stopped

Services start

All nodes in a Vespa system has at least these running services:

config-proxy proxies config requests between Vespa applications and the configserver node. All configuration is cached locally so that this node can maintain its current configuration, even if the configserver shuts down
config-sentinel registers itself with the config-proxy and subscribes to and enforces node configuration, meaning the configuration of what services should be run locally, and with what parameters
vespa-logd monitors $VESPA_HOME/logs/vespa/vespa.log, which is used by all other services, and relays everything to the log-server

Sequence:

  1. config-proxy is started. The environment variables VESPA_CONFIGSERVERS and VESPA_CONFIGSERVER_RPC_PORT are used to connect to the config-server(s). It will retry all config servers in case some are down.
  2. config-sentinel is started, and subscribes to node configuration (i.e. a service list) from config-proxy using its hostname as the config id. See Node and Network setup - Concepts and Requirements for details about how the hostname is detected and how to override it. The config for the config-sentinel (the service list) lists the processes to be started, along with the config id to assign to each, typically the logical name of that service instance.
  3. config-proxy subscribes to node configuration from config-server, caches it, and returns the result to config-sentinel
  4. config-sentinel starts the services given in the node configuration, with the config id as argument - see example output below, like id="search/qrservers/qrserver.0". Each service:
    1. Subscribes to configuration from config-proxy
    2. config-proxy subscribes to configuration from config-server, caches it and returns result to the service
    3. The service runs according to its configuration, logging to $VESPA_HOME/logs/vespa/vespa.log. The processes instantiate internal components, each assigned the same or another config id, and instantiating further components.
When the config-server receives updated configuration for a running system, it propagates the changed configuration to nodes subscribing to it. In turn, these nodes reconfigure themselves accordingly.

User interface

The config sentinel runs a RPC service which can be used to list, start and stop the services supposed to run on that node. This can be useful for testing and debugging. Use the vespa-sentinel-cmd command line tool to trigger these actions. Example output from vespa-sentinel-cmd list:

vespa-sentinel-cmd 'sentinel.ls' OK.
container state=RUNNING mode=AUTO pid=27993 exitstatus=0 id="default/container.0"
container-clustercontroller state=RUNNING mode=AUTO pid=27997 exitstatus=0 id="admin/cluster-controllers/0"
distributor state=RUNNING mode=AUTO pid=27996 exitstatus=0 id="search/distributor/0"
logd state=RUNNING mode=AUTO pid=5751 exitstatus=0 id="hosts/r6-3/logd"
logserver state=RUNNING mode=AUTO pid=27994 exitstatus=0 id="admin/logserver"
searchnode state=RUNNING mode=AUTO pid=27995 exitstatus=0 id="search/search/cluster.search/0"
slobrok state=RUNNING mode=AUTO pid=28000 exitstatus=0 id="admin/slobrok.0"
To learn more about the processes and services, see files and processes. Use vespa-model-inspect host hostname to list services running on a node.