• [+] expand all

Files, Processes, Ports, Environment

This is a reference of directories used in a Vespa installation, processes that run on the Vespa nodes and ports used. It is not necessary to know the information in this file to operate Vespa in production as the system automatically manages its files and processes. See also log files.

Directories

DirectoryDescription
$VESPA_HOME/bin/ Command line utilities and scripts
$VESPA_HOME/libexec/vespa/ Command line utilities and scripts
$VESPA_HOME/sbin/ Server programs, daemons, etc
$VESPA_HOME/lib64/ Dynamically linked libraries, typically third-party libraries
$VESPA_HOME/lib/jars/ Java archives
$VESPA_HOME/logs/vespa/ Log files
$VESPA_HOME/var/db/vespa/config_server/serverdb/ Config server database and user applications
$VESPA_HOME/share/vespa/ A directory with config definitions and XML schemas for application package validation
$VESPA_HOME/conf/vespa Various config files used by Vespa or libraries Vespa depend on

Processes and ports

The following is an overview of which ports and port ranges are used by the different services in a Vespa system. Note that for services capable of running multiple instances on the same node, all instances will run within the listed port range.

Many services are allocated ports dynamically. So even though the allocation is deterministic, i.e. the same system will get the same ports on subsequent startups, a particular service instance may get different ports when the overall system setup is changed through services.xml. Use vespa-model-inspect to see port allocations.

  • The number of ports used in a range depends on number of instances that are running
  • Not all ports within a range are used, but they are assigned each service to support future extensions
  • The range from 19100 is used for internal communication ports, i.e. ports that are not necessary to use from an external API
  • See Configuring Http Servers and Filters for how to configure Container ports and services.xml for how to configure other ports
ProcessHostPort/rangepsFunction
Config server Config server nodes 19070-19071 java (...) -jar $VESPA_HOME/lib/jars/standalone-container-jar-with-dependencies.jar Vespa Configuration server
2181-2183 Embedded Zookeeper cluster ports, see zookeeper-server.def
Config sentinel All nodes 19098 $VESPA_HOME/sbin/vespa-config-sentinel Sentinel that starts and stops vespa services and makes sure they are running unless they are manually stopped
Config proxy All nodes 19090 java (…) com.yahoo.vespa.config.proxy.ProxyServer Communication liaison between Vespa processes and config server. Caches config in memory
Slobrok Admin nodes 19099 for RPC port, HTTP port dynamically allocated in the 19100-19899 range $VESPA_HOME/sbin/vespa-slobrok Service location object broker
logd All nodes 19089 $VESPA_HOME/sbin/vespa-logd Reads local log files and sends them to log server
Log server Log server node 19080 java (...) -jar lib/jars/logserver-jar-with-dependencies.jar Vespa Log server
Metrics proxy All nodes 19092-19095 java (...) -jar $VESPA_HOME/lib/jars/container-disc-with-dependencies.jar Provides a single access point for metrics from all services on a Vespa node
Distributor Content cluster dynamically allocated in the 19100-19899 range $VESPA_HOME/sbin/vespa-distributord-bin Content layer distributor processes
Cluster controller Content cluster dynamically allocated in the 19100-19899 range java (...) -jar $VESPA_HOME/lib/jars/container-disc-jar-with-dependencies.jar Cluster controller processes, manages state for content nodes
proton Content cluster dynamically allocated in the 19100-19899 range $VESPA_HOME/sbin/vespa-proton-bin Searchnode process, receives queries from the container and returns results from the indexes. Also receives feed and indexes documents
container Container cluster 8080 java (...) -jar $VESPA_HOME/lib/jars/container-disc-with-dependencies.jar Container running servers, handlers and processing components

System limits

The startup scripts in the next section ensures that some of the system limits are set to a value greater than or equal to the table below.

NameLimitCommand executed
Max user processes 409600 ulimit -u 409600
Open files 262144 ulimit -n 262144
Core file size unlimited ulimit -c unlimited

If the startup scripts are not able to set the maximum number of open files Vespa will not start.

Vespa start / stop

Start and stop all services on a node:

$ $VESPA_HOME/bin/vespa-start-services
$ $VESPA_HOME/bin/vespa-stop-services

Likewise, for the config server:

$ $VESPA_HOME/bin/vespa-start-configserver
$ $VESPA_HOME/bin/vespa-stop-configserver

Learn more about which processes are run at Vespa startup.

Pre-start check

To run an automated node check before Vespa is started, provide a check script, returning 0 if service vespa-services start is allowed to run. This is useful when autostarting:

$ export VESPA_VALIDATIONSCRIPT=/tmp/foo.bar.sh
$ cat /tmp/foo.bar.sh
#!/bin/sh
echo detect some problem
exit 1
$ $VESPA_HOME/bin/vespa-start-services
... starting ...
detect some problem
validation script '/tmp/foo.bar.sh' failed
$VESPA_HOME/bin/vespa-start-services failed: exit code 1 from command:
$VESPA_HOME/libexec/vespa/start-vespa-base.sh

Core dumps

Example settings:

$ mkdir -p /tmp/cores && chmod a+rwx /tmp/cores
$ echo "/tmp/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

This will write files like /tmp/cores/core.vespa-proton-bi.1721.localhost.1580387387.

Environment variables

Vespa configuration is set in application packages. Some configuration is used to bootstrap nodes - this is set in environment variables. Environment variables are only read at startup.

$VESPA_HOME/conf/vespa/default-env.txt is read in Vespa start scripts - use this to modify variables. Each line has the format action variablename value where the items are:

ItemDescription
action One of fallback, override, or unset. fallback sets the variable if it is unset (or empty). override set the value regardless. unset unsets the variable.
variablename The name of the variable, e.g. VESPA_CONFIGSERVERS
value The rest of the line is the variable's value.
Environment variableDescription
VESPA_CONFIGSERVERS A comma-separated list of hosts to run configservers, use fully qualified hostnames. Should always be set to the same value on all hosts in a multi-host setup. If not set, localhost is assumed. Refer to configuration server operations.
VESPA_CONFIG_SOURCES Used by libraries like the Document API to set config server endpoints. Refer to configuration server operations for example use.
VESPA_WEB_SERVICE_PORT The port number where REST apis will run, default 8080. This isn't strictly needed, as the port number can be set for each HTTP server in services.xml, but with a big application it can be easier to set the default port number just once. Also note that this needs to be set when starting the configserver, since the REST api implementation gets its port number from there.
VESPA_TLS_CONFIG_FILE Absolute path to TLS configuration file.
VESPA_CONFIGSERVER_JVMARGS JVM arguments for the config server - see tuning.
VESPA_CONFIGPROXY_JVMARGS JVM arguments for the config proxy - see tuning.