• [+] expand all

Docker containers in production

This document describes tuning and adaptions for running Vespa Docker containers in production.

Mounting persistent volumes

The quick start and AWS ECS multi node guides show how to run Vespa in Docker containers. In these examples, all the data is stored inside the container - the data is lost if the container is deleted. When running Vespa inside Docker containers in production, volume mappings to the parent host should be added to persist data and logs.

  • /opt/vespa/var
  • /opt/vespa/logs
$ mkdir -p /tmp/vespa/var;  export VESPA_VAR_STORAGE=/tmp/vespa/var
$ mkdir -p /tmp/vespa/logs; export VESPA_LOG_STORAGE=/tmp/vespa/logs
$ docker run --detach --name vespa --hostname vespa-container --volume $VESPA_VAR_STORAGE:/opt/vespa/var \
  --volume $VESPA_LOG_STORAGE:/opt/vespa/logs --publish 8080:8080 vespaengine/vespa

System limits

When Vespa starts inside Docker containers, the startup scripts will set system limits. Make sure that the environment starting the Docker engine is setup in such a way that these limits can be set inside the containers.

For a CentOS/RHEL base host, Docker is usually started by systemd. In this case, LimitNOFILE, LimitNPROC and LimitCORE should be set to meet the minimum requirements in system limits.

In general when using Docker or Podman to run Vespa the --ulimit option should be used to set limits according to system limits. The --pids-limit should be set to unlimited (-1 for Docker and 0 for Podman).

Controlling which services to start

The Docker image vespaengine/vespa takes a parameter that controls which services are started inside the container.

Starting a configserver container:

$ docker run <other arguments> \
  --env VESPA_CONFIGSERVERS=<comma separated list of config servers> \
  vespaengine/vespa configserver

Starting a services container (configserver will not be started):

$ docker run <other arguments> \
  --env VESPA_CONFIGSERVERS=<comma separated list of config servers> \
  vespaengine/vespa services

Starting a container with both configserver and services:

$ docker run <other arguments> \
  --env VESPA_CONFIGSERVERS=<comma separated list of config servers> \
  vespaengine/vespa

This is required in the case where the configserver container should run other services like an adminserver or logserver (see services.html)

If the VESPA_CONFIGSERVERS environment variable is not specified it will be set to the container hostname.

Graceful stop

Stopping a running vespaengine/vespa container triggers a graceful shutdown, which saves time when starting the container again (i.e. data structures are fl). If the container is shutdown forcefully, the content nodes might need to restore the state from the transaction log which might be time consuming. There is no chance of data loss or data corruption as the data is always written and sync'ed to persistent storage.

The default timeout for the Docker daemon to wait for the shutdown might be too low for larger number of documents per node. Below stop will wait at least 120 seconds before terminating the running container forcefully, if the stop is successfully performed before the timeout has passed the command takes less than the timeout.

$ docker stop name -t 120

It is also possible to configure the default Docker daemon timeout, see --shutdown-timeout.