Docker Containers in Production

This document describes tuning and adaptions that is useful when running Vespa Docker containers in production.

Mounting persistent volumes for container nodes

The quick start guide and AWS ECS multi node guide show how to run Vespa in docker containers. In these examples all the data get stored inside the container. This means that the data is lost if the container is deleted. When running Vespa inside Docker containers in production, volume mappings should be added to persist data and logs.

Two directories have to be mounted when creating the container:
  • /opt/vespa/var
  • /opt/vespa/logs
Example mounting directories from the Docker host in the container instance:
$ mkdir -p /tmp/vespa/var
$ export VESPA_VAR_STORAGE=/tmp/vespa/var
$ mkdir -p /tmp/vespa/logs
$ export VESPA_LOG_STORAGE=/tmp/vespa/logs
$ docker run --detach --name vespa --hostname vespa-container --privileged --volume $VESPA_VAR_STORAGE:/opt/vespa/var \
  --volume $VESPA_LOG_STORAGE:/opt/vespa/logs --publish 8080:8080 vespaengine/vespa

System limits in docker containers

When Vespa starts inside Docker containers the startup scripts will set certain system limits.

Make sure that the environment starting the Docker engine is setup in such a way that these limits can be set inside the containers.

For a CentOS/RHEL base host Docker is usually started by systemd. In this case LimitNOFILE, LimitNPROC and LimitCORE should be set to meet the minimum requirements in system limits.

Controlling which services to start

The docker image vespaengine/vespa can take a command that controls which services are started inside the container.

Starting a configserver container:
$ docker run <other arguments> --env VESPA_CONFIGSERVERS=<comma separated list of config servers> vespaengine/vespa configserver
Starting a services container (configserver will not be started):
$ docker run <other arguments> --env VESPA_CONFIGSERVERS=<comma separated list of config servers> vespaengine/vespa services

Omitting the command will result in both the configserver and services being started. The VESPA_CONFIGSERVERS environment variable will then be set to the local host.