• [+] expand all

Multinode systems

A Vespa system consists of one or more stateless and stateful clusters configured by an application package. A Vespa system is configured and managed through an admin cluster as shown below.

Vespa Overview

All nodes of a Vespa system have the same software installed. Which processes are started on each node and how they are configured is determined by the admin cluster from the specification given in services.xml in the application package.

Creating a multinode system from a sample application

To create a fully functional production ready multinode system from a single-node sample application, follow these steps (also see next steps):

  1. Add an admin cluster in services.xml:
    <services version="1.0">
        <admin version="2.0">
            <adminserver hostalias="admin0"/>
                <configserver hostalias="admin0"/>
    This cluster can have multiple nodes as config servers, but it is not strictly necessary as the other nodes will continue to run when the admin cluster is unreachable.
  2. Install the Vespa image or Docker image on all the nodes and start Vespa on the nodes.
  3. Run
    $ echo "override VESPA_CONFIGSERVERS [adminserver-hostname]" >> $VESPA_HOME/conf/vespa/default-env.txt
    where [adminserver-hostname] is replaced by the full hostname of the admin server (or a comma-separated list if multiple).
  4. Add these nodes to the container and content clusters by adding more node tags in src/main/application/services.xml (to avoid traffic disruption on restarts or node failure at least two nodes are needed in each cluster). One can add the same nodes to multiple clusters.
  5. Add the same nodes to src/main/application/hosts.xml.

With this, the application is fully production ready!

Refer to configuration server operations for troubleshooting.

Log collection

Logs are automatically collected from all nodes in real time to the admin node listed as adminserver. To view log messages from the system, run vespa-logfmt on this node.

Making changes to live systems

To change the system, deploy the changed application to the admin cluster. The admin cluster will automatically change the participating nodes as necessary. It is safe to do this while serving live query and write traffic. In some cases the admin cluster will report that some processes must be restarted to make the change effective. To avoid query or write traffic disruption, such restarts must be done on one node at the time, waiting until the node is fully up before restarting the next one.

Next steps

  • The multinode sample application is a useful starting point for setting up a system and run some basic tests to get familiar with Vespa APIs and interfaces.
  • multinode-HA is a high-availability multi-node template - use this as a basis for the final configuration.