Live-upgrading Vespa

This document describes how to live-upgrade a Vespa system. Follwing this procedure ensures that the upgrade can be accomplished without disruption to read or write traffic.

  1. Consider before upgrading
    • If you are upgrading to a new major version, you should first upgrade to the latest version on the current major, then make sure there are no deprecation warnings when building and deploying your application.
    • Redundancy: You should always have capacity to take one node per cluster out of service at the time. This procedure relies on that. If your content clusters have redundancy=1, or searchable-copies=1, you will have some data unavailability (reduced coverage) during live upgrade.
    • To reduce node downtime, you can download the new Vespa version to all hosts in advance.
    • To reduce content node node downtime, consider use flush-on-shutdown. This flushes the transaction before stopping nodes, which reduces startup time.
  2. Upgrade config servers
    • Detach all nodes from the config servers by running this command on all hosts:
      $ vespa-configproxy-cmd -m setmode memorycache
      This causes services to keep their current configuration until both config server and the services themselves have been upgraded.
    • Install the new Vespa version on the config servers and restart them.
    • Redeploy and activate the application:
      $ vespa-deploy prepare <app> && vespa-deploy activate
    • Restart services on the config servers - the config proxy will then go back to using the default mode and serve config from the upgraded config servers.
  3. Upgrade all other nodes one by one For each of the other nodes in the system:
    • Stop Vespa services on the node.
    • Install the new Vespa version.
    • Start Vespa services on the node.
    • Reattach the node to the config servers:
      $ vespa-configproxy-cmd -m setmode default
    • Wait until the node is fully up before doing the next node.