This document describes how to live-upgrade a Vespa system. Follwing this procedure ensures that the upgrade can be accomplished without disruption to read or write traffic.
- Consider before upgrading
- If you are upgrading to a new major version, you should first upgrade to the latest version on the current major, then make sure there are no deprecation warnings when building and deploying your application.
- Redundancy: You should always have capacity to take one node per cluster out of service at the time. This procedure relies on that. If your content clusters have redundancy=1, or searchable-copies=1, you will have some data unavailability (reduced coverage) during live upgrade.
- To reduce node downtime, you can download the new Vespa version to all hosts in advance.
- To reduce content node node downtime, consider use flush-on-shutdown. This flushes the transaction before stopping nodes, which reduces startup time.
- Upgrade config servers
Detach all nodes from the
config servers by running this command on all hosts:
$ vespa-configproxy-cmd -m setmode memorycacheThis causes services to keep their current configuration until both config server and the services themselves have been upgraded.
- Install the new Vespa version on the config servers and restart them.
Redeploy and activate the application:
$ vespa-deploy prepare <app> && vespa-deploy activate
- Restart services on the config servers - the config proxy will then go back to using the default mode and serve config from the upgraded config servers.
- Detach all nodes from the config servers by running this command on all hosts:
- Upgrade all other nodes one by one
For each of the other nodes in the system:
- Stop Vespa services on the node.
- Install the new Vespa version.
- Start Vespa services on the node.
- Reattach the node to the config servers:
$ vespa-configproxy-cmd -m setmode default
- Wait until the node is fully up before doing the next node.