This document describes how to live upgrade a Vespa instance.
Use this procedure to upgrade without disruption to read or write traffic.
- Before upgrading
-
If upgrading to a new major version:
Upgrade to the latest version on the current major first, then read the release notes
for the new major before progressing.
-
Redundancy: For availability,
there must be sufficient capacity to take one node per cluster out of service at the time.
If the clusters have redundancy=1, or searchable-copies=1,
some data will not be available during the upgrade (reduced coverage).
-
To reduce node downtime, download the new Vespa version to all hosts in advance.
- Detach the application nodes
Not necessary in Vespa 8, for upgrading between Vespa 7 versions see Vespa 8 release notes.
- Upgrade config servers
- Upgrade all other nodes one by one
- for each of the other nodes in the system:
- Stop services
on the node.
- Install the new Vespa version.
- Start services
on the node.
- Wait until the node is fully up before doing the next node -
metrics/interfaces to be used to evaluate if the next node can be stopped:
-
Check if a node is up using
/state/v1/health.
-
Check the
vds.idealstate.merge_bucket.pending.average
metric on content nodes.
When 0, all buckets are in sync - see example.
Troubleshooting
See
config server troubleshooting.