• [+] expand all

Live-upgrading Vespa

This document describes how to live-upgrade a Vespa instance. Use this procedure to upgrade without disruption to read or write traffic.

  1. Before upgrading
    • If upgrading to a new major version, first upgrade to the latest version on the current major, then make sure there are no deprecation warnings when building and deploying the application
      • Also make sure to detach nodes as part of the procedure
    • Redundancy: for data availability, there must be capacity to take one node per cluster out of service at the time. This procedure relies on that. If the clusters have redundancy=1, or searchable-copies=1, some data is not available (reduced coverage) during the upgrade
    • To reduce node downtime, download the new Vespa version to all hosts in advance
  2. Upgrade config servers
    • Install the new Vespa version on the config servers and restart them one by one: Restart config server. Wait until it is up again, look in vespa log for "Changing health status code from 'initializing' to 'up'" or use health checks.
    • Redeploy and activate the application:
      $ vespa-deploy prepare <app> && vespa-deploy activate
  3. Upgrade all other nodes one by one - for each of the other nodes in the system:
    • Stop Vespa services on the node
    • Install the new Vespa version
    • Start Vespa services on the node
    • Wait until the node is fully up before doing the next node - metrics/interfaces to be used to evaluate if the next node can be stopped:
      • .idealstate.merge_bucket.pending: When 0, all buckets are in sync
      • Check if a node is up using the health API


See config server troubleshooting.