Pod Lifecycle

This page describes the lifecycle of the Pods in a Vespa on Kubernetes deployment: how they are reconciled to the desired state, and how failures are handled. The ConfigServer and Application Pods have built-in resilience and recovery capabilities, and are gracefully rotated during maintenance to preserve availability and data integrity.

Reconciliation

The ConfigServer Pods, their PersistentVolumeClaims, and their dependent networking resources are provisioned by the Operator and bootstrapped when a VespaSet resource is detected. From then on, the Operator is responsible for reconciling the ConfigServer resources to the desired state — including during upgrades, changes to the podTemplate, changes to the storageClass, and various other maintenance.

The ConfigServers are responsible for deploying the Application Pods when an application package arrives. From then on, they are responsible for reconciling the Application Pods to the desired state — likewise during upgrades, changes to the podTemplate, changes to the storageClass, and various other maintenance.

During a reconciliation task, each Pod goes through the following flow. This is performed in an orchestrated fashion, depending on the topology configured, which is encoded into both the Operator and the ConfigServer:

1. The Pod is disabled as a backend.
2. The Pod is drained of traffic.
3. The Pod is deleted.
4. A new Pod with the updated spec is created.
5. The Pod waits until healthy.
6. The Pod is enabled as a backend.

During this process, the volume may be recreated, depending on what is being reconciled — refer to Configure Custom Overrides.

If a Pod is voluntarily deleted by force, the reconciler sees the missing Pod as having no podTemplate, detects a diff against the desired state, and reconciles towards the target by recreating the Pod.

Failure Handling

Failures are detected and recovered from automatically, without requiring administrator intervention. This applies to Application Pods and ConfigServer Pods alike — Application Pods are failed out by the ConfigServers, and ConfigServer Pods by the Operator:

1. A Pod encounters a failure — for example, a hardware failure — and becomes unresponsive.
2. The unresponsive Pod is detected and marked as failed: the ConfigServers fail out Application Pods, and the Operator fails out ConfigServer Pods.
3. The failed Pod and its Persistent Volume are cleaned up and removed from the cluster.
4. A replacement Pod is automatically provisioned to restore capacity — data is redistributed to a replacement Application Pod, while a replacement ConfigServer Pod rejoins the ConfigServer quorum.

Throughout this process, the cluster keeps serving traffic: queries and feeding continue on the remaining Pods, the remaining ConfigServers continue serving configuration, and Vespa's data replication ensures no data is lost while the replacement is brought up.

Refer to Production Serving for how to reduce the impact of involuntary failures.