Production Serving

This page describes how to harden a Vespa on Kubernetes deployment for production serving: reducing the impact of involuntary failures and keeping the cluster available through node outages and maintenance.

Spread Pods Across Distinct Nodes

In production, the ConfigServer Pods and Content Pods, especially if they are using a local storage type, should be scheduled on distinct Kubernetes Nodes, so that a single Node failure takes down at most one Pod of each kind. Node spreading is configured with a podAntiAffinity rule with topologyKey: kubernetes.io/hostname in the podTemplate — see Configure Custom Overrides.

ConfigServer Pods

The ConfigServers are backed by ZooKeeper, thus the quorum requires a majority to remain available. If two of the three ConfigServer Pods share a Node and that Node goes down, a split-brain scenario can occur. Placing each ConfigServer Pod on its own node ensures that no two ConfigServers go down at the same time.

apiVersion: k8s.ai.vespa/v1
kind: VespaSet
metadata:
  name: my-vespa-cluster
spec:
  configServer:
    podTemplate:
      spec:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: configserver
                topologyKey: kubernetes.io/hostname

Content Pods

For Content Pods, spreading across nodes ensures data integrity. Vespa maintains multiple copies of each document across content nodes, and colocating those Pods on the same node would allow a single node failure to take out multiple data copies at once.

Node spreading for Application Pods is configured per profile — the example below spreads the Content Pods of the music cluster:

apiVersion: k8s.ai.vespa/v1
kind: VespaSet
metadata:
  name: my-vespa-cluster
spec:
  profiles:
    music:
      podTemplate:
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchLabels:
                      vespa.ai/cluster-id: music
                  topologyKey: kubernetes.io/hostname

Data Redundancy

Beyond Node placement, data integrity for content clusters is configured in the application package:

• min-redundancy sets the minimum total number of data copies the content cluster will maintain, ensuring a minimum number of copies regardless of other configuration.
• Grouped distribution places content nodes in groups, where each group holds a full copy of the data. Combined with node spreading, this allows entire groups to fail — or be taken down for maintenance — while the remaining groups keep serving queries.

The Vespa serving scaling guide also has good recommendations for sizing and topology of production content clusters.