• [+] expand all

Using Kubernetes with Vespa

This article outlines how to run Vespa using Kubernetes. Find a quickstart for running Vespa in a single pod in singlenode quickstart with minikube.

Setting up a multi-pod Vespa cluster is a bit more complicated, and requires knowledge about how Vespa configures its services. Use the multinode-HA sample application as a basis for configuration.

Vespa overview illustration
  • A Vespa cluster is made of one or more config servers in a config server cluster. This cluster keeps configuration for the services running in the service pods. The config server cluster pods should hence be started first.
  • Config servers use Apache Zookeeper for shared state. The config servers will not set their /state/v1/health to UP before Zookeeper quorum is reached. This means that all config server pods must be running before quorum is reached, and one cannot use a readinessProbe probe for the config servers for a staggered start.
  • See a practical example at config server cluster startup - once completed it should look like:
    $ kubectl get pods
    NAME                   READY   STATUS    RESTARTS   AGE
    vespa-configserver-0   1/1     Running   0          2m45s
    vespa-configserver-1   1/1     Running   0          107s
    vespa-configserver-2   1/1     Running   0          62s
  • Once the config server cluster is started successfully, the application package can be deployed, and the pods for the services nodes started. The application package maps services to pods (nodes), so this must be deployed successfully before the services in the pods can start. It does not matter whether one deploys the application package before or after starting the service pods, as the pods will idle, waiting for configuration.
  • multinode-HA starts the pods first, see Vespa startup. As the application package is not yet deployed, the service inside the pods is not started (as it is not configured). The Vespa infrastructure is started, however, see config sentinel - so the pod is started with the config-proxy waiting for services config at this point.
  • The cluster startup feature is good to know. This is a setting to not start a service before enough services can run - see the Connectivity check log messages.
  • Deploy the application package. At this point, the pods will know which service to run, and start a container or content node service. Shortly after, the /state/v1/health endpoint is enabled on the pods.
  • Note that ports are allocated dynamically, but the defaults will get you started - see the illustration with services and ports for /state/v1/health:
    • Config server: 19071
    • Container node: 8080
    • Content node: 19107

The list above is an overview of the config server -> application package -> service /state/v1/health dependency chain. This sequence of steps must be considered when building the Kubernetes cluster configuration.

A good next step is running the multinode-HA for Kubernetes - there you will also find useful troubleshooting tools.

Singlenode quickstart with minikube

This section describes how to install and run Vespa on a single machine using Kubernetes (K8s). See Getting Started for troubleshooting, next steps and other guides. Also see Vespa example on GKE.

  1. Validate environment:

    Refer to Docker memory for details and troubleshooting:

    $ docker info | grep "Total Memory"
  2. Start Kubernetes cluster with minikube:
    $ minikube start --driver docker --memory 4096
  3. Clone the Vespa sample apps:
    $ git clone --depth 1 https://github.com/vespa-engine/sample-apps.git
    $ export VESPA_SAMPLE_APPS=`pwd`/sample-apps
  4. Create Kubernetes configuration files:
    $ cat << EOF > service.yml
    apiVersion: v1
    kind: Service
      name: vespa
        app: vespa
        app: vespa
      type: NodePort
      - name: container
        port: 8080
        targetPort: 8080
        protocol: TCP
      - name: config
        port: 19071
        targetPort: 19071
        protocol: TCP
    $ cat << EOF > statefulset.yml
    apiVersion: apps/v1
    kind: StatefulSet
      name: vespa
        app: vespa
      replicas: 1
      serviceName: vespa
          app: vespa
            app: vespa
          - name: vespa
            image: vespaengine/vespa
            imagePullPolicy: Always
            - name: VESPA_CONFIGSERVERS
              value: vespa-0.vespa.default.svc.cluster.local
              runAsUser: 1000
            - containerPort: 8080
              protocol: TCP
                path: /state/v1/health
                port: 19071
                scheme: HTTP
  5. Start the service:
    $ kubectl apply -f service.yml -f statefulset.yml
  6. Wait for the service to enter a running state:
    $ kubectl get pods --watch
    Wait for STATUS Running:
        NAME      READY   STATUS              RESTARTS   AGE
        vespa-0   0/1     ContainerCreating   0          8s
        vespa-0   0/1     Running             0          2m4s
  7. Start port forwarding to pods:
    $ kubectl port-forward vespa-0 19071 8080 &
  8. Wait for configserver start - wait for 200 OK:
    $ curl -s --head http://localhost:19071/state/v1/health
  9. Deploy and activate the application package:
    $ vespa deploy ${VESPA_SAMPLE_APPS}/album-recommendation
  10. Ensure the application is active - wait for 200 OK:

    This normally takes a minute or so:

    $ curl -s --head http://localhost:8080/state/v1/health
  11. Feed documents:
    $ vespa feed sample-apps/album-recommendation/ext/documents.jsonl
  12. Make a query:

    $ vespa query 'select * from music where true'
  13. Run a document get request:
    $ vespa document get id:mynamespace:music::love-is-here-to-stay
  14. Clean up:

    Stop the running container:

    $ kubectl delete service,statefulsets vespa

    Stop port forwarding:

    $ killall kubectl

    Stop minikube:

    $ minikube stop

At any point during the procedure, dump logs for troubleshooting:

$ kubectl logs vespa-0