• [+] expand all

State API

The cluster controller has an API for viewing and modifying a content cluster state. To find the URL to access the State API, identify the cluster controller services in the application. Only the master cluster controller will be able to respond. The master cluster controller is the cluster controller alive that has the lowest index. Thus, one will typically use cluster controller 0, but if contacting it fails, try number 1 and so on. Using vespa-model-inspect:

$ vespa-model-inspect service -u container-clustercontroller

container-clustercontroller @ hostname.domain.com : admin
admin/cluster-controllers/0
    http://hostname.domain.com:19050/ (STATE EXTERNAL QUERY HTTP)
    http://hostname.domain.com:19117/ (EXTERNAL HTTP)
    tcp/hostname.domain.com:19118 (MESSAGING RPC)
    tcp/hostname.domain.com:19119 (ADMIN RPC)

In this example, there is only one clustercontroller, and the State Rest API is available on the port marked STATE and HTTP, 19050 in this example. This information can also be retrieved through the model config in the config server.

Find examples of API usage in content nodes.

Types

Type Spec Example Description
cluster <identifier> music The name given to a content cluster in a Vespa application.
description .* Some \"JSON escaped\" text Description can contain anything that is valid JSON. However, as the information is presented in various interfaces, some which may present reasons for all the states in a cluster or similar, keeping it short and to the point makes it easier to fit the information neatly into a table and get a better cluster overview.
group-spec <identifier>(\.<identifier>)* asia.switch0 The hierarchical group assignment of a given content node. This is a dot separated list of identifiers given in the application services.xml configuration.
node [0-9]+ 0 The index or distribution key identifying a given node within the context of a content cluster and a service type.
service-type (distributor|storage) distributor The type of the service to look at state for, within the context of a given content cluster.
state-disk (up|down) up One of the valid disk states.
state-generated (initializing|up|down|retired|maintenance) up One of the valid node generated states.
state-unit (initializing|up|stopping|down) up One of the valid node unit states.
state-user (up|down|retired|maintenance) up One of the valid node user states.

Errors

Errors are indicated using HTTP status codes. An error response from the State API includes a JSON encoded error response with extra information. As a request may fail outside of the State API, it is not guaranteed that such a JSON representation exist though. To make it simpler for clients, all errors they need to handle specifically should be specified in HTTP error codes. Thus, the content of the JSON error report, if it exists, can be left unspecified, and just used to improve an error report if needed.

Cluster controller not master - master known

This error means communicating with the wrong cluster controller. This returns a standard HTTP redirect, so the HTTP client can automatically redo the request on the correct cluster controller.

As the cluster controller available with the lowest index will be the master, the cluster controllers are normally queried in index order. Hence, it is unlikely to ever get this error, but rather fail to connect to the cluster controller if it is not the current master.

HTTP/1.1 303 See Other
Location: http://<master>/<current-request>
Content-Type: application/json

{
    "message" : "Cluster controller index not master. Use master at index index.
}

Cluster controller not master - unknown or no master

This error is used if the cluster controller asked is not master, and it doesn't know who the master is. This can happen, e.g. in a network split where cluster controller 0 no longer can reach cluster controller 1 and 2, in which case cluster controller 0 knows it is not master, as it can't see the majority, and cluster controller 1 and 2 will vote 1 to master.

HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{
    "message" : "No known master cluster controller currently exist."
}

Recursive mode

To use recursive mode, specify the recursive URL parameter, and give it a numeric value for number of levels. A value of true is also valid, this returns all levels. Examples:

  • Use recursive=1 for a node request to also see all data
  • use recursive=2 to see all the node data within each service type

In recursive mode, you will see the same output as found in the spec below. However, where there is a { "link" : "<url-path>" } element, this element will be replaced by the content of that request, given a recursive value of one less than the request above.

Functions

Following is a list of the available functions, with sample successful responses. All requests with content have a content length response header, and more headers are generally returned.

List content clusters

HTTP GET /cluster/v2/

{
    "cluster" : {
        "music" : {
            "link" : "/cluster/v2/music"
        },
        "books" : {
            "link" : "/cluster/v2/books"
        }
    }
}

Get cluster state and list service types within cluster

HTTP GET /cluster/v2/<cluster>

{
    "state" : {
        "generated" : {
            "state" : "state-generated",
            "reason" : "description"
        }
    }
    "service" : {
        "distributor" : {
            "link" : "/cluster/v2/mycluster/distributor"
        },
        "storage" : {
            "link" : "/cluster/v2/mycluster/storage"
        }
    }
 }

List nodes per service type for cluster

HTTP GET /cluster/v2/cluster/service-type

{
    "node" : {
        "0" : {
            "link" : "/cluster/v2/mycluster/storage/0"
        },
        "1" : {
            "link" : "/cluster/v2/mycluster/storage/1"
        }
    }
}

Get node state

HTTP GET /cluster/v2/cluster/service-type/node

{
    "attributes" : {
        "hierarchical-group" : "group-spec"
    },
    "partition" : {
        "0" : {
            "link" : "/cluster/v2/mycluster/storage/0/0"
        }
    },
    "state" : {
        "unit" : {
            "state" : "state-unit",
            "reason" : "description"
        },
        "generated" : {
            "state" : "state-generated",
            "reason" : "description"
        },
        "user" : {
            "state" : "state-user",
            "reason" : "description"
        }
    },
    "metrics" : {
        "bucket-count" : 0,
        "unique-document-count" : 0,
        "unique-document-total-size" : 0
    }
}

Set node user state

HTTP PUT /cluster/v2/cluster/service-type/node

{
    "state" : {
        "user" : {
            "state" : "retired",
            "reason" : "This colo will be removed soon"
        }
    }
}
{
    "wasModified": true,
    "reason": "ok"
}