Content nodes, states and metrics

Refer to administrative procedures for configuration and state monitoring / management.

Try the Multinode testing and observability sample app to get familiar with interfaces and behavior.

Refer to components for an overview of the Document API flow. Content cluster processes are distributor, proton and cluster controller:

Node state

Content and distributor nodes have state:

Up	The node is up and available to keep buckets and serve requests.
Down	The node is not available, and can not be used.
Initializing	The node is starting up. It knows what buckets it stores data for, and may serve requests, but it does not know the metadata for all its buckets yet, such as checksums and document counts. The node is available for bucket placement.
Stopping	This node is stopping and is expected to be down soon. This state is typically only exposed to the cluster controller to tell why the node stopped. The cluster controller will expose the node as down or in maintenance mode for the rest of the cluster. This state is thus not seen by the distribution algorithm.
Maintenance	This node is temporarily unavailable. The node is available for bucket placement, redundancy is hence lower. Using this mode, new replicas of the documents stored on this node will not be created, allowing the node to be down with less of a performance impact on the rest of the cluster. This mode is typically used to mask a down state during controlled node restarts, or by an administrator that need to do some short maintenance work, like upgrading software or restart the node.
Retired	A retired node is available and serves requests. This state is used to remove nodes while keeping redundancy. Buckets are moved to other nodes (with low priority), until empty. Special considerations apply when using grouped distribution as buckets are not necessarily removed.

Distributor nodes start / transfer buckets quickly and are hence not in maintenance or retired.

Cluster state

There are three kinds of state:

Unit state	The cluster controller fetches states from all nodes, called unit states. States reported from the nodes are either initializing, up or stopping. If the node can not be reached, a down state is assumed. This means, the cluster controller detects failed nodes. The subsequent generated states will hence have nodes in down, and the ideal state algorithm will redistribute buckets of documents.
User state	user state must be one of up, down, maintenance or retired: Retire a node from a cluster - use retired to move buckets to other nodes Short-lived maintenance work - use maintenance to avoid merging buckets to other nodes Fail a bad node. The cluster controller or an operator can set a node down Use tools for user state management.
Generated state	The cluster controller generates the cluster state from the unit and user states, over time. The generated state is called the cluster state.

For new cluster states, the cluster state version is upped, and the new cluster state is broadcasted to all nodes. There is a minimum time between each cluster state change.

It is possible to set a minimum capacity for the cluster state to be up. If a cluster has so many nodes unavailable that it is considered down, the state of each nodes is irrelevant, and thus new cluster states will not be created and broadcasted before enough nodes are back for the cluster to come back up. A cluster state indicating the entire cluster is down, may thus have outdated data on the node level.

Cluster controller

The main task of the cluster controller is to maintain the cluster state. This is done by polling nodes for state, generating a cluster state, which is then broadcast to all the content nodes in the cluster. Note that clients do not interface with the cluster controller - they get the cluster state from the distributors - details.

Node state polling	The cluster controller polls nodes, sending the current cluster state. If the cluster state is no longer correct, the node returns correct information immediately. If the state is correct, the request lingers on the node, such that the node can reply to it immediately if its state changes. After a while, the cluster controller will send a new state request to the node, even with one pending. This triggers a reply to the lingering request and makes the new one linger instead. Hence, nodes have a pending state request. During a controlled node shutdown, it starts the shutdown process by responding to the pending state request that it is now stopping. Note: As controlled restarts or shutdowns are implemented as TERM signals from the config-sentinel, the cluster controller is not able to differ between controlled and other shutdowns.
Cluster state generation	The cluster controller translates unit and user states into the generated cluster state
Cluster state broadcast	When node unit states are received, a cluster controller internal cluster state is updated. New cluster states are distributed with a minimum interval between. A grace period per unit state too - e.g, distributors and content nodes that are on the same node often stop at the same time. The version number is upped, and the new cluster state is broadcast. If cluster state version is reset, clients to distributors can temporarily fail operations in the transition, but will eventually converge on the new (lower) cluster state version.

Master election

Vespa can be configured with one cluster controller. Reads and writes will work well in case of cluster controller down, but other changes to the cluster (like a content node down) will not be handled. It is hence recommended to configure a set of cluster controllers.

The cluster controller nodes elect a master, which does the node polling and cluster state broadcast. The other cluster controller nodes only exist to do master election and potentially take over if the master dies.

All cluster controllers will vote for cluster controller with the lowest index that says it is ready. If a cluster controller has more than half of the votes, it will be elected master. As a majority vote is required, the number of cluster controllers should be an odd number of 3 or greater. A fresh master will not broadcast states before a transition time is passed, allowing an old master to have some time to realize it is no longer the master.

Constantly restarting node

A broken node may end up with processes constantly restarting. It may die during initialization due to accessing corrupt files, or it may die when it starts receiving requests of a given type triggering a node local bug. This is bad for distributor nodes, as these restarts create constant ownership transfer between distributors, causing windows where buckets are unavailable.

The cluster controller has functionality for detecting such nodes. If a node restarts in a way that is not detected as a controlled shutdown, more than max_premature_crashes, the cluster controller will set the wanted state of this node to be down.

Detecting a controlled restart is currently a bit tricky. A controlled restart is typically initiated by sending a TERM signal to the process. Not having any other sign, the content layer has to assume that all TERM signals are the cause of controlled shutdowns. Thus, if the process keep being killed by kernel due to using too much memory, this will look like controlled shutdowns to the content layer.

Distributor

Buckets are mapped to distributors using the ideal state algorithm. As the cluster state changes, buckets are re-mapped immediately. The mapping does not overlap - a bucket is owned by one distributor.

Distributors do not persist the bucket database, the bucket-to-content-node mapping is kept in memory in the distributor. Document count, persisted size and a metadata checksum per bucket is stored as well. At distributor (re)start, content nodes are polled for bucket information, and return which buckets are owned by this distributor (using the ideal state algorithm). There is hence no master / name node in Vespa. Likewise, at any distributor cluster state change, content nodes are polled for bucket handover - a distributor will then handle a new set of buckets.

Document operations are mapped to content nodes based on bucket locations - each put/update/get/remove is mapped to a bucket and sent to the right content nodes. To manage the document set as it grows and nodes change, buckets move between content nodes.

Document API clients (i.e. container nodes with <document-api>) do not interface with the cluster controller, and do not know the cluster state at startup. Clients hence do not know which distributor to use at startup. A random distributor is therefore used first. If the document operation hit the wrong distributor, WRONG_DISTRIBUTION is returned, with the current cluster state in the response. WRONG_DISTRIBUTION is hence expected and normal at cold start / state change events.

Timestamps

Write operations have a last modified time timestamp assigned when passing through the distributor. The timestamp is guaranteed to be unique within the bucket where it is stored. The timestamp is used by the content layer to decide which operation is newest. These timestamps can be used when visiting, to process/retrieve documents within a given time range. To guarantee unique timestamps, they are in microseconds - the microsecond part is generated to avoid conflicts with other documents.

If documents are migrated between clusters, the target cluster will have new timestamps for their entries. Also, when reprocessing documents within a cluster, documents will have new timestamps, even if not modified.

Ordering

The Document API uses the document ID to order operations. A Document API client ensures that only one operation is pending at the same time. This ensures that if a client sends multiple operations for the same document, they will be processed in a defined order.

Note: If sending two write operations to the same document, and the first operation fails, the second operation that was enqueued is sent. If the client chooses to just resend the failed request, the order of operations has been switched.

If different clients have pending operations on the same document, the order is undefined.

Maintenance operations

Distributors track which content nodes have which buckets in their bucket batabase. Distributors then use the ideal state algorithm to generate bucket maintenance operations. A stable system has all buckets located per the ideal state:

• If buckets have too few replicas, new are generated on other content nodes.
• If the replicas differ, a bucket merge is issued to get replicas consistent.
• If a buckets has too many replicas, superfluous are deleted. Buckets are merged, if inconsistent, before deletion.
• If two buckets exist, such that both may contain the same document, the buckets are split or joined to remove such overlapping buckets. Read more on inconsistent buckets.
• If buckets are too small/large, they will be joined or split.

The maintenance operations have different priorities. If no maintenance operations are needed, the cluster is said to be in the ideal state. The distributors synchronize maintenance load with user load, e.g. to remap requests to other buckets after bucket splitting and joining.

Metrics

.idealstate.idealstate_diff	This metric tries to create a single value indicating distance to the ideal state. A value of zero indicates that the cluster is in the ideal state. Graphed values of this metric gives a good indication for how fast the cluster gets back to the ideal state after changes. Note that some issues may hide other issues, so sometimes the graph may appear to stand still or even go a bit up again, as resolving one issue may have detected one or several others.
.idealstate.buckets_toofewcopies	Specifically lists how many buckets have too few copies. Compare to the buckets metric to see how big a portion of the cluster this is.
.idealstate.buckets_toomanycopies	Specifically lists how many buckets have too many copies. Compare to the buckets metric to see how big a portion of the cluster this is.
.idealstate.buckets	The total number of buckets managed. Used by other metrics reporting bucket counts to know how big a part of the cluster they relate to.
.idealstate.buckets_notrusted	Lists how many buckets have no trusted copies. Without trusted buckets operations against the bucket may have poor performance, having to send requests to many copies to try and create consistent replies.
.idealstate.delete_bucket.pending	Lists how many buckets that needs to be deleted.
.idealstate.merge_bucket.pending	Lists how many buckets there are, where we suspect not all copies store identical document sets.
.idealstate.split_bucket.pending	Lists how many buckets are currently being split.
.idealstate.join_bucket.pending	Lists how many buckets are currently being joined.
.idealstate.set_bucket_state.pending	Lists how many buckets are currently altered for active state. These are high priority requests which should finish fast, so these requests should seldom be seen as pending.

Distributor node restart

When a distributor stops, it will try to respond to any pending cluster state request first. New incoming requests after shutdown is commenced will fail immediately, as the socket is no longer accepting requests. Cluster controllers will thus detect processes stopping almost immediately.

The cluster state will be updated with the new state internally in the cluster controller. Then the cluster controller will wait for maximum min_time_between_new_systemstates before publishing the new cluster state - this to reduce short-term state fluctuations.

The cluster controller has the option of setting states to make other distributors take over ownership of buckets, or mask the change, making the buckets owned by the distributor restarting unavailable for the time being. Distributors restart fast, so the restarting distributor may transition directly from up to initializing. If it doesn't, current default behavior is to set it down immediately.

If transitioning directly from up to initializing, requests going through the remaining distributors will be unaffected. The requests going through the restarting distributor will immediately fail when it shuts down, being resent automatically by the client. The distributor typically restart within seconds, and syncs up with the service layer nodes to get metadata on buckets it owns, in which case it is ready to serve requests again.

If the distributor transitions from up to down, and then later to initializing, other distributors will request metadata from the service layer node to take over ownership of buckets previously owned by the restarting distributor. Until the distributors have gathered this new metadata from all the service layer nodes, requests for these buckets can not be served, and will fail back to client. When the restarting node comes back up and is marked initializing or up in the cluster state again, the additional nodes will dump knowledge of the extra buckets they previously acquired.

For requests with timeouts of several seconds, the transition should be invisible due to automatic client resending. Requests with a lower timeout might fail, and it is up to the application whether to resend or handle failed requests.

Requests to buckets not owned by the restarting distributor will not be affected. The other distributors will start to do some work though, affecting latency, and distributors will refetch metadata for all buckets they own, not just the additional buckets, which may cause some disturbance.

Content node restart

When a content node restarts in a controlled fashion, it marks itself in the stopping state and rejects new requests. It will process its pending request queue before shutting down. Consequently, client requests are typically unaffected by content node restarts. The currently pending requests will typically be completed. New copies of buckets will be created on other nodes, to store new requests in appropriate redundancy. This happens whether node transitions through down or maintenance state. The difference being that if transitioning through maintenance state, the distributor will not start any effort of synchronizing new copies with existing copies. They will just store the new requests until the maintenance node comes back up.

When coming back up, content nodes will start with gathering information on what buckets it has data stored for. While this is happening, the service layer will expose that it is initializing, but not done with the bucket list stage. During this time, the cluster controller will not mark it initializing in cluster state yet. Once the service layer node knows what buckets it has, it reports that it is calculating metadata for the buckets, at which time the node may become visible as initializing in cluster state. At this time it may start process requests, but as bucket checksums have not been calculated for all buckets yet, there will exist buckets where the distributor doesn't know if they are in sync with other copies or not.

The background load to calculate bucket checksums has low priority, but load received will automatically create metadata for used buckets. With an overloaded cluster, the initializing step may not finish before all buckets have been initialized by requests. With a cluster close to max capacity, initializing may take quite some time.

The cluster is mostly unaffected during restart. During the initializing stage, bucket metadata is unknown. Distributors will assume other copies are more appropriate for serving read requests. If all copies of a bucket are in an initializing state at the same time, read requests may be sent to a bucket copy that does not have the most updated state to process it.