Content cluster processes are distributor, proton and cluster controller.
The distributor calculates the correct content node using
the distribution algorithm and the cluster state.
With no known cluster state, the client library will send requests to a random node,
which replies with the updated cluster state if the node was incorrect.
Cluster states are versioned, such that clients hitting outdated distributors do not override
updated states with old states.
The distributor keeps track of
which content nodes that stores replicas of each bucket (maximum one replica each),
based on redundancy
and information from the cluster controller.
A bucket maps to one distributor only.
A distributor keeps a bucket database with bucket metadata.
The metadata holds which content nodes store replicas of the buckets,
the checksum of the bucket content and the number of documents and meta entries within the bucket.
Each document is algorithmically mapped to a bucket and forwarded to the correct content nodes.
The distributors detect whether there are enough bucket replicas on the
content nodes and add/remove as needed.
Write operations wait for replies from every replica
and fail if less than redundancy are persisted within timeout.
The cluster controller
manages the state of the distributor and content nodes.
This cluster state is used by the document processing chains
to know which distributor to send documents to,
as well as by the distributor to know which content nodes should have which bucket.
Node state
Content and distributor nodes have state:
State
Description
Up
The node is up and available to keep buckets and serve requests.
Down
The node is not available, and can not be used.
Initializing
The node is starting up.
It knows what buckets it stores data for, and may serve requests,
but it does not know the metadata for all its buckets yet,
such as checksums and document counts.
The node is available for bucket placement.
Stopping
This node is stopping and is expected to be down soon.
This state is typically only exposed to the cluster controller
to tell why the node stopped.
The cluster controller will expose the node as down
or in maintenance mode for the rest of the cluster.
This state is thus not seen by the distribution algorithm.
Maintenance
This node is temporarily unavailable.
The node is available for bucket placement, redundancy is hence lower.
Using this mode, new replicas of the documents stored on this node will not be created,
allowing the node to be down with less of a performance impact on the rest of the cluster.
This mode is typically used to mask a down state during controlled node restarts,
or by an administrator that need to do some short maintenance work,
like upgrading software or restart the node.
Retired
A retired node is available and serves requests.
This state is used to remove nodes while keeping redundancy.
Buckets are moved to other nodes (with low priority), until empty.
Special considerations apply when using
grouped distribution
as buckets are not necessarily removed.
Distributor nodes start / transfer buckets quickly
and are hence not in maintenance or retired.
The cluster controller fetches states from all nodes, called unit states.
States reported from the nodes are either initializing, up or stopping.
If the node can not be reached, a down state is assumed.
This means, the cluster controller detects failed nodes.
The subsequent generated states will hence have nodes in down,
and the ideal state algorithm will redistribute
buckets of documents.
User state
user state must be one of
up, down, maintenance or retired:
Retire a node from a cluster -
use retired to move buckets to other nodes
Short-lived maintenance work -
use maintenance to avoid merging buckets to other nodes
Fail a bad node. The cluster controller or an operator can set a node down
The cluster controller generates the cluster state
from the unit and user states, over time.
The generated state is called the cluster state.
For new cluster states, the cluster state version is upped,
and the new cluster state is broadcast to all nodes.
There is a minimum time between each cluster state change.
It is possible to set a minimum capacity for the cluster state to be up.
If a cluster has so many nodes unavailable that it is considered down,
the state of each node is irrelevant,
and thus new cluster states will not be created and broadcast
before enough nodes are back for the cluster to come back up.
A cluster state indicating the entire cluster is down,
may thus have outdated data on the node level.
Cluster controller
The main task of the cluster controller is to maintain the cluster state.
This is done by polling nodes for state,
generating a cluster state,
which is then broadcast to all the content nodes in the cluster.
Note that clients do not interface with the cluster controller -
they get the cluster state from the distributors - details.
Task
Description
Node state polling
The cluster controller polls nodes, sending the current cluster state.
If the cluster state is no longer correct, the node returns correct information immediately.
If the state is correct, the request lingers on the node,
such that the node can reply to it immediately if its state changes.
After a while, the cluster controller will send a new state request to the node,
even with one pending.
This triggers a reply to the lingering request and makes the new one linger instead.
Hence, nodes have a pending state request.
During a controlled node shutdown, it starts the shutdown process
by responding to the pending state request that it is now stopping.
Note: As controlled restarts or shutdowns are implemented as TERM signals
from the config-sentinel,
the cluster controller is not able to differ between controlled and other shutdowns.
Cluster state generation
The cluster controller translates unit and user states
into the generated cluster state
Cluster state broadcast
When node unit states are received, a cluster controller internal cluster state is updated.
New cluster states are distributed with a minimum interval between.
A grace period per unit state too -
e.g, distributors and content nodes that are on the same node often stop at the same time.
The version number is upped, and the new cluster state is broadcast.
If cluster state version is reset,
clients to distributors can temporarily fail operations in the transition,
but will eventually converge on the new (lower) cluster state version.
Vespa can be configured with one cluster controller.
Reads and writes will work well in case of cluster controller down,
but other changes to the cluster (like a content node down) will not be handled.
It is hence recommended configuring a set of cluster controllers.
The cluster controller nodes elect a master,
which does the node polling and cluster state broadcast.
The other cluster controller nodes only exist to do master election
and potentially take over if the master dies.
All cluster controllers will vote for cluster controller with the lowest index that says it is ready.
If a cluster controller has more than half of the votes, it will be elected master.
As a majority vote is required,
the number of cluster controllers should be an odd number of 3 or greater.
A fresh master will not broadcast states before a transition time is passed,
allowing an old master to have some time to realize it is no longer the master.
Distributor
Buckets are mapped to distributors using the ideal state algorithm.
As the cluster state changes, buckets are re-mapped immediately.
The mapping does not overlap -
a bucket is owned by one distributor.
Distributors do not persist the bucket database,
the bucket-to-content-node mapping is kept in memory in the distributor.
Document count, persisted size and a metadata checksum per bucket is stored as well.
At distributor (re)start, content nodes are polled for bucket information,
and return which buckets are owned by this distributor (using the ideal state algorithm).
There is hence no master / name node in Vespa.
Likewise, at any distributor cluster state change,
content nodes are polled for bucket handover -
a distributor will then handle a new set of buckets.
Document operations are mapped to content nodes based on bucket locations -
each put/update/get/remove is mapped to a bucket
and sent to the right content nodes.
To manage the document set as it grows and nodes change, buckets move between content nodes.
Document API clients (i.e. container nodes with
<document-api>)
do not interface with the cluster controller,
and do not know the cluster state at startup.
Clients hence do not know which distributor to use at startup.
A random distributor is therefore used first.
If the document operation hit the wrong distributor,
WRONG_DISTRIBUTION is returned, with the current cluster state in the response.
WRONG_DISTRIBUTION is hence expected and normal at cold start / state change events.
Timestamps
Write operations
have a last modified time timestamp assigned when passing through the distributor.
The timestamp is guaranteed to be unique within the
bucket where it is stored.
The timestamp is used by the content layer to decide which operation is newest.
These timestamps can be used when visiting,
to process/retrieve documents within a given time range.
To guarantee unique timestamps, they are in microseconds -
the microsecond part is generated to avoid conflicts with other documents.
If documents are migrated between clusters,
the target cluster will have new timestamps for their entries.
Also, when reprocessing documentswithin a cluster,
documents will have new timestamps, even if not modified.
Ordering
The Document API uses the document ID to order operations.
A Document API client ensures that only one operation is pending at the same time.
This ensures that if a client sends multiple operations for the same document,
they will be processed in a defined order.
Note: If sending two write operations to the same document,
and the first operation fails,
the second operation that was enqueued is sent.
If the client chooses to just resend the failed request,
the order of operations has been switched.
If different clients have pending operations on the same document,
the order is undefined.
Maintenance operations
Distributors track which content nodes have which buckets in their bucket database.
Distributors then use the ideal state algorithm
to generate bucket maintenance operations.
A stable system has all buckets located per the ideal state:
If buckets have too few replicas, new are generated on other content nodes.
If the replicas differ, a bucket merge is issued to get replicas consistent.
If a buckets has too many replicas, superfluous are deleted.
Buckets are merged, if inconsistent, before deletion.
If two buckets exist, such that both may contain the same document,
the buckets are split or joined to remove such overlapping buckets.
Read more on inconsistent buckets.
If buckets are too small/large, they will be joined or split.
The maintenance operations have different priorities.
If no maintenance operations are needed, the cluster is said to be in the ideal state.
The distributors synchronize maintenance load with user load,
e.g. to remap requests to other buckets after bucket splitting and joining.
Restart
When a distributor stops, it will try to respond to any pending cluster state request first.
New incoming requests after shutdown is commenced will fail immediately,
as the socket is no longer accepting requests.
Cluster controllers will thus detect processes stopping almost immediately.
The cluster state will be updated with the new state internally in the cluster controller.
Then the cluster controller will wait for maximum
min_time_between_new_systemstates
before publishing the new cluster state - this to reduce short-term state fluctuations.
The cluster controller has the option of setting states to make other
distributors take over ownership of buckets, or mask the change, making the
buckets owned by the distributor restarting unavailable for the time being.
Distributors restart fast, so the restarting distributor may transition
directly from up to initializing.
If it doesn't, current default behavior is to set it down immediately.
If transitioning directly from up to initializing, requests going through
the remaining distributors will be unaffected.
The requests going through the restarting distributor will immediately fail when it shuts down,
being resent automatically by the client.
The distributor typically restart within seconds,
and syncs up with the service layer nodes to get metadata on buckets it owns,
in which case it is ready to serve requests again.
If the distributor transitions from up to down, and then later to initializing,
other distributors will request metadata from the service layer node to take
over ownership of buckets previously owned by the restarting distributor.
Until the distributors have gathered this new metadata from all the service
layer nodes, requests for these buckets can not be served, and will fail back to client.
When the restarting node comes back up and is marked initializing or up in the cluster state again,
the additional nodes will dump knowledge of the extra buckets they previously acquired.
For requests with timeouts of several seconds,
the transition should be invisible due to automatic client resending.
Requests with a lower timeout might fail,
and it is up to the application whether to resend or handle failed requests.
Requests to buckets not owned by the restarting distributor will not be affected.
The other distributors will start to do some work though, affecting latency,
and distributors will re-fetch metadata for all buckets they own,
not just the additional buckets, which may cause some disturbance.
Content node
The content node runs proton, which is the query backend.
Restart
When a content node restarts in a controlled fashion,
it marks itself in the stopping state and rejects new requests.
It will process its pending request queue before shutting down.
Consequently, client requests are typically unaffected by content node restarts.
The currently pending requests will typically be completed.
New copies of buckets will be created on other nodes, to store new requests in appropriate redundancy.
This happens whether node transitions through down or maintenance state.
The difference being that if transitioning through maintenance state,
the distributor will not start any effort of synchronizing new copies with existing copies.
They will just store the new requests until the maintenance node comes back up.
When coming back up, content nodes will start with gathering information
on what buckets it has data stored for.
While this is happening, the service layer will expose that it is initializing,
but not done with the bucket list stage.
During this time, the cluster controller will not mark it initializing in cluster state yet.
Once the service layer node knows what buckets it has,
it reports that it is calculating metadata for the buckets,
at which time the node may become visible as initializing in cluster state.
At this time it may start process requests,
but as bucket checksums have not been calculated for all buckets yet,
there will exist buckets where the distributor doesn't know if they are in sync with other copies or not.
The background load to calculate bucket checksums has low priority,
but load received will automatically create metadata for used buckets.
With an overloaded cluster, the initializing step may not finish before all buckets
have been initialized by requests.
With a cluster close to max capacity, initializing may take quite some time.
The cluster is mostly unaffected during restart.
During the initializing stage, bucket metadata is unknown.
Distributors will assume other copies are more appropriate for serving read requests.
If all copies of a bucket are in an initializing state at the same time,
read requests may be sent to a bucket copy that does not have the most updated state to process it.
Metrics
Metric
Description
.idealstate.idealstate_diff
This metric tries to create a single value indicating distance to the ideal state.
A value of zero indicates that the cluster is in the ideal state.
Graphed values of this metric gives a good indication for how
fast the cluster gets back to the ideal state after changes.
Note that some issues may hide other issues, so sometimes the graph
may appear to stand still or even go a bit up again, as resolving one
issue may have detected one or several others.
.idealstate.buckets_toofewcopies
Specifically lists how many buckets have too few copies.
Compare to the buckets metric to see how big a portion of the cluster this is.
.idealstate.buckets_toomanycopies
Specifically lists how many buckets have too many copies.
Compare to the buckets metric to see how big a portion of the cluster this is.
.idealstate.buckets
The total number of buckets managed. Used by other metrics reporting
bucket counts to know how big a part of the cluster they relate to.
.idealstate.buckets_notrusted
Lists how many buckets have no trusted copies.
Without trusted buckets operations against the bucket may have poor performance,
having to send requests to many copies to try and create consistent replies.
.idealstate.delete_bucket.pending
Lists how many buckets that needs to be deleted.
.idealstate.merge_bucket.pending
Lists how many buckets there are,
where we suspect not all copies store identical document sets.
.idealstate.split_bucket.pending
Lists how many buckets are currently being split.
.idealstate.join_bucket.pending
Lists how many buckets are currently being joined.
.idealstate.set_bucket_state.pending
Lists how many buckets are currently altered for active state.
These are high priority requests which should finish fast,
so these requests should seldom be seen as pending.
Example, using the quickstart -
find the distributor port (look for HTTP):
