# Distributor Metrics [](/en/reference/operations/metrics/distributor.html.md "View as Markdown") | Name | Unit | Description | | --- | --- | --- | | vds.idealstate.buckets\_rechecking | bucket | The number of buckets that we are rechecking for ideal state operations | | vds.idealstate.idealstate\_diff | bucket | A number representing the current difference from the ideal state. This is a number that decreases steadily as the system is getting closer to the ideal state | | vds.idealstate.buckets\_toofewcopies | bucket | The number of buckets the distributor controls that have less than the desired redundancy | | vds.idealstate.buckets\_toomanycopies | bucket | The number of buckets the distributor controls that have more than the desired redundancy | | vds.idealstate.buckets | bucket | The number of buckets the distributor controls | | vds.idealstate.buckets\_notrusted | bucket | The number of buckets that have no trusted copies. | | vds.idealstate.bucket\_replicas\_moving\_out | bucket | Bucket replicas that should be moved out, e.g. retirement case or node added to cluster that has higher ideal state priority. | | vds.idealstate.bucket\_replicas\_copying\_out | bucket | Bucket replicas that should be copied out, e.g. node is in ideal state but might have to provide data other nodes in a merge | | vds.idealstate.bucket\_replicas\_copying\_in | bucket | Bucket replicas that should be copied in, e.g. node does not have a replica for a bucket that it is in ideal state for | | vds.idealstate.bucket\_replicas\_syncing | bucket | Bucket replicas that need syncing due to mismatching metadata | | vds.idealstate.max\_observed\_time\_since\_last\_gc\_sec | second | Maximum time (in seconds) since GC was last successfully run for a bucket. Aggregated max value across all buckets on the distributor. | | vds.idealstate.delete\_bucket.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.delete\_bucket.done\_failed | operation | The number of operations that failed | | vds.idealstate.delete\_bucket.pending | operation | The number of operations pending | | vds.idealstate.delete\_bucket.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.delete\_bucket.throttled | operation | The number of operations throttled by throttling operation starter | | vds.idealstate.merge\_bucket.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.merge\_bucket.done\_failed | operation | The number of operations that failed | | vds.idealstate.merge\_bucket.pending | operation | The number of operations pending | | vds.idealstate.merge\_bucket.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.merge\_bucket.throttled | operation | The number of operations throttled by throttling operation starter | | vds.idealstate.merge\_bucket.source\_only\_copy\_changed | operation | The number of merge operations where source-only copy changed | | vds.idealstate.merge\_bucket.source\_only\_copy\_delete\_blocked | operation | The number of merge operations where delete of unchanged source-only copies was blocked | | vds.idealstate.merge\_bucket.source\_only\_copy\_delete\_failed | operation | The number of merge operations where delete of unchanged source-only copies failed | | vds.idealstate.split\_bucket.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.split\_bucket.done\_failed | operation | The number of operations that failed | | vds.idealstate.split\_bucket.pending | operation | The number of operations pending | | vds.idealstate.split\_bucket.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.split\_bucket.throttled | operation | The number of operations throttled by throttling operation starter | | vds.idealstate.join\_bucket.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.join\_bucket.done\_failed | operation | The number of operations that failed | | vds.idealstate.join\_bucket.pending | operation | The number of operations pending | | vds.idealstate.join\_bucket.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.join\_bucket.throttled | operation | The number of operations throttled by throttling operation starter | | vds.idealstate.garbage\_collection.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.garbage\_collection.done\_failed | operation | The number of operations that failed | | vds.idealstate.garbage\_collection.pending | operation | The number of operations pending | | vds.idealstate.garbage\_collection.documents\_removed | document | Number of documents removed by GC operations | | vds.idealstate.garbage\_collection.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.garbage\_collection.throttled | operation | The number of operations throttled by throttling operation starter | | vds.distributor.puts.latency | millisecond | The latency of put operations | | vds.distributor.puts.ok | operation | The number of successful put operations performed | | vds.distributor.puts.failures.total | operation | Sum of all failures | | vds.distributor.puts.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.puts.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.puts.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.puts.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.puts.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.puts.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.puts.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.puts.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.puts.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.puts.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.puts.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.removes.latency | millisecond | The latency of remove operations | | vds.distributor.removes.ok | operation | The number of successful removes operations performed | | vds.distributor.removes.failures.total | operation | Sum of all failures | | vds.distributor.removes.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.removes.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.removes.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.removes.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.removes.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.removes.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.removes.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.removes.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.removes.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.removes.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.removes.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.updates.latency | millisecond | The latency of update operations | | vds.distributor.updates.ok | operation | The number of successful updates operations performed | | vds.distributor.updates.failures.total | operation | Sum of all failures | | vds.distributor.updates.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.updates.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.updates.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.updates.diverging\_timestamp\_updates | operation | Number of updates that report they were performed against divergent version timestamps on different replicas | | vds.distributor.updates.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.updates.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.updates.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.updates.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.updates.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.updates.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.updates.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.updates.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.updates.fast\_path\_restarts | operation | Number of safe path (write repair) updates that were restarted as fast path updates because all replicas returned documents with the same timestamp in the initial read phase | | vds.distributor.removelocations.ok | operation | The number of successful removelocations operations performed | | vds.distributor.removelocations.failures.total | operation | Sum of all failures | | vds.distributor.removelocations.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.removelocations.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.removelocations.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.removelocations.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.removelocations.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.removelocations.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.removelocations.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.removelocations.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.removelocations.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.removelocations.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.removelocations.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.removelocations.latency | millisecond | The average latency of removelocations operations | | vds.distributor.gets.latency | millisecond | The average latency of gets operations | | vds.distributor.gets.ok | operation | The number of successful gets operations performed | | vds.distributor.gets.failures.total | operation | Sum of all failures | | vds.distributor.gets.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.gets.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.gets.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.gets.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.gets.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.gets.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.gets.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.gets.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.gets.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.gets.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.gets.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.visitor.latency | millisecond | The average latency of visitor operations | | vds.distributor.visitor.ok | operation | The number of successful visitor operations performed | | vds.distributor.visitor.failures.total | operation | Sum of all failures | | vds.distributor.visitor.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.visitor.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.visitor.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.visitor.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.visitor.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.visitor.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.visitor.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.visitor.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.visitor.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.visitor.bytes\_per\_visitor | operation | The number of bytes visited on content nodes as part of a single client visitor command | | vds.distributor.visitor.docs\_per\_visitor | operation | The number of documents visited on content nodes as part of a single client visitor command | | vds.distributor.visitor.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.visitor.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.docsstored | document | Number of documents stored in all buckets controlled by this distributor | | vds.distributor.bytesstored | byte | Number of bytes stored in all buckets controlled by this distributor | | metricmanager.periodichooklatency | millisecond | Time in ms used to update a single periodic hook | | metricmanager.resetlatency | millisecond | Time in ms used to reset all metrics. | | metricmanager.sleeptime | millisecond | Time in ms worker thread is sleeping | | metricmanager.snapshothooklatency | millisecond | Time in ms used to update a single snapshot hook | | metricmanager.snapshotlatency | millisecond | Time in ms used to take a snapshot | | vds.distributor.activate\_cluster\_state\_processing\_time | millisecond | Elapsed time where the distributor thread is blocked on merging pending bucket info into its bucket database upon activating a cluster state | | vds.distributor.bucket\_db.memory\_usage.allocated\_bytes | byte | The number of allocated bytes | | vds.distributor.bucket\_db.memory\_usage.dead\_bytes | byte | The number of dead bytes (\<= used\_bytes) | | vds.distributor.bucket\_db.memory\_usage.onhold\_bytes | byte | The number of bytes on hold | | vds.distributor.bucket\_db.memory\_usage.used\_bytes | byte | The number of used bytes (\<= allocated\_bytes) | | vds.distributor.getbucketlists.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.getbucketlists.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.getbucketlists.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.getbucketlists.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.getbucketlists.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.getbucketlists.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.getbucketlists.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.getbucketlists.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.getbucketlists.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.getbucketlists.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.getbucketlists.failures.total | operation | Total number of failures | | vds.distributor.getbucketlists.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.getbucketlists.latency | millisecond | The average latency of getbucketlists operations | | vds.distributor.getbucketlists.ok | operation | The number of successful getbucketlists operations performed | | vds.distributor.recoverymodeschedulingtime | millisecond | Time spent scheduling operations in recovery mode after receiving new cluster state | | vds.distributor.set\_cluster\_state\_processing\_time | millisecond | Elapsed time where the distributor thread is blocked on processing its bucket database upon receiving a new cluster state | | vds.distributor.state\_transition\_time | millisecond | Time it takes to complete a cluster state transition. If a state transition is preempted before completing, its elapsed time is counted as part of the total time spent for the final, completed state transition | | vds.distributor.stats.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.stats.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.stats.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.stats.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.stats.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.stats.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.stats.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.stats.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.stats.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.stats.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.stats.failures.total | operation | The total number of failures | | vds.distributor.stats.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.stats.latency | millisecond | The average latency of stats operations | | vds.distributor.stats.ok | operation | The number of successful stats operations performed | | vds.distributor.update\_gets.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.update\_gets.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.update\_gets.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.update\_gets.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.update\_gets.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.update\_gets.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.update\_gets.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.update\_gets.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.update\_gets.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.update\_gets.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.update\_gets.failures.total | operation | The total number of failures | | vds.distributor.update\_gets.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.update\_gets.latency | millisecond | The average latency of update\_gets operations | | vds.distributor.update\_gets.ok | operation | The number of successful update\_gets operations performed | | vds.distributor.update\_metadata\_gets.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.update\_metadata\_gets.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.update\_metadata\_gets.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.update\_metadata\_gets.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.update\_metadata\_gets.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.update\_metadata\_gets.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.update\_metadata\_gets.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.update\_metadata\_gets.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.update\_metadata\_gets.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.update\_metadata\_gets.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.update\_metadata\_gets.failures.total | operation | The total number of failures | | vds.distributor.update\_metadata\_gets.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.update\_metadata\_gets.latency | millisecond | The average latency of update\_metadata\_gets operations | | vds.distributor.update\_metadata\_gets.ok | operation | The number of successful update\_metadata\_gets operations performed | | vds.distributor.update\_puts.failures.busy | operation | The number of messages from storage that failed because the storage node was busy | | vds.distributor.update\_puts.failures.concurrent\_mutations | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID | | vds.distributor.update\_puts.failures.inconsistent\_bucket | operation | The number of operations failed due to buckets being in an inconsistent state or not found | | vds.distributor.update\_puts.failures.notconnected | operation | The number of operations discarded because there were no available storage nodes to send to | | vds.distributor.update\_puts.failures.notfound | operation | The number of operations that failed because the document did not exist | | vds.distributor.update\_puts.failures.notready | operation | The number of operations discarded because distributor was not ready | | vds.distributor.update\_puts.failures.safe\_time\_not\_reached | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed | | vds.distributor.update\_puts.failures.storagefailure | operation | The number of operations that failed in storage | | vds.distributor.update\_puts.failures.test\_and\_set\_failed | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document | | vds.distributor.update\_puts.failures.timeout | operation | The number of operations that failed because the operation timed out towards storage | | vds.distributor.update\_puts.failures.total | operation | The total number of put failures | | vds.distributor.update\_puts.failures.wrongdistributor | operation | The number of operations discarded because they were sent to the wrong distributor | | vds.distributor.update\_puts.latency | millisecond | The average latency of update\_puts operations | | vds.distributor.update\_puts.ok | operation | The number of successful update\_puts operations performed | | vds.distributor.mutating\_op\_memory\_usage | byte | Estimated amount of memory used by active mutating operations across all distributor stripes, in bytes | | vds.idealstate.nodes\_per\_merge | node | The number of nodes involved in a single merge operation. | | vds.idealstate.set\_bucket\_state.blocked | operation | The number of operations blocked by blocking operation starter | | vds.idealstate.set\_bucket\_state.done\_failed | operation | The number of operations that failed | | vds.idealstate.set\_bucket\_state.done\_ok | operation | The number of operations successfully performed | | vds.idealstate.set\_bucket\_state.pending | operation | The number of operations pending | | vds.idealstate.set\_bucket\_state.throttled | operation | The number of operations throttled by throttling operation starter | | vds.bouncer.clock\_skew\_aborts | operation | Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range | Copyright © 2026 - [Cookie Preferences](#)