• [+] expand all

Distributor Metrics

NameDescriptionUnit
vds.idealstate.buckets_rechecking The number of buckets that we are rechecking for ideal state operations bucket
vds.idealstate.idealstate_diff A number representing the current difference from the ideal state. This is a number that decreases steadily as the system is getting closer to the ideal state bucket
vds.idealstate.buckets_toofewcopies The number of buckets the distributor controls that have less than the desired redundancy bucket
vds.idealstate.buckets_toomanycopies The number of buckets the distributor controls that have more than the desired redundancy bucket
vds.idealstate.buckets The number of buckets the distributor controls bucket
vds.idealstate.buckets_notrusted The number of buckets that have no trusted copies. bucket
vds.idealstate.bucket_replicas_moving_out Bucket replicas that should be moved out, e.g. retirement case or node added to cluster that has higher ideal state priority. bucket
vds.idealstate.bucket_replicas_copying_out Bucket replicas that should be copied out, e.g. node is in ideal state but might have to provide data other nodes in a merge bucket
vds.idealstate.bucket_replicas_copying_in Bucket replicas that should be copied in, e.g. node does not have a replica for a bucket that it is in ideal state for bucket
vds.idealstate.bucket_replicas_syncing Bucket replicas that need syncing due to mismatching metadata bucket
vds.idealstate.max_observed_time_since_last_gc_sec Maximum time (in seconds) since GC was last successfully run for a bucket. Aggregated max value across all buckets on the distributor. second
vds.idealstate.delete_bucket.done_ok The number of operations successfully performed operation
vds.idealstate.delete_bucket.done_failed The number of operations that failed operation
vds.idealstate.delete_bucket.pending The number of operations pending operation
vds.idealstate.delete_bucket.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.delete_bucket.throttled The number of operations throttled by throttling operation starter operation
vds.idealstate.merge_bucket.done_ok The number of operations successfully performed operation
vds.idealstate.merge_bucket.done_failed The number of operations that failed operation
vds.idealstate.merge_bucket.pending The number of operations pending operation
vds.idealstate.merge_bucket.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.merge_bucket.throttled The number of operations throttled by throttling operation starter operation
vds.idealstate.merge_bucket.source_only_copy_changed The number of merge operations where source-only copy changed operation
vds.idealstate.merge_bucket.source_only_copy_delete_blocked The number of merge operations where delete of unchanged source-only copies was blocked operation
vds.idealstate.merge_bucket.source_only_copy_delete_failed The number of merge operations where delete of unchanged source-only copies failed operation
vds.idealstate.split_bucket.done_ok The number of operations successfully performed operation
vds.idealstate.split_bucket.done_failed The number of operations that failed operation
vds.idealstate.split_bucket.pending The number of operations pending operation
vds.idealstate.split_bucket.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.split_bucket.throttled The number of operations throttled by throttling operation starter operation
vds.idealstate.join_bucket.done_ok The number of operations successfully performed operation
vds.idealstate.join_bucket.done_failed The number of operations that failed operation
vds.idealstate.join_bucket.pending The number of operations pending operation
vds.idealstate.join_bucket.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.join_bucket.throttled The number of operations throttled by throttling operation starter operation
vds.idealstate.garbage_collection.done_ok The number of operations successfully performed operation
vds.idealstate.garbage_collection.done_failed The number of operations that failed operation
vds.idealstate.garbage_collection.pending The number of operations pending operation
vds.idealstate.garbage_collection.documents_removed Number of documents removed by GC operations document
vds.idealstate.garbage_collection.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.garbage_collection.throttled The number of operations throttled by throttling operation starter operation
vds.distributor.puts.latency The latency of put operations millisecond
vds.distributor.puts.ok The number of successful put operations performed operation
vds.distributor.puts.failures.total Sum of all failures operation
vds.distributor.puts.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.puts.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.puts.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.puts.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.puts.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.puts.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.puts.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.puts.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.puts.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.puts.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.puts.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.removes.latency The latency of remove operations millisecond
vds.distributor.removes.ok The number of successful removes operations performed operation
vds.distributor.removes.failures.total Sum of all failures operation
vds.distributor.removes.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.removes.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.removes.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.removes.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.removes.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.removes.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.removes.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.removes.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.removes.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.removes.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.removes.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.updates.latency The latency of update operations millisecond
vds.distributor.updates.ok The number of successful updates operations performed operation
vds.distributor.updates.failures.total Sum of all failures operation
vds.distributor.updates.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.updates.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.updates.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.updates.diverging_timestamp_updates Number of updates that report they were performed against divergent version timestamps on different replicas operation
vds.distributor.updates.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.updates.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.updates.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.updates.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.updates.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.updates.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.updates.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.updates.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.updates.fast_path_restarts Number of safe path (write repair) updates that were restarted as fast path updates because all replicas returned documents with the same timestamp in the initial read phase operation
vds.distributor.removelocations.ok The number of successful removelocations operations performed operation
vds.distributor.removelocations.failures.total Sum of all failures operation
vds.distributor.removelocations.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.removelocations.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.removelocations.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.removelocations.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.removelocations.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.removelocations.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.removelocations.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.removelocations.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.removelocations.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.removelocations.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.removelocations.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.removelocations.latency The average latency of removelocations operations millisecond
vds.distributor.gets.latency The average latency of gets operations millisecond
vds.distributor.gets.ok The number of successful gets operations performed operation
vds.distributor.gets.failures.total Sum of all failures operation
vds.distributor.gets.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.gets.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.gets.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.gets.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.gets.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.gets.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.gets.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.gets.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.gets.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.gets.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.gets.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.visitor.latency The average latency of visitor operations millisecond
vds.distributor.visitor.ok The number of successful visitor operations performed operation
vds.distributor.visitor.failures.total Sum of all failures operation
vds.distributor.visitor.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.visitor.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.visitor.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.visitor.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.visitor.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.visitor.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.visitor.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.visitor.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.visitor.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.visitor.bytes_per_visitor The number of bytes visited on content nodes as part of a single client visitor command operation
vds.distributor.visitor.docs_per_visitor The number of documents visited on content nodes as part of a single client visitor command operation
vds.distributor.visitor.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.visitor.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.docsstored Number of documents stored in all buckets controlled by this distributor document
vds.distributor.bytesstored Number of bytes stored in all buckets controlled by this distributor byte
metricmanager.periodichooklatency Time in ms used to update a single periodic hook millisecond
metricmanager.resetlatency Time in ms used to reset all metrics. millisecond
metricmanager.sleeptime Time in ms worker thread is sleeping millisecond
metricmanager.snapshothooklatency Time in ms used to update a single snapshot hook millisecond
metricmanager.snapshotlatency Time in ms used to take a snapshot millisecond
vds.distributor.activate_cluster_state_processing_time Elapsed time where the distributor thread is blocked on merging pending bucket info into its bucket database upon activating a cluster state millisecond
vds.distributor.bucket_db.memory_usage.allocated_bytes The number of allocated bytes byte
vds.distributor.bucket_db.memory_usage.dead_bytes The number of dead bytes (<= used_bytes) byte
vds.distributor.bucket_db.memory_usage.onhold_bytes The number of bytes on hold byte
vds.distributor.bucket_db.memory_usage.used_bytes The number of used bytes (<= allocated_bytes) byte
vds.distributor.getbucketlists.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.getbucketlists.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.getbucketlists.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.getbucketlists.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.getbucketlists.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.getbucketlists.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.getbucketlists.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.getbucketlists.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.getbucketlists.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.getbucketlists.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.getbucketlists.failures.total Total number of failures operation
vds.distributor.getbucketlists.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.getbucketlists.latency The average latency of getbucketlists operations millisecond
vds.distributor.getbucketlists.ok The number of successful getbucketlists operations performed operation
vds.distributor.recoverymodeschedulingtime Time spent scheduling operations in recovery mode after receiving new cluster state millisecond
vds.distributor.set_cluster_state_processing_time Elapsed time where the distributor thread is blocked on processing its bucket database upon receiving a new cluster state millisecond
vds.distributor.state_transition_time Time it takes to complete a cluster state transition. If a state transition is preempted before completing, its elapsed time is counted as part of the total time spent for the final, completed state transition millisecond
vds.distributor.stats.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.stats.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.stats.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.stats.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.stats.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.stats.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.stats.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.stats.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.stats.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.stats.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.stats.failures.total The total number of failures operation
vds.distributor.stats.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.stats.latency The average latency of stats operations millisecond
vds.distributor.stats.ok The number of successful stats operations performed operation
vds.distributor.update_gets.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.update_gets.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.update_gets.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.update_gets.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.update_gets.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.update_gets.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.update_gets.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.update_gets.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.update_gets.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.update_gets.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.update_gets.failures.total The total number of failures operation
vds.distributor.update_gets.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.update_gets.latency The average latency of update_gets operations millisecond
vds.distributor.update_gets.ok The number of successful update_gets operations performed operation
vds.distributor.update_metadata_gets.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.update_metadata_gets.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.update_metadata_gets.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.update_metadata_gets.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.update_metadata_gets.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.update_metadata_gets.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.update_metadata_gets.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.update_metadata_gets.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.update_metadata_gets.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.update_metadata_gets.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.update_metadata_gets.failures.total The total number of failures operation
vds.distributor.update_metadata_gets.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.update_metadata_gets.latency The average latency of update_metadata_gets operations millisecond
vds.distributor.update_metadata_gets.ok The number of successful update_metadata_gets operations performed operation
vds.distributor.update_puts.failures.busy The number of messages from storage that failed because the storage node was busy operation
vds.distributor.update_puts.failures.concurrent_mutations The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID operation
vds.distributor.update_puts.failures.inconsistent_bucket The number of operations failed due to buckets being in an inconsistent state or not found operation
vds.distributor.update_puts.failures.notconnected The number of operations discarded because there were no available storage nodes to send to operation
vds.distributor.update_puts.failures.notfound The number of operations that failed because the document did not exist operation
vds.distributor.update_puts.failures.notready The number of operations discarded because distributor was not ready operation
vds.distributor.update_puts.failures.safe_time_not_reached The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed operation
vds.distributor.update_puts.failures.storagefailure The number of operations that failed in storage operation
vds.distributor.update_puts.failures.test_and_set_failed The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document operation
vds.distributor.update_puts.failures.timeout The number of operations that failed because the operation timed out towards storage operation
vds.distributor.update_puts.failures.total The total number of put failures operation
vds.distributor.update_puts.failures.wrongdistributor The number of operations discarded because they were sent to the wrong distributor operation
vds.distributor.update_puts.latency The average latency of update_puts operations millisecond
vds.distributor.update_puts.ok The number of successful update_puts operations performed operation
vds.idealstate.nodes_per_merge The number of nodes involved in a single merge operation. node
vds.idealstate.set_bucket_state.blocked The number of operations blocked by blocking operation starter operation
vds.idealstate.set_bucket_state.done_failed The number of operations that failed operation
vds.idealstate.set_bucket_state.done_ok The number of operations successfully performed operation
vds.idealstate.set_bucket_state.pending The number of operations pending operation
vds.idealstate.set_bucket_state.throttled The number of operations throttled by throttling operation starter operation
vds.bouncer.clock_skew_aborts Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range operation