• [+] expand all

Storage Metrics

NameDescriptionUnit

vds.datastored.alldisks.buckets

Number of buckets managed bucket

vds.datastored.alldisks.docs

Number of documents stored document

vds.datastored.alldisks.bytes

Number of bytes stored byte

vds.datastored.alldisks.activebuckets

Number of active buckets on the node bucket

vds.datastored.alldisks.readybuckets

Number of ready buckets on the node bucket

vds.visitor.allthreads.averagevisitorlifetime

Average lifetime of a visitor millisecond

vds.visitor.allthreads.averagequeuewait

Average time an operation spends in input queue. millisecond

vds.visitor.allthreads.queuesize

Size of input message queue. operation

vds.visitor.allthreads.completed

Number of visitors completed operation

vds.visitor.allthreads.created

Number of visitors created. operation

vds.visitor.allthreads.failed

Number of visitors failed operation

vds.visitor.allthreads.averagemessagesendtime

Average time it takes for messages to be sent to their target (and be replied to) millisecond

vds.visitor.allthreads.averageprocessingtime

Average time used to process visitor requests millisecond

vds.visitor.allthreads.aborted

Number of visitors aborted. instance

vds.visitor.allthreads.averagevisitorcreationtime

Average time spent creating a visitor instance millisecond

vds.visitor.allthreads.destination_failure_replies

Number of failure replies received from the visitor destination instance

vds.filestor.queuesize

Size of input message queue. operation

vds.filestor.averagequeuewait

Average time an operation spends in input queue. millisecond

vds.filestor.active_operations.size

Number of concurrent active operations operation

vds.filestor.active_operations.latency

Latency (in ms) for completed operations millisecond

vds.filestor.throttle_window_size

Current size of async operation throttler window size operation

vds.filestor.throttle_waiting_threads

Number of threads waiting to acquire a throttle token thread

vds.filestor.throttle_active_tokens

Current number of active throttle tokens instance

vds.filestor.allthreads.mergemetadatareadlatency

Time spent in a merge step to check metadata of current node to see what data it has. millisecond

vds.filestor.allthreads.mergedatareadlatency

Time spent in a merge step to read data other nodes need. millisecond

vds.filestor.allthreads.mergedatawritelatency

Time spent in a merge step to write data needed to current node. millisecond

vds.filestor.allthreads.mergeavgdatareceivedneeded

Amount of data transferred from previous node in chain that we needed to apply locally. byte

vds.filestor.allthreads.mergebuckets.count

Number of requests processed. request

vds.filestor.allthreads.mergebuckets.failed

Number of failed requests. request

vds.filestor.allthreads.mergebuckets.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.mergelatencytotal

Latency of total merge operation, from master node receives it, until merge is complete and master node replies. millisecond

vds.filestor.allthreads.merge_put_latency

Latency of individual puts that are part of merge operations millisecond

vds.filestor.allthreads.merge_remove_latency

Latency of individual removes that are part of merge operations millisecond

vds.filestor.allstripes.throttled_rpc_direct_dispatches

Number of times an RPC thread could not directly dispatch an async operation directly to Proton because it was disallowed by the throttle policy instance

vds.filestor.allstripes.throttled_persistence_thread_polls

Number of times a persistence thread could not immediately dispatch a queued async operation because it was disallowed by the throttle policy instance

vds.filestor.allstripes.timeouts_waiting_for_throttle_token

Number of times a persistence thread timed out waiting for an available throttle policy token instance

vds.filestor.allstripes.averagequeuewait

Average time an operation spends in input queue. millisecond

vds.filestor.allthreads.put.count

Number of requests processed. operation

vds.filestor.allthreads.put.failed

Number of failed requests. operation

vds.filestor.allthreads.put.test_and_set_failed

Number of operations that were skipped due to a test-and-set condition not met operation

vds.filestor.allthreads.put.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.put.request_size

Size of requests, in bytes byte

vds.filestor.allthreads.remove.count

Number of requests processed. operation

vds.filestor.allthreads.remove.failed

Number of failed requests. operation

vds.filestor.allthreads.remove.test_and_set_failed

Number of operations that were skipped due to a test-and-set condition not met operation

vds.filestor.allthreads.remove.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.remove.request_size

Size of requests, in bytes byte

vds.filestor.allthreads.remove.not_found

Number of requests that could not be completed due to source document not found. request

vds.filestor.allthreads.get.count

Number of requests processed. operation

vds.filestor.allthreads.get.failed

Number of failed requests. operation

vds.filestor.allthreads.get.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.get.request_size

Size of requests, in bytes byte

vds.filestor.allthreads.get.not_found

Number of requests that could not be completed due to source document not found. request

vds.filestor.allthreads.update.count

Number of requests processed. request

vds.filestor.allthreads.update.failed

Number of failed requests. request

vds.filestor.allthreads.update.test_and_set_failed

Number of requests that were skipped due to a test-and-set condition not met request

vds.filestor.allthreads.update.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.update.request_size

Size of requests, in bytes byte

vds.filestor.allthreads.update.latency_read

Latency of the source read in the request. millisecond

vds.filestor.allthreads.update.not_found

Number of requests that could not be completed due to source document not found. request

vds.filestor.allthreads.createiterator.count

Number of requests processed. request

vds.filestor.allthreads.createiterator.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.createiterator.failed

Number of failed requests. request

vds.filestor.allthreads.visit.count

Number of requests processed. request

vds.filestor.allthreads.visit.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.visit.docs

Number of entries read per iterate call document

vds.filestor.allthreads.visit.failed

Number of failed requests. request

vds.filestor.allthreads.remove_location.count

Number of requests processed. request

vds.filestor.allthreads.remove_location.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.remove_location.failed

Number of failed requests. request

vds.filestor.allthreads.splitbuckets.count

Number of requests processed. request

vds.filestor.allthreads.splitbuckets.failed

Number of failed requests. request

vds.filestor.allthreads.splitbuckets.latency

Latency of successful requests. request

vds.filestor.allthreads.joinbuckets.count

Number of requests processed. request

vds.filestor.allthreads.joinbuckets.failed

Number of failed requests. request

vds.filestor.allthreads.joinbuckets.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.deletebuckets.count

Number of requests processed. request

vds.filestor.allthreads.deletebuckets.failed

Number of failed requests. request

vds.filestor.allthreads.deletebuckets.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.remove_by_gid.count

Number of requests processed. request

vds.filestor.allthreads.remove_by_gid.failed

Number of failed requests. request

vds.filestor.allthreads.remove_by_gid.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.setbucketstates.count

Number of requests processed. request

vds.filestor.allthreads.setbucketstates.failed

Number of failed requests. request

vds.filestor.allthreads.setbucketstates.latency

Latency of successful requests. millisecond

vds.mergethrottler.averagequeuewaitingtime

Time merges spent in the throttler queue millisecond

vds.mergethrottler.queuesize

Length of merge queue instance

vds.mergethrottler.active_window_size

Number of merges active within the pending window size instance

vds.mergethrottler.estimated_merge_memory_usage

An estimated upper bound of the memory usage (in bytes) of the merges currently in the active window byte

vds.mergethrottler.bounced_due_to_back_pressure

Number of merges bounced due to resource exhaustion back-pressure instance

vds.mergethrottler.locallyexecutedmerges.ok

The number of successful merges for 'locallyexecutedmerges' instance

vds.mergethrottler.locallyexecutedmerges.failures.aborted

The number of merges that failed because the storage node was (most likely) shutting down operation

vds.mergethrottler.locallyexecutedmerges.failures.bucketnotfound

The number of operations that failed because the bucket did not exist operation

vds.mergethrottler.locallyexecutedmerges.failures.busy

The number of merges that failed because the storage node was busy operation

vds.mergethrottler.locallyexecutedmerges.failures.exists

The number of merges that were rejected due to a merge operation for their bucket already being processed operation

vds.mergethrottler.locallyexecutedmerges.failures.notready

The number of merges discarded because distributor was not ready operation

vds.mergethrottler.locallyexecutedmerges.failures.other

The number of other failures operation

vds.mergethrottler.locallyexecutedmerges.failures.rejected

The number of merges that were rejected operation

vds.mergethrottler.locallyexecutedmerges.failures.timeout

The number of merges that failed because they timed out towards storage operation

vds.mergethrottler.locallyexecutedmerges.failures.total

Sum of all failures operation

vds.mergethrottler.locallyexecutedmerges.failures.wrongdistribution

The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current operation

vds.mergethrottler.mergechains.ok

The number of successful merges for 'mergechains' operation

vds.mergethrottler.mergechains.failures.busy

The number of merges that failed because the storage node was busy operation

vds.mergethrottler.mergechains.failures.total

Sum of all failures operation

vds.mergethrottler.mergechains.failures.exists

The number of merges that were rejected due to a merge operation for their bucket already being processed operation

vds.mergethrottler.mergechains.failures.notready

The number of merges discarded because distributor was not ready operation

vds.mergethrottler.mergechains.failures.other

The number of other failures operation

vds.mergethrottler.mergechains.failures.rejected

The number of merges that were rejected operation

vds.mergethrottler.mergechains.failures.timeout

The number of merges that failed because they timed out towards storage operation

vds.mergethrottler.mergechains.failures.wrongdistribution

The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current operation

vds.server.network.tls-handshakes-failed

Number of client or server connection attempts that failed during TLS handshaking operation

vds.server.network.peer-authorization-failures

Number of TLS connection attempts failed due to bad or missing peer certificate credentials failure

vds.server.network.client.tls-connections-established

Number of secure mTLS connections established connection

vds.server.network.server.tls-connections-established

Number of secure mTLS connections established connection

vds.server.network.client.insecure-connections-established

Number of insecure (plaintext) connections established connection

vds.server.network.server.insecure-connections-established

Number of insecure (plaintext) connections established connection

vds.server.network.tls-connections-broken

Number of TLS connections broken due to failures during frame encoding or decoding connection

vds.server.network.failed-tls-config-reloads

Number of times background reloading of TLS config has failed failure

vds.bouncer.unavailable_node_aborts

Number of operations that were aborted due to the node (or target bucket space) being unavailable operation

vds.changedbucketownershiphandler.avg_abort_processing_time

Average time spent aborting operations for changed buckets millisecond

vds.changedbucketownershiphandler.external_load_ops_aborted

Number of outdated external load operations aborted operation

vds.changedbucketownershiphandler.ideal_state_ops_aborted

Number of outdated ideal state operations aborted operation

vds.communication.bucket_space_mapping_failures

Number of messages that could not be resolved to a known bucket space operation

vds.communication.convertfailures

Number of messages that failed to get converted to storage API messages operation

vds.communication.exceptionmessageprocesstime

Time transport thread uses to process a single message that fails with an exception thrown into communication manager millisecond

vds.communication.messageprocesstime

Time transport thread uses to process a single message millisecond

vds.communication.messagequeue

Size of input message queue. item

vds.communication.sendcommandlatency

Average ms used to send commands to MBUS millisecond

vds.communication.sendreplylatency

Average ms used to send replies to MBUS millisecond

vds.communication.toolittlememory

Number of messages failed due to too little memory available operation

vds.datastored.bucket_space.active_buckets

Number of active buckets in the bucket space bucket

vds.datastored.bucket_space.bucket_db.memory_usage.allocated_bytes

The number of allocated bytes byte

vds.datastored.bucket_space.bucket_db.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

vds.datastored.bucket_space.bucket_db.memory_usage.onhold_bytes

The number of bytes on hold byte

vds.datastored.bucket_space.bucket_db.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

vds.datastored.bucket_space.buckets_total

Total number buckets present in the bucket space (ready + not ready) bucket

vds.datastored.bucket_space.entries

Number of entries (documents + tombstones) stored in the bucket space document

vds.datastored.bucket_space.bytes

Bytes stored across all documents in the bucket space byte

vds.datastored.bucket_space.docs

Documents stored in the bucket space document

vds.datastored.bucket_space.ready_buckets

Number of ready buckets in the bucket space bucket

vds.datastored.fullbucketinfolatency

Amount of time spent to process a full bucket info request millisecond

vds.datastored.fullbucketinforeqsize

Amount of distributors answered at once in full bucket info requests. node

vds.datastored.simplebucketinforeqsize

Amount of buckets returned in simple bucket info requests bucket

vds.filestor.allthreads.applybucketdiff.count

Number of requests processed. request

vds.filestor.allthreads.applybucketdiff.failed

Number of failed requests. request

vds.filestor.allthreads.applybucketdiff.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.applybucketdiffreply

Number of applybucketdiff replies that have been processed. request

vds.filestor.allthreads.bucketfixed

Number of times bucket has been fixed because of corruption bucket

vds.filestor.allthreads.bucketverified.count

Number of requests processed. request

vds.filestor.allthreads.bucketverified.failed

Number of failed requests. request

vds.filestor.allthreads.bucketverified.latency

Latency of successful requests. request

vds.filestor.allthreads.bytesmerged

Total number of bytes merged into this node. byte

vds.filestor.allthreads.createbuckets.count

Number of requests processed. request

vds.filestor.allthreads.createbuckets.failed

Number of failed requests. request

vds.filestor.allthreads.createbuckets.latency

Latency of successful requests. request

vds.filestor.allthreads.failedoperations

Number of operations throwing exceptions. operation

vds.filestor.allthreads.getbucketdiff.count

Number of requests processed. request

vds.filestor.allthreads.getbucketdiff.failed

Number of failed requests. request

vds.filestor.allthreads.getbucketdiff.latency

Latency of successful requests. request

vds.filestor.allthreads.getbucketdiffreply

Number of getbucketdiff replies that have been processed. request

vds.filestor.allthreads.internaljoin.count

Number of requests processed. request

vds.filestor.allthreads.internaljoin.failed

Number of failed requests. request

vds.filestor.allthreads.internaljoin.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.movedbuckets.count

Number of requests processed. request

vds.filestor.allthreads.movedbuckets.failed

Number of failed requests. request

vds.filestor.allthreads.movedbuckets.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.operations

Number of operations processed. operation

vds.filestor.allthreads.readbucketinfo.count

Number of requests processed. request

vds.filestor.allthreads.readbucketinfo.failed

Number of failed requests. request

vds.filestor.allthreads.readbucketinfo.latency

Latency of successful requests. request

vds.filestor.allthreads.readbucketlist.count

Number of requests processed. request

vds.filestor.allthreads.readbucketlist.failed

Number of failed requests. request

vds.filestor.allthreads.readbucketlist.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.recheckbucketinfo.count

Number of requests processed. request

vds.filestor.allthreads.recheckbucketinfo.failed

Number of failed requests. request

vds.filestor.allthreads.recheckbucketinfo.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.revert.count

Number of requests processed. request

vds.filestor.allthreads.revert.failed

Number of failed requests. request

vds.filestor.allthreads.revert.latency

Latency of successful requests. millisecond

vds.filestor.allthreads.revert.not_found

Number of requests that could not be completed due to source document not found. request

vds.filestor.allthreads.stat_bucket.count

Number of requests processed. request

vds.filestor.allthreads.stat_bucket.failed

Number of failed requests. request

vds.filestor.allthreads.stat_bucket.latency

Latency of successful requests. request

vds.filestor.bucket_db_init_latency

Time taken (in ms) to initialize bucket databases with information from the persistence provider millisecond

vds.filestor.directoryevents

Number of directory events received. operation

vds.filestor.diskevents

Number of disk events received. operation

vds.filestor.partitionevents

Number of partition events received. operation

vds.filestor.pendingmerge

Number of buckets currently being merged. bucket

vds.filestor.waitingforlockrate

Amount of times a filestor thread has needed to wait for lock to take next message in queue. operation

vds.mergethrottler.mergechains.failures.aborted

The number of merges that failed because the storage node was (most likely) shutting down operation

vds.mergethrottler.mergechains.failures.bucketnotfound

The number of operations that failed because the bucket did not exist operation

vds.server.memoryusage

Amount of memory used by the storage subsystem byte

vds.server.memoryusage_visiting

Message use from visiting byte

vds.server.message_memory_use.highpri

Message use from high priority storage messages byte

vds.server.message_memory_use.lowpri

Message use from low priority storage messages byte

vds.server.message_memory_use.normalpri

Message use from normal priority storage messages byte

vds.server.message_memory_use.total

Message use from storage messages byte

vds.server.message_memory_use.veryhighpri

Message use from very high priority storage messages byte

vds.state_manager.invoke_state_listeners_latency

Time spent (in ms) propagating state changes to internal state listeners millisecond

vds.visitor.cv_queueevictedwaittime

Milliseconds waiting in create visitor queue, for visitors that was evicted from queue due to higher priority visitors coming millisecond

vds.visitor.cv_queuefull

Number of create visitor messages failed as queue is full operation

vds.visitor.cv_queuesize

Size of create visitor queue item

vds.visitor.cv_queuetimeoutwaittime

Milliseconds waiting in create visitor queue, for visitors that timed out while in the visitor quueue millisecond

vds.visitor.cv_queuewaittime

Milliseconds waiting in create visitor queue, for visitors that was added to visitor queue but scheduled later millisecond

vds.visitor.cv_skipqueue

Number of times we could skip queue as we had free visitor spots operation

vds.server.network.rpc-capability-checks-failed

Number of RPC operations that failed to due one or more missing capabilities failure

vds.server.network.status-capability-checks-failed

Number of status page operations that failed to due one or more missing capabilities failure

vds.server.fnet.num-connections

Total number of connection objects connection