• [+] expand all

Storage Metrics

NameDescriptionUnit
vds.datastored.alldisks.buckets Number of buckets managed bucket
vds.datastored.alldisks.docs Number of documents stored document
vds.datastored.alldisks.bytes Number of bytes stored byte
vds.datastored.alldisks.activebuckets Number of active buckets on the node bucket
vds.datastored.alldisks.readybuckets Number of ready buckets on the node bucket
vds.visitor.allthreads.averagevisitorlifetime Average lifetime of a visitor millisecond
vds.visitor.allthreads.averagequeuewait Average time an operation spends in input queue. millisecond
vds.visitor.allthreads.queuesize Size of input message queue. operation
vds.visitor.allthreads.completed Number of visitors completed operation
vds.visitor.allthreads.created Number of visitors created. operation
vds.visitor.allthreads.failed Number of visitors failed operation
vds.visitor.allthreads.averagemessagesendtime Average time it takes for messages to be sent to their target (and be replied to) millisecond
vds.visitor.allthreads.averageprocessingtime Average time used to process visitor requests millisecond
vds.visitor.allthreads.aborted Number of visitors aborted. instance
vds.visitor.allthreads.averagevisitorcreationtime Average time spent creating a visitor instance millisecond
vds.visitor.allthreads.destination_failure_replies Number of failure replies received from the visitor destination instance
vds.filestor.queuesize Size of input message queue. operation
vds.filestor.averagequeuewait Average time an operation spends in input queue. millisecond
vds.filestor.active_operations.size Number of concurrent active operations operation
vds.filestor.active_operations.latency Latency (in ms) for completed operations millisecond
vds.filestor.throttle_window_size Current size of async operation throttler window size operation
vds.filestor.throttle_waiting_threads Number of threads waiting to acquire a throttle token thread
vds.filestor.throttle_active_tokens Current number of active throttle tokens instance
vds.filestor.allthreads.mergemetadatareadlatency Time spent in a merge step to check metadata of current node to see what data it has. millisecond
vds.filestor.allthreads.mergedatareadlatency Time spent in a merge step to read data other nodes need. millisecond
vds.filestor.allthreads.mergedatawritelatency Time spent in a merge step to write data needed to current node. millisecond
vds.filestor.allthreads.mergeavgdatareceivedneeded Amount of data transferred from previous node in chain that we needed to apply locally. byte
vds.filestor.allthreads.mergebuckets.count Number of requests processed. request
vds.filestor.allthreads.mergebuckets.failed Number of failed requests. request
vds.filestor.allthreads.mergebuckets.latency Latency of successful requests. millisecond
vds.filestor.allthreads.mergelatencytotal Latency of total merge operation, from master node receives it, until merge is complete and master node replies. millisecond
vds.filestor.allthreads.put_latency Latency of individual puts that are part of merge operations millisecond
vds.filestor.allthreads.remove_latency Latency of individual removes that are part of merge operations millisecond
vds.filestor.allstripes.throttled_rpc_direct_dispatches Number of times an RPC thread could not directly dispatch an async operation directly to Proton because it was disallowed by the throttle policy instance
vds.filestor.allstripes.throttled_persistence_thread_polls Number of times a persistence thread could not immediately dispatch a queued async operation because it was disallowed by the throttle policy instance
vds.filestor.allstripes.timeouts_waiting_for_throttle_token Number of times a persistence thread timed out waiting for an available throttle policy token instance
vds.filestor.allstripes.averagequeuewait Average time an operation spends in input queue. millisecond
vds.filestor.allthreads.put.count Number of requests processed. operation
vds.filestor.allthreads.put.failed Number of failed requests. operation
vds.filestor.allthreads.put.test_and_set_failed Number of operations that were skipped due to a test-and-set condition not met operation
vds.filestor.allthreads.put.latency Latency of successful requests. millisecond
vds.filestor.allthreads.put.request_size Size of requests, in bytes byte
vds.filestor.allthreads.remove.count Number of requests processed. operation
vds.filestor.allthreads.remove.failed Number of failed requests. operation
vds.filestor.allthreads.remove.test_and_set_failed Number of operations that were skipped due to a test-and-set condition not met operation
vds.filestor.allthreads.remove.latency Latency of successful requests. millisecond
vds.filestor.allthreads.remove.request_size Size of requests, in bytes byte
vds.filestor.allthreads.remove.not_found Number of requests that could not be completed due to source document not found. request
vds.filestor.allthreads.get.count Number of requests processed. operation
vds.filestor.allthreads.get.failed Number of failed requests. operation
vds.filestor.allthreads.get.latency Latency of successful requests. millisecond
vds.filestor.allthreads.get.request_size Size of requests, in bytes byte
vds.filestor.allthreads.get.not_found Number of requests that could not be completed due to source document not found. request
vds.filestor.allthreads.update.count Number of requests processed. request
vds.filestor.allthreads.update.failed Number of failed requests. request
vds.filestor.allthreads.update.test_and_set_failed Number of requests that were skipped due to a test-and-set condition not met request
vds.filestor.allthreads.update.latency Latency of successful requests. millisecond
vds.filestor.allthreads.update.request_size Size of requests, in bytes byte
vds.filestor.allthreads.update.latency_read Latency of the source read in the request. millisecond
vds.filestor.allthreads.update.not_found Number of requests that could not be completed due to source document not found. request
vds.filestor.allthreads.createiterator.count Number of requests processed. request
vds.filestor.allthreads.createiterator.latency Latency of successful requests. millisecond
vds.filestor.allthreads.createiterator.failed Number of failed requests. request
vds.filestor.allthreads.visit.count Number of requests processed. request
vds.filestor.allthreads.visit.latency Latency of successful requests. millisecond
vds.filestor.allthreads.visit.docs Number of entries read per iterate call document
vds.filestor.allthreads.visit.failed Number of failed requests. request
vds.filestor.allthreads.remove_location.count Number of requests processed. request
vds.filestor.allthreads.remove_location.latency Latency of successful requests. millisecond
vds.filestor.allthreads.remove_location.failed Number of failed requests. request
vds.filestor.allthreads.splitbuckets.count Number of requests processed. request
vds.filestor.allthreads.splitbuckets.failed Number of failed requests. request
vds.filestor.allthreads.splitbuckets.latency Latency of successful requests. request
vds.filestor.allthreads.joinbuckets.count Number of requests processed. request
vds.filestor.allthreads.joinbuckets.failed Number of failed requests. request
vds.filestor.allthreads.joinbuckets.latency Latency of successful requests. millisecond
vds.filestor.allthreads.deletebuckets.count Number of requests processed. request
vds.filestor.allthreads.deletebuckets.failed Number of failed requests. request
vds.filestor.allthreads.deletebuckets.latency Latency of successful requests. millisecond
vds.filestor.allthreads.setbucketstates.count Number of requests processed. request
vds.filestor.allthreads.setbucketstates.failed Number of failed requests. request
vds.filestor.allthreads.setbucketstates.latency Latency of successful requests. millisecond
vds.mergethrottler.averagequeuewaitingtime Time merges spent in the throttler queue millisecond
vds.mergethrottler.queuesize Length of merge queue instance
vds.mergethrottler.active_window_size Number of merges active within the pending window size instance
vds.mergethrottler.bounced_due_to_back_pressure Number of merges bounced due to resource exhaustion back-pressure instance
vds.mergethrottler.locallyexecutedmerges.ok The number of successful merges for 'locallyexecutedmerges' instance
vds.mergethrottler.locallyexecutedmerges.failures.aborted The number of merges that failed because the storage node was (most likely) shutting down operation
vds.mergethrottler.locallyexecutedmerges.failures.bucketnotfound The number of operations that failed because the bucket did not exist operation
vds.mergethrottler.locallyexecutedmerges.failures.busy The number of merges that failed because the storage node was busy operation
vds.mergethrottler.locallyexecutedmerges.failures.exists The number of merges that were rejected due to a merge operation for their bucket already being processed operation
vds.mergethrottler.locallyexecutedmerges.failures.notready The number of merges discarded because distributor was not ready operation
vds.mergethrottler.locallyexecutedmerges.failures.other The number of other failures operation
vds.mergethrottler.locallyexecutedmerges.failures.rejected The number of merges that were rejected operation
vds.mergethrottler.locallyexecutedmerges.failures.timeout The number of merges that failed because they timed out towards storage operation
vds.mergethrottler.locallyexecutedmerges.failures.total Sum of all failures operation
vds.mergethrottler.locallyexecutedmerges.failures.wrongdistribution The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current operation
vds.mergethrottler.mergechains.ok The number of successful merges for 'mergechains' operation
vds.mergethrottler.mergechains.failures.busy The number of merges that failed because the storage node was busy operation
vds.mergethrottler.mergechains.failures.total Sum of all failures operation
vds.mergethrottler.mergechains.failures.exists The number of merges that were rejected due to a merge operation for their bucket already being processed operation
vds.mergethrottler.mergechains.failures.notready The number of merges discarded because distributor was not ready operation
vds.mergethrottler.mergechains.failures.other The number of other failures operation
vds.mergethrottler.mergechains.failures.rejected The number of merges that were rejected operation
vds.mergethrottler.mergechains.failures.timeout The number of merges that failed because they timed out towards storage operation
vds.mergethrottler.mergechains.failures.wrongdistribution The number of merges that were discarded (flushed) because they were initiated at an older cluster state than the current operation
vds.server.network.tls-handshakes-failed Number of client or server connection attempts that failed during TLS handshaking operation
vds.server.network.peer-authorization-failures Number of TLS connection attempts failed due to bad or missing peer certificate credentials failure
vds.server.network.client.tls-connections-established Number of secure mTLS connections established connection
vds.server.network.server.tls-connections-established Number of secure mTLS connections established connection
vds.server.network.client.insecure-connections-established Number of insecure (plaintext) connections established connection
vds.server.network.server.insecure-connections-established Number of insecure (plaintext) connections established connection
vds.server.network.tls-connections-broken Number of TLS connections broken due to failures during frame encoding or decoding connection
vds.server.network.failed-tls-config-reloads Number of times background reloading of TLS config has failed failure
vds.bouncer.unavailable_node_aborts Number of operations that were aborted due to the node (or target bucket space) being unavailable operation
vds.changedbucketownershiphandler.avg_abort_processing_time Average time spent aborting operations for changed buckets millisecond
vds.changedbucketownershiphandler.external_load_ops_aborted Number of outdated external load operations aborted operation
vds.changedbucketownershiphandler.ideal_state_ops_aborted Number of outdated ideal state operations aborted operation
vds.communication.bucket_space_mapping_failures Number of messages that could not be resolved to a known bucket space operation
vds.communication.convertfailures Number of messages that failed to get converted to storage API messages operation
vds.communication.exceptionmessageprocesstime Time transport thread uses to process a single message that fails with an exception thrown into communication manager millisecond
vds.communication.messageprocesstime Time transport thread uses to process a single message millisecond
vds.communication.messagequeue Size of input message queue. item
vds.communication.sendcommandlatency Average ms used to send commands to MBUS millisecond
vds.communication.sendreplylatency Average ms used to send replies to MBUS millisecond
vds.communication.toolittlememory Number of messages failed due to too little memory available operation
vds.datastored.bucket_space.active_buckets Number of active buckets in the bucket space bucket
vds.datastored.bucket_space.bucket_db.memory_usage.allocated_bytes The number of allocated bytes byte
vds.datastored.bucket_space.bucket_db.memory_usage.dead_bytes The number of dead bytes (<= used_bytes) byte
vds.datastored.bucket_space.bucket_db.memory_usage.onhold_bytes The number of bytes on hold byte
vds.datastored.bucket_space.bucket_db.memory_usage.used_bytes The number of used bytes (<= allocated_bytes) byte
vds.datastored.bucket_space.buckets_total Total number buckets present in the bucket space (ready + not ready) bucket
vds.datastored.bucket_space.bytes Bytes stored across all documents in the bucket space byte
vds.datastored.bucket_space.docs Documents stored in the bucket space document
vds.datastored.bucket_space.ready_buckets Number of ready buckets in the bucket space bucket
vds.datastored.fullbucketinfolatency Amount of time spent to process a full bucket info request millisecond
vds.datastored.fullbucketinforeqsize Amount of distributors answered at once in full bucket info requests. node
vds.datastored.simplebucketinforeqsize Amount of buckets returned in simple bucket info requests bucket
vds.filestor.allthreads.applybucketdiff.count Number of requests processed. request
vds.filestor.allthreads.applybucketdiff.failed Number of failed requests. request
vds.filestor.allthreads.applybucketdiff.latency Latency of successful requests. millisecond
vds.filestor.allthreads.applybucketdiffreply Number of applybucketdiff replies that have been processed. request
vds.filestor.allthreads.bucketfixed Number of times bucket has been fixed because of corruption bucket
vds.filestor.allthreads.bucketverified.count Number of requests processed. request
vds.filestor.allthreads.bucketverified.failed Number of failed requests. request
vds.filestor.allthreads.bucketverified.latency Latency of successful requests. request
vds.filestor.allthreads.bytesmerged Total number of bytes merged into this node. byte
vds.filestor.allthreads.createbuckets.count Number of requests processed. request
vds.filestor.allthreads.createbuckets.failed Number of failed requests. request
vds.filestor.allthreads.createbuckets.latency Latency of successful requests. request
vds.filestor.allthreads.failedoperations Number of operations throwing exceptions. operation
vds.filestor.allthreads.getbucketdiff.count Number of requests processed. request
vds.filestor.allthreads.getbucketdiff.failed Number of failed requests. request
vds.filestor.allthreads.getbucketdiff.latency Latency of successful requests. request
vds.filestor.allthreads.getbucketdiffreply Number of getbucketdiff replies that have been processed. request
vds.filestor.allthreads.internaljoin.count Number of requests processed. request
vds.filestor.allthreads.internaljoin.failed Number of failed requests. request
vds.filestor.allthreads.internaljoin.latency Latency of successful requests. millisecond
vds.filestor.allthreads.movedbuckets.count Number of requests processed. request
vds.filestor.allthreads.movedbuckets.failed Number of failed requests. request
vds.filestor.allthreads.movedbuckets.latency Latency of successful requests. millisecond
vds.filestor.allthreads.operations Number of operations processed. operation
vds.filestor.allthreads.readbucketinfo.count Number of requests processed. request
vds.filestor.allthreads.readbucketinfo.failed Number of failed requests. request
vds.filestor.allthreads.readbucketinfo.latency Latency of successful requests. request
vds.filestor.allthreads.readbucketlist.count Number of requests processed. request
vds.filestor.allthreads.readbucketlist.failed Number of failed requests. request
vds.filestor.allthreads.readbucketlist.latency Latency of successful requests. millisecond
vds.filestor.allthreads.recheckbucketinfo.count Number of requests processed. request
vds.filestor.allthreads.recheckbucketinfo.failed Number of failed requests. request
vds.filestor.allthreads.recheckbucketinfo.latency Latency of successful requests. millisecond
vds.filestor.allthreads.revert.count Number of requests processed. request
vds.filestor.allthreads.revert.failed Number of failed requests. request
vds.filestor.allthreads.revert.latency Latency of successful requests. millisecond
vds.filestor.allthreads.revert.not_found Number of requests that could not be completed due to source document not found. request
vds.filestor.allthreads.stat_bucket.count Number of requests processed. request
vds.filestor.allthreads.stat_bucket.failed Number of failed requests. request
vds.filestor.allthreads.stat_bucket.latency Latency of successful requests. request
vds.filestor.bucket_db_init_latency Time taken (in ms) to initialize bucket databases with information from the persistence provider millisecond
vds.filestor.directoryevents Number of directory events received. operation
vds.filestor.diskevents Number of disk events received. operation
vds.filestor.partitionevents Number of partition events received. operation
vds.filestor.pendingmerge Number of buckets currently being merged. bucket
vds.filestor.waitingforlockrate Amount of times a filestor thread has needed to wait for lock to take next message in queue. operation
vds.mergethrottler.mergechains.failures.aborted The number of merges that failed because the storage node was (most likely) shutting down operation
vds.mergethrottler.mergechains.failures.bucketnotfound The number of operations that failed because the bucket did not exist operation
vds.server.memoryusage Amount of memory used by the storage subsystem byte
vds.server.memoryusage_visiting Message use from visiting byte
vds.server.message_memory_use.highpri Message use from high priority storage messages byte
vds.server.message_memory_use.lowpri Message use from low priority storage messages byte
vds.server.message_memory_use.normalpri Message use from normal priority storage messages byte
vds.server.message_memory_use.total Message use from storage messages byte
vds.server.message_memory_use.veryhighpri Message use from very high priority storage messages byte
vds.state_manager.invoke_state_listeners_latency Time spent (in ms) propagating state changes to internal state listeners millisecond
vds.visitor.cv_queueevictedwaittime Milliseconds waiting in create visitor queue, for visitors that was evicted from queue due to higher priority visitors coming millisecond
vds.visitor.cv_queuefull Number of create visitor messages failed as queue is full operation
vds.visitor.cv_queuesize Size of create visitor queue item
vds.visitor.cv_queuetimeoutwaittime Milliseconds waiting in create visitor queue, for visitors that timed out while in the visitor quueue millisecond
vds.visitor.cv_queuewaittime Milliseconds waiting in create visitor queue, for visitors that was added to visitor queue but scheduled later millisecond
vds.visitor.cv_skipqueue Number of times we could skip queue as we had free visitor spots operation
vds.server.network.rpc-capability-checks-failed Number of RPC operations that failed to due one or more missing capabilities failure
vds.server.network.status-capability-checks-failed Number of status page operations that failed to due one or more missing capabilities failure
vds.server.fnet.num-connections Total number of connection objects connection