SearchNode Metrics

NameDescriptionUnit

content.proton.config.generation

The oldest config generation used by this search node version

content.proton.documentdb.documents.total

The total number of documents in this documents db (ready + not-ready) document

content.proton.documentdb.documents.ready

The number of ready documents in this document db document

content.proton.documentdb.documents.active

The number of active / searchable documents in this document db document

content.proton.documentdb.documents.removed

The number of removed documents in this document db document

content.proton.documentdb.index.docs_in_memory

Number of documents in memory index document

content.proton.documentdb.disk_usage

The total disk usage (in bytes) for this document db byte

content.proton.documentdb.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.heart_beat_age

How long ago (in seconds) heart beat maintenace job was run second

content.proton.docsum.count

Docsum requests handled request

content.proton.docsum.docs

Total docsums returned document

content.proton.docsum.latency

Docsum request latency millisecond

content.proton.search_protocol.query.latency

Query request latency (seconds) second

content.proton.search_protocol.query.request_size

Query request size (network bytes) byte

content.proton.search_protocol.query.reply_size

Query reply size (network bytes) byte

content.proton.search_protocol.docsum.latency

Docsum request latency (seconds) second

content.proton.search_protocol.docsum.request_size

Docsum request size (network bytes) byte

content.proton.search_protocol.docsum.reply_size

Docsum reply size (network bytes) byte

content.proton.search_protocol.docsum.requested_documents

Total requested document summaries document

content.proton.executor.proton.queuesize

Size of executor proton task queue task

content.proton.executor.proton.accepted

Number of executor proton accepted tasks task

content.proton.executor.proton.wakeups

Number of times a executor proton worker thread has been woken up wakeup

content.proton.executor.proton.utilization

Ratio of time the executor proton worker threads has been active fraction

content.proton.executor.proton.rejected

Number of rejected tasks task

content.proton.executor.flush.queuesize

Size of executor flush task queue task

content.proton.executor.flush.accepted

Number of accepted executor flush tasks task

content.proton.executor.flush.wakeups

Number of times a executor flush worker thread has been woken up wakeup

content.proton.executor.flush.utilization

Ratio of time the executor flush worker threads has been active fraction

content.proton.executor.flush.rejected

Number of rejected tasks task

content.proton.executor.match.queuesize

Size of executor match task queue task

content.proton.executor.match.accepted

Number of accepted executor match tasks task

content.proton.executor.match.wakeups

Number of times a executor match worker thread has been woken up wakeup

content.proton.executor.match.utilization

Ratio of time the executor match worker threads has been active fraction

content.proton.executor.match.rejected

Number of rejected tasks task

content.proton.executor.docsum.queuesize

Size of executor docsum task queue task

content.proton.executor.docsum.accepted

Number of executor accepted docsum tasks task

content.proton.executor.docsum.wakeups

Number of times a executor docsum worker thread has been woken up wakeup

content.proton.executor.docsum.utilization

Ratio of time the executor docsum worker threads has been active fraction

content.proton.executor.docsum.rejected

Number of rejected tasks task

content.proton.executor.shared.queuesize

Size of executor shared task queue task

content.proton.executor.shared.accepted

Number of executor shared accepted tasks task

content.proton.executor.shared.wakeups

Number of times a executor shared worker thread has been woken up wakeup

content.proton.executor.shared.utilization

Ratio of time the executor shared worker threads has been active fraction

content.proton.executor.shared.rejected

Number of rejected tasks task

content.proton.executor.warmup.queuesize

Size of executor warmup task queue task

content.proton.executor.warmup.accepted

Number of accepted executor warmup tasks task

content.proton.executor.warmup.wakeups

Number of times a warmup executor worker thread has been woken up wakeup

content.proton.executor.warmup.utilization

Ratio of time the executor warmup worker threads has been active fraction

content.proton.executor.warmup.rejected

Number of rejected tasks task

content.proton.executor.field_writer.queuesize

Size of executor field writer task queue task

content.proton.executor.field_writer.accepted

Number of accepted executor field writer tasks task

content.proton.executor.field_writer.wakeups

Number of times a executor field writer worker thread has been woken up wakeup

content.proton.executor.field_writer.utilization

Ratio of time the executor fieldwriter worker threads has been active fraction

content.proton.executor.field_writer.saturation

Ratio indicating the max saturation of underlying worker threads. A higher saturation than utilization indicates a bottleneck in one of the worker threads. fraction

content.proton.executor.field_writer.rejected

Number of rejected tasks task

content.proton.documentdb.job.total

The job load average total of all job metrics fraction

content.proton.documentdb.job.attribute_flush

Flushing of attribute vector(s) to disk fraction

content.proton.documentdb.job.memory_index_flush

Flushing of memory index to disk fraction

content.proton.documentdb.job.disk_index_fusion

Fusion of disk indexes fraction

content.proton.documentdb.job.document_store_flush

Flushing of document store to disk fraction

content.proton.documentdb.job.document_store_compact

Compaction of document store on disk fraction

content.proton.documentdb.job.bucket_move

Moving of buckets between 'ready' and 'notready' sub databases fraction

content.proton.documentdb.job.lid_space_compact

Compaction of lid space in document meta store and attribute vectors fraction

content.proton.documentdb.job.removed_documents_prune

Pruning of removed documents in 'removed' sub database fraction

content.proton.documentdb.threading_service.master.queuesize

Size of threading service master task queue task

content.proton.documentdb.threading_service.master.accepted

Number of accepted threading service master tasks task

content.proton.documentdb.threading_service.master.wakeups

Number of times a threading service master worker thread has been woken up wakeup

content.proton.documentdb.threading_service.master.utilization

Ratio of time the threading service master worker threads has been active fraction

content.proton.documentdb.threading_service.master.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.index.queuesize

Size of threading service index task queue task

content.proton.documentdb.threading_service.index.accepted

Number of accepted threading service index tasks task

content.proton.documentdb.threading_service.index.wakeups

Number of times a threading service index worker thread has been woken up wakeup

content.proton.documentdb.threading_service.index.utilization

Ratio of time the threading service index worker threads has been active fraction

content.proton.documentdb.threading_service.index.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.summary.queuesize

Size of threading service summary task queue task

content.proton.documentdb.threading_service.summary.accepted

Number of accepted threading service summary tasks task

content.proton.documentdb.threading_service.summary.wakeups

Number of times a threading service summary worker thread has been woken up wakeup

content.proton.documentdb.threading_service.summary.utilization

Ratio of time the threading service summary worker threads has been active fraction

content.proton.documentdb.threading_service.summary.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.attribute_field_writer.accepted

Number of accepted tasks task

content.proton.documentdb.threading_service.attribute_field_writer.queuesize

Size of task queue task

content.proton.documentdb.threading_service.attribute_field_writer.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.attribute_field_writer.utilization

Ratio of time the worker threads has been active fraction

content.proton.documentdb.threading_service.attribute_field_writer.wakeups

Number of times a worker thread has been woken up wakeup

content.proton.documentdb.threading_service.index_field_inverter.accepted

Number of accepted tasks task

content.proton.documentdb.threading_service.index_field_inverter.queuesize

Size of task queue task

content.proton.documentdb.threading_service.index_field_inverter.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.index_field_inverter.utilization

Ratio of time the worker threads has been active fraction

content.proton.documentdb.threading_service.index_field_inverter.wakeups

Number of times a worker thread has been woken up wakeup

content.proton.documentdb.threading_service.index_field_writer.accepted

Number of accepted tasks task

content.proton.documentdb.threading_service.index_field_writer.queuesize

Size of task queue task

content.proton.documentdb.threading_service.index_field_writer.rejected

Number of rejected tasks task

content.proton.documentdb.threading_service.index_field_writer.utilization

Ratio of time the worker threads has been active fraction

content.proton.documentdb.threading_service.index_field_writer.wakeups

Number of times a worker thread has been woken up wakeup

content.proton.documentdb.ready.lid_space.lid_bloat_factor

The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) fraction

content.proton.documentdb.ready.lid_space.lid_fragmentation_factor

The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) fraction

content.proton.documentdb.ready.lid_space.lid_limit

The size of the allocated lid space documentid

content.proton.documentdb.ready.lid_space.highest_used_lid

The highest used lid documentid

content.proton.documentdb.ready.lid_space.used_lids

The number of lids used documentid

content.proton.documentdb.ready.lid_space.lowest_free_lid

The lowest free local document id documentid

content.proton.documentdb.notready.lid_space.lid_bloat_factor

The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) fraction

content.proton.documentdb.notready.lid_space.lid_fragmentation_factor

The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) fraction

content.proton.documentdb.notready.lid_space.lid_limit

The size of the allocated lid space documentid

content.proton.documentdb.notready.lid_space.highest_used_lid

The highest used lid documentid

content.proton.documentdb.notready.lid_space.used_lids

The number of lids used documentid

content.proton.documentdb.notready.lid_space.lowest_free_lid

The lowest free local document id documentid

content.proton.documentdb.removed.lid_space.lid_bloat_factor

The bloat factor of this lid space, indicating the total amount of holes in the allocated lid space ((lid_limit - used_lids) / lid_limit) fraction

content.proton.documentdb.removed.lid_space.lid_fragmentation_factor

The fragmentation factor of this lid space, indicating the amount of holes in the currently used part of the lid space ((highest_used_lid - used_lids) / highest_used_lid) fraction

content.proton.documentdb.removed.lid_space.lid_limit

The size of the allocated lid space documentid

content.proton.documentdb.removed.lid_space.highest_used_lid

The highest used lid documentid

content.proton.documentdb.removed.lid_space.used_lids

The number of lids used documentid

content.proton.documentdb.removed.lid_space.lowest_free_lid

The lowest free local document id documentid

content.proton.documentdb.bucket_move.buckets_pending

The number of buckets left to move bucket

content.proton.resource_usage.disk

The relative amount of disk used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller fraction

content.proton.resource_usage.disk_usage.total

The total relative amount of disk used by this content node (value in the range [0, 1]) fraction

content.proton.resource_usage.disk_usage.total_utilization

The relative amount of disk used compared to the content node disk resource limit fraction

content.proton.resource_usage.disk_usage.transient

The relative amount of transient disk used by this content node (value in the range [0, 1]) fraction

content.proton.resource_usage.memory

The relative amount of memory used by this content node (transient usage not included, value in the range [0, 1]). Same value as reported to the cluster controller fraction

content.proton.resource_usage.memory_usage.total

The total relative amount of memory used by this content node (value in the range [0, 1]) fraction

content.proton.resource_usage.memory_usage.total_utilization

The relative amount of memory used compared to the content node memory resource limit fraction

content.proton.resource_usage.memory_usage.transient

The relative amount of transient memory used by this content node (value in the range [0, 1]) fraction

content.proton.resource_usage.memory_mappings

The number of memory mapped files file

content.proton.resource_usage.open_file_descriptors

The number of open files file

content.proton.resource_usage.feeding_blocked

Whether feeding is blocked due to resource limits being reached (value is either 0 or 1) binary

content.proton.resource_usage.malloc_arena

Size of malloc arena byte

content.proton.documentdb.attribute.resource_usage.address_space

The max relative address space used among components in all attribute vectors in this document db (value in the range [0, 1]) fraction

content.proton.documentdb.attribute.resource_usage.feeding_blocked

Whether feeding is blocked due to attribute resource limits being reached (value is either 0 or 1) binary

content.proton.documentdb.attribute.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.attribute.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.attribute.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.attribute.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.resource_usage.cpu_util.setup

cpu used by system init and (re-)configuration fraction

content.proton.resource_usage.cpu_util.read

cpu used by reading data from the system fraction

content.proton.resource_usage.cpu_util.write

cpu used by writing data to the system fraction

content.proton.resource_usage.cpu_util.compact

cpu used by internal data re-structuring fraction

content.proton.resource_usage.cpu_util.other

cpu used by work not classified as a specific category fraction

content.proton.transactionlog.entries

The current number of entries in the transaction log record

content.proton.transactionlog.disk_usage

The disk usage (in bytes) of the transaction log byte

content.proton.transactionlog.replay_time

The replay time (in seconds) of the transaction log during start-up second

content.proton.documentdb.ready.document_store.disk_usage

Disk space usage in bytes byte

content.proton.documentdb.ready.document_store.disk_bloat

Disk space bloat in bytes byte

content.proton.documentdb.ready.document_store.max_bucket_spread

Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) fraction

content.proton.documentdb.ready.document_store.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.ready.document_store.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.ready.document_store.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.ready.document_store.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.notready.document_store.disk_usage

Disk space usage in bytes byte

content.proton.documentdb.notready.document_store.disk_bloat

Disk space bloat in bytes byte

content.proton.documentdb.notready.document_store.max_bucket_spread

Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) fraction

content.proton.documentdb.notready.document_store.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.notready.document_store.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.notready.document_store.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.notready.document_store.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.removed.document_store.disk_usage

Disk space usage in bytes byte

content.proton.documentdb.removed.document_store.disk_bloat

Disk space bloat in bytes byte

content.proton.documentdb.removed.document_store.max_bucket_spread

Max bucket spread in underlying files (sum(unique buckets in each chunk)/unique buckets in file) fraction

content.proton.documentdb.removed.document_store.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.removed.document_store.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.removed.document_store.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.removed.document_store.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.ready.document_store.cache.elements

Number of elements in the cache item

content.proton.documentdb.ready.document_store.cache.memory_usage

Memory usage of the cache (in bytes) byte

content.proton.documentdb.ready.document_store.cache.hit_rate

Rate of hits in the cache compared to number of lookups fraction

content.proton.documentdb.ready.document_store.cache.lookups

Number of lookups in the cache (hits + misses) operation

content.proton.documentdb.ready.document_store.cache.invalidations

Number of invalidations (erased elements) in the cache. operation

content.proton.documentdb.notready.document_store.cache.elements

Number of elements in the cache item

content.proton.documentdb.notready.document_store.cache.memory_usage

Memory usage of the cache (in bytes) byte

content.proton.documentdb.notready.document_store.cache.hit_rate

Rate of hits in the cache compared to number of lookups fraction

content.proton.documentdb.notready.document_store.cache.lookups

Number of lookups in the cache (hits + misses) operation

content.proton.documentdb.notready.document_store.cache.invalidations

Number of invalidations (erased elements) in the cache. operation

content.proton.documentdb.removed.document_store.cache.elements

Number of elements in the cache item

content.proton.documentdb.removed.document_store.cache.hit_rate

Rate of hits in the cache compared to number of lookups fraction

content.proton.documentdb.removed.document_store.cache.invalidations

Number of invalidations (erased elements) in the cache. item

content.proton.documentdb.removed.document_store.cache.lookups

Number of lookups in the cache (hits + misses) operation

content.proton.documentdb.removed.document_store.cache.memory_usage

Memory usage of the cache (in bytes) byte

content.proton.documentdb.ready.attribute.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.ready.attribute.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.ready.attribute.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.ready.attribute.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.documentdb.ready.attribute.disk_usage

Disk space usage (in bytes) of the flushed snapshot of this attribute for this document type byte

content.proton.documentdb.notready.attribute.memory_usage.allocated_bytes

The number of allocated bytes byte

content.proton.documentdb.notready.attribute.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) byte

content.proton.documentdb.notready.attribute.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) byte

content.proton.documentdb.notready.attribute.memory_usage.onhold_bytes

The number of bytes on hold byte

content.proton.index.cache.postinglist.elements

Number of elements in the cache. Contains disk index posting list files across all document types item

content.proton.index.cache.postinglist.memory_usage

Memory usage of the cache (in bytes). Contains disk index posting list files across all document types byte

content.proton.index.cache.postinglist.hit_rate

Rate of hits in the cache compared to number of lookups. Contains disk index posting list files across all document types fraction

content.proton.index.cache.postinglist.lookups

Number of lookups in the cache (hits + misses). Contains disk index posting list files across all document types operation

content.proton.index.cache.postinglist.invalidations

Number of invalidations (erased elements) in the cache. Contains disk index posting list files across all document types operation

content.proton.index.cache.bitvector.elements

Number of elements in the cache. Contains disk index bitvector files across all document types item

content.proton.index.cache.bitvector.memory_usage

Memory usage of the cache (in bytes). Contains disk index bitvector files across all document types byte

content.proton.index.cache.bitvector.hit_rate

Rate of hits in the cache compared to number of lookups. Contains disk index bitvector files across all document types fraction

content.proton.index.cache.bitvector.lookups

Number of lookups in the cache (hits + misses). Contains disk index bitvector files across all document types operation

content.proton.index.cache.bitvector.invalidations

Number of invalidations (erased elements) in the cache. Contains disk index bitvector files across all document types operation

content.proton.documentdb.index.memory_usage.allocated_bytes

The number of allocated bytes for the memory index for this document type byte

content.proton.documentdb.index.memory_usage.used_bytes

The number of used bytes (<= allocated_bytes) for the memory index for this document type byte

content.proton.documentdb.index.memory_usage.dead_bytes

The number of dead bytes (<= used_bytes) for the memory index for this document type byte

content.proton.documentdb.index.memory_usage.onhold_bytes

The number of bytes on hold for the memory index for this document type byte

content.proton.documentdb.index.disk_usage

Disk space usage (in bytes) of all disk indexes for this document type byte

content.proton.documentdb.index.io.search.read_bytes

Bytes read from disk index posting list and bitvector files as part of search for this document type byte

content.proton.documentdb.index.io.search.cached_read_bytes

Bytes read from cached disk index posting list and bitvector files as part of search for this document type byte

content.proton.documentdb.ready.index.memory_usage.allocated_bytes

The number of allocated bytes for this index field in the memory index for this document type byte

content.proton.documentdb.ready.index.disk_usage

Disk space usage (in bytes) of this index field in all disk indexes for this document type byte

content.proton.documentdb.matching.queries

Number of queries executed query

content.proton.documentdb.matching.soft_doomed_queries

Number of queries hitting the soft timeout query

content.proton.documentdb.matching.query_latency

Total average latency (sec) when matching and ranking a query second

content.proton.documentdb.matching.query_setup_time

Average time (sec) spent setting up and tearing down queries second

content.proton.documentdb.matching.docs_matched

Number of documents matched document

content.proton.documentdb.matching.docs_ranked

Number of documents ranked (first phase) document

content.proton.documentdb.matching.docs_reranked

Number of documents re-ranked (second phase) document

content.proton.documentdb.matching.rank_profile.queries

Number of queries executed query

content.proton.documentdb.matching.rank_profile.soft_doomed_queries

Number of queries hitting the soft timeout query

content.proton.documentdb.matching.rank_profile.soft_doom_factor

Factor used to compute soft-timeout fraction

content.proton.documentdb.matching.rank_profile.query_latency

Total average latency (sec) when matching and ranking a query second

content.proton.documentdb.matching.rank_profile.query_setup_time

Average time (sec) spent setting up and tearing down queries second

content.proton.documentdb.matching.rank_profile.grouping_time

Average time (sec) spent on grouping second

content.proton.documentdb.matching.rank_profile.rerank_time

Average time (sec) spent on 2nd phase ranking second

content.proton.documentdb.matching.rank_profile.docs_matched

Number of documents matched document

content.proton.documentdb.matching.rank_profile.docs_ranked

Number of documents ranked (first phase) document

content.proton.documentdb.matching.rank_profile.docs_reranked

Number of documents re-ranked (second phase) document

content.proton.documentdb.matching.rank_profile.limited_queries

Number of queries limited in match phase query

content.proton.documentdb.matching.rank_profile.docid_partition.active_time

Time (sec) spent doing actual work second

content.proton.documentdb.matching.rank_profile.docid_partition.docs_matched

Number of documents matched document

content.proton.documentdb.matching.rank_profile.docid_partition.docs_ranked

Number of documents ranked (first phase) document

content.proton.documentdb.matching.rank_profile.docid_partition.docs_reranked

Number of documents re-ranked (second phase) document

content.proton.documentdb.matching.rank_profile.docid_partition.wait_time

Time (sec) spent waiting for other external threads and resources second

content.proton.documentdb.matching.rank_profile.match_time

Average time (sec) for matching a query (1st phase) second

content.proton.documentdb.feeding.commit.operations

Number of operations included in a commit operation

content.proton.documentdb.feeding.commit.latency

Latency for commit in seconds second

content.proton.session_cache.grouping.num_cached

Number of currently cached sessions session

content.proton.session_cache.grouping.num_dropped

Number of dropped cached sessions session

content.proton.session_cache.grouping.num_insert

Number of inserted sessions session

content.proton.session_cache.grouping.num_pick

Number if picked sessions session

content.proton.session_cache.grouping.num_timedout

Number of timed out sessions session

content.proton.session_cache.search.num_cached

Number of currently cached sessions session

content.proton.session_cache.search.num_dropped

Number of dropped cached sessions session

content.proton.session_cache.search.num_insert

Number of inserted sessions session

content.proton.session_cache.search.num_pick

Number if picked sessions session

content.proton.session_cache.search.num_timedout

Number of timed out sessions session

metricmanager.periodichooklatency

Time in ms used to update a single periodic hook millisecond

metricmanager.resetlatency

Time in ms used to reset all metrics. millisecond

metricmanager.sleeptime

Time in ms worker thread is sleeping millisecond

metricmanager.snapshothooklatency

Time in ms used to update a single snapshot hook millisecond

metricmanager.snapshotlatency

Time in ms used to take a snapshot millisecond