• [+] expand all

Vespa Command-line Tools

This is the reference for the Vespa command-line tools.


Use vespa-attribute-inspect to inspect the content of an attribute field - useful when troubleshooting query rewriting. Related: vespa-index-inspect.

Synopsis: vespa-attribute-inspect [-p attribute] [-a] [-s attribute] <attribute>

Example (make sure to flush the attribute before using):

$ vespa-proton-cmd --local triggerFlush && \
  vespa-attribute-inspect -p /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/attribute/year/snapshot-10/year && \
  cat /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/attribute/year/snapshot-10/year.out
Option Description
-p print content to <attribute>.out
-s save attribute to <attribute>.save.dat


vespa-configproxy-cmd is a status and control tool for the config proxy. The config proxy runs on all nodes as a proxy for one or more config servers. It connects to the config proxy on localhost by default.

Run it without arguments to dump a list of active configIds - format specification:

<config definition name>,<config id>,<config generation>,<MD5 checksum>,<xxHash checksum>

Synopsis: vespa-configproxy-cmd [args]


$ vespa-configproxy-cmd -m sources

Find a more comprehensive example in inspecting config.

Option Description
-m method, available methods are:
cache Output the config proxy cache content (overview)
getConfig Get config (see vespa-get-config)
getmode Outputs the current mode of the config proxy
setmode Use default or memorycache - example
invalidatecache Clears the current cache in config proxy
cachefull Output the config proxy cache content (including config payload)
sources Output the config proxy's upstream config sources
updatesources Updates the config proxy's upstream config sources to the supplied ones
-p port number, optional
-s hostname, optional
-h help text and usage


vespa-configserver-remove-state removes config server state on the host.

Synopsis: vespa-configserver-remove-state [-force]

Option Description
-force Do not ask for confirmation before removal


vespa-config-status can run on any machine in a Vespa cluster and outputs a list of all running services which is running with an outdated application package. It can be invoked without any arguments, or with optional arguments.

Synopsis: vespa-config-status [-v] [-c host] [-c host:port] [-f host0,...,hostN]


$ vespa-config-status
Option Description
-v Verbose - show all services, even if they are up to date
-c arg Get Vespa cluster configuration from the config server specified by host and port. Use host or host:port for config server
-f Filter to only query config status for the given comma-separated set of hosts


vespa-deploy deploys an application package, using the deploy API. Refer to the deploy reference for details.

Synopsis: vespa-deploy [-h] [-v] [-n] [-f] [-t timeout] [-c hostname] [-p port] prepare|activate|upload|fetch|help [args]


$ vespa-deploy prepare [application-path|zip-file] && vespa-deploy activate
$ vespa-deploy prepare app.zip && vespa-deploy activate
Command Description
prepare vespa-deploy prepare combines the upload and prepare steps.
activate vespa-deploy activate invokes the activate step.
upload vespa-deploy upload uploads an application package
fetch vespa-deploy fetch fetches an application package. Useful to get the active configuration for an instance.
help Same as -h
Option Description
-h Show help text
-v Verbose
-n Dry-run deployment
-f Force - ignore validation errors
-t Timeout
-c Config server hostname
-p Config server port


vespa-destination is a simple receiver for messagebus messages. It outputs messages received on stdout. Also see vespa-visit-target.

Synopsis: vespa-destination [options]


$ vespa-destination --name msg_sink
Option Description
--instant Reply in message thread
--name arg Slobrok name to register
--maxqueuetime arg Adjust the in queue size to have a maximum queue wait period of this many ms (default -1 = unlimited)
--silent #nummsg Do not dump anything, but progress every #nummsg
--sleeptime arg The number of milliseconds to sleep per message, to simulate processing time
--threads arg The number of threads to process the incoming data
--verbose Dump the contents of certain messages to stdout


vespa-fbench is used for benchmarking the capacity of a Vespa system. Refer to vespa-benchmarking for usage and examples.

Multiple hostnames and ports can be used, to distribute load round-robin to clients.

Synopsis: vespa-fbench [options] <hostname> <port>


$ vespa-fbench -n 10 -q query%03d.txt -s 300 -c 0 -o output%03d.txt -xy test.domain.com 8080
Option Description
-H header append extra header to each get request.
-A assign authority hostname:port. Overrides Host: header sent.
-a str append string to each query
-n numClients Run vespa-fbench with numClients clients in parallel. If not specified, vespa-fbench will use a default value of 10 clients.
-c cycleTime each client will make a request each <num> milliseconds [1000] ('-1' -> cycle time should be twice the response time)
-l limit minimum response size for successful requests [0]
-i ignoreCount do not log the <num> first results. -1 means no logging [0]
-s seconds run the test for <num> seconds. -1 means forever [60]
-q queryFilePattern pattern defining input query files, e.g. query%03d.txt (the pattern is used with sprintf to generate filenames). Unless using POST, a query file has one query per line, each line starting with /search/:
-P use POST for requests instead of GET. Two lines per query, format:
{"yql" : "select * from sources * where true"}
Any line starting with "/" will be taken as a URL path, with the following lines taken as the content (these lines can NOT start with "/"). With json body payload, also specify the content type using -H "Content-Type: application/json"
-o outputFilePattern save query results to output files with the given pattern (default is not saving.)
-r restartLimit number of times to re-use each query file. -1 means no limit [-1]
-m maxLineSize max line size in input query files [8192]. Can not be less than the minimum [1024].
-p seconds print summary every <num> seconds. only available when installing vespa-fbench from test branch,
-k enable HTTP keep-alive.
-d Base64 decode POST request content
-x write benchmarkdata-reporting to output file:
NumHits Number of hits returned
NumFastHits Number of actual document hits returned
TotalHitCount Total number of hits for query
QueryHits Hits as specified in query
QueryOffset Offset as specified in query
NumErrors Number of error hits returned
NumGroupHits Number of grouping hits returned
SearchTime Time used for searching. Entire query time for one phase search, first phase for two-phase search
AttributeFetchTime Time used for attribute fetching, or 0 for one phase search
FillTime Time used for summary fetching, or 0 for one phase search
-y write data on coverage to output file (must be used with -x).
DocsSearched Total number of documents in nodes searched
NodesSearched Total number of search nodes which were used
FullCoverage 1 if true, 0 if false
-z use single query file to be distributed between clients.
-T file CA certificate file to verify peer against. Use to benchmark https enabled port. (e.g -T /etc/ssl/certs/ca-bundle.crt)
-C file client certificate file name
-K file client private key file name
-D use TLS configuration from environment if T/C/K is not used
Default output:
connection reuse count Indicates how many times HTTP connections were reused to issue another request. Note that this number will only be displayed if the -k switch (enable HTTP keep-alive) is used.
clients Echo of the -n parameter.
cycle time Echo of the -c parameter.
lower response limit Echo of the -l parameter.
skipped requests Number of requests that was skipped by vespa-fbench. vespa-fbench will typically skip a request if the line containing the query url exceeds a pre-defined limit. Skipped requests will have minimal impact on the statistical results.
failed requests The number of failed requests. A request will be marked as failed if en error occurred while reading the result or if the result contained fewer bytes than 'lower response limit'.
successful requests Number of successful requests. Each performed request is counted as either successful or failed. Skipped requests (see above) are not performed and therefore not counted.
cycles not held Number of cycles not held. The cycle time is specified with the -c parameter. It defines how often a client should perform a new request. However, a client may not perform another request before the result from the previous request has been obtained. Whenever a client is unable to initiate a new request 'on time' due to not being finished with the previous request, this value will be increased.
minimum response time The minimum response time. The response time is measured as the time period from just before the request is sent to the server, till the result is obtained from the server.
maximum response time The maximum response time. The response time is measured as the time period from just before the request is sent to the server, till the result is obtained from the server.
average response time The average response time. The response time is measured as the time period from just before the request is sent to the server, till the result is obtained from the server.
X percentile The X percentile of the response time samples; a value selected such that X percent of the response time samples are below this value. In order to calculate percentiles, a histogram of response times is maintained for each client at runtime and merged after the test run ends. If a percentile value exceeds the upper bound of this histogram, it will be approximated (and thus less accurate) and marked with '(approx)'.
actual query rate The average number of queries per second; QPS.
utilization The percentage of time used waiting for the server to complete (successful) requests. Note that if a request fails, the utilization will drop since the client has 'wasted' the time spent on the failed request.
zero hit queries The number of queries that gave zero hits in Vespa


usage: vespa-fbench-filter-file [-a] [-h] [-m maxLineSize]

Read concatenated fastserver logs from stdin and write
extracted query urls to stdout.

 -a       : all parameters to the original query urls are preserved.
            If the -a switch is not given, only 'query' and 'type'
            parameters are kept in the extracted query urls.
 -h       : print this usage information.
 -m <num> : max line size for input/output lines.
            Can not be less than the default [10240]


usage: vespa-fbench-geturl <host> <port> <url>


usage: vespa-fbench-split-file [-p pattern] [-m maxLineSize] <numparts> [<file>]

Reads from <file> (stdin if <file> is not given) and
randomly distributes each line between <numpart> output
files. The names of the output files are generated by
combining the <pattern> with sequential numbers using
the sprintf function.

 -p pattern : output name pattern ['query%03d.txt']
 -m <num>   : max line size for input/output lines.
              Can not be less than the default [10240]
 <numparts> : number of output files to generate.


vespa-feeder is a feeding client that parses JSON input as Vespa document operations and sends to a Vespa application. It parses the content of the input sequentially, and feeds each operation in order. However, since many operations will be pending at any time, and because the processing time of an operation varies, there is no guarantee as to which order operations reach the content nodes. As this can be important when it comes to operations that apply to the same document id, there is logic in place to not send an operation for a document id to which there is already a pending operation.

vespa-feeder prints a report at end of feed. To print this report once a minute, use --verbose:

Messages sent to vespa (route default) :
PutDocument: ok: 999997 msgs/sec: 411.38 failed: 0 ignored: 0 latency(min, max, avg): 2, 4360, 99

ignored reports the number of documents that could not be routed to any content clusters because they did not match any of the configured document types or selections - examples are:

  • A document type is removed from the application and the feed file contains documents with this type
  • One or more selection expressions restrict the documents the cluster accepts, and the feed file contains documents that are excluded. An example is feeding expired documents - a selection for documents that are less than 30 days old and the feed file contains documents that are 30+ days old

Synopsis: vespa-feeder [--abortondataerror true|false] [--abortonsenderror true|false] [--file arg] [--maxpending arg] [--maxpendingsize arg] [--maxfeedrate arg] [mode standard|benchmark] [--noretry] [--retrydelay arg] [--route arg] [--timeout arg] [--trace arg] [--validate] [--dumpDocuments filename] [--numthreads arg] [create-if-non-existent] [-v,--verbose] filename


$ vespa-feeder file.json
Option Description
--abortondataerror arg Abort if the input has errors (true|false) - default true. Set to false in case the input has errors (e.g. invalid characters). vespa-feeder notifies on parsing errors at the end of the feed, but it will not abort
--abortonsenderror arg Abort if an error occurred while sending operations to Vespa (true|false) - default true
--file arg Input files to read. These can also be passed as arguments without the option prefix. If none is given, this tool parses identifiers from stdin
--maxpending arg Maximum number of pending operations. This disables dynamic throttling, use with care
--maxpendingsize arg Maximum size (in bytes) of pending operations
--maxfeedrate arg Limits the feed rate to the given number (operations/second)
--mode The mode to run vespa-feeder in (standard|benchmark) - default standard
--noretry Disables retries of recoverable failures
--retrydelay arg The time (in seconds) to wait between retries of a failed operation. Default 1
--route arg The route to send the data to. Default the the default route
--timeout arg Time (in seconds) allowed for sending operations. Default 180
--trace arg Trace level of network traffic. Default 0
--validate Run validation tool on input files - do not feed
--dumpDocuments <filename> File where documents in the put are serialized
--numthreads arg How many threads to use for sending. Default 1
--create-if-non-existent Enable setting of create-if-non-existent to true on all document updates in the given feed
-v, --verbose Enable verbose output of progress


vespa-get retrieves documents from a Vespa content cluster, and prints to stdout. vespa-get retrieves documents identified by the document ids passed as command line arguments. If no document ids are passed through the command line interface, ids will be read from stdin - separated by line breaks.

Synopsis: vespa-get <options> [documentid...]

Option Description
-a,--trace tracelevel Trace level to use (default 0)
-c,--configid configid Use the specified config id for messagebus configuration
-f,--fieldset fieldset Retrieve the specified fields only (see Document field sets) (default '[all]')
-h,--help Show this syntax page
-i,--printids Show only identifiers of retrieved documents
-j,--jsonoutput JSON output (default)
-l,--loadtype loadtype Load type (default "")
-n,--noretry Do not retry operation on transient errors, as is default
-r,--route route Send request to the given messagebus route
-s,--showdocsize Show binary size of document
-t,--timeout timeout Set timeout for the request in seconds (default 0)
-u,--cluster cluster Send request to the given content cluster
-x,--xmloutput XML output


Get cluster state - refer to content nodes.

Synopsis: vespa-get-cluster-state [options]

Option Description
-h, --help Show help
-v More verbose output
-s Less verbose output
--show-hidden Also show hidden undocumented debug options
-c, --cluster Cluster name of cluster to query. If unspecified, and vespa is installed on current node, information will be attempted auto-extracted
-f, --force Force execution
--config-server Host name of config server to query
--config-server-port Port to connect to config server on
--config-request-timeout Timeout of config request


vespa-get-config is a command line tool to get configuration from a config server or config proxy. By default, it connects to the config proxy on localhost, fetches config from its cache and prints the config payload on stdout. Configuration is addressed using name and configId. If configId is omitted, the global and default data for that name is returned. The default port number is 19090, the config proxy's port - use 19070 to access a config server. Also check ports.

Synopsis: vespa-get-config -n defName -i configId <option> [args]


$ vespa-get-config -n container.statistics -i search/cluster.search

Find a more comprehensive example in inspecting config.

Option Description
-n config definition name, including namespace (on the form <namespace>.<name>)
-i config id, optional
-a config def schema file, optional (if you want to use another schema than known for the config server)
-m defMd5, optional
-c configMd5, optional
-t server timeout, in seconds, default value 3, optional
-w timeout, default value 10, optional
-s server hostname, default localhost, optional
-p port, default 19090, optional
-d debug mode, optional
-h help text and usage


Get the state of one or more storage services from the fleet controller - refer to content nodes:

State Description
Unit state The state of the node seen from the cluster controller.
User state The state the administrator want the node to be in, default "up". Can be set by using vespa-set-node-state or by the cluster controller
Generated state The state of a given node in the current cluster state. This is the state all the other nodes know about. This state is a product of the other two states and cluster controller logic to keep the cluster stable.

Synopsis: vespa-get-node-state [options]

Option Description
-h, --help Show help
-v More verbose output
-s Less verbose output
--show-hidden Also show hidden undocumented debug options
-c, --cluster Cluster name of cluster to query. If unspecified, and vespa is installed on current node, information will be attempted auto-extracted
-f, --force Force execution
-t, --type Node type - can either be 'storage' or 'distributor'. If not specified, the operation will use state for both types
-i, --index Node index. If not specified, all nodes found running on this host will be used
--config-server Host name of config server to query
--config-server-port Port to connect to config server on
--config-request-timeout Timeout of config request


Use vespa-index-inspect to inspect indexed data - useful when troubleshooting query rewriting. Related: vespa-attribute-inspect.

It shows posting list information (per token or all/range), or dumps the indexed tokens:

vespa-index-inspect showpostings [--indexdir indexDir] --field field word

vespa-index-inspect showpostings [--indexdir indexDir] [--field field] --transpose \
  [--docidlimit docIdLimit] [--mindocid mindocid]

vespa-index-inspect dumpwords [--indexdir indexDir] --field field \
  [--minnumdocs minnumdocs] [--verbose] [--wordnum]

Synopsis:vespa-index-inspect showpostings|dumpwords [--indexdir path] [--field fieldname] [--transpose] [--minnumdocs count] [--docidlimit docIdLimit] [--mindocid mindocid] [--verbose] [--wordnum] [word]

Example (make sure to flush the index before using):

$ vespa-proton-cmd --local triggerFlush && \
  vespa-index-inspect dumpwords \
  --indexdir /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/index/index.flush.1 \
  --field artist
bad	2
so	1
Option Description
--indexdir path Index location
--field fieldname Field to analyze
--transpose Dump all tokens
--minnumdocs count Minimum number of documents to analyze
--docidlimit docid Dump up to this doc id
--mindocid docid Start from this docid
--wordnum Also dump token numbers
--verbose Verbose output


Dump JVM heap, thread stacks and other debugging information from a Java-based Vespa service.

Invoke binary without arguments to print help and list the services running on this node.

$ vespa-jvm-dumper

Produce JVM debugging information by invoking binary with config id of target service and output directory.

$ vespa-jvm-dumper default/container.1 /opt/vespa/tmp/jvm-dump


Print or modify log levels for a VESPA service, stored in $VESPA_HOME/var/db/vespa/logcontrol/service.logcontrol. Refer to controlling log levels for details. component-specification specifies which sub-components of the service should be controlled. If empty, all components are controlled:

  • x. : Matches only component x
  • x : Matches component x and all its sub-components

Synopsis (show log levels): vespa-logctl [OPTION] <service>[:component-specification]

Synopsis (set log levels): vespa-logctl [OPTION] <service>[:component-specification] <level-mods>

level-mods are defined as : <level>=<on|off>[,<level>=<on|off>]...

level is one of: all, fatal, error, warning, info, event, config, debug, spam

Example: For service container, set com.yahoo.search.searchchain and all sub-components of com.yahoo.search.searchchain to enable all except spam and debug:

$ vespa-logctl container:com.yahoo.search.searchchain all=on,spam=off,debug=off
Option Description
-c Create the control file if it does not exist (implies -n)
-a Update all .logcontrol files
-r Reset to default levels
-n Create the component entry if it does not exist
-f file Use <file> as the log control file
-d dir Look in <dir> for log control files


vespa-logfmt reads Vespa log files, selects messages, and writes a formatted version of these messages to standard output. If no file argument is given, vespa-logfmt will read the last Vespa log file $VESPA_HOME/logs/vespa/vespa.log (this also works with the -f option). Otherwise, reads only the files given as arguments. To read standard input, supply a single dash ’-’ as a file argument. Refer to the logs reference.

Synopsis: vespa-logfmt [-l levellist ] [-s fieldlist ] [-p pid ] [-S service ] [-H host ] [-c regex ] [-m regex ] [-f ] [-N ] [-t ] [-ts ] [file …]


Display only messages with log level "event", printing a human-readable time (without any fractional seconds), the service generating the event and the event message:

$ vespa-logfmt -l event -s fmttime,service,message
[2017-09-05 06:16:16] config-sentinel  stopped/1 name="sbin/vespa-config-sentinel -c hosts/vespa-container (pid 1558)" pid=1558 exitcode=1
[2017-09-05 06:16:16] config-sentinel  starting/1 name="sbin/vespa-config-sentinel -c hosts/vespa-container (pid 1564)"
[2017-09-05 06:16:16] config-sentinel  started/1 name="config-sentinel"
[2017-09-05 06:17:00] configserver     count/1 name=configserver.failedRequests value=0
[2017-09-05 06:17:00] configserver     count/1 name=procTime value=0
[2017-09-05 06:17:00] configserver     count/1 name=configserver.requests value=0

Display messages with log levels that are not any of info, debug, or event, printing the time in seconds and microseconds, the log level, the component name, and the message text:

$ vespa-logfmt -l all-info,-debug -s level -s time,usecs,component,message -t -l -event
1504592294.738000 WARNING : configproxy.com No config found for name=sentinel,namespace=cloud.config,configId=hosts/vespa-container within timeout, will retry
1504592296.388000 WARNING : configproxy.com Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/localhost:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/,port=37806,localport=19070] }
1504592307.949461 WARNING : config-sentinel Connection to tcp/localhost:19090 failed or timed out
1504592307.949587 WARNING : config-sentinel FRT Connection tcp/localhost:19090 suspended until 2017-09-05 06:19:07 GMT
Option Description
-l levellist (--level=levellist) Filter messages by log level. By default, only messages of level fatal, error, warning, and info will be included, while messages of level config, event, debug, and spam will be ignored. This option allows you to replace or modify the list of log levels to be included. levellist is a comma separated list of level names.
  • The name all may be used to add all known levels
  • You may use + or - in front of terms to add or remove from the current (or default) list of levels instead of replacing it
  • Adding term
-s fieldlist Select which fields of log messages to show. The output field order is fixed. When using this option, only the named fields will be printed. The default fields are as [-s fmttime,msecs,level,service,component,message]. The fieldlist is a comma separated list of field names. The name all may be used to add all possible fields. Prepending a minus sign will turn off display of the named field. Starting the list with a plus sign will add and remove fields from the current (or default) list of fields instead of replacing it. Using this option several times works as if the given fieldlist arguments had been concatenated into one comma-separated list. Fields:
time Print the time in seconds since the epoch. Ignored if fmttime is shown
fmttime Print the time in human-readable [YYYY-MM-DD HH:mm:ss] format. Note that the time is printed in the local timezone; to get GMT output, use env TZ=GMT vespa-logfmt
msecs Add milliseconds after the seconds in time and fmttime output. Ignored if usecs is in effect
usecs Add microseconds after the seconds in time and fmttime output
host Print the hostname field
level Print the level field (upper-cased)
pid Print the pid field
service Print the service field
component Print the component field
message Print the message text field. You probably always want to add this
-p pid Select messages where the pid field matches the pid string
-S service Select messages where the service field matches the service string
-H host Select messages where the hostname field matches the host string
-c regex Select messages where the component field matches the regex, using perlre regular expression matching
-m regex Select messages where the message text field matches the regex, using perlre regular expression matching
-f Invoke tail -F to follow input file
-N De-quote quoted newlines in the message text field to an actual newline plus tab
-t Format the component field (if shown) as a fixed-width string, truncating if necessary
-ts Format the service field (if shown) as a fixed-width string, truncating if necessary


Use vespa-makefsa to compile a list of phrases into a finite state automation (FSA) file. FSA files are used in query phrasing and rewriting.

If input file is not specified, standard input is used.

Synopsis: vespa-makefsa [-h] [-b] [-B] [-e] [-n] [-s bytes] [-z bytes] [-t] [-p] [-i] [-S serial] [-v] [-V] [input_file] output_file

Option Description
-h Help text
-b Use binary input format with Base64 encoded info
-B Use binary input format with raw
-e Use text input format with no info (default)
-n Use text input format with (unsigned) numerical info
-s bytes Data size for numerical info: 1,2 or 4(default)
-z bytes Data size for binary info (-B) (0 means NUL terminated)
-t Use text input format
-p Build automaton with perfect hash
-i Ignore info string, regardless of input format
-S serial Serial number
-v Verbose
-V Display version


vespa-model-inspect is a tool for inspecting the topology and services of a Vespa system. Hosts, services, clusters, ports, URLs and config ids can be inspected. It can run on any machine in a Vespa cluster that is running a Vespa configuration server.

Synopsis: vespa-model-inspect [-c host | host:port] [-t tag] [-h] [-u] [-v] command

Command Description
hosts Show hostnames of all hosts in the Vespa system
services Show a list of all service types in the Vespa system
clusters Show a list of all named clusters in the Vespa system
configids Show a list of all config ids in the Vespa system
filter:ports List ports matching filter options
host hostname Show host details: What services are running, and what ports have they allocated
service servicetype Show service details: What instances of the service are running, on what hosts, and what ports have they allocated
cluster clustername Show all services in the cluster, with details on hostname and allocated ports
configid configid Show all services using this configid
get-index-of servicetype host Show all indexes for instances of the service type on the given host
Option Description
-c host | host:port Specify host and port (or just host) to use for getting the config that this tool displays. Default is to use the configserver. You might want to use localhost:19090 if you are on a host with a running Vespa system without a config server
-h Show usage
-t tag to filter on a port tag
-u Show URLs for services
-v Verbose mode


$ vespa-model-inspect hosts

$ vespa-model-inspect services

$ vespa-model-inspect -u service distributor
distributor @ myhost.mydomain.com : content
    tcp/myhost1.mydomain.com:19112 (MESSAGING)
    tcp/myhost1.mydomain.com:19113 (STATUS RPC)
    http://myhost1.mydomain.com:19114/ (STATE STATUS HTTP)
distributor @ myhost2.mydomain.com : content
    tcp/myhost2.mydomain.com:19112 (MESSAGING)
    tcp/myhost2.mydomain.com:19113 (STATUS RPC)
    http://myhost2.mydomain.com:19114/ (STATE STATUS HTTP)
distributor @ myhost3.mydomain.com : content


Internal script used by other scripts to find hostname, config server addresses / ports, version and more. Not intended for end-user usage.


Use vespa-proton-cmd to send commands to proton.

Synopsis: vespa-proton-cmd HOSTSPEC COMMAND [ARGS]

The hostspec argument is one of port|spec|--local|--id=name. Use vespa-model-inspect to locate the search node ADMIN RPC port:

$ vespa-model-inspect service searchnode
searchnode @ /mynode.myhost.com : search
    tcp/mynode.myhost.com:19108 (STATUS ADMIN RTC RPC)
    tcp/mynode.myhost.com:19109 (FS4)
    tcp/mynode.myhost.com:19110 (TEST HACK SRMP)
    tcp/mynode.myhost.com:19111 (ENGINES-PROVIDER RPC)
    tcp/mynode.myhost.com:19112 (STATE HEALTH JSON HTTP)


$ vespa-proton-cmd 19108 triggerFlush
  OK: flush trigger enabled

Unless the -h or --help option is used, one of these commands must be present:

Command Description
die Exit proton without cleanup.
getProtonStatus Get the current proton state and its components.
getState Get the current proton state.
triggerFlush Trigger flush as soon as possible for all document types.
prepareRestart Estimates the cost of transaction log replay, and flushes data structures if that will speed up a subsequent start. If this is not called before stopping proton, there is no estimation and no flush.


Dumps all resolved query profile properties for a set of dimension values

Synopsis:vespa-query-profile-dump-tool dump [query-profile] [dir]? [parameters]?


dump default                           # dumps the 'default' profile non-variant values in the current dir

dump default x=x1&y=y1                 # dumps the 'default' profile resolved with dimensions values x=x1 and y=y1 in the current dir

dump default myapppackage              # dumps the 'default' profile non-variant values in myapppackage/search/query-profiles

dump default dev/myprofiles x=x1&y=y1  # dumps the 'default' profile resolved with dimensions values x=x1 and y=y1 in dev/myprofiles
Option Description
query-profile Name of the query profile to dump the values of
dir Path to an application package or query profile directory. Default: current dir
parameters HTTP request encoded dimension keys used during resolving. Default: none


vespa-remove-index is a command line tool to remove index data on a Vespa search node, by wiping out selected files and subdirectories found in $VESPA_HOME/var/db/vespa/. This process is irreversible and the indexes deleted can not be recovered.

Stop services before running it - example:

$ vespa-stop-services && vespa-remove-index -force && vespa-start-services

Synopsis:vespa-remove-index [-force] [-cluster name]


$ vespa-remove-index
[info] For cluster music distribution key 0 you have:
[info] 156 kilobytes of data in var/db/vespa/search/cluster.music/n0
Really to remove this vespa index? Type "yes" if you are sure ==> yes
[info] removing data:  rm -rf var/db/vespa/search/cluster.music/n0
[info] removed.
Option Description
-force Do not require verification from the user before really removing index data
-cluster name Only remove data for given cluster name


vespa-route is a tool to inspect Vespa routing configurations. If file is set, it will be parsed as a feed and the output will look similar to when using /document/v1/ with trace enabled.

Synopsis: vespa-route [options] [file]


$ vespa-route
There are 5 route(s):
    1. default
    2. music
    3. music-direct
    4. music-index
    5. storage/cluster.music

There are 2 hop(s):
    1. docproc/cluster.music.indexing/chain.indexing
    2. indexing
Option Description
--documentmanagerconfigid <id> Sets the config id that supplies document configuration
--dump Prints the complete content of the routing table
--help Prints this help
--hop <name> Prints detailed information about hop <name>
--hops Prints a list of all available hops
--identity <id> Sets the identity of message bus
--listenport <num> Sets the port message bus will listen to
--oosserverpattern <id> Sets the out-of-service server pattern for message bus
--protocol <name> Sets the name of the protocol whose routing to inspect
--route <name> Prints detailed information about route <name>
--routes Prints a list of all available routes
--routingconfigid <id> Sets the config id that supplies the routing tables
--services Prints a list of all available services
--slobrokconfigid <id> Sets the config id that supplies the slobrok server list
--trace <num> Sets the trace level to use when visualizing route
--verify All hops and routes are verified when routing


Use vespa-sentinel-cmd to list, start and stop services - refer to config sentinel for examples. It can also check for connectivity between nodes.

Synopsis: vespa-sentinel-cmd [-h] list|start <service>|restart <service>|stop <service>|connectivity

Option Description
-hHelp text
Command Description
list Lists the services running on this node and their status:
service name
  • RUNNING: Service is running
  • FINISHED: Service has been stopped
  • FAILED: Service has crashed and failed to restart
  • TERMINATING: Service is stopping
  • MANUAL: Service has to be started and stopped manually
  • AUTO: Service will restart automatically if it stops
pid Pid of the process (main thread)
exitstatus Exit code last time service stopped.
id Config ID of the service
restart [name] Restarts the service with the given name. The name is the first string in the service list given by list
stop [name] Stops the service with the given name
start [name] Starts the service with the given name
connectivity Use to troubleshoot startup issues / network configuration / ACLs / iptables:
$ vespa-sentinel-cmd connectivity
vespa-sentinel-cmd 'connectivity' OK.
node0.vespanet -> ok
node1.vespanet -> ok
node2.vespanet -> ok


Set the user state of a node. This will set the generated state to the user state if the user state is "better" than the generated state that would have been created if the user state was up. For instance, a node that is in up state can be forced into down state, while a node that is currently down can not be forced into retired state, but can be forced into maintenance state.

Synopsis: vespa-set-node-state [options] up|down|maintenance|retired [description]


$ vespa-set-node-state -i 0 maintenance "Set to maintenance for software upgrade"
Option Description
-h, --help Show help
-v More verbose output
-s Less verbose output
--show-hidden Also show hidden undocumented debug options
-n, --no-wait Do not wait for node state changes to be visible in the cluster before returning
-c, --cluster Cluster name of cluster to query. If unspecified, and vespa is installed on current node, information will be attempted auto-extracted
-f, --force Force execution
-t, --type Node type - can either be 'storage' or 'distributor'. If not specified, the operation will use state for both types
-i, --index Node index. If not specified, all nodes found running on this host will be used
--config-server Host name of config server to query
--config-server-port Port to connect to config server on
--config-request-timeout Timeout of config request


Start a config server on a node, details. See vespa-start-services for node setup steps, before startup.

Synopsis: vespa-start-configserver


Start all services on a node, details.

As part of starting Vespa, the startup script calls rhel-prestart.sh to set up directories and system limits - this requires privileges to do so - run this command with sudo, see example. Refer to vespa-start-services.sh and environment variables.

Synopsis: vespa-start-services

When debugging a failed start, use vespa-logfmt to inspect the log. It is also useful to read up on the start sequence and make sure the config server is running - Vespa will not start with a running config server.

Vespa has a Safe Cluster Startup mode to only start vespa services after X% of nodes are running - see cluster startup.


vespa-stat is a tool to fetch statistics about a specific user, group, bucket, gid or document.

vespa-stat works in two stages. The first stage is to figure out the actual buckets we want to look at. In the second stage, it can dump the located buckets. For each command line option, only the relevant documents will be dumped (the document for --document/--gid, or the user/group's documents for --user/--group). This stage can be turned on by adding --dump, but is default on for the case of --document/--gid.

Synopsis: vespa-stat [options]


$ vespa-stat --document id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085
Bucket maps to the following actual files:
	BucketInfo(BucketId(0x4000000000004800): [distributor:17] [node(idx=17,crc=0xe5ce35c7,docs=57/57,bytes=478040/478040,trusted=true,active=true,ready=true),  node(idx=15,crc=0xe5ce35c7,docs=57/57,bytes=478040/478040,trusted=true,active=true,ready=true)])

Details for BucketId(0x4000000000004800):
	Bucket information from node 15:
Persistence bucket BucketId(0x4000000000004800), partition 0
  Timestamp: 1452598747000000, Doc(id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085), gid(0x0048e840a48002b12abbb0a0), size: 101

	Bucket information from node 17:
Persistence bucket BucketId(0x4000000000004800), partition 0
  Timestamp: 1452598747000000, Doc(id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085), gid(0x0048e840a48002b12abbb0a0), size: 101
Option Description
-b, --bucket <bucketid> Dump list of buckets that are contained in the given bucket, or that contain it
-d, --dump Dump list of documents for all buckets matching the selection command.
-g, --group <groupid> Dump list of buckets that can contain the given group
-h, --help Help text
-l, --gid <globalid> Dump information about one specific document, as given by the GID (implies --dump)
-o, --document <docid> Dump information about one specific document (implies --dump)
-r, --route <route> Route to send the messages to, usually the name of the storage cluster
-s, --bucketspace <space> Bucket space (default or global). If not specified, default is used
-u, --user <userid> Dump list of buckets that can contain the given user


Use vespa-status-filedistribution to get status from file distribution. Should by run on a config server, it connects to config server on localhost to get status.

Synopsis: vespa-status-filedistribution [--application <applicationNameArg>] [--debug] [--environment <environmentArg>] [(-h | --help)] [--instance <instanceNameArg>] [--region <regionArg>] [--tenant <tenantNameArg>] [--timeout <timeoutArg>]

Option Description
--application <applicationName> Application name
--debug Print debug log
--environment <environment> Environment name
-h, --help Display help information
--instance <instanceName> Instance name
--region <regionName> Region name
--tenant <tenantName> Tenant name
--timeout <timeout> timeout (in seconds)


Stop a config server on a node, details.

Synopsis: vespa-stop-configserver


Stop all services on a node, details. Running vespa-stop-services on a content node will call prepareRestart to optimize restart time.

Synopsis: vespa-stop-services


Tool for testing and benchmarking rpc docsum interface. Refer to VespaSummaryBenchmark.java


Refer to visiting.

Synopsis: vespa-visit [options]

Option Description
--abortonclusterdown Abort if cluster is down
-b, --maxbuckets <num> Maximum buckets per visitor
--bucketspace <space> Bucket space to visit (default or global). If not specified, default is used
-c, --cluster <cluster> Visit the given cluster
-d, --datahandler <target> Send results to the given target - see vespa-visit-target
-f, --from <timestamp> Only visit from the given timestamp (microseconds)
-h, --help Show help text
-i, --printids Display only document identifiers
--jsonoutput Output documents as JSON (default format)
-l, --fieldset <fieldset> Retrieve the specified fields only, see fieldsets. Default is [all]
--libraryparam <key> <val> Send parameter to the visitor library
-m, --maxpending <num> Maximum pending messages to data handlers per storage visitor
--maxpendingsuperbuckets <num> Maximum pending visitor messages from the vespa-visit client. If set, dynamic throttling of visitors is disabled
--maxtotalhits <num> Abort visiting when received this many total documents. This is only an approximate number, all pending work will be completed and those documents will also be returned
-o, --timeout <milliseconds> Time out visitor after given time
-p, --progress <file> Use given file to track progress
--processtime <num> Sleep this amount of milliseconds before processing message. (Debug option for pretending to be slow client)
-r, --visitremoves Include information of removed documents
-s, --selection <selection> Selection string for which documents to visit
--skipbucketsonfatalerrors Skip visiting super buckets with fatal error codes
-t, --to <timestamp> Only visit up to the given timestamp (microseconds)
--tracelevel <level> Tracelevel ([0-9]), for debugging
-u, --buckettimeout <milliseconds> Fail visitor if visiting a single bucket takes longer than this (default same as timeout)
-v, --verbose Show progress and info on STDERR
--visitinconsistentbuckets Don't wait for inconsistent buckets to become consistent
--visitlibrary <string> Use the given visitor library
-x, --xmloutput Output documents as XML.


Output results from a visitor. Also see vespa-destination.

Synopsis: vespa-visit-target [options]

Option Description
-c, --visithandler <classname> Use the given class as a visit handler (defaults to StdOutVisitorHandler)
-h, --help Show help page
-i, --printids Display document IDs only
-o, --visitoptions <args> Option arguments to pass through to the visitor handler instance
-p, --processtime <msecs> Sleep msecs milliseconds before processing message. (Debug option for pretending to be slow client)
-s, --bindtoslobrok <address> Bind to slobrok address. One, and only one, of the binding options must be set
-t, --bindtosocket <port> Bind to TCP port. One, and only one, of the binding options must be set
-v, --verbose Indent output, show progress and info on STDERR