Vespa Command-line Tools

This is the reference for the Vespa command-line tools.

You can run these tools in Vespa Docker image:

docker run --entrypoint bash vespaengine/vespa ./opt/vespa/bin/[tool] [args]

vespa-configproxy-cmd

vespa-configproxy-cmd is a status and control tool for the config proxy. The config proxy runs on all nodes as a proxy for one or more config servers. It connects to the config proxy on localhost by default.

Run it without arguments to dump a list of active configIds - format specification:

<config definition name>,<config id>,<config generation>,<MD5 checksum>,<xxHash checksum>

Synopsis: vespa-configproxy-cmd [args]

Example:

$ vespa-configproxy-cmd -m sources

Find a more comprehensive example in inspecting config.

vespa-configserver-remove-state

vespa-configserver-remove-state removes config server state on the host.

Synopsis: vespa-configserver-remove-state [-force]

vespa-config-status

vespa-config-status can run on any machine in a Vespa cluster and outputs a list of all running services which is running with an outdated application package. It can be invoked without any arguments, or with optional arguments.

Synopsis: vespa-config-status [-v] [-c host] [-c host:port] [-f host0,...,hostN]

Example:

$ vespa-config-status

vespa-deploy

vespa-deploy is a standalone tool to deploy an application package. Prefer the Vespa CLI instead. Under the hood deployment uses the the deploy REST API, which you can also use directly. Refer to the deploy reference for details.

Synopsis: vespa-deploy [-h] [-v] [-n] [-f] [-t timeout] [-c hostname] [-p port] prepare|activate|upload|fetch|help [args]

Example:

$ vespa-deploy prepare [application-path|zip-file] && vespa-deploy activate

$ vespa-deploy prepare app.zip && vespa-deploy activate

vespa-destination

vespa-destination is a simple receiver for messagebus messages. It outputs messages received on stdout. Also see vespa-visit-target.

Synopsis: vespa-destination [options]

Example:

$ vespa-destination --name msg_sink

vespa-fbench-filter-file

usage: vespa-fbench-filter-file [-a] [-h] [-m maxLineSize]

Read concatenated fastserver logs from stdin and write
extracted query urls to stdout.

 -a       : all parameters to the original query urls are preserved.
            If the -a switch is not given, only 'query' and 'type'
            parameters are kept in the extracted query urls.
 -h       : print this usage information.
 -m <num> : max line size for input/output lines.
            Can not be less than the default [10240]

vespa-fbench-geturl

usage: vespa-fbench-geturl <host> <port> <url>

vespa-fbench-split-file

usage: vespa-fbench-split-file [-p pattern] [-m maxLineSize] <numparts> [<file>]

Reads from <file> (stdin if <file> is not given) and
randomly distributes each line between <numpart> output
files. The names of the output files are generated by
combining the <pattern> with sequential numbers using
the sprintf function.

 -p pattern : output name pattern ['query%03d.txt']
 -m <num>   : max line size for input/output lines.
              Can not be less than the default [10240]
 <numparts> : number of output files to generate.

vespa-feeder

vespa-feeder is a feeding client that parses JSON input as Vespa document operations and sends to a Vespa application. It parses the content of the input sequentially, and feeds each operation in order. However, since many operations will be pending at any time, and because the processing time of an operation varies, there is no guarantee as to which order operations reach the content nodes. As this can be important when it comes to operations that apply to the same document id, there is logic in place to not send an operation for a document id to which there is already a pending operation.

vespa-feeder prints a report at end of feed. To print this report once a minute, use --verbose:

Messages sent to vespa (route default) :
----------------------------------------
PutDocument: ok: 999997 msgs/sec: 411.38 failed: 0 ignored: 0 latency(min, max, avg): 2, 4360, 99

ignored reports the number of documents that could not be routed to any content clusters because they did not match any of the configured document types or selections - examples are:

• A document type is removed from the application and the feed file contains documents with this type
• One or more selection expressions restrict the documents the cluster accepts, and the feed file contains documents that are excluded. An example is feeding expired documents - a selection for documents that are less than 30 days old and the feed file contains documents that are 30+ days old

Synopsis: vespa-feeder [--abortondataerror true|false] [--abortonsenderror true|false] [--file arg] [--maxpending arg] [--maxpendingsize arg] [--maxfeedrate arg] [mode standard|benchmark] [--noretry] [--retrydelay arg] [--route arg] [--timeout arg] [--trace arg] [--validate] [--dumpDocuments filename] [--numthreads arg] [create-if-non-existent] [-v,--verbose] filename

Example:

$ vespa-feeder file.json

vespa-get

vespa-get retrieves documents from a Vespa content cluster, and prints to stdout. vespa-get retrieves documents identified by the document ids passed as command line arguments. If no document ids are passed through the command line interface, ids will be read from stdin - separated by line breaks.

Synopsis: vespa-get <options> [documentid...]

vespa-get-cluster-state

Get cluster state - refer to content nodes.

Synopsis: vespa-get-cluster-state [options]

vespa-get-config

vespa-get-config is a command line tool to get configuration from a config server or config proxy. By default, it connects to the config proxy on localhost, fetches config from its cache and prints the config payload on stdout. Configuration is addressed using name and configId. If configId is omitted, the global and default data for that name is returned. The default port number is 19090, the config proxy's port - use 19070 to access a config server. Also check ports.

Synopsis: vespa-get-config -n defName -i configId <option> [args]

Example:

$ vespa-get-config -n container.statistics -i search/cluster.search

Find a more comprehensive example in inspecting config.

vespa-get-node-state

Get the state of one or more storage services from the fleet controller - refer to content nodes:

Synopsis: vespa-get-node-state [options]

vespa-index-inspect

Use vespa-index-inspect to inspect indexed data on a content node. To troubleshoot query rewriting and linguistic transformations use this guide instead.

It shows posting list information (per token or all/range), or dumps the indexed tokens:

vespa-index-inspect showpostings [--indexdir indexDir] --field field word

vespa-index-inspect showpostings [--indexdir indexDir] [--field field] --transpose \
  [--docidlimit docIdLimit] [--mindocid mindocid]

vespa-index-inspect dumpwords [--indexdir indexDir] --field field \
  [--minnumdocs minnumdocs] [--verbose] [--wordnum]

Synopsis:vespa-index-inspect showpostings|dumpwords [--indexdir path] [--field fieldname] [--transpose] [--minnumdocs count] [--docidlimit docIdLimit] [--mindocid mindocid] [--verbose] [--wordnum] [word]

Example (make sure to flush the index before using):

$ vespa-proton-cmd --local triggerFlush && \
  vespa-index-inspect dumpwords \
  --indexdir /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/index/index.flush.1 \
  --field artist
bad	2
so	1

vespa-attribute-inspect

Use vespa-attribute-inspect to inspect the content of an attribute field on a content node. To troubleshoot query rewriting and linguistic transformations use this guide instead.

Synopsis: vespa-attribute-inspect [-p attribute] [-a] [-s attribute] <attribute>

Example (make sure to flush the attribute before using):

$ vespa-proton-cmd --local triggerFlush && \
  vespa-attribute-inspect -p /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/attribute/year/snapshot-10/year && \
  cat /opt/vespa/var/db/vespa/search/cluster.music/n0/documents/music/0.ready/attribute/year/snapshot-10/year.out

vespa-jvm-dumper

Dump JVM heap, thread stacks and other debugging information from a Java-based Vespa service.

Invoke binary without arguments to print help and list the services running on this node.

$ vespa-jvm-dumper

Produce JVM debugging information by invoking binary with config id of target service and output directory.

$ vespa-jvm-dumper default/container.1 /opt/vespa/tmp/jvm-dump

vespa-logctl

Print or modify log levels for a VESPA service, stored in $VESPA_HOME/var/db/vespa/logcontrol/service.logcontrol. Refer to controlling log levels for details. component-specification specifies which subcomponents of the service should be controlled. If empty, all components are controlled:

• x. : Matches only component x
• x : Matches component x and all its sub-components

Synopsis (show log levels): vespa-logctl [OPTION] <service>[:component-specification]

Synopsis (set log levels): vespa-logctl [OPTION] <service>[:component-specification] <level-mods>

level-mods are defined as : <level>=<on|off>[,<level>=<on|off>]...

level is one of: all, fatal, error, warning, info, event, config, debug, spam

Example: For service container, set com.yahoo.search.searchchain and all subcomponents of com.yahoo.search.searchchain to enable all except spam and debug:

$ vespa-logctl container:com.yahoo.search.searchchain all=on,spam=off,debug=off

vespa-logfmt

vespa-logfmt reads Vespa log files, selects messages, and writes a formatted version of these messages to standard output. If no file argument is given, vespa-logfmt will read the last Vespa log file $VESPA_HOME/logs/vespa/vespa.log (this also works with the -f option). Otherwise, reads only the files given as arguments. To read standard input, supply a single dash ’-’ as a file argument. Refer to the logs reference.

Synopsis: vespa-logfmt [-l levellist ] [-s fieldlist ] [-p pid ] [-S service ] [-H host ] [-c regex ] [-m regex ] [-f ] [-N ] [-t ] [-ts ] [file …]

Examples:

Display only messages with log level "event", printing a human-readable time (without any fractional seconds), the service generating the event and the event message:

$ vespa-logfmt -l event -s fmttime,service,message
...
[2017-09-05 06:16:16] config-sentinel  stopped/1 name="sbin/vespa-config-sentinel -c hosts/vespa-container (pid 1558)" pid=1558 exitcode=1
[2017-09-05 06:16:16] config-sentinel  starting/1 name="sbin/vespa-config-sentinel -c hosts/vespa-container (pid 1564)"
[2017-09-05 06:16:16] config-sentinel  started/1 name="config-sentinel"
[2017-09-05 06:17:00] configserver     count/1 name=configserver.failedRequests value=0
[2017-09-05 06:17:00] configserver     count/1 name=procTime value=0
[2017-09-05 06:17:00] configserver     count/1 name=configserver.requests value=0

Display messages with log levels that are not any of info, debug, or event, printing the time in seconds and microseconds, the log level, the component name, and the message text:

$ vespa-logfmt -l all-info,-debug -s level -s time,usecs,component,message -t -l -event
...
1504592294.738000 WARNING : configproxy.com No config found for name=sentinel,namespace=cloud.config,configId=hosts/vespa-container within timeout, will retry
1504592296.388000 WARNING : configproxy.com Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/localhost:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/127.0.0.1,port=37806,localport=19070] }
1504592307.949461 WARNING : config-sentinel Connection to tcp/localhost:19090 failed or timed out
1504592307.949587 WARNING : config-sentinel FRT Connection tcp/localhost:19090 suspended until 2017-09-05 06:19:07 GMT

vespa-model-inspect

vespa-model-inspect is a tool for inspecting the topology and services of a Vespa system. Hosts, services, clusters, ports, URLs and config ids can be inspected. It can run on any machine in a Vespa cluster that is running a Vespa configuration server.

Synopsis: vespa-model-inspect [-c host | host:port] [-t tag] [-h] [-u] [-v] command

Examples:

$ vespa-model-inspect hosts
mynode.mydomain.com

$ vespa-model-inspect services
config-sentinel
configproxy
configserver
container
container-clustercontroller
distributor
docprocservice
filedistributorservice
logd
logserver
searchnode
slobrok
storagenode

$ vespa-model-inspect -u service distributor
distributor @ myhost.mydomain.com : content
myapp/distributor/4
    tcp/myhost1.mydomain.com:19112 (MESSAGING)
    tcp/myhost1.mydomain.com:19113 (STATUS RPC)
    http://myhost1.mydomain.com:19114/ (STATE STATUS HTTP)
distributor @ myhost2.mydomain.com : content
myapp/distributor/5
    tcp/myhost2.mydomain.com:19112 (MESSAGING)
    tcp/myhost2.mydomain.com:19113 (STATUS RPC)
    http://myhost2.mydomain.com:19114/ (STATE STATUS HTTP)
distributor @ myhost3.mydomain.com : content

vespa-print-default

Internal script used by other scripts to find hostname, config server addresses / ports, version and more. Not intended for end-user usage.

vespa-proton-cmd

Use vespa-proton-cmd to send commands to proton.

Synopsis: vespa-proton-cmd HOSTSPEC COMMAND [ARGS]

The hostspec argument is one of port|spec|--local|--id=name. Use vespa-model-inspect to locate the search node ADMIN RPC port:

$ vespa-model-inspect service searchnode
searchnode @ /mynode.myhost.com : search
music/search/cluster.music/0
    tcp/mynode.myhost.com:19108 (STATUS ADMIN RTC RPC)
    tcp/mynode.myhost.com:19109 (FS4)
    tcp/mynode.myhost.com:19110 (TEST HACK SRMP)
    tcp/mynode.myhost.com:19111 (ENGINES-PROVIDER RPC)
    tcp/mynode.myhost.com:19112 (STATE HEALTH JSON HTTP)

Example:

$ vespa-proton-cmd 19108 triggerFlush
  OK: flush trigger enabled

Unless the -h or --help option is used, one of these commands must be present:

vespa-remove-index

vespa-remove-index is a command line tool to remove index data on a Vespa search node, by wiping out selected files and subdirectories found in $VESPA_HOME/var/db/vespa/. This process is irreversible and the indexes deleted can not be recovered.

Stop services before running it - example:

$ vespa-stop-services && vespa-remove-index -force && vespa-start-services

Synopsis:vespa-remove-index [-force] [-cluster name]

Example:

$ vespa-remove-index
[info] For cluster music distribution key 0 you have:
[info] 156 kilobytes of data in var/db/vespa/search/cluster.music/n0
Really to remove this vespa index? Type "yes" if you are sure ==> yes
[info] removing data:  rm -rf var/db/vespa/search/cluster.music/n0
[info] removed.

vespa-route

vespa-route is a tool to inspect Vespa routing configurations. If file is set, it will be parsed as a feed and the output will look similar to when using /document/v1/ with trace enabled.

Synopsis: vespa-route [options] [file]

Example:

$ vespa-route
There are 5 route(s):
    1. default
    2. music
    3. music-direct
    4. music-index
    5. storage/cluster.music

There are 2 hop(s):
    1. docproc/cluster.music.indexing/chain.indexing
    2. indexing

vespa-sentinel-cmd

Use vespa-sentinel-cmd to list, start and stop services - refer to config sentinel for examples. It can also check for connectivity between nodes.

Synopsis: vespa-sentinel-cmd [-h] list|start <service>|restart <service>|stop <service>|connectivity

$ vespa-sentinel-cmd connectivity
vespa-sentinel-cmd 'connectivity' OK.
node0.vespanet -> ok
node1.vespanet -> ok
node2.vespanet -> ok

vespa-set-node-state

Set the user state of a node. This will set the generated state to the user state if the user state is "better" than the generated state that would have been created if the user state was up. For instance, a node that is in up state can be forced into down state, while a node that is currently down can not be forced into retired state, but can be forced into maintenance state.

Synopsis: vespa-set-node-state [options] up|down|maintenance|retired [description]

Example:

$ vespa-set-node-state -i 0 maintenance "Set to maintenance for software upgrade"

vespa-significance

Generates a significance model file from Vespa documents. Available in Vespa as of version 8.426.8.

The generated model uses the same tokenizer as the default query processor, see linguistics in Vespa for details. When using a custom tokenizer, the model generator needs to be modified accordingly. Tokens are converted to lower-case without stemming. This corresponds to how the model is applied to query terms.

Synopsis: vespa-significance generate [options]

Example:

$ vespa-significance generate --in vespa-dump.jsonl --out en_model.json --field text --language en

When running in Docker, it is useful to mount a folder with vespa-feed documents and to store the model file, e.g.:

$ podman run -it --entrypoint bash -v $PWD/data:/data -w /data vespaengine/vespa:latest /opt/vespa/bin/vespa-significance generate --in docs.jsonl --out model.zst --field text --language en

vespa-start-configserver

Start a config server on a node, details. See vespa-start-services for node setup steps, before startup.

Synopsis: vespa-start-configserver

vespa-start-services

Start all services on a node, details.

As part of starting Vespa, the startup script calls rhel-prestart.sh to set up directories and system limits - this requires privileges to do so - run this command with sudo, see example. Refer to vespa-start-services.sh and environment variables.

Synopsis: vespa-start-services

When debugging a failed start, use vespa-logfmt to inspect the log. It is also useful to read up on the start sequence and make sure the config server is running - Vespa will not start with a running config server.

Vespa has a Safe Cluster Startup mode to only start vespa services after X% of nodes are running - see cluster startup.

vespa-stat

vespa-stat is a tool to fetch statistics about a specific user, group, bucket, gid or document.

vespa-stat works in two stages. The first stage is to figure out the actual buckets we want to look at. In the second stage, it can dump the located buckets. For each command line option, only the relevant documents will be dumped (the document for --document/--gid, or the user/group's documents for --user/--group). This stage can be turned on by adding --dump, but is default on for the case of --document/--gid.

Synopsis: vespa-stat [options]

Example:

$ vespa-stat --document id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085
Bucket maps to the following actual files:
	BucketInfo(BucketId(0x4000000000004800): [distributor:17] [node(idx=17,crc=0xe5ce35c7,docs=57/57,bytes=478040/478040,trusted=true,active=true,ready=true),  node(idx=15,crc=0xe5ce35c7,docs=57/57,bytes=478040/478040,trusted=true,active=true,ready=true)])

Details for BucketId(0x4000000000004800):
	Bucket information from node 15:
Persistence bucket BucketId(0x4000000000004800), partition 0
  Timestamp: 1452598747000000, Doc(id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085), gid(0x0048e840a48002b12abbb0a0), size: 101

	Bucket information from node 17:
Persistence bucket BucketId(0x4000000000004800), partition 0
  Timestamp: 1452598747000000, Doc(id:my_namespace:my_search::12345678-4fb7-3797-ae9a-d4d7a4e6e085), gid(0x0048e840a48002b12abbb0a0), size: 101

vespa-status-filedistribution

Use vespa-status-filedistribution to get status from file distribution. Should be run on a config server, it connects to config server on localhost to get status.

Synopsis: vespa-status-filedistribution [--application <applicationNameArg>] [--debug] [--environment <environmentArg>] [(-h | --help)] [--instance <instanceNameArg>] [--region <regionArg>] [--tenant <tenantNameArg>] [--timeout <timeoutArg>]

vespa-stop-configserver

Stop a config server on a node, details.

Synopsis: vespa-stop-configserver

vespa-stop-services

Stop all services on a node, details. Running vespa-stop-services on a content node will call prepareRestart to optimize restart time.

Synopsis: vespa-stop-services

vespa-summary-benchmark

Tool for testing and benchmarking rpc docsum interface. Refer to VespaSummaryBenchmark.java

vespa-visit

Used to run a visit operation, with more options than vespa visit. It uses the Vespa Message Bus and must be run inside a Vespa application - it does not use the Vespa HTTP APIs.

By default, vespa-visit gets visited documents and emits to stdout. However, the tool may specify a vespa-visit-target and be used as a tool to run reprocessing or migration. It supports keeping a progress file on disk, such that you can restart it if it should fail in the middle for some reason.

To migrate a set of documents from one cluster to another, use visiting - as the data is transferred directly, using a compact serialization format, from the source nodes to the targets, this is performance optimal (data is not piped through the visit client). Implement backup this way, or dump to file.

Search node recovery: Feed the documents directly to a search cluster. Example, selecting documents of type music:

$ vespa-visit --selection music --datahandler indexing

This feeds from the source into the search cluster in the same application. Note that simultaneous feed can make updates go lost.

Include remove-entries in the visit operation using --visitremove - this dumps the tombstones of documents recently removed.

The content policy can be configured to use a set of configuration servers from another cluster to configure itself. This is specified with the config parameter. As an example, the following route routes to the content cluster mycluster with a configuration server on myconfigserver.mydomain.com:12345:

[Content:config=tcp/myconfigserver.mydomain.com:12345;cluster=mycluster]

The following examples illustrate how to copy all data from a source cluster to another cluster using vespa-visit:

# Copies all data in the local cluster, routing it to the remote mycluster
$ vespa-visit --datahandler '[Content:config=tcp/myconfigserver.mydomain.com:12345;cluster=mycluster]'

# Limit to 'music' documents only
$ vespa-visit --datahandler '[Content:config=tcp/myconfigserver.mydomain.com:19070;cluster=mycluster]' \
  --selection music

# Limit to all documents for user '1234'
$ vespa-visit --datahandler '[Content:config=tcp/myconfigserver.mydomain.com:12345;cluster=mycluster]' \
  --selection id.user=1234

Visitor processor types:

Requests sent from the visitor processor are sent to a visitor target - types:

Also see vespa-destination.

Synopsis: vespa-visit [options]

vespa-visit-target

vespa-visit-target is a tool to set up an endpoint for visiting data. It binds to a socket or to a slobrok address, which is specified as a target in the visit client. Also see vespa-destination.

Synopsis: vespa-visit-target [options]