This is the /document/v1 API reference documentation.
Use this API for synchronous Document operations to a Vespa endpoint -
refer to reads and writes for other options.
Some examples use number and groupdocument id modifiers.
These are special cases that only work as expected for document types
with mode=streaming or mode=store-only.
Do not use group or number modifiers with regular indexed mode document types.
Configuration
To enable the API, add document-api in the serving container cluster -
services.xml:
<services>
<container>
<document-api/>
HTTP requests
HTTP request
document/v1 operation
Description
GET
Get a document by ID or Visit a set of documents by selection.
Iterate over and get all documents, or a selection of documents,
in chunks, using continuation tokens to track progress.
Visits are a linear scan over the documents in the cluster.
/document/v1/
It is possible to specify namespace and document type with the visit path:
/document/v1/<namespace>/<document-type>/docid
Documents can be grouped to limit accesses to a subset.
A group is defined by a numeric ID or string — see id scheme.
cluster -
Visits can only retrieve data from one content cluster,
so clustermust be specified
for requests at the root /document/v1/ level, or when there is ambiguity.
This is required even if the application has only one content cluster.
Optional parameters:
bucketSpace -
Parent documents are global
and in the globalbucket space.
By default, visit will visit non-global documents
in the default bucket space, unless document type is indicated,
and is a global document type.
concurrency -
Use to configure backend parallelism for each visit HTTP request.
slices -
Split visiting of the document corpus across more than one HTTP
request—thus allowing the concurrent use of more HTTP containers—use the
slices and sliceId parameters.
stream -
It's recommended enabling streamed HTTP responses,
with the stream parameter,
as this reduces memory consumption and reduces HTTP overhead.
Write documents visited in source cluster to the
destinationCluster in the same application.
A selection is mandatory — typically the document type.
Supported paths (see visit above for semantics):
Update visited documents in cluster with the partial update
contained in the request body in the
document update JSON format.
Supported paths (see visit above for semantics):
Specify the bucket space to visit.
Document types marked as global exist in a separate bucket space from non-global document types.
When visiting a particular document type,
the bucket space is automatically deduced based on the provided type name.
When visiting at a root /document/v1/ level this information is not available,
and the non-global ("default") bucket space is visited by default.
Specify global to visit global documents instead.
Supported values: default (for non-global documents) and global.
Sends the given number of visitors in parallel to the backend,
improving throughput at the cost of resource usage.
Default is 1.
When stream=true, concurrency limits the maximum concurrency,
which is otherwise unbounded, but controlled by a dynamic throttle policy.
Important:
Given a concurrency parameter of N,
the worst case for memory used while processing the request grows linearly with N,
unless stream mode is turned on.
This is because the container currently buffers all response data in memory before sending them to the client,
and all sent visitors must complete before the response can be sent.
condition
String
For test-and-set.
Run a document operation conditionally — if the condition fails,
a 412 Precondition Failed is returned.
See example.
continuation
String
When visiting, a continuation token is returned as the "continuation" field
in the JSON response, as long as more documents remain.
Use this token as the continuation parameter to visit the next chunk of documents.
See example.
create
Boolean
If true, updates to non-existent documents will create an empty document to update.
See create if nonexistent.
Used by the vespa-feed-client
using --speed-test for bandwidth testing,
by setting to true.
fieldSet
String
A field set string
with the set of document fields to fetch from the backend.
Default is the special [document] fieldset, returning all document fields.
To fetch specific fields, use the name of the document type, followed by a comma-separated list of fields
(for example music:artist,song to fetch two fields declared in music.sd).
route
String
The route for single document operations, and for operations generated
by copy, update or
deletion visits. Default value is default.
See routes.
The slice number of the visit represented by this HTTP request. This number must be non-negative
and less than the number of slices specified for the visit -
e.g., if the number of slices is 10, sliceId is in the range [0-9].
Note:
If the number of distribution bits change during a sliced visit,
the results are undefined.
Thankfully, this is a very rare occurrence and is only triggered when adding content nodes.
slices
Integer
Split the document corpus into this number of independent slices. This lets multiple, concurrent series of HTTP
requests advance the same logical visit independently, by specifying a different sliceId for each.
stream
Boolean
Whether to stream the HTTP response, allowing data to flow as soon as documents arrive from the backend.
This obsoletes the wantedDocumentCount parameter.
The HTTP status code will always be 200 if the visit is successfully initiated. Default value is false.
format.tensors
String
Controls how tensors are rendered in the result.
Value
Description
short
Default. Render the tensor value in an object having two keys, "type" containing the value,
and "cells"/"blocks"/"values" (depending on the type) containing the tensor content.
Render the tensor content in the type-appropriate short form.
long
Render the tensor value in an object having two keys, "type" containing the value,
and "cells" containing the tensor content.
Render the tensor content in the general verbose form.
Render the tensor content directly.
Render the tensor content in the general verbose form.
timeChunk
String
Target time to spend on one chunk of a copy, update or remove visit; with optional ks, s, ms or µs unit.
Default value is 60.
timeout
String
Request timeout in seconds, or with optional ks, s, ms or µs unit. Default value is 180s.
tracelevel
Integer
Number in the range [0,9], where higher gives more details.
The trace dumps which nodes and chains the document operation has touched.
See routes.
wantedDocumentCount
Integer
Best effort attempt to not respond to the client before wantedDocumentCount
number of documents have been visited.
Response may still contain fewer documents if there are not enough matching documents left
to visit in the cluster, or if the visiting times out.
This parameter is intended for the case when you have relatively few documents in your cluster
and where each visit request would otherwise process only a handful of documents.
The maximum value of wantedDocumentCount is bounded
by an implementation-specific limit to prevent excessive resource usage.
If the cluster has many documents (on the order of tens of millions),
there is no need to set this value.
fromTimestamp
Integer
Filters the returned document set to only include documents that were last modified
at a time point equal to or higher to the specified value, in microseconds from UTC epoch.
Default value is 0 (include all documents).
toTimestamp
Integer
Filters the returned document set to only include documents that were last modified
at a time point lower than the specified value, in microseconds from UTC epoch.
Default value is 0 (sentinel value; include all documents). If non-zero, must be
greater than, or equal to, fromTimestamp.
includeRemoves
Boolean
Include recently removed document IDs, along with the set of returned documents.
By default, only documents currently present in the corpus are returned in the
"documents" array of the response; when this parameter is set to
"true", documents that were recently removed, and whose tombstones
still exist, are also included in that array, as entries on the form
{ "remove": "id:ns:type::foobar" }. See
here
for specifics on tombstones, including their lifetime.
HTTP request headers
Header
Values
Description
Accept
application/json or application/jsonl
The Accept header
lets the client specify to the server what
media (MIME) types it
accepts as the response format.
All Document V1 API calls support application/json for returning JSON responses.
Streaming visiting additionally supports application/jsonl for returning
JSON Lines (JSONL) since Vespa 8.593.
To ensure compatibility with older versions, make sure to check the Content-TypeHTTP response header. A JSONL response will always have a
Content-Type media type of application/jsonl, and JSON wil always have
a media type of application/json.
Multiple acceptable types can be specified. JSONL will be returned if (and only if)
application/jsonl is part of the list and no other media
types have a higher quality value.
Example:
Accept: application/jsonl
If the client accepts both JSON and JSONL, the server will respond with JSONL:
Accept: application/json, application/jsonl
For backwards compatibility, if no Accept header is provided
(or if no provided media types are acceptable) application/json
is assumed.
Request body
POST and PUT requests must include a body for single document operations; PUT must
also include a body for update where visits.
A field has a value for a POST and an update operation object for PUT.
Documents and operations use the document JSON format.
The document fields must match the schema:
The update-operation is most often assign - see
update operations for the full list.
Values for id / put / update in the request body are silently dropped.
The ID is generated from the request path, regardless of request body data - example:
This makes it easier to generate a feed file that can be used for both the
vespa-feed-client and this API.
HTTP status codes
Code
Description
200
OK. Attempts to remove or update a non-existent document also yield this status code (see 412 below).
204
No Content. Successful response to OPTIONS request.
400
Bad request. Returned for undefined document types + other request errors.
See 13465
for defined document types not assigned to a content cluster when using PUT.
Inspect message for details.
404
Not found; the document was not found. This is only used when getting documents.
405
Method Not Allowed. HTTP method is not supported by the endpoint. Valid combinations are listed
above
412
condition is not met.
Inspect message for details. This is also the result when
a condition if specified, but the document does not exist.
413
Content too large; used for POST and PUT requests that are above the request size limit.
429
Too many requests; the document API has too many inflight feed operations, retry later.
500
Server error; an unspecified error occurred when processing the request/response.
503
Service unavailable; the document API was unable to produce a response at this time.
504
Gateway timeout; the document API failed to respond within the given (or default 180s) timeout.
507
Insufficient storage; the content cluster is out of memory or disk space.
HTTP response headers
Header
Values
Description
X-Vespa-Ignored-Fields
true
Will be present and set to 'true' only when a put or update contains one or more fields
which were ignored since they are not present in the document type.
Such operations will be applied exactly as if they did not contain the field operations
referencing non-existing fields.
References to non-existing fields in field paths are not detected.
A continuation indicates the client should make further requests to get more data, while lack of a
continuation indicates an error occurred, and that visiting should cease, or that there are no more documents.
A message can be returned for failed operations:
{"pathId":"<pathid>","message":"<message text>"}
JSON Lines
A JSON Lines (JSONL) response is a stream of newline-separated JSON objects. Each line contains
exactly one JSON object, and each JSON object takes up exactly one line. No line breaks are
allowed within an object.
JSONL is an optional response format for streaming visiting, enabling
efficient client-side parsing and fine-grained, continuous tracking of visitor progress.
The JSONL response format is currently not supported for any other operations than streaming visiting.
The JSONL response format is enabled by providing a HTTP Accept request
header that specifies application/jsonl as the preferred response type, and will
have a Content-Type of application/jsonl if the server
is on a version that supports JSONL visiting. Clients must check the Content-Type
header to ensure they are getting the format they expect.
An opaque string value representing the current visitor progress through the data space.
This value can be provided as part of a subsequent visitor request to continue visiting
from where the last request left off. Clients should not attempt to parse the contents
of this string, as it's considered an internal implementation detail and may be changed
(in a backwards compatible way) without any prior announcement.
percentFinished
A floating point number between 0 and 100 (inclusive) that gives an approximation of how
far the visitor has progressed through the data space.
The last line of a successful request should always be a continuation object.
If (and only if) visiting has completed, the last continuation object will have a
percentFinished value of 100 and will not have a token field.
message
A message received from the backend visitor session. Can be used by clients to report
problems encountered during visiting.
Possible sub-object fields:
Field name
Description
text
The actual message, in unstructured text
severity
The severity of the message. One of info, warning or error.
sessionStats
Statistics from the backend visitor session.
Possible sub-object fields:
Field name
Description
documentCount
The number of visited and selected documents. If includeRemoves
is true, this also includes the number of returned removes (tombstones).
Note that it's possible for a successful response to contain zero put or remove
objects if the selection did not match any documents.
Differences from the JSON format
The biggest difference in semantics between the JSON and JSONL response formats is when, and how,
continuation objects are returned.
In the JSON format a continuation is included once at the very end of the response object
and covers the progress made by the entire request. If the request somehow fails after receiving
99% of all documents but prior to receiving the continuation field, the client must retry the
entire request from the previously known continuation value. This can result in getting many requested
documents twice; once from the incomplete first request and once more from the second request that
covers the same part of the data space.
In the JSON Lines format, a contination object is emitted to the stream every time a backend
data bucket has been fully visited, as well as at the end of the
response stream. This may happen many times in a response. Each continuation object subsumes
the progress of previously emitted continuations, meaning that a client only needs to remember the
most recent continuation value it observed in the response. If the request fails prior to
completion, the client can specify the most recent continuation in the next request; it will then only
receive duplicates for the data buckets that were actively being processed when the request failed.