Vespa models data as documents.
A document has a string identifier, set by the application, unique across all documents.
A document is a set of key-value pairs.
A document has a schema (i.e. type),
defined in the schema.
When configuring clusters, a documents
element set what document types a cluster is to store.
This configuration is used to configure the garbage collector if it is enabled.
Additionally, it is used to define default routes for documents sent into the application.
By default, a document will be sent to all clusters having the document type defined.
Refer to routing for details.
Vespa uses the document ID to distribute documents to nodes.
From the document identifier, the content layer calculates a numeric location.
A bucket contains all the documents,
where a given amount of least-significant bits of the location are all equal.
This property is used to enable co-localized storage of documents -
read more in buckets and
content cluster elasticity.
The document identifiers are URIs, represented by a string,
which must conform to a defined URI scheme for document identifiers.
The document identifier string may only contain text characters,
as defined by isTextCharacter in
com.yahoo.text.Text.
Schemes have two parts:
Namespace: Used to separate data for application-specific purposes
Modifiers to the id scheme, used to configure document distribution to
buckets.
With no modifiers, the id scheme distributes all documents uniformly.
The key/value-pairs field contains a comma-separated list of lexicographically sorted key/value pairs.
n and g are mutually exclusive:
n=<number>
Number in the range [0,2^63-1]
g=<groupname>
Just like n=, the string is hashed to a number
Important:
Do not use unless really required -
there are few use cases where a non-uniform document distribution to buckets is useful.
user-specified
Yes
A unique ID string
Fieldsets
Use fieldset to limit the fields that are returned from a read operation,
like get or visit.
Fieldsets should be considered hints to Vespa, used to optimize.
It should not be considered an error if Vespa returns more fields than specified.
The timestamp-field must have values in seconds since EPOCH:
field timestamp type long {
indexing: attribute
attribute {
fast-access
}
}
When garbage-collection="true", Vespa iterates over the document space to purge expired documents.
Vespa will invoke the configured GC selection for each stored document at most once every
garbage-collection-interval seconds.
Important:
This is a best-effort garbage collection feature to conserve CPU and space.
Use query filters if it is important to exclude documents based on a criterion.
Using a selection with now can have side effects
when re-feeding or re-processing documents, as timestamps can be stale.
A common problem is feeding with too old timestamps,
resulting in no documents being indexed.
Normally, documents that are already expired at write time are not persisted.
When using create
(Create if nonexistent), it is possible to create documents that are expired and will be removed in next cycle.
Deploying a configuration where the selection string selects no documents
will cause all documents to be garbage collected.
Use visit to test the selection string.
Garbage collected documents are not to be expected to be recoverable.
The fields that are referenced in the selection expression should be attributes.
Also, either the fields should be set with "fast-access" or the number of
searchable copies in the content cluster should be the same as
the redundancy.
Otherwise, the document selection maintenance will be slow
and have a major performance impact on the system.
Imported fields
can be used in the selection string to expire documents, but special care needs to be
taken when using these.
See
using imported fields in selections for more information and restrictions.
Document garbage collection is a low priority background operation that runs continuously
unless preempted by higher priority operations.
If the cluster is too heavily loaded by client feed operations, there's a risk of starving
GC from running. To verify that garbage collection is not starved, check the
vds.idealstate.max_observed_time_since_last_gc_sec.average
distributor metric.
If it significantly exceeds garbage-collection-interval it is an indication that GC is starved.
To batch remove, set a selection that matches no documents, like "not music"
Use vespa-visit to test the selection.
Dump the IDs of all documents that would be preserved:
To process documents, use Document processing.
Examples are enriching documents (look up data from other sources),
transform content (like linguistic transformations, tokenization),
filter data and trigger other events based on the input data.
Both sample apps also use the Document API to GET/PUT/UPDATE other documents as part of processing,
using asynchronous
DocumentAccess.
Use this as a starting point for applications that enrich data when writing.