Schema Reference

This documents the syntax and content of schemas, document types and fields. This is a reference, see schemas for an overview and examples.

Syntax

Throughout this document, a string in square brackets represents some argument. The whole string, including the brackets, is replaced by a concrete string in a schema.

Constructs in schemas have a regular syntax. Each element starts by the element identifier, possibly followed by the name of this particular occurrence of the element, possibly followed by a space-separated list of interleaved attribute names and attribute values, possibly followed by the element body. Thus, one will find elements of these varieties:

[element-identifier] : [element-body]
[element-identifier] [element-name] : [element-body]
[element-identifier] [element-name] [attribute-name] [attribute-value]
[element-identifier] [element-name] [attribute-name] [attribute-value] { [element-body] }

One-line element values starts by a colon and ends by newline.

Multiline values (for fields supporting them) are any block of text enclosed in curly brackets.

Comments may be inserted anywhere and start with a hash (#).

Names are identifiers, they must match ["a"-"z","A"-"Z", "_"]["a"-"z","A"-"Z","0"-"9","_"]*.

A schema-file is not sensitive to indentation.

Elements

Elements and structure of a schema file:

schema
    document
        struct
            field
                match
        field 
            annotationreference<annotationtype>
            array<type>
            bool
            byte
            double
            float
            int
            long
            map<key-type,value-type>
            position
            predicate
            raw
            reference<document-type>
            string
            struct-name
            tensor(dimension-1,...,dimension-N)
            uri
            weightedset<element-type>
                weightedset
            alias
            attribute
                distance-metric
            bolding
            dictionary
            id
            index
                hnsw
            indexing
            indexing-rewrite
            match
            normalizing
            query-command
            rank
            rank-type
            sorting
            stemming
            struct-field
                indexing
                match
                query-command
                struct-field
                 …
                summary
                summary-to DEPRECATED
            summary
            summary-to DEPRECATED
            weight
        compression
    index
    field
    fieldset
    rank-profile
        match-phase
            attribute
            order
            max-hits
            diversity
                attribute
                min-groups
        first-phase
            keep-rank-count
            rank-score-drop-limit
            expression
        second-phase
            expression
            rerank-count
        global-phase
            expression
            rerank-count
        function [name] 
        inputs
        constants
        onnx-model
        rank-properties
        match-features
        mutate
            on-match
            on-first-phase
            on-second-phase
            on-summary
        summary-features
        rank-features
        ignore-default-rank-features
        num-threads-per-search
        num-search-partitions
        min-hits-per-thread
        termwise-limit
        post-filter-threshold
        approximate-threshold
        target-hits-max-adjustment-factor
        rank
        rank-type
    constant
    onnx-model
    stemming
    document-summary
        summary
    annotation
        field
    import field
    raw-as-base64-in-summary

schema

The root element of schemas. A schema describes a type of data and what we should compute over it. A schema must be defined in a file named [schema-name].sd.

schema [name] inherits [name] {
    [body]
}

The inherits attribute is optional. If a schema is inherited, this schema will include all the constructs of it as if they were defined in this (except the parent document type). The document type in this must declare that it inherits the document type of the parent schema.

The body is mandatory and may contain:

document

Contained in schema and describes a document type. This can also be the root of the schema, if the document is not to be queried directly.

document [name] inherits [name-list] {
    [body]
}

The document name is optional, it defaults to the containing schema element's name. If there is no containing schema element, the document name is required. If the document with a name is defined inside a schema, the document name must match the schema element's name. The reference to document type in the documentation refers to the document name defined here.

The inherits attribute is optional and has as value a comma-separated list of names of other document types. A document type may inherit the fields of one or more other document types, see document inheritance for examples. If no document types are explicitly inherited, the document inherits the generic document type.

The body of a document type is optional and may contain:

struct

Contained in document. Defines a composite type. A struct consists of zero or more fields that the user can access together as one. The struct has to be defined before it is used as a type in a field specification.

struct [name] {
    [body]
}

The body of a struct is optional and may contain:

field

Contained in schema, document, struct or annotation. Defines a named value with a type and (optionally) how this field should be stored, indexed, searched, presented and how it should influence ranking.

field [name] type [type-name] {
    [body]
}

• attribute
• body
• case
• context
• documentid
• else
• header
• hit
• host
• if
• index
• position
• reference
• relevancy
• sddocname
• summary
• switch
• tokenize

Other names not to use include any words that start with a number or includes special characters.

The type attribute is mandatory - supported types:

annotation foo {
    field baz type annotationreference<bar> { }
}

annotation bar { }

struct person {
    field first_name type string {}
    field last_name  type string {}
}

field people type array<person> {
    indexing: summary
    struct-field first_name {
        indexing: attribute
        attribute: fast-search
    }
}

field alive type bool {
    indexing: summary | attribute
}

field smallnumber type byte {
    indexing: summary | attribute
}

field mydouble type double {
    indexing: summary | attribute
}

field myfloat type float {
    indexing: summary | attribute
}

field release_year type int {
    indexing: summary | attribute
}

field bignumber type long {
    indexing: summary | attribute
}

struct person {
    field first_name type string {}
    field last_name type string {}
}
field identities type map<string, person> {
    indexing: summary
    struct-field key {
        indexing: attribute
        attribute: fast-search
    }
}

field my_map type map<string, int> {
    indexing: summary
    struct-field key { indexing: attribute }
    struct-field value { indexing: attribute }
}

struct mystruct {
    field key type string { }
    field value type int { }
}
field my_array type array<mystruct> {
    indexing: summary
    struct-field key {
        indexing: attribute
        attribute: fast-search
    }
}

field location type position {
    indexing: attribute
}

field predicate_field type predicate {
    indexing: attribute
    index {
        arity: 2  # mandatory
        lower-bound: 3
        upper-bound: 200
        dense-posting-list-threshold: 0.25
    }
}

field rawfield type raw {
    indexing: summary | attribute
}

field artist_ref type reference<artist> {
    indexing: attribute
}

field surname type string {
    indexing: summary | index
}

struct person {
    field first_name type string {}
    field last_name type string {}
}
field my_person type person {
    indexing: summary
}

field tensorfield type tensor<float>(x{},y{}) {
    indexing: attribute | summary
}

field tensorfield type tensor<float>(x[2],y[2]) {
    indexing: attribute | summary
}

http://mysite.mydomain.com:8080/path/shop?d=hab&id=1804905709&cat=100#frag1

[field-name].[component-name]:term

query=sourceurl.hostname:slashdot

alias [component]: [alias]

field surl type uri {
    indexing: summary | index
    alias: url
    alias hostname: site
}

field tag type weightedset<string> {
    indexing: attribute | summary
}

The body of a field is optional for schema, document and struct, and disallowed for annotation. It may contain the following elements:

field artist type string {
    alias: artist_name
}

Fields can not have default values. See the document guide for how to auto-set field values.

It is not possible to query for fields without value (i.e. query for NULL) - see the query language reference. Fields without value are not returned in query results.

Fields can be declared outside the document block in the schema. These fields are not part of the document type but behave like regular fields for queries. Since they are not part of the document they cannot be written directly, but instead take their values from document fields, using the input expression: indexing: input my_document_field | embed | summary | index

This is useful e.g. to index a field in multiple ways, or to change the field value, something which is not allowed with document fields. When the document field(s) used as input are updated, these fields are updated with them.

struct-field

Contained in field or struct-field. Defines how this struct field (a subfield of a struct) should be stored, indexed, searched, presented and how it should influence ranking. The field in which this struct field is contained must be of type struct or a collection of type struct:

struct-field [name] {
    [body]
}

The body of a struct field is optional and may contain the following elements:

If this struct field is of type struct (i.e. a nested struct), only indexing:summary may be specified. See array<type> for example use.

fieldset

Contained in schema.

Fieldsets provide a way to group fields with the same match settings together for searching. To search multiple fields - example:

fieldset myfieldset {
    fields: a,b,c
}

Using the query yql=select from sources * where myfieldset contains "foo" will return all the documents for which one or more of the fields a, b or c contain "foo". By naming the field set default, those fields are searched without specifying the field set in unstructured queries: query=foo.

The fields making up the fieldset should be as similar as possible in terms of indexing clause and match mode. If they are not, test the application thoroughly. Having different match modes for the fields in the fieldset will generate a warning during application deployment.

Adding a fieldset will not create extra index structures in memory / on disk, it is just a mapping.

If specific match settings for the field set is needed, such as exact, specify it using a match clause:

fieldset myfieldset {
    fields: a,b,c
    match {
        exact
    }
}

Use query-commands in the field set to set search settings. Example:

fieldset myfieldset {
    fields: a,b,c
    query-command:"exact @@"
}

compression

Contained in document. If a compression level is set within this element, lz4 compression is enabled for whole documents.

compression {
    [body]
}

rank-profile

Contained in schema or equivalently in separate files in the application package, named [profile-name].profile in any directory below schemas/[schema-name]/. A rank profile is a named set of ranking expression functions and settings which can be selected in the query).

Whether defined inline in the schema or in a separate .profile file, the syntax of a rank profile is

rank-profile [name] inherits [rank-profile1], [rank-profile2], ...  {
    [body]
}

The inherits list is optional and may contain the name of other rank profiles in this schema or one it inherits. Elements not defined in this rank profile will then be inherited from those profiles. Inheriting multiple profiles which define the same elements leads to an error at deployment.

The body of a rank-profile may contain:

    adjustedTargetHits = min(targetHits / estimatedFilterHitRatio, targetHits * targetHitsMaxAdjustmentFactor).
    

match-phase

Contained in rank-profile. The match-phase feature lets you increase performance by limiting hits exposed to first-phase ranking to the highest (lowest) values of some attribute. The performance gain may be substantial, especially with an expensive first-phase function. The quality loss is dependent on how well the chosen attribute correlates with the first-phase score.

Documents which have no value of the chosen attribute will be taken as having the value 0.

See also graceful degradation.

match-phase {
    attribute: [numeric single value attribute]
    order: [ascending | descending]
    max-hits: [integer]
    diversity
}

strict

Contained in rank-profile. True or false. By default, Vespa uses loose type checking, where any query feature used but not defined in a query profile type is assumed to be a floats. Set true to cause a deploy failure on missing query property type definitions instead.

strict: true

diversity

Contained in match-phase. Diversity is used to specify diversity in different phases - supported in match-phase. It is used to guarantee a minimum result set diversity among the hits selected by match-phase and is only relevant in that context - not as a general way of achieving diversity.

Specify the name of an attribute that will be used to provide diversity. Result sets are guaranteed to get at least min-groups unique values from the diversity attribute from this phase. A document is considered as a candidate if:

• The query has not yet reached the max-hits number produced from this phase.
• The query has not yet reached the max number of candidates in one group. This is computed by the max-hits of the phase divided by min-groups

diversity {
    attribute: [numeric attribute]
    min-groups: [integer]
}

first-phase

Contained in rank-profile. The config specifying the first phase of ranking. See phased ranking with Vespa. This is the initial ranking performed on all matching documents, you should therefore avoid doing computationally expensive relevancy calculations here. By default, this will use the ranking feature nativeRank.

first-phase {
    [body]
}

expression

Contained in first-phase or second-phase or global-phase. Specify a ranking expression. The expression can either be written directly or loaded from a file. When writing it directly the syntax is:

expression: [ranking expression]

expression {
    [ranking expression]
    [ranking expression]
    [ranking expression]
}

The second format is primarily a convenience feature when using long expressions, enabling them to be split over multiple lines.

Expressions can also be loaded from a separate file. This is useful when dealing with the long expressions generated by e.g. MLR. The syntax is:

expression: file:[path-to-expressionfile]

The path is relative to the location of the schema definition file. The file itself must end with .expression. This suffix is optional in the sd-file. Therefore expression: file:mlrranking.expression and expression: file:mlrranking are identical. Both refer to a file called mlrranking.expression in the schemas directory.

rank-features

Contained in rank-profile. List of extra rank features to be dumped when using the query-argument rankfeatures.

rank-features: [feature] [feature]

rank-features {
    [feature]
    [feature]
}

Any number of ranking features can be listed on each line, separated by space.

inputs

Contained in rank-profile. List of inputs: Query features consumed by ranking expressions in this profile.

Query features are set either as a request property, or equivalently from a Searcher, by calling query.getRanking().getFeatures().put("query(myInput)", myValue).

Query feature types can also be declared in query profile types, but declaring inputs in the profile needing them is usually preferable.

Inputs are inherited from inherited profiles.

inputs {
    name [type]? (: value)?
}

Input examples:

inputs {
    myDouble: 0.5
    query(myOtherDouble) double
    query(myArray) tensor(x[3])
    query(myMap) tensor(key{}]):{key1: 1.0, key2: 2.0}
}

constants

Contained in rank-profile. List of constants available in ranking expressions, resolved and optimized at configuration time.

Constants are inherited from inherited profiles, and from the schema itself.

constants {
    name [type]?: value|file:[path]
}

Constant examples:

constants {
    myDouble: 0.5
    constant(myOtherDouble) double: 0.6
    constant(myArray) tensor(x[3]):[1, 2, 3]
    constant(myMap) tensor(key{}]):{key1: 1.0, key2: 2.0}
    constant(myLargeTensor) tensor(x[10000]): file:constants/myTensor.json.lz4
}

rank-properties

Contained in rank-profile. List of generic properties, in the form of key/value pairs to be used by ranking features. Examples.

rank-properties {
    key: value
}

function (inline)? [name]

Contained in rank-profile. Define a named function that can be referenced as a part of the ranking expression, or (if having no arguments) as a feature. A function accepts any number of arguments.

function [name]([arg1], [arg2], [arg3]) {
    expression: …
}

function [name] ([arg1], [arg2], [arg3]) {
    expression {
        [ranking expression]
        [ranking expression]
        …
}

rank-profile default inherits default {
    function myfeature() {
        expression: fieldMatch(title) + freshness(timestamp)
    }
    function otherfeature(foo) {
        expression{ nativeRank(foo, body) }
    }

    first-phase {
        expression: myfeature * 10
    }
    second-phase {
        expression: otherfeature(title) * myfeature
    }
    summary-features: myfeature
}

You can not include functions that accept arguments in summary features.

Adding the inline modifier will inline this function in the calling expression if it also has no arguments. This is faster for small and cheap functions (and more expensive for others).

second-phase

Contained in rank-profile. The config specifying the second phase of ranking. See phased ranking with Vespa. This is the optional re-ranking phase performed on the top ranking hits from the first-phase, and where you should put any advanced relevancy calculations. For example Machine Learned Ranking (MLR) models. By default, no second-phase ranking is performed.

second-phase {
    [body]
}

global-phase

Contained in rank-profile. The config specifying the global phase of ranking. See phased ranking with Vespa. This is an optional re-ranking phase performed on the top ranking hits in the stateless container after merging hits from all the content nodes. The "top ranking" here means as scored by the first-phase ranking expression or (if specified) second-phase ranking expression. Typically used for computing large ONNX models which would be expensive to compute on all content nodes. By default, no global-phase ranking is performed.

global-phase {
    [body]
}

summary-features

Contained in rank-profile. List of rank features to be included with each result hit, in the summaryfeatures field. Also see feature values in results.

If not specified, the features are as specified in the parent profile (if any). To inherit the features from the parent profile and specify additional features, specify explicitly that the features should be inherited from the parent as shown below. Refer to schema inheritance for examples.

The rank features specified here are computed in the fill phase of multiphased queries.

summary-features: [feature] [feature]…

or

summary-features [inherits parent-profile]? {
    [feature]
    [feature]
}

Any number of rank features separated by space can be listed on each line.

match-features

Contained in rank-profile. List of rank features to be included with each result hit, in the matchfeatures field. Also see feature values in results.

If not specified, the features are as specified in the parent profile (if any). To inherit the features from the parent profile and specify additional features, specify explicitly that the features should be inherited from the parent as shown below, also see schema inheritance.

match-features is similar to summary-features, but the rank features specified here are computed in the first phase of multiphase searching, also called the match phase. This gives a different performance trade-off, for details see feature values in results.

match-features: [feature] [feature]…

or

match-features [inherits parent-profile]? {
    [feature]
    [feature]
}

Any number of ranking features separated by space can be listed on each line.

mutate

Contained in rank-profile. Specifies mutating operations you can do to each of the documents that make it through the 4 query phases, on-match, on-first-phase, on-second-phase and on-summary.

mutate {
    [phase name] { [attribute name] [operation] [numeric_value] }
}

The attribute must be a single value numeric attribute, enabled as mutable. It must also be defined outside of the document clause.

Find examples and use cases in rank phase statistics.

constant

Prefer to define constants in the rank profiles that need them, with rank profile inheritance to avoid repetition. See constants.

Contained in schema. This defines a named constant tensor located in a file with a given type that can be used in ranking expressions using the rank feature constant(name):

constant [name] {
    [body]
}

constant my_constant_tensor {
    file: constants/my_constant_tensor_file.json
    type: tensor<float>(x{},y{})
}

{
    "cells": [
        { "address": { "x": "a", "y": "b"}, "value": 2.0 },
        { "address": { "x": "c", "y": "d"}, "value": 3.0 }
    ]
}

When an application with tensor constants is deployed, the files are distributed to the content nodes before the new configuration is being used by the search nodes. Incremental changes to constant tensors is not supported. When changed, replace the old file with a new one and re-deploy the application or create a new constant with a new name in a new file.

raw-as-base64-in-summary

Contained in schema. Whether raw fields should be rendered as a base64 encoded string in summary, the same way as in json feed format, rather than an escaped string. This is default true.

onnx-model

Contained in rank-profile or schema. This defines a named ONNX model located in a file that can be used in ranking expressions using the "onnx" rank feature.

Prefer to define onnx models in the rank profiles using them. Onnx models are inherited from parent profiles, and from the schema.

onnx-model [name] {
    [body]
}

The body of an ONNX model must contain:

For more details including examples, see ranking with ONNX models.

document-summary

Contained in schema. An explicitly defined document summary. By default, a document summary named default is created. Using this element, other document summaries containing a different set of fields can be created.

document-summary [name] inherits [document-summary] {
    [body]
}

The inherits attribute is optional. If defined, it contains the name of one other document summary in the same schema. This will cause summary fields to be inherited from the referenced document summary. Refer to schema inheritance for examples.

The body of a document summary consists of:

Use the summary query parameter to choose a document summary in searches. See also document summaries.

stemming

Contained in field, schema or index. Sets how to stem a field or an index, or how to stem by default. Read more on stemming.

stemming: [stemming-type]

normalizing

Contained in field. Sets normalizing to be done on this field. Default is to normalize.

normalizing: [normalizing-type]

dictionary

Contained in field, and specifies details of the dictionary used in the inverted index of the field. Applies only to attributes annotated with fast-search. You can specify either btree or hash, or both.

Note that prefix search for strings and range search for numeric fields will fall back to full scan if using hash. It is primarily intended for use when you have many unique terms with few occurrences (short postinglists), where the dictionary lookup cost is significant. Use hash for fields with high uniqueness (high cardinality), for example an 'id' field which is unique in the corpus where the posting list is always of size 1.

Normally, btree is your best choice as it offers reasonable performance for both exact, prefix and range type of dictionary lookups. This is also the default. Find more details in attribute index structures.

In addition, one can specify uncased or cased dictionary for string attributes, default is uncased. This setting is sanity checked against match casing. In an uncased dictionary, casing is normalized by lowercasing so that 'bear' equals 'Bear' equals 'BEAR'. In a cased dictionary, they will all be different.

attribute

Contained in field or struct-field. Specifies a property of an index structure attribute:

attribute [attribute-name]: [property]

attribute [attribute-name] {
    [property]
    [property]
    …
}

An attribute is multivalued if assigning it multiple values during indexing, by using a multivalued field type like array or map, or by using e.g. split / for_each or by letting multiple fields write their value to the attribute field.

Note that normalizing and tokenization is not supported for attribute fields. Queries in attribute fields are not normalized, nor stemmed. Use index on fields to enable. Both index and attribute can be set on a field.

sorting

Contained in attribute or field. Specifies how sorting should be done.

sorting : [property]

sorting {
    [property]
    …
}

distance-metric

Specifies the distance metric to use with the nearestNeighbor query operator to calculate the distance between document positions and the query position. Only relevant for tensor attribute fields, where each tensor holds one or multiple vectors.

Which distance metric to use depends on the model used to produce the vectors; it must match the distance metric used during representation learning (model learning). If you are using an "off-the-shelf" model to vectorize your data, please ensure that the distance metric matches the distance metric suggested for use with the model. Different type of vectorization models use different type of distance metrics.

The calculated distance will be used to select the closest hits for nearestNeighbor query operator, but also to build the HNSW index (if specified) and to produce the distance and closeness ranking features.

distance-metric: [metric]

euclidean

The default metric is euclidean distance which is just the length of a line segment between the two points. To compute the euclidean distance directly in a ranking expression instead of fetching one already computed in a nearestNeighbor query operator, use the euclidean_distance helper function:

    function mydistance() {
        expression: euclidean_distance(attribute(myembedding), query(myqueryvector), mydim))
    }
  

angular

The angular distance metric computes the angle between the vectors. Its range is [0,pi], which is the angular distance. This is also known as ordering by cosine similarity where the score function is just the cosine of the angle. To compute the angular distance directly in a ranking expression, use the cosine_similarity helper function:

    function angle() {
        expression: acos(cosine_similarity(attribute(myembedding), query(myqueryvector), mydim))
    }
  

    rank-profile cosine {
        first-phase {
            expression: cos(distance(field, myembedding))
        }
    }
  

dotproduct

The dotproduct distance metric is used to mathematically transform a "maximum inner product" search into a form where it can be solved by nearest neighbor search, where the dotproduct is used as a score directly (large positive dotproducts are considered "nearby"). Internally an extra dimension is added (ensuring that all vectors are normalized to the same length) and a distance similar to prenormalized-angular is used to build the HNSW index. For details, see this high level guide based on section 3.1 Order Preserving Transformations in this paper.

Note that the distance and closeness rank-features will not have the usual semantic meanings when using the dotproduct distance metric. In particular, closeness will just return the dot product $\sum _n(x_i*y_i)$ which may have any negative or positive value, and distance is just the negative dot product. If a normalized closeness in range [0,1] is needed, an appropriate sigmoid function must be applied. For example, if your attribute is named "foobar", and the maximum dotproduct seen is around 4000, the expression sigmoid(0.001*closeness(field,foobar)) could be a possible choice.

The dotproduct distance metric is very useful for some vectorization models, including matrix factorization, that use "maximum inner product" (MIP), with vectors that aren't normalized. These models use both direction and magnitude. For example, the Dense passage retrieval sample app uses the mentioned transformation as the original text to vector model was trained using MIP. In contrast, the MS Marco ranking sample app uses a text to vector model which used cosine similarity during training so no transformation was required.

prenormalized-angular

The prenormalized-angular distance metric must only be used when both query and document vectors are normalized. This metric was previously named "innerproduct" and required unit length vectors; the new version computes the length of the query vector once and assumes all other vectors are of the same length.

Using prenormalized-angular with vectors that are not normalized causes unpredictable nearest neighbor search, and is observed to give very bad results both for performance and quality.

The length, magnitude, or norm of a vector x is calculated as length = sqrt(sum(pow(xi,2))). The unit length normalized vector is then given by [xi/length]. Zero vectors may not be used at all.

The Vespa prenormalized-angular computes the cosine similarity and uses 1.0 - cos(angle) as the distance metric. It gives exactly the same ordering as angular distance, but with a distance in the range [0,2], since cosine similarity has range [1,-1], so the end result is 0.0 for same direction vectors, 1.0 for a right angle, and 2.0 for vectors with exactly opposite directions. Getting the cosine score (or angle) is therefore easy:

    rank-profile cosine {
        first-phase {
          expression: 1.0 - distance(field, embedding)
        }
        function angle() {
          expression: acos(1.0 - distance(field, embedding))
        }
    }
  

    function mysimilarity() {
        expression: cosine_similarity(attribute(myembedding), query(myqueryvector), mydim))
    }
  

geodegrees

The geodegrees distance metric is only valid for geographical coordinates (two-dimensional vectors containing latitude and longitude on Earth, in degrees). It computes the great-circle distance (in kilometers) between two geographical points using the Haversine formula. See geodegrees system test for an example.

hamming

The Hamming distance metric counts the number of dimensions where the vectors have different coordinates. This isn't useful for floating-point data since it means you only get 1 bit of information from each floating-point number. Instead, it should be used for binary data where each bit is considered a separate coordinate. Practically, this means you should use the int8 cell value type for your tensor, with the usual encoding from bit pattern to numerical value, for example:

• 00000000 → 0 (hex 00)
• 00010001 → 17 (hex 11)
• 00101010 → 42 (hex 2A)
• 01111111 → 127 (hex 7F)
• 10000000 → -128 (hex 80)
• 10000001 → -127 (hex 81)
• 11111110 → -2 (hex FE)
• 11111111 → -1 (hex FF)

Feeding data for this use case may be done with "hex dump" format instead of numbers in range [-128,127] both to have a more natural format for representing binary data, and to avoid the overhead of parsing a large json array of numbers.

bolding

Contained in field or summary. Highlight matching query terms in the summary:

bolding: on

The default is no bolding, set bolding: on to enable it. Note that this command is overridden by summary: dynamic. If both are specified, bolding will be ignored. The difference between using bolding instead of summary: dynamic is the latter will provide a dynamic abstract in addition to highlighting query terms, while the first only highlights. Bolding is only supported for index fields of type string or array<string>.

The default XML element used to highlight the search terms is <hi> - to override, set container.qr-searchers configuration. Example using <strong>:

<container>
    <search>
        <config name="container.qr-searchers">
            <tag>
                <bold>
                    <open>&lt;strong&gt;</open>
                    <close>&lt;/strong&gt;</close>
                </bold>
                <separator>...</separator>
            </tag>
        </config>
    </search>
</container>

Maximum field byte length for bolding is 64Mb - field values larger than this will be represented as a snippet as in summary: dynamic.

id

Contained in field. Sets the numerical id of this field. All fields have a document-internal id internally for transfer and storage. Ids are usually determined programmatically as a 31-bit number. Some storage and transfer space can be saved by instead explicitly setting id's to a 7-bit number.

id: [positive integer]

An id must satisfy these requirements:

• Must be a positive integer
• Must be less than 100 or larger than 127
• Must be unique within the document and all documents this document inherits

index

Contained in field or schema. Sets index parameters. Content in fields with index are normalized and tokenized by default. This element can be single- or multivalued:

index [index-name]: [property]

index [index-name] {
    [property]
    [property]
    …
}

hnsw

Contained in index. Specifies that an HNSW index should be built to speed up approximate nearest neighbor search. Only useful for tensor attribute fields with one indexed dimension using the nearestNeighbor query operator. This implements a modified version of the Hierarchical Navigable Small World (HNSW) graphs algorithm (paper).

hnsw {
    [parameter]: [value]
    [parameter]: [value]
    ...
}

The following parameters are used when building the index graph:

The distance metric specified on the attribute is used when building and searching the graph. Example:

index {
    hnsw {
        max-links-per-node: 16
        neighbors-to-explore-at-insert: 100
    }
}

See Approximate Nearest Neighbor Search using HNSW Index for examples of use, and see Approximate Nearest Neighbor Search in Vespa - Part 1 blog post for how the Vespa team selected HNSW as the baseline algorithm for extension and integration in Vespa.

indexing

Contained in field or struct-field. One or more Indexing Language instructions used to produce index, attribute and summary data from this field. Indexing instructions has pipeline semantics similar to unix shell commands. The value of the field enters the pipeline during indexing and the pipeline puts the value into the desired index structures, possibly doing transformations and pulling in other values along the way.

indexing: [index-statement]

indexing {
    [indexing-statement];
    [indexing-statement];
    …
}

If the field containing this is defined outside the document, it must start by an indexing statement which outputs a value (either "input [fieldname]" to fetch a field value, or a literal, e.g "some-value" ). Fields in documents will use the value of the enclosing field as input (input [fieldname]) if one isn't explicitly provided.

Specify the operations separated by the pipe (|) character. For advanced processing needs, use the indexing language, or write a document processor. Supported expressions for fields are:

When combining both index and attribute in the indexing statement for a field, e.g indexing: summary | attribute | index, the match mode becomes text for the field. So searches in this field will not search the contents in the attribute but the index.

Find examples and more details in the Text Matching guide.

indexing-rewrite

Contained in field. Vespa will normally rewrite indexing statements extensively to implement the technical tasks which are required to carry out the intentions of the indexing statement. The rewriting done can be controlled using this element.

indexing-rewrite: none

Include this to let an indexing statement pass through unaltered. Note that such statements must begin with an input <fieldname>, get_var or constant expression. You should understand which rewrites Vespa does, and be certain that your indexing statement can do without them to use this. This statement must be placed somewhere below the indexing statement in the field.

match

Contained in field, fieldset or struct-field. Sets the matching method to use for this field to something else than the default token matching.

match: [property]

match {
    [property]
    [property]
    …
}

Whether the match type is text, word or exact, all term matching will be done after normalization and locale independent lowercasing (in that order).

Find examples and more details in the Text Matching guide. Also see search using regular expressions.

match {
    exact
    exact-terminator: "@%"
}

someword AND (tag:!*!@@ OR tag:(kanoo)@@)

foo AND (artist:"'N Sync" OR artist:"*NSYNC" OR artist:A*teens OR artist:"Wham!")

match {
    gram
    gram-size: 3
}

rank

Contained in field or rank-profile. Set the kind of ranking calculations which will be done for the field. Even though the actual ranking expressions decide the ranking, this setting tells Vespa which preparatory calculations and which data structures are needed for the field.

rank [field-name]: [ranking settings]

rank {
    [ranking setting]
}

Related: See the filter query annotation for how to annotate query terms as filters.

query-command

Contained in fieldset, field or struct-field. Specifies a function to be performed on query terms to the indexes of this field when searching. The Search Container server has support for writing Vespa Searcher plugins which processes these commands.

query-command: [an identifier or quoted string]

If you write a plugin searcher which needs some index-specific configuration parameter, that parameter can be set here.

There is one built-in query-command available: phrase-segmenting. If this is set, terms connected by non-word characters in user queries (such as "a.b") will be parsed to a phrase item, instead of by default, an AND item where these terms have connectivity set to 1.

rank-type

Contained in field or rank-profile. Selects the low-level rank settings to be used for this field when using nativeRank.

rank-type [field-name]: [rank-type-name]

For nativeRank one can specify a rank type per field. If the supported rank types do not meet requirements, one can explicitly configure the native rank features using rank-properties. See the native rank reference for more information.

summary-to

summary-to: [summary-name], [summary-name], …

Fields with summary will always be part of the default summary regardless of this setting. Use explicit document-summary instead. See also document summaries.

summary

Contained in field or document-summary or struct-field. Declares a summary field.

summary: [property]

summary [name] type [type] {
    [body]
}

source: [field-name], [field-name], …

to: [document-summary-name], [document-summary-name], …

Read more about document summaries.

weight

Contained in field. The weight of a field - the default is 100. The field weight is used when calculating the rank scores.

weight: [positive integer]

weightedset

Contained in field of type weightedset. Properties of a weighted set.

weightedset: [property]

weightedset {
    [property]
    [property]
    …
}

field tag type weightedset<string> {
    indexing: attribute | summary
    weightedset {
        create-if-nonexistent
        remove-if-zero
    }
}

annotation

Contained in schema. Defines an annotation type, to be used by the Annotations API. A name of the annotation is mandatory, the body is optional.

annotation [name] {
    [body]
}

import field

Contained in schema. Using a reference to a document type, import a field from that document type into this schema to be used for matching, ranking, grouping and sorting. Only attribute fields can be imported. The imported field inherits all but the following properties from the parent field:

• attribute: fast-access

Refer to parent/child for a complete example. Note that the imported field is put outside of the document type:

schema myschema {
    document myschema {
        field parentschema_ref type reference<parentschema> {
            indexing: attribute
        }
    }
    import field parentschema_ref.name as parent_name {}
}

Extra restrictions apply for some of the field types:

To use an imported field in summary, create an explicit document summary containing the field.

Imported fields can be used to expire documents, but read this first.

Document and search field types

Note that it is possible to make a document field of one type into one or more instances of another search field, by declaring a field outside the document, which uses other fields as input. For example, to create an integer attribute for a string containing a comma-separated list of integers in the document, do like this:

schema example {
    document example {
        field yearlist type string { # Comma-separated years
        }
    }

    field year type array<int> { # Search field using the yearlist value
        indexing: input yearlist | split "," | attribute
    }
}

Modifying schemas

This section describes how a schema in a live application can be modified—categories:

1. Valid changes without restart or re-feed
2. Changes that require restart but not re-feed
3. Changes that require reindexing
4. Changes that require re-feed

When running vespa-deploy prepare on a new application package, the changes in the schema files are compared with the files in the current active package. If some of the changes require restart or re-feed, the output from vespa-deploy prepare specifies which actions are needed.

Valid changes without restart or re-feed

Procedure:

1. Run vespa-deploy prepare on the changed application
2. Run vespa-deploy activate. The changes will take effect immediately

    document-summary short-summary {
        summary artist type string {}
    }

    field artist type string {
            indexing: summary | attribute
    }

    document-summary rename-summary {
        summary artist_name type string {
            source: artist
        }
    }

Changes that require restart but not re-feed

Procedure:

1. Run vespa-deploy prepare on the changed application. Output specifies which restart actions are needed
2. Run vespa-deploy activate
3. Restart services on the services specified in the prepare output

Example: Given a content cluster mycluster with mode index:

schema test {
    document test {
        field f1 type string { indexing: summary }
    }
}

schema test {
    document test {
        field f1 type string { indexing: attribute | summary }
    }
}

WARNING: Change(s) between active and new application that require restart:
In cluster 'mycluster' of type 'search':
    Restart services of type 'searchnode' because:
        1) Document type 'test': Field 'f1' changed: add attribute aspect

Changes that require reindexing

All of the changes listed below require reindexing of all documents. Unlike re-feed, which requires an external source of data, reindexing is done using documents stored in Vespa, and is automatic (once triggered). It can also run concurrently with feed and serving, but until reindexing is complete, affected fields will be empty or have potentially wrong annotations not matching the query processing. Procedure:

1. Run vespa-deploy prepare on the changed application. Output specifies which reindexing actions are needed
2. Run vespa-deploy activate
3. Trigger reindexing for the indicated document types and clusters

Changes:

    field artist type string {
            indexing: summary | index
    }

    document-summary rename-summary {
        summary artist_name type string {
            source: artist
        }
    }

schema test {
    document test {
        field f1 type string { indexing: summary }
    }
}

schema test {
    document test {
        field f1 type string { indexing: index | summary }
    }
}

WARNING: Change(s) between active and new application that require re-index:
Reindex document type 'test' in cluster 'mycluster' because:
    1) Document type 'test': Field 'f1' changed: add index aspect, indexing script: '{ input f1 | summary f1; }' -> '{ input f1 | tokenize normalize stem:"SHORTEST" | index f1 | summary f1; }'

Changes that require re-feed

All of the changes listed below require re-feeding of all documents. Unless a change is listed in the above sections treat it as if it was listed here. Until re-feed is complete, affected fields will be empty or have potentially wrong annotations not matching the query processing. Procedure:

1. Run vespa-deploy prepare on the changed application. Output specifies which re-feed actions are needed
2. Stop feeding, wait until done
3. Run vespa-deploy activate
4. Re-feed all documents

schema test {
    document test {
        field f1 type string { indexing: summary }
    }
}

schema test {
    document test {
        field f1 type int { indexing: summary }
    }
}

WARNING: Change(s) between active and new application that require re-feed:
Re-feed document type 'test' in cluster 'mycluster' because:
    1) Document type 'test': Field 'f1' changed: data type: 'string' -> 'int'