Search Definition Reference

edit page

This document lists the syntax and content of search definitions, document types and fields. This is a reference, read search definitions first for an overview. Find an example at the end.

There must be at least one search definition (.sd) file containing a search element in an application package.

Syntax

Throughout this document, a string in square brackets represents some argument. The whole string, including the brackets, is replaced by a concrete string in a search definition.

Constructs in search definitions have a regular syntax. Each element starts by the element identifier, possibly followed by the name of this particular occurrence of the element, possibly followed by a space-separated list of interleaved attribute names and attribute values, possibly followed by the element body. Thus, one will find elements of these varieties:

[element-identifier] : [element-body]

[element-identifier] [element-name] : [element-body]

[element-identifier] [element-name] [attribute-name] [attribute-value]

[element-identifier] [element-name] [attribute-name] [attribute-value] {
    [element-body]
}

Elements

A search definition must contain no more than one search clause - elements:

search
    document
        struct
            field
                match
        field
            alias
            attribute
            bolding
            id
            index
            indexing
            indexing-rewrite
            match
            normalizing
            query-command
            rank
            rank-type
            sorting
            stemming
            struct-field
                indexing
                match
                query-command
                struct-field
                 …
                summary
                summary-to DEPRECATED
            summary
            summary-to DEPRECATED
            weight
            weightedset
        compression
    index
    field
    fieldset
    rank-profile
        match-phase
            attribute
            order
            max-hits
            diversity
                attribute
                min-groups
        first-phase
            keep-rank-count
            rank-score-drop-limit
            expression
        ignore-default-rank-features
        num-threads-per-search
        rank
        rank-type
        rank-features
        constants
        rank-properties
        second-phase
            expression
            rerank-count
        summary-features
    constant
    stemming
    document-summary
        summary
    annotation
        field
    import field

search

The root element of search definitions. A search definition describes how some data should be stored, indexed, ranked and presented in results. A search definition must be defined in a file named [search-definition-name].sd.

search [name] {
    [body]
}

document

Contained in search. Describes a document type. This can also be the root of the search definition, if the document is not to be searched directly. A document type may inherit the fields of one or more other document types. If no document types are explicitly inherited, the document inherits the generic document type.

document [name] inherits [name-list] {
    [body]
}

The inherits attribute is optional and has as value a comma-separated list of names of other document types.

The body of a document type is optional and may contain:

struct

Contained in document. Defines a composite type. A struct consists of zero or more fields that the user can access together as one. The struct has to be defined before it is used as a type in a field specification.

struct [name] {
    [body]
}

Note that struct types are supported differently in indexed search and streaming search mode. Take a look at struct type, struct array type and map type for more details.

The body of a struct is optional and may contain:

field

Contained in search, document, struct or annotation. Defines a named value with a type and (optionally) how this field should be stored, indexed, searched, presented and how it should influence ranking.

field [name] type [type-name] {
    [body]
}

• attribute
• body
• case
• context
• documentid
• else
• header
• hit
• host
• if
• index
• position
• reference
• relevancy
• sddocname
• summary
• switch
• tokenize

The type attribute is mandatory - see field type for details and indexing restrictions. Supported types:

The body of a field is optional for search, document and struct, and disallowed for annotation. It may contain the following elements:

If the field is part of a struct definition, i.e. contained in the struct element, only match may be specified.

If the field is of type struct, only indexing, match and query-command may be specified.

A field declared outside of a document tag (i.e. immediately within a search tag) is referred to as an extra-field. Such fields may not be set directly, not programmatically and not through a feed - doing so will cause the document to be rejected by the indexer. Extra-field may only be populated using indexing statements that input the value of proper fields (e.g. indexing: input my_document_field | normalize | summary | index).

struct-field

Contained in field or struct-field. Defines how this struct field (a subfield of a struct) should be stored, indexed, searched, presented and how it should influence ranking. The field in which this struct field is contained must be of type struct or a collection of type struct. Note that struct fields are supported differently in indexed search and streaming search:

struct-field [name] {
    [body]
}

fieldset

Contained in search. Note: this is not related to the Document fieldset.

Fieldsets provide a way to group fields together for searching, to search multiple fields - example:

fieldset myfieldset {
  fields: a,b,c
}

The fields making up the field set should be as similar as possible in terms of indexing clause, matching etc. If they are not, test the application thoroughly. For example, it will work for a mix of attributes and indexes, but the matching for attribute fields will always be exact unless in streaming mode.

If specific match settings for the field set is needed, such as exact, specify it using a match clause:

fieldset myfieldset {
  fields: a,b,c
  match {
    exact
  }
}

fieldset myfieldset {
  fields: a,b,c
  query-command:"exact @@"
}

compression

DEPRECATED - DO NOT USE - see deprecations.

Contained in document. If a compression level is set within this element, lz4 compression is enabled for whole documents.

compression {
    [body]
}

The body of a compression specification is optional and may contain:

Name	Description	Occurrence
type	LZ4 is the only valid compression method.	Zero to one
level	Enable compression. LZ4 is linear and 9 means HC(high compression)	Zero to one
threshold	A percentage (multiplied by 100) giving the maximum size that compressed data can have to keep the compressed value. If the resulting compressed data is higher than this, the document will be stored uncompressed. Default value is 95.	Zero to one

rank-profile

Contained in search. A rank profile is a named set of rank settings which can be specified during queries (see the ranking parameter in the search API).

Rank profiles are used to specify an alternative ranking of the same data for different purposes, and to experiment with new rank settings. If no explicit rank profile is specified, one called "default" is implicitly created to hold the rank settings from each field. The "default" rank profile is always selected for queries which does not specify one. It is possible to add additional settings to the default rank profile by explicitly defining it.

rank-profile [name] inherits [rank-profile] {
    [body]
}

In addition to the default rank profile, a profile named unranked is implicitly created. This rank-profile makes sure that the rank phases in the search backend are skipped and should be used for queries that only require matching and do not use ranking. If you are sorting on something different than rank score this is also the profile to use. Note that this profile should not be used if the query contains Wand search operators. Also note that using this profile will give better performance as the rank phases are skipped.

The body of a rank-profile may contain:

match-phase

Contained in rank-profile. The config specifying ranking to be used during matching. This is used to limit the result set in order to cut latency. It is particularly useful if the first-phase ranking is expensive. It can be used for sorting on numeric values to limit the evaluated result set.

match-phase {
    attribute: [numeric single value attribute]
    order: [ascending | descending]
    max-hits: [integer]
    diversity
}

diversity

Contained in match-phase. Diversity is used to specify diversity in different phases - supported in match-phase. It is used to guarantee a minimum result set diversity.

Specify the name of an attribute that will be used to provide diversity. Result sets are guaranteed to get at least min-groups unique values from the diversity attribute from this phase. A document is considered as a candidate if:

• The query has not yet reached the max-hits number produced from this phase.
• The query has not yet reached the max number of candidates in one group. This is computed by the max-hits of the phase divided by min-groups

diversity {
    attribute: [numeric attribute]
    min-groups: [integer]
}

first-phase

Contained in rank-profile. The config specifying the first phase of ranking. This is the initial ranking performed on all hits, and you should therefore avoid doing heavy rank-calculations here. By default, this will use the ranking feature nativeRank.

first-phase {
    [body]
}

expression

Contained in first-phase or second-phase. Specify a ranking expression. The expression can either be written directly or loaded from a file. When writing it directly the syntax is:

expression: [ranking expression]

expression {
    [ranking expression]
    [ranking expression]
    [ranking expression]
}

Expressions can also be loaded from a separate file. This is useful when dealing with the very long expressions generated by e.g. MLR. The syntax is:

expression: file:[path-to-expressionfile]

rank-features

Contained in rank-profile. List of extra rank features to be dumped when using the query-argument rankfeatures.

rank-features: [feature] [feature]

rank-features {
    [feature]
    [feature]
}

constants

Contained in rank-profile. List of constants available in ranking expressions, resolved and optimized at configuration time.

constants {
    key: value
}

rank-properties

Contained in rank-profile. List of generic properties, in the form of key/value pairs to be used by ranking features.

rank-properties {
    key: value
}

function (inline)? [name]

Contained in rank-profile. Define a named function that can be referenced as a part of the ranking expression, or (if having no arguments) as a feature. A function accepts any number of arguments.

function [name]([arg1], [arg2], [arg3]) {
  expression: …
}

function [name] ([arg1], [arg2], [arg3]) {
  expression {
      [ranking expression]
      [ranking expression]
      …
}

rank-profile default inherits default {
  function myfeature() {
    expression: fieldMatch(title) + freshness(timestamp)
  }
  function otherfeature(foo) {
    expression{ nativeRank(foo, body) }
  }

  first-phase {
    expression: myfeature * 10
  }
  second-phase {
    expression: otherfeature(title) * myfeature
  }
  summary-features: myfeature
}

Adding the inline modifier will inline this function in the calling expression if it also has no arguments. This is faster for very small and cheap functions (and more expensive for others).

second-phase

Contained in rank-profile. The config specifying the second phase of ranking. This is the optional reranking performed on the best hits from the first phase, and where you should put any advanced ranking calculations (e.g. MLR). By default, no second-phase ranking is performed. In streaming search we perform the second phase ranking on all hits. You can therefore put all the rank calculation in the first phase rank expression and just skip second phase.

second-phase {
    [body]
}

summary-features

Contained in rank-profile. List of rank features to be dumped for every query. Using many items will have a performance impact, a larger list to be returned only when requested can be specified in rank-features.

summary-features: [feature] [feature]…

summary-features {
    [feature]
    [feature]
}

constant

Contained in search. This defines a named constant tensor located in a file with a given type that can be used in ranking expressions via the rank feature constant. A constant with a given name is defined as follows:

constant [name] {
    [body]
}

constant my_constant_tensor {
    file: constants/my_constant_tensor_file.json
    type: tensor<float>(x{},y{})
}

{
    "cells": [
        { "address": { "x": "a", "y": "b"}, "value": 2.0 },
        { "address": { "x": "c", "y": "d"}, "value": 3.0 }
    ]
}

document-summary

Contained in search. An explicitly defined document summary. By default, a document summary named default is created. Using this element, other document summaries containing a different set of fields can be created.

document-summary [name] {
    [body]
}

stemming

Contained in field, search or index. Sets how to stem a field or an index, or how to stem by default. Read more on stemming.

stemming: [stemming-type]

normalizing

Contained in field. Sets normalizing to be done on this field. Default is to normalize.

normalizing: [normalizing-type]

alias

Contained in attribute, field or index. Makes an index or attribute available under an additional name:

alias [index/attr-name]: [alias]

attribute

Contained in field or struct-field. Specifies a property of an index structure attribute:

attribute [attribute-name]: [property]

attribute [attribute-name] {
    [property]
    [property]
    …
}

Note that normalizing and tokenization is not enabled by default for attribute fields. Queries in attribute fields are hence not normalized. Use index on fields to enable. Both index and attribute can be set on a field.

sorting

Contained in attribute or field. Specifies how sorting should be done.

sorting : [property]

sorting {
    [property]
    [property]
    …
}

bolding

Contained in field. Highlight matching query terms in the summary:

bolding: on

The default is no bolding, set bolding: on to enable it. Note that this command is overridden by summary: dynamic, if both are specified, bolding will be ignored. The difference between using bolding instead of summary: dynamic is the latter will provide a dynamic abstract in addition to highlighting search terms while the first only does highlighting.

The default XML element used to highlight the search terms is <hi> - to override, set container.qr-searchers configuration. Example using <strong>:

<container>
  <search>
    <config name="container.qr-searchers">
      <tag>
        <bold>
          <open>&lt;strong&gt;</open>
          <close>&lt;/strong&gt;</close>
        </bold>
        <separator>...</separator>
      </tag>
    </config>
  <search>
<container>

id

Contained in field. Sets the numerical id of this field. All fields have a document-internal id internally for transfer and storage. Id's are usually determined programmatically as a 31-bit number. Some storage and transfer space can be saved by instead explicitly setting id's to a 7-bit number.

id: [positive integer]

• Must be a positive integer
• Must be less than 100 or larger than 127
• Must be unique within the document and all documents this document inherits

index

Contained in field or search. Sets index parameters. Content in fields with index are normalized and tokenized by default. This element can be single- or multivalued:

index [index-name]: [property]

index [index-name] {
    [property]
    [property]
    …
}

indexing

Contained in field or struct-field. One or more Indexing Language instructions used to produce index, attribute and summary data from this field. Indexing instructions has pipeline semantics similar to unix shell commands. The value of the field enters the pipeline during indexing and the pipeline puts the value into the desired index structures, possibly doing transformations and pulling in other values along the way.

indexing: [index-statement]

indexing {
    [indexing-statement];
    [indexing-statement];
    …
}

Specify the operations separated by the pipe (|) character. For advanced processing needs, use the indexing language, or write a document processor. Supported expressions for fields are:

When combining both index and attribute in the indexing statement for a field, e.g indexing: summary|attribute|index, the match mode becomes text for the field. So searches in this field will not search the contents in the attribute but the index

indexing-rewrite

Contained in field. Vespa will normally rewrite indexing statements extensively to implement the technical tasks which are required to carry out the intentions of the indexing statement. The rewriting done can be controlled using this element.

indexing-rewrite: none

match

Contained in field, fieldset or struct-field. Sets the matching method to use for this field to something else than the default token matching.

match: [property]

match {
    [property]
    [property]
    …
}

Also see search using regular expressions.

match {
    exact
    exact-terminator: "@%"
}

someword AND (tag:!*!@@ OR tag:(kanoo)@@)

foo AND (artist:"'N Sync" OR artist:"*NSYNC" OR artist:A*teens OR artist:"Wham!")

match {
    gram
    gram-size: 3
}

rank

Contained in field or rank-profile. Set the kind of ranking calculations which will be done for the field. Even though the actual ranking expressions decide the ranking, this settings tells Vespa which preparatory calculations and which data structures are needed for the field.

rank [field-name]: [ranking settings]

rank {
    [ranking setting]
}

query-command

Contained in fieldset, field or struct-field. Specifies a function to be performed on query terms to the indexes of this field when searching. The Search Container server has support for writing Vespa Searcher plugins which processes these commands.

query-command: [any string]

rank-type

Contained in field or rank-profile. Selects the low-level rank settings to be used for this field when using nativeRank.

rank-type [field-name]: [rank-type-name]

summary-to

DEPRECATED Contained in field or struct-field. Specifies the name of the document summaries which should contain this field.

summary-to: [summary-name], [summary-name], …

summary

Contained in field or document-summary or struct-field. Declares a summary field.

summary: [property]

summary [name] type [type] {
    [body]
}

source: [field-name], [field-name], …

to: [document-summary-name], [document-summary-name], …

weight

Contained in field. The weight of a field - the default is 100. The field weight is used when calculating the rank scores.

weight: [positive integer]

weightedset

Contained in field of type weightedset. Properties of a weighted set.

weightedset: [property]

weightedset {
    [property]
    [property]
    …
}

annotation

Contained in search. Defines an annotation type, to be used by the Annotations API. A name of the annotation is mandatory, the body is optional.

annotation [name] {
    [body]
}

import field

Contained in search. Using a reference to a document type, import a field from that document type into this search definition to be used for matching, ranking, grouping and sorting. Refer to parent/child.

Only attribute fields can be imported. The imported field inherit all but the following properties from the parent field:

• attribute: fast-access

To use an imported field in summary, create an explicit document summary containing the field.

Field types

field surname type string {
    indexing: summary | index
}

field release_year type int {
    indexing: summary | attribute
}

field bignumber type long {
    indexing: summary | attribute
}

field alive type bool {
    indexing: summary | attribute
}

field smallnumber type byte {
    indexing: summary | attribute
}

field myfloat type float {
    indexing: summary | attribute
}

field mydouble type double {
    indexing: summary | attribute
}

field location type position {
    indexing: attribute
}

field predicate_field type predicate {
    indexing: attribute
    index {
        arity: 2  # mandatory
        lower-bound: 3
        upper-bound: 200
        dense-posting-list-threshold: 0.25
    }
}

field rawfield type raw {
    indexing: summary
}

http://mysite.mydomain.com:8080/path/shop?d=hab&id=1804905709&cat=100#frag1

[field-name].[component-name]:term

query=sourceurl.hostname:slashdot

alias [component]: [alias]

field surl type uri {
    indexing: summary | index
    alias: url
    alias hostname: site
}

struct person {
    field first_name type string {}
    field last_name  type string {}
}

field people type array<person> {
    indexing: summary
    struct-field first_name { indexing: attribute }
    struct-field last_name  { indexing: attribute }
}

field tag type weightedset<string> {
    indexing: attribute | summary
}

field tensorfield type tensor<float>(x{},y{}) {
    indexing: attribute | summary
}

field tensorfield type tensor<float>(x[2],y[2]) {
    indexing: attribute | summary
}

struct person {
    field first_name type string {}
    field last_name type string {}
}
field my_person type person {
    indexing: summary
}

struct person {
    field first_name type string {}
    field last_name type string {}
}
field identities type map<string, person> {
    indexing: summary
    struct-field key { indexing: attribute }
    struct-field value.last_name { indexing: attribute }
}

field my_map type map<string, int> {
    indexing: summary
    struct-field key { indexing: attribute }
    struct-field value { indexing: attribute }
}

struct mystruct {
    field key type string { }
    field value type int { }
}
field my_array type array<mystruct> {
    indexing: summary
    struct-field key { indexing: attribute }
    struct-field value { indexing: attribute }
}

annotation foo {
    field baz type annotationreference<bar> { }
}

annotation bar { }

field artist_ref type reference<artist> {
    indexing: attribute
}

Document and search field types

Note that it is possible to make a document field of one type into one or more instances of another search field, by declaring a field outside the document, which uses other fields as input. For example, to create an integer attribute for a string containing a comma-separated list of integers in the document, do like this:

search example {
  document example {
    field yearlist type string { # Comma-separated years
      …
    }
    …
  }

  field year type array<int> { # Search field using the yearlist value
    indexing: input yearlist | split "," | attribute
  }
}

Config overrides affecting searchdefinition

maxtermoccurrences

Limits the number of occurrences of a term that will indexed. By default it is 100. This is a global setting that will be used for all fields in all document types.

<config name='vespa.configdefinition.ilscripts'>
    <maxtermoccurrences>100</maxtermoccurrences>
</config>

fieldmatchmaxlength

This is the max length of a field that is tokenized and indexed. By default it is 1000000. This is a global setting that will be used for all fields in all document types. For individual control see max-length

<config name='vespa.configdefinition.ilscripts'>
    <fieldmatchmaxlength>1000000</fieldmatchmaxlength>
</config>

Example

search example {

  document example {

    field title type string {
      indexing: summary | index
      alias: analias.totitle
      alias default: analias_todefault
    }

    field description type string {
      indexing: summary | index
    }

    field author type string {
      indexing: summary | index
      # author names only, so no stemming
      stemming: none
    }

    field category type string {
      indexing: summary | attribute
      attribute: fast-search
      match: exact #Don't tokenize  
      rank:filter # Only for matching. Most efficient search of a string type
    }

    field popularity type int {
      indexing: summary | attribute
      attribute:fast-search
    }

    field measurement type int {
      indexing: summary | attribute
    }

    # Categories as an array - preferable
    field morecategories type array<string> { indexing: index
    }

  }

  fieldset default {
    fields: title, description
  }

  # Additional default settings
  rank-profile default inherits default {
    first-phase {
        expression: nativeRank
    }
    second-phase {
      expression {
          0.5 * 0.5 * (0.1 * attribute(popularity) + fieldMatch(description))
          + 0.2 * attributeMatch(category)
          + 0.3 * fieldMatch(title)
      }
      rerank-count: 200
    }
  }

  # Some experimental ranking changes
  rank-profile experimental inherits default {
    second-phase {
      expression {
          0.5 * 0.5 * (attribute(measurement) * attribute(popularity) + fieldMatch(description))
          + 0.2 * attributeMatch(category)
          + 0.3 * fieldMatch(title)
      }
    }
  }

  # Ranking expression from separate file (filename mlrrank.expression, in the same directory as the sd-file)
  rank-profile mlrraning inherits default {
    second-phase {
      expression: file:mlrrank
    }
  }

  rank-profile other inherits experimental {
    second-phase {
      rerank-count: 100
    }
  }

  rank-profile justthebest {
    match-phase {
      attribute: popularity
      max-hits: 10000
      diversity {
        attribute: category
        min-groups: 20
      }
    }
    first-phase {
      expression: nativeRank + 0.1 * attribute(popularity)
    }
    second-phase {
      expression {
          0.5 * 0.5 * (0.1 * attribute(popularity) + fieldMatch(description))
          + 0.2 * attributeMatch(category)
          + 0.3 * fieldMatch(title)
      }
    }
  }

}

Modify Search Definitions

This section describes how a search definition in a live application can be modified - categories:

1. Valid changes without restart or re-feed
2. Changes that require restart but not re-feed
3. Changes that require re-feed

NOTE: If there are changes to perform on a live system that are not covered by this document and no output is given from vespa-deploy prepare, their impact is undefined and in no way guaranteed to allow a system to stay live until re-feeding. Changes not related to the search definition are discussed in admin procedures.

It is best practice to try changes in a staging system first.

Valid changes without restart or re-feed

Procedure:

1. Run vespa-deploy prepare on the changed application
2. Run vespa-deploy activate. The changes will take effect immediately

Changes that require restart but not re-feed

Procedure:

1. Run vespa-deploy prepare on the changed application. Output specifies which restart actions are needed
2. Run vespa-deploy activate
3. Restart services on the services specified in the prepare output

search test {
  document test {
    field f1 type string { indexing: summary }
  }
}

search test {
  document test {
    field f1 type string { indexing: attribute | summary }
  }
}

WARNING: Change(s) between active and new application that require restart:
In cluster 'mycluster' of type 'search':
    Restart services of type 'searchnode' because:
        1) Document type 'test': Field 'f1' changed: add attribute aspect

Changes that require re-feed

All of the changes listed below require re-feeding of all documents. Unless a change is listed in the above sections treat it as if it was listed here. Until re-feed is complete, affected fields will be empty or have potentially wrong annotations not matching the query processing. Procedure:

1. Run vespa-deploy prepare on the changed application. Output specifies which re-feed actions are needed
2. Stop feeding, wait until done
3. Run vespa-deploy activate
4. Re-feed all documents

search test {
  document test {
    field f1 type string { indexing: summary }
  }
}

search test {
  document test {
    field f1 type string { indexing: index | summary }
  }
}

WARNING: Change(s) between active and new application that require re-feed:
Re-feed document type 'test' in cluster 'mycluster' because:
    1) Document type 'test': Field 'f1' changed: add index aspect, indexing script: '{ input f1 | summary f1; }' -> '{ input f1 | tokenize normalize stem:"SHORTEST" | index f1 | summary f1; }'