Rank Feature Reference

Query features

An application specific feature submitted with the query, see using the query feature.

A normalized number (between 0.0 and 1.0) describing the significance of the term; used as a multiplier or weighting factor by many other text matching rank features.

This should ideally be set by a searcher in the container for global correctness as each node will estimate the significance values from the local corpus. Use the Java API for significance or YQL annotation for significance.

As a fallback, a significance based on Robertson-Sparck-Jones term weighting is used; it is logarithmic from 1.0 for rare terms down to 0.5 for common terms (those occurring in every document seen).

Note that "rare" is defined as a frequency of 0.000001 or less. This is the term document frequency (how many documents contain the term out of all documents that can be observed), so you cannot get 1.0 as the fallback until you actually have a large number of documents (minimum 1 million) in the same search process.

See numTerms config.

The importance of matching this query term given in the query

The normalized strength with which this term is connected to the previous term in the query. Must be assigned to query terms in a searcher using the Java API for connectivity or YQL annotation for connectivity.

The total number of terms in this query, including both user and synthetic terms in all fields.

Document features

The number of terms in this field if one or more query term matched the field, 1000000 if no query term matched the field.

The value of a tensor or single value numeric attribute or null/NaN if not set. Use isNan() to check if value is not defined. Using undefined values in ranking expressions leads to undefined behavior.

The value at index n (base 0) of a numeric array attribute with the given name. Note that the index number must be explicit, it cannot be the output of an expression function. The order of the items in an array attribute is the same as the order they have in the input feed. If items are added using partial updates they are added to the end of the existing items list.

The weight found at a given key in a weighted set attribute

1 if the given key is present in a weighted set attribute, 0 otherwise

The number of elements in the attribute with the given name.

Creates a tensor<double> with one mapped dimension from the given integer or string weighted set attribute. The attribute is specified as the full feature name, attribute(name). The dimension parameter is optional. If omitted the dimension name will be the attribute name.

Example: Given the weighted set:

{key1:0, key2:1, key3:2.5}

tensorFromWeightedSet(attribute(myField), dim) produces:

tensor<double>(dim{}):{ {dim:key1}:0.0, {dim:key2}:1.0, {dim:key3}:2.5} }

Creates a tensor<double> with one mapped dimension from the given single value or array attribute. The value(s) must be integers or strings. The attribute is specified as the full feature name, attribute(name). The dimension parameter is optional. If omitted the dimension name will be the attribute name.

Example: Given an attribute field myField containing the array value:

[v1, v2, v3]

tensorFromLabels(attribute(myField), dim) produces:

tensor<double>(dim{}):{ {dim:v1}:1.0, {dim:v2}:1.0, {dim:v3}:1.0} }

See tensorFromWeightedSet for performance notes.

Field match features - normalized

A normalized measure of the degree to which this query and field matched (default, the long name of this is match). Use this if you do not want to create your own combination function of more fine-grained fieldmatch features.

Normalized proximity - a value which is close to 1 when matched terms are close inside each segment, and close to zero when they are far apart inside segments. Relatively more connected terms influence this value more. This is absoluteProximity/average connectedness for the query terms for this field.

Note that if all the terms are far apart, the proximity will be 1, but the number of segments will be high. Proximity is only concerned with closeness within segments, a total score must also take the number of segments into account.

The normalized total completeness, where field completeness is more important:

queryCompleteness * ( 1 - fieldCompletenessImportance ) + fieldCompletenessImportance * fieldCompleteness

The normalized ratio of query tokens matched in the field:

The normalized ratio of query tokens which was matched in the field:

A normalized metric of how well the order of the terms agrees in the chosen segments:

1-outOfOrder/pairs

A normalized measure of the degree to which different terms are related (occurring in the same segment):

1-(segments-1)/(matches-1)

A normalized measure of how early the first segment occurs in this field.

A normalized metric of the relative size of the longest sequence:

longestSequence/matches

A normalized metric of the closeness (inverse of spread) of segments in the field:

1-segmentDistance/fieldLength

The normalized proximity of the matched terms, not taking term connectedness into account. This number is close to 1 if all the matched terms are following each other in sequence, and close to 0 if they are far from each other or out of order.

Returns the normalized proximity of the matched terms, weighted by the connectedness of the query terms. This number is 0.1 if all the matched terms are and have default or lower connectedness, close to 1 if they are following in sequence and have a high connectedness, and close to 0 if they are far from each other in the segments or out of order.

Returns a normalized measure of the number of occurrences of the terms of the query. This is 1 if there are many occurrences of the query terms in absolute terms, or relative to the total content of the field, and 0 if there are none.

This is suitable for occurrence in fields containing regular text.

Returns a normalized measure of the number of occurrence of the terms of the query:

$$\frac{\sum_{\text{all query terms}}(min(\text{number of occurrences of the term},maxOccurrences))}{(\text{query term count} × 100)}$$

This is 1 if there are many occurrences of the query terms, and 0 if there are none.

This number is not relative to the field length, so it is suitable for uses of occurrence to denote relative importance between matched terms (i.e. fields containing keywords, not normal text).

Returns a normalized measure of the number of occurrence of the terms of the query, weighted by term weight. This number is close to 1 if there are many occurrences of highly weighted query terms, in absolute terms, or relative to the total content of the field, and 0 if there are none.

Returns a normalized measure of the number of occurrence of the terms of the query, taking weights into account so that occurrences of higher weighted query terms has more impact than lower weighted terms.

This is 1 if there are many occurrences of the highly weighted terms, and 0 if there are none.

This number is not relative to the field length, so it is suitable for uses of occurrence to denote relative importance between matched terms (i.e. fields containing keywords, not normal text).

Returns a normalized measure of the number of occurrence of the terms of the query in absolute terms, or relative to the total content of the field, weighted by term significance.

This number is 1 if there are many occurrences of the highly significant terms, and 0 if there are none.

Field match features - normalized and relative to the whole query

The normalized weight of this match relative to the whole query: The sum of the weights of all matched terms/the sum of the weights of all query terms. If all the query terms were matched, this is 1. If no terms were matched, or these matches has weight zero this is 0.

As the sum of this number over all the terms of the query is always 1, sums over all fields of normalized rank features for each field multiplied by this number for the same field will produce a normalized number.

Note that this scales with the number of matched query terms in the field. If you want a component which does not, divide by matches.

Returns the normalized term significance of the terms of this match relative to the whole query: The sum of the significance of all matched terms/the sum of the significance of all query terms. If all the query terms were matched, this is 1. If no terms were matched, or if the significance of all the matched terms is zero, this number is zero.

This metric has the same properties as weight.

See the term(n).significance feature for how the significance for a single term is calculated.

Returns the average of significance and weight. This has the same properties as those metrics.

Field match features - not normalized

The number of field text segments which are needed to match the query as completely as possible

The total number of query terms which was matched in this field

The number of degraded query terms which was matched in this field. A degraded term is a term where no occurrence information is available during calculation. The number of degraded matches is less than or equal to the total number of matches.

The total number of out of order token sequences within matched field segments

The total number of position jumps (backward or forward) within field segments

The summed length of all gaps within segments

The size of the longest matched continuous, in-order sequence in the field

The number of tokens in the field preceding the start of the first matched segment

The number of tokens in the field following the end of the last matched segment

The sum of the distance between all segments making up a match to the query, measured as the sum of the number of token positions separating the start of each field adjacent segment.

Query and field similarity

A weighted sum of the individual similarity measures.

A measure of how close together the query terms appear in the field.

A measure of the order in which the query terms appear in the field compared to the query.

A measure of how much of the query the field covers when a single term from the field can only cover a single term in the query. Query term weights are used during normalization.

A measure of how much of the field the query covers when a single term from the query can only cover a single term in the field.

Query term and field match features

The position of the first occurrence of this query term in this index field. numTerms configuration

The number of occurrences of this query term in this index field

Returns number of times any term in the query matches this index/attribute field.

Returns 1 if the index/attribute field with the given name is matched by the query.

Returns 1 if the index/attribute field with the given name is matched by the query term with position n.

The minimum distance between the occurrences of term x and term y in this index field. Term x occurs before term y.

The position of the occurrence of term x in this index field used for the forward distance.

The minimum distance between the occurrences of term y and term x in this index field. Term y occurs before term x.

The position of the occurrence of term y in this index field used for the reverse distance.

Features for indexed multivalue string fields

A weighted combination of fieldCompleteness and queryCompleteness for the element in the field that produces the highest value for this output after the elements weight is factored in. The weighting can be adjusted using elementCompleteness(name).fieldCompletenessImportance.

The field completeness of the best matching element. This is calculated as:

max( (number of query terms matched in the element) / (element size), 1.0).

The query completeness of the best matching element. This is calculated as:

(sum of weight for query terms matched in the element) / (sum of weight for query terms searching the field).

The weight of the best matching element.

Aggregated similarity between the query and individual field elements. The same sub-scores used by the textSimilarity feature are calculated for each individual element in the field. The final output is calculated as the maximum of the combined element similarity measures (similarity measures are combined the same way as the default output of the textSimilarity feature) multiplied with the element weight which is 1 for arrays, and the supplied weights for indexed weighted sets.

This is a flexible feature; how sub-scores are combined for each element and how element scores are aggregated may be configured. You may also add additional outputs if you want to capture multiple signals from a single field. Use elementSimilarity to customize this feature.

Attribute match features - normalized

A normalized measure of the degree to which this query and field matched. This is currently the same as completeness. Note that depending on what the attribute is used for, this may or may not be a suitable metric. If the attribute is a weighted set representing counts of items (like tags), normalizedWeight is probably a better metric.

The normalized total completeness, where field completeness is more important:

queryCompleteness * ( 1 - fieldCompletenessImportance + fieldCompletenessImportance * fieldCompleteness )

The query completeness for this attribute:

matches/the number of query terms searching this attribute

The normalized ratio of query tokens which was matched in the field. For arrays: matches/array length For weighted sets: sum of weight of matched terms/sum of weights of entire set. This is relatively expensive to calculate for large weighted sets.

A number which is close to 1 if the attribute weights of most matches in a weighted set are high (relative to maxWeight), 0 otherwise

A number which is close to 1 if the attribute weights of most matches in a weighted set are high (relative to maxWeight), and where highly weighted query terms has more impact, 0 otherwise

Used with the nearestNeighbor query operator. A number which is close to 1 when a point vector in the document is close to a matching point vector in the query. The document vectors and the query vector must be the same tensor type, with one indexed dimension of size N, representing a point in an N-dimensional space.

• dimension: Specifies the dimension of name. This must be either the string field or the string label.

  When using field, the name given must be a field with a tensor attribute of appropriate type. Often used when the document type has only one vector field, see example.

  When using label, queries are assumed to contain a nearestNeighbor query item with a label that matches the given name. This is useful when having multiple vector fields, where closeness() then maps to the nearestNeighbor operator with the field configured. Example.

• name: The value of the field name or label.

The output value is $$ closeness(dimension,name) = \frac{1.0}{1.0 + distance(dimension,name)}$$

When the tensor field stores multiple vectors per document, the minimum distance between the vectors of a document and the query vector is used in the calculation above.

A number which is close to 1 if the timestamp in attribute name is recent compared to the current time compared to maxAge:

max( 1-age(name)/maxAge , 0 )

Scales linearly with age, see freshness plot.

A logarithmic-shaped freshness; also goes from 1 to 0, but looks like freshness plot. The function is based on -log(age(name) + scale) and is calculated as:

$$\frac{log(maxAge + scale) - log(age(name) + scale)}{log(maxAge + scale) - log(scale)}$$

where scale is defined using halfResponse and maxAge:

$$\frac{-halfResponse^2}{2 × halfResponse - maxAge}$$

When age(name) == halfResponse the function output is 0.5.

Attribute match features - normalized and relative to the whole query

This has the same semantics as fieldMatch(name).weight.

This has the same semantics as fieldMatch(name).significance.

Returns the average of significance and weight. This has the same properties as those metrics.

Attribute match features - not normalized

The number of query terms which was matched in this attribute

The sum of the weights of the attribute keys matched in a weighted set attribute

totalWeight/matches

The maximum weight of the attribute keys matched in a weighted set attribute

Used with the nearestNeighbor query operator and a tensor field attribute name storing multiple vectors per document. This feature returns a tensor with one mapped dimension and one point with a value of 1.0, where the label of that point indicates which document vector was closest to the query vector in the nearest neighbor search.

Given a tensor field with type tensor<float>(m{},x[3]) used with the nearestNeighbor operator, an example output of this feature is:

    tensor<float>(m{}):{ 3: 1.0 }

In this example, the document vector with label 3 in the mapped m dimension was closest to the query vector.

Used with the nearestNeighbor query operator tagged with a label label and a tensor field attribute name storing multiple vectors per document.

See closest(name) for details.

Used with the nearestNeighbor query operator. A number which is close to 0 when a point vector in the document is close to a matching point vector in the query. The document vectors and the query vector must be the same tensor type, with one indexed dimension of size N, representing a point in an N-dimensional space.

• dimension: Specifies the dimension of name. This must be either the string field or the string label.

  When using field, the name given must be a field with a tensor attribute of appropriate type. Often used when the document type has only one vector field, see example.

  When using label, queries are assumed to contain a nearestNeighbor query item with a label that matches the given name. This is useful when having multiple vector fields, where distance() then maps to the nearestNeighbor operator with the field configured. Example.

• name: The value of the field name or label.

The output value depends on the distance metric used. The default is the Euclidean distance between the "n"-dimensional query point "d" and the point "d" in the document tensor field: $$ distance = \sqrt{\sum_{i=1}^n (q_i - d_i)^2} $$

When the tensor field stores multiple vectors per document, the minimum distance between the vectors of a document and the query vector is used in the calculation above.

The document age in seconds relative to the unit time value stored in the attribute having this name

Features combining multiple fields and attributes

A normalized average of the fieldMatch and attributeMatch scores of all the searched fields and attributes, where the contribution of each field and attribute is weighted by its weight setting.

The sum of the weight settings of all the field and attributes searched by the query

The (schema) weight setting of a field or attribute

Rank scores

Calculates the Okapi BM25 ranking function over the given indexed string field. This feature is cheap to compute, about 3-4 times faster than nativeRank, while still providing a good rank score quality wise. This feature is a good candidate for usage in a first phase ranking function when ranking text documents. Note that the field must be enabled to be used with the bm25 feature; set the enable-bm25 flag in the index section of the field definition. See the BM25 Reference for more detailed information.

A reasonably good rank score which is computed cheaply by Vespa. This value only is a good candidate first phase ranking function, and is the default used in the default rank profile. The value computed by this function may change between Vespa versions. See the native rank reference for more information.

Same as nativeRank, but only the given set of fields are used in the calculation.

Captures how well query terms match in index fields. Used by nativeRank. See the native rank reference for more information.

Same as nativeFieldMatch, but only the given set of index fields are used in the calculation.

Captures how near matched query terms occur in index fields. Used by nativeRank. See the native rank reference for more information.

Same as nativeProximity, but only the given set of index fields are used in the calculation.

Captures how well query terms match in attribute fields. Used by nativeRank. See the native rank reference for more information.

Same as nativeAttributeMatch, but only the given set of attribute fields are used in the calculation.

Calculates the sparse dot product between query term weights and match weights for the given field. Example: A weighted set string field X:

"X": {
    "x": 10,
    "y": 20,
    "z": 30
}

For the query (x!2 OR y!4), the nativeDotProduct(X) feature will have the value 100 (10*2+20*4) for that document.

Calculates the sparse dot product between query term weights and match weights as above, but for all term/field combinations.

The value of the rank score calculated in the first phase (unavailable in first phase ranking expressions)

Global features

A global sequence number computed as (1 << 48) - (LocalDocumentId << 16 || distribution-key). This will give a global sequence to documents. This is a cheap way of having stable ordering of documents. Note the large range of this value. Also note that if the system is not stable, e.g. if documents move around due to new nodes coming in, or nodes being removed, it will no longer be stable as documents might be found in a different replica. If you need true global ordering we suggest assigning a unique numeric id to your documents as an attribute field and use the attribute(name) feature.

Time at which the query is executed in unix-time (seconds since epoch)

A pseudorandom number in the range [0,1> which is drawn once per document during rank evaluation. By default, the current time in microseconds is used as a seed value. Users can specify a seed value by setting random.seed in the rank profile. If you need several independent random numbers the feature can be named like this: random(foo), random(bar).

A pseudorandom number in the range [0,1> that is stable for a given hit. This means that a hit will always receive the same random score (on a single node). If it is required that the scores be different between different queries, specify a seed value dependent upon the query. By default, the seed value is 1024. Users can specify a seed value by adding the query parameter rankproperty.random.match.seed=<value>. If you need several independent random numbers the feature can be named like this: random(foo).match, random(bar).match.

Same as random, except the random number is drawn from the Gaussian distribution using the supplied mean and stddev parameters. Can be called without parameters; default values are assumed. Seed is set similarly as random. If you need several independent random numbers with the same parameters, the feature can be named like this: randomNormal(0.0,1.0,foo), randomNormal(0.0,1.0,bar). If the parameters to randomNormal are not the same, you do not need to supply an additional name, e.g. randomNormal(0.0, 0.1) and randomNormal(0.0, 0.5) results in two independent values.

Same as randomNormal, except that the generated number is stable for a given hit, similar to random.match.

Returns the constant tensor value.

Match operator scores

The sum of all raw scores produced by match operators for this field.

The raw score produced by the query item with the given label.

Geo search

A number which is close to 1 if the position in attribute name is close to the query position compared to maxDistance:

max(1-distance(name)/maxDistance , 0)

Scales linearly with distance, see closeness plot.

A logarithmic-shaped closeness; like normal closeness it goes from 1 to 0, but looks like closeness plot. The function is a logarithmic fall-off based on log(distance + scale) and is calculated as:

$$closeness(name).logscale = \frac{log(maxDistance + scale) - log(distance(name) + scale))}{(log(maxDistance + scale) - log(scale))}$$

where scale is defined using halfResponse and maxDistance:

$$scale = \frac{halfResponse^2}{(maxDistance - 2 × halfResponse)}$$

When distance(name) == halfResponse the function output is 0.5; halfResponse should be less than maxDistance/2 since that means adding a certain distance when you are close matters more than adding the same distance when you're already far away.

The euclidian distance from the query position to the given position attribute in millionths of degrees (about 10 cm). If there are multiple positions in the query, items that actually search in name is preferred. Also: if multiple query items search in name, or name is an array of positions, or both, the closest distance found is returned.

As above, but scaled, so it uses the kilometer as unit of distance, instead of "micro-degrees".

The array index of the closest position found. Useful when name is of array<position> type.

The latitude (geographical north-south coordinate) of the closest position found. In range from -90.0 (South Pole) to +90.0 (North Pole). Useful when name is of array<position> type.

The latitude (geographical east-west coordinate) of the closest position found. In range from -180.0 (extreme west) to +180.0 (extreme east). Useful when name is of array<position> type.

The euclidian distance from a path through 2d space given in the query to the given position attribute in millionths of degrees. This is useful e.g. for finding the closest locations to a given road. The query path is set in the rankproperty.distanceToPath(name).path query parameter, using syntax "(x1,y1,x2,y2,..)" also in millionth of degrees, see the distance to path example. The closest point along the path is referred to as the intersection.

The normalized distance along the query path traveled before intersection (0.0 indicates start of path, 0.5 is middle, and 1.0 is end of path).

The cross-product of the intersected path segment and the intersection-to-document vector. Given that the document was found to lie closest to the path element A->B, the intersected path segment vector is [ B.x - A.x, B.y - A.y ]. Furthermore, given that the intersection of that path element occurred at point I for document location D, the intersection-to-document vector is [ I.x - D.x, I.y - D.y]. This is useful e.g. for finding what side of a path a document exists by looking at the sign of this value.

Utility features

foreach iterates over a set of feature output values and performs an operation on them. Only the values where the condition evaluates to true are considered for the operation. The result of this operation is returned.

• dimension: Specifies what to iterate over. This can be:
  • terms: All query term indices, from 0 and up to maxTerms.
  • fields: All index field names.
  • attributes: All attribute field names.
• variable: The name of the variable 'storing' each of the items you are iterating over.
• feature: The name of the feature you want to use the output value from. Use the variable as part of the feature name, and for each item you iterate over this variable is replaced with the actual item. Note that the variable replacement is a simple string replace, so you should use a variable name that is not in conflict with the feature name.
• condition: The condition used on each feature output value to find out if the value should be considered by the operation. The condition can be:
  • >a: Use feature output if greater than number a.
  • <a: Use feature output if less than number a.
  • true: Use all feature output values.
• operation: The operation you want to perform on the feature output values. This can be:
  • sum: Calculate the sum of the values.
  • product: Calculate the product of the values.
  • average: Calculate the average of the values.
  • max: Find the max of the values.
  • min: Find the min of the values.
  • count: Count the number of values.

Lets say you want to calculate the average score of the fieldMatch feature for all index fields, but only consider the scores larger than 0. Then you can use the following setup of the foreach feature:

foreach(fields,N,fieldMatch(N), ">0", average).

Note that when using the conditions >a and <a the arguments must be quoted.

You can also specify a ranking expression in the foreach feature by using the rankingExpression feature. The rankingExpression feature takes the expression as the first and only parameter and outputs the result of evaluating this expression. Let's say you want to calculate the average score of the squared fieldMatch feature score for all index fields. Then you can use the following setup of the foreach feature:

foreach(fields, N, rankingExpression("fieldMatch(N)*fieldMatch(N)"), true, average)

Note that you must quote the expression passed in to the rankingExpression feature.

The sparse dot product of the vector represented by the given weighted set attribute and the vector sent down with the query.

You can also do an ordinary full dotproduct by using arrays instead of weighted sets. This will be a lot faster when you have full vectors in the document with more than 5-10% non-zero values. You are also then not limited to integer weights. All the numeric datatypes can be used with arrays, so you have full floating point support. The 32 bit floating point type yields the fastest execution.

• name: The name of the weighted set string/integer or array of numeric attribute.
• vector: The name of the vector sent down with the query.

Each unique string/integer in the weighted set corresponds to a dimension and the belonging weight is the vector component for that dimension. The query vector is set in the rankproperty.dotProduct.vector query parameter, using syntax {d1:c1,d2:c2,…} where d1 and d2 are dimensions matching the strings/integers in the weighted set and c1 and c2 are the vector components (floating point numbers). The number of dimensions in the weighted set and the query vector do not need to be the same. When calculating the dot product we only use the dimensions present in both the weighted set and the query vector.

When using an array the dimensions is a positive integer starting at 0. If the query is sparse all non given dimensions are zero. That also goes for dimensions that outside of the array size in each document.

Assume a weighted set string attribute X with:

"X": {
    "x": 10,
    "y": 20
}

for a particular document. The result of using the feature dotProduct(X,Y) with the query vector rankproperty.dotProduct.Y={x:2,y:4} will then be 100 (10*2+20*4) for this document.

Arrays can be passed down as [w1 w2 w3 …] or on sparse form {d1:c1,d2:c2,…} as is already supported for weighted sets.

Convenience function for generating token sequence input to Transformer models. Creates a tensor with dimensions d0[1], d1[length], where d0 is the batch dimension and d1 is the maximum length of the token sequence. Assumes the inputs are zero-padded tensors representing token sequences. The result is the token sequence:

CLS + input_1 + SEP + input_2 + SEP + ... + 0's

• length: The maximum length of the token sequence
• input_N: Where to retrieve input from. At least one is required.

The inputs are typically retrieved from the query, document attributes or constants. For instance, tokenInputIds(128, query(my_input), attribute(my_field)) where input types are:

• query(my_input): tensor(d0[32])
• attribute(my_field): tensor(d0[128])

will create a tensor of type d0[1],d1[128] consisting of the CLS token 101, the tokens from the query, the SEP token 102, the tokens from the document field, the SEP token 102, and 0's for the rest of the tensor.

Convenience function for generating token sequence input to Transformer models. Creates a tensor with dimensions d0[1], d1[length], where d0 is the batch dimension and d1 is the maximum length of the token sequence. Assumes the inputs are zero-padded tensors representing token sequences. The result is the token sequence:

start_sequence_id + input_1 + sep_sequence_id + input_2 + sep_sequence_id + ... + 0's

• start_sequence_idThe start sequence id, typically 1
• sep_sequence_idThe separator sequence id, typically 2
• length: The maximum length of the token sequence
• input_N: Where to retrieve input from. At least one is required.

The inputs are typically retrieved from the query, document attributes or constants. For instance, customTokenInputIds(1,2,128, query(my_input), attribute(my_field)) where input types are:

• query(my_input): tensor(d0[32])
• attribute(my_field): tensor(d0[128])

Convenience function for generating token sequence input to Transformer models. Similar to the tokenInputIds, creates a tensor of type d0[1],d1[length] which represents a mask with zeros for the first input including CLS and SEP token, ones for the rest of the inputs (up to and including the final SEP token), and 0's for the rest of the tensor.

Convenience function for generating token sequence input to Transformer models. Similar to the tokenInputIds, creates a tensor of type d0[1],d1[length] which represents a mask with ones for all tokens that are set (CLS and SEP and all inputs), and zeros for the rest.