Vespa 8 Release Notes

This document lists the changes between Vespa major versions 7 and 8. As documented in Vespa versions, new functionality in Vespa is introduced in minor versions, while major versions are used to mark releases breaking compatibility. As Vespa 8 does not introduce any new functionality, it is as safe and mature as the versions of Vespa 7 preceding it. No further releases will be made of Vespa 7, except possible critical security fixes.

Overview

The compatibility breaking changes in Vespa 8 fall into these categories:

• Changes to default behaviour
• Application package structure and settings - deprecated settings and constructs in e.g. schemas and services.xml are removed.
• Java APIs - deprecated APIs are removed or revoked from Vespa's public API surface.
• Container runtime environment - incompatible changes to the Java build and runtime environments.
• HTTP API changes
• Removed command line tools
• Removed or renamed metrics
• Changes to the document selection language
• Security related changes
• Operating system support
• Other changes, not covered by any of the above categories.

To ensure their applications are compatible with Vespa 8, application owners must:

• Review the list of changes to defaults and add the necessary options if you want to preserve behavior from Vespa 7.
• Make sure there are no deprecation warnings when compiling against Vespa 7.
• Review the application package changes and make sure there are no deployment warnings when deploying on Vespa 7.
• Review the list of HTTP API changes and update any clients of the application.
• Review the remaining sections of this document, and update the application and its environment accordingly.

Usage of deprecated Java APIs produce warnings during compilation, while deployment warnings are produced for application package deprecations and most changes to the container runtime environment. In hosted Vespa or Vespa Cloud, deployment warnings are shown in the application's console view. However, for other types of changes, there is no way to emit deprecation warnings, so these are only described in this document and other Vespa documentation.

The following sections lists all the changes from Vespa 7 to Vespa 8 in detail.

Changed defaults

These changes may break clients, and impact both performance and user experience. Applications that are in production and relies on these defaults should make configuration changes to keep the existing behavior when upgrading to Vespa 8. This can be done on Vespa 7, before upgrading - using bucket tests can be useful.

The following defaults have changed:

      <config name="container.core.access-log">
        <fileHandler>
          <compressionFormat>GZIP<compressionFormat>
    

Application package changes

Removed settings from schemas

The following settings are removed from schema:

             struct weightedfloat {
                 field value type float {}
                 field weight type int {}
             }
             field myfield type array<weightedfloat> {
                 ...
          

TensorFlow import

Vespa 8 removes support for direct import of TensorFlow models. ONNX is now the preferred ML model format, and works both for ranking as well as stateless model evaluation. ONNX contains tools to convert models from TensorFlow to ONNX, but Vespa will no longer provide this.

Changed semantics in services.xml

The following elements and attributes in services.xml have new semantics:

Removed constructs from services.xml

The following elements and attributes are removed from services.xml:

application/ folder support removed

Application package content (services.xml, the schemas/ folder, etc.) is supposed to be put at the root level in the application zip, such that the unzipped application package has ./services.xml, etc.

But it used to be that the application package content could be placed inside an application directory. This support is removed on Vespa 8.

searchdefinitions/ folder is deprecated

Search definition schemas should now be placed in the schemas/ folder. The old folder will still work on Vespa 8, but causes a deprecation warning upon deployment.

Java API changes

Removed Java packages

Removed Java Classes and methods

Classes and methods that were marked as deprecated in Vespa 7 are removed. If deprecation warnings are emitted for Vespa APIs when building the application, these must be fixed before migrating to Vespa 8. The sections below contain only the most notable changes.

The following classes are no longer public API and have been moved to Vespa internal packages:

The following methods are removed:

Breaking changes to Java APIs

The Javadoc of the deprecated types/members should document the replacement API. The below list is not exhaustive - some smaller and trivial changes are not listed.

Removed support for built-in XML factories

The Jdisc container has historically supported injection of built-in providers for the following XML factories:

• javax.xml.datatype.DatatypeFactory
• javax.xml.parsers.DocumentBuilderFactory and SAXParserFactory
• javax.xml.stream.XMLEventFactory, XMLInputFactory and XMLOutputFactory
• javax.xml.transform.TransformerFactory
• javax.xml.validation.SchemaFactory
• javax.xml.xpath.XPathFactory

These are now removed. Please check for more recent alternatives if you need this type of XML processing.

Deprecated Java APIs

A few redundant APIs have been deprecated because they have replacements that provide the same, or better, functionality. We advise you switch to the replacement to reduce future maintenance cost.

Container Runtime Environment

JDK version

Vespa 8 upgrades the JDK version from 11 to 17. To ensure full compatibility, all container components should be rebuilt with JDK 17 before being deployed on Vespa 8.

Changes to provided maven artifacts

Guava has been upgraded from version 20.0 to 27.1. If you are using APIs that have been removed from the library since version 20, your code must be updated. In most cases, it should be trivial to find replacement APIs in Java's standard library.

The following Maven artifacts are no longer provided runtime to user application plugins by the Jdisc container:

Make sure your application OSGi bundle embeds the required artifacts from the above list. An artifact can be embedded by adding it in scope compile to the dependencies section in pom.xml. Typically, these artifacts have until now been used in scope provided. Use mvn dependency:tree to check whether any of the listed artifacts are directly or transitively included as dependencies.

As always, remove any dependencies that are not required by your project. Consult the Maven documentation on Dependency Exclusions for how to remove a transitively included dependency.

<dependencies>
  ...
  <dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20211205</version>
    <scope>compile</scope>
  </dependency>
  ...
</dependencies>

Removed HTTP API parameters

The following HTTP API parameters are removed from the query API:

Removed command line tools

vespa-http-client

The vespa-http-client command line tool is removed on Vespa 8 and is replaced by the new vespa-feed-client. The new client uses HTTP/2 and the Document v1 API.

The underlying rest API used by the vespa-http-client will still be available and supported on Vespa 8. You can therefore continue to use an old client distributed with Vespa 7 to feed to a Vespa 8 installation. Note that there will not be released any updates for vespa-http-client after the initial Vespa 8 release, while fixes and security updates to the rest API implementation will continue as part of Vespa 8. We strongly recommend that you migrate away from vespa-http-client in a timely manner.

Removed or renamed metrics

The following metrics are renamed:

The following metrics are removed:

Exact matching of document types in selection language

The document selection language now uses exact matching for document types rather than inheritance ("is-a") matching.

Example with two minimal document schemas:

• document my_doc_type {}
• document my_extended_doc_type inherits my_doc_type {}

Previously, the selection expression my_doc_type would match both a document instance of type my_doc_type and my_extended_doc_type. It will now only match a document of type my_doc_type.

Security

Strict mode enabled by default

Strict mode for request filtering in the jdisc container is enabled by default in Vespa 8. See documentation on the strict-mode attribute in services.xml for details.

Request headers controlling remote host/port in access log

The jdisc container will use the X-Forwarded-For and X-Forwarded-Port request headers to set the remote host and port respectively in the access log. The following request headers will no longer be handled by default:

• y-ra
• yahooremoteip
• client-ip
• y-rp

Operating system support for Vespa artifacts

RPMs

The supported OS for Vespa RPMs changes from CentOS Linux 7 to CentOS Stream 8 for Vespa 8. RPMs will still be built and distributed on Fedora Copr. If you install Vespa RPMs you will have to upgrade your OS to CentOS Stream 8.

OCI containers (Docker containers)

The base image used in our OCI containers changes from docker.io/centos:7 to quay.io/centos/centos:stream8 for Vespa 8. This means that the container image is built and tested on systems running kernel version 4.18.0 (current kernel for CentOS Stream 8). If you use Vespa's container image, you should upgrade the hosts running the containers to the same or a newer kernel version.

Other changes

Unknown rank profiles

Queries that specify a rank profile which does not exist in all the schemas being queried will now fail instead of falling back to using the default profile. Queries to multiple schemas must use a rank profile that exists in all of them, which can be ensured by inheriting a common schema.

Unknown summary classes

Queries that specify a non-existent summary class will now fail, instead of being rendered empty. Queries to multiple schemas must use a summary class that exists in all of them, which can be ensured by inheriting a common schema.

The "qrserver" service name

Vespa containers are in general using "container" as their service name. However, a container cluster that has declared neither document-processing nor document-api used to be named "qrserver". On Vespa 8 all container clusters uses the service name "container". This affects the output of all metrics APIs, as well as the Vespa log output.

Container access logs

The folder for container access logs has been moved from $VESPA_HOME/logs/vespa/qrs/ to $VESPA_HOME/logs/vespa/access/.

The default compression format has changed from gzip to zstd, see changed defaults.

ONNX output in summary features

When defining an ONNX model output in summary features, Vespa 8 ensures that the summary feature name is onnx rather than onnxModel as in previous version.

Changes in rankfeatures

Vespa can calculate and return all rank-features in the rankfeatures summary field. Vespa 8 contains some changes to this list:

• now is removed
• bm25(field) is added
• matches(field) is added

The "storage" message bus routing policy is removed

The "storage" routing policy was removed in early Vespa 7, and clients specifying it have been forwarded to the content routing policy for backwards compatibility. The forwarding is removed on Vespa 8 and clients needs be updated.

Replace all usages of the "storage" policy with "content", which behaves identically.

vespa-dispatch-bin process is removed

The dispatch functionality is moved into the Vespa Container and the vespa-dispatch-bin process is removed. As this is not a public interface, the default was switched to not using vespa-dispatch-bin in Vespa-7.109.10. The process was removed in subsequent Vespa releases:

Rolling upgrade note: A rolling upgrade over Vespa-7.109.10 should work with no extra steps.

YQL format

• When Vespa outputs an YQL statement, it will now not end the string by a semicolon. Terminating statements with semicolon continue to be optional and legal input.
• Annotations are not enclosed in [] brackets (still valid input).
• The annotation name is not quoted (still valid input).

Example Vespa 7 / Vespa 8:

where text contains ([{"distance": 5}]near("a", "b"));
where text contains ({distance: 5}near("a", "b"))

Upgrade procedure

• When upgrading the config servers, the nodes of the application cannot receive config until they are upgraded themselves. We need to set all of them in standalone mode before continuing by running this command on each node:

      $ vespa-configproxy-cmd -m setmode memorycache
      

  Each node will automatically reattach itself when it is upgraded.
• Install the new Vespa version on the config servers and restart them one by one. Wait until it is up again, look in vespa log for "Changing health status code from 'initializing' to 'up'" or use health checks.
• Redeploy and activate the application:

  $ vespa deploy