• [+] expand all

Vespa 7 Release Notes

These notes documents the changes between Vespa major versions 6 and 7. As documented in Vespa versions, major versions are used to mark breaking compatibility, not to release significant new functionality (which instead happens on minor versions). However, even on major versions, compatibility is broken only in two specific ways: Changes to defaults means that applications may need to set some option explicitly to preserve earlier behavior, and removal of deprecated functionality means that applications which use functionality that has earlier been deprecated need to change to keep working.

Most deprecated functionality causes warning during compilation (Java API deprecations) or deployment (application package deprecations), however with web service API's there is no way to emit deprecation warnings, and we have to rely on marking these as deprecated in the documentation.

Given this, application owners need to do 3 tasks to be compatible with Vespa 7:

  • Review whether changes to defaults requires additional settings in the application (note that this is likely on changing from 6 to 7 due to the changes to tokenization and stemming).
  • Make sure there are no deprecation warnings on compilation and deployment on Vespa 6.
  • Review the list of removed web service API's and API parameters and make sure these are not used by clients of the application.

As Vespa 7 does not introduce new functionality, it is as safe and mature as the versions of Vespa 6 preceding it. Upon release of Vespa 7, no further releases will be made of Vespa 6 for any reason.

Changes

The following sections lists the changes on moving from Vespa 6 to Vespa 7 which must be reviewed by applications.

Changed defaults

The following defaults have changed:

ChangeConfiguration required to avoid change on Vespa 7
stemming: shortest changed to stemming: best Add stemming: shortest to the schema block of all schemas.
Default linguistics component changed from SimpleLinguistics to OpenNlpLinguistics, including language detection using Optimaize turned on by default. Configure com.yahoo.language.simple.SimpleLinguistics as a component in services.xml as described in linguistics in Vespa
The default format accepted by the Java HTTP client is changed from XML to JSON To keep using XML:
  • Java API: When calling FeedClientFactory.create(sessionParams, ...), pass a SessionParams instance which has a FeedParams instance which have dataFormat set to FeedParams.DataFormat.XML_UTF8
  • Command line: Pass the --xmloutput option.
Query timeout changed from 5000 ms to 500 ms. Set the timeout parameter explicitly in requests or query profiles.
ranking.softtimeout.enable changed to default true Set to false in requests or a query profile.
The default access log format is changed to JSON. To keep the old proprietary format, set accesslog type=vespa in services.xml as described in the accesslog reference.
Default return format in vespa-visit and vespa-get is changed to JSON To get XML output specify the --xmloutput method

JDK version

Java components must be rebuilt with JDK 11 for the Vespa bundle-plugin to generate the correct set of imported packages for your OSGi bundles.

Removed Java API's

Classes and methods that were marked as deprecated in Vespa 6 are removed. If you get deprecation warnings for Vespa API's when building your application, they must be fixed before migrating to Vespa 7.

Container Runtime Environment

The following maven artifacts are no longer provided runtime:

  • commons-codec:commons-codec
  • org.apache.httpcomponents:httpclient
  • org.apache.httpcomponents:httpcore

If you need any of these dependencies, they must be embedded in your bundle by adding them in scope 'compile' in pom.xml.

Removed HTTP API's

The following HTTP APIs are removed:

NameReplacement
Legacy HTTP apis for document feeding:
  • /feed
  • /remove
  • /removelocation
  • /get
  • /visit
  • /document
The /document/v1/ web service API, or (for high throughput) the vespa-http-client.

Removed HTTP API parameters

The following HTTP API parameters are removed

NameReplacement
The defidx parameter in the search API Use a custom searcher if this functionality is needed.

Removed command line tools

The following command line tools are removed:

NameReplacement
Vespa spooler Custom client using the Java HTTP client

Removed settings from schemas

The following settings are removed from schemas:

NameReplacement
header None. This setting doesn't have any effect
body None. This setting doesn't have any effect

Removed constructs from services.xml

The following tags and attributes are removed from services.xml:

NameReplacement
‘rotationScheme’ attribute in <container><accesslog> None, rotation scheme ‘date’ will always be used
<container><filter> tag <container><http><filtering>

Removed metrics

The following metrics are removed:

NameReplacement
free/used/totalMemoryBytes mem.heap.free/used/total
http.in.bytes serverBytesReceived
http.out.bytes serverBytesSent
http.requests serverNumRequests
http.latency serverTotalSuccessfulResponseLatency
http.out.firstbytetime serverTimeToFirstByte
proc.uptime serverStartedMillis
proton.* content.proton.* (note that metrics might have different structure and names in new namespace)
vds.filestor.spi.* vds.filestor.alldisks.allthreads.*

Empty fields

Fields containing no value will not be included in responses on Vespa 7.

Allowed characters in request URIs

Vespa 6 allowed some special characters in raw form in the query component of request URIs. Vespa 7 requires these characters to be properly percent-encoded (RFC 2396).
CharacterPercent-encoding
\ %5C
^ %5E
` %60
{ %7B
| %7C
} %7D

Changes to the default JSON result format

The content of fields of type position in the default JSON query result format was rendered as XML on Vespa 6 but is rendered as JSON.

Specifically, the content of a position field was rendered as a string like

<position x="-121996000" y="37401000" latlong="N37.401000;W121.996000"/>

but is now instead rendered as a JSON map:


{
  "y": 37401000,
  "x": -121996000,
  "latlong": "N37.401000;W121.996000"
}

Renamed metrics

The following metrics are renamed:

Old NameNew Name
95p_query_latencyquery_latency.95percentile
99p_query_latencyquery_latency.99percentile
active_queriesactive_queries.average
athenz-tenant-cert.expiry.secondsathenz-tenant-cert.expiry.seconds.last
bytesvds.datastored.alldisks.bytes.average
configserver.cacheChecksumElemsconfigserver.cacheChecksumElems.last
configserver.cacheConfigElemsconfigserver.cacheConfigElems.last
configserver.delayedResponsesconfigserver.delayedResponses.count
configserver.failedRequestsconfigserver.failedRequests.count
configserver.hostsconfigserver.hosts.last
configserver.latencyconfigserver.latency.average
configserver.requestsconfigserver.requests.count
configserver.sessionChangeErrorsconfigserver.sessionChangeErrors.count
configserver.zkAvgLatencyconfigserver.zkAvgLatency.last
configserver.zkConnectionsconfigserver.zkConnections.last
configserver.zkMaxLatencyconfigserver.zkMaxLatency.last
configserver.zkOutstandingRequestsconfigserver.zkOutstandingRequests.last
configserver.zkZNodesconfigserver.zkZNodes.last
content.cluster-controller.cluster-state-change.countcluster-controller.cluster-state-change.count
content.proton.memoryusage.maxcontent.proton.documentdb.memory_usage.allocated_bytes.max
content.proton.transport.docsum.latency.averagecontent.proton.docsum.latency.average
degraded_queriesdegraded_queries.rate
deletefailedvds.idealstate.delete_bucket.done_failed.rate
deleteokvds.idealstate.delete_bucket.done_ok.rate
deletependingvds.idealstate.delete_bucket.pending.average
diskqueuesizevds.filestor.alldisks.queuesize.average
diskqueuewaitvds.filestor.alldisks.averagequeuewait.sum.average
diskusagecontent.proton.documentdb.disk_usage.last
docsvds.datastored.alldisks.docs.average
document_requestscontent.proton.docsum.docs.rate
documents_activecontent.proton.documentdb.documents.active.last
documents_inmemorycontent.proton.documentdb.index.docs_in_memory.last
documents_processeddocuments_processed.rate
documents_readycontent.proton.documentdb.documents.ready.last
documents_removedcontent.proton.documentdb.documents.removed.last
documents_totalcontent.proton.documentdb.documents.total.last
empty_resultsempty_results.rate
error.backend_communication_errorerror.backend_communication_error.rate
error.backends_ooserror.backends_oos.rate
error.empty_document_summarieserror.empty_document_summaries.rate
error.internal_server_errorerror.internal_server_error.rate
error.invalid_query_parametererror.invalid_query_parameter.rate
error.invalid_query_transformationerror.invalid_query_transformation.rate
error.misconfigured_servererror.misconfigured_server.rate
error.plugin_failureerror.plugin_failure.rate
error.result_with_errorserror.result_with_errors.rate
error.timeouterror.timeout.rate
error.unhandled_exceptionerror.unhandled_exception.rate
error.unspecifiederror.unspecified.rate
failed_queriesfailed_queries.rate
handled.requestshandled.requests.count
hits_per_queryhits_per_query.average
joinfailedvds.idealstate.join_bucket.done_failed.rate
joinokvds.idealstate.join_bucket.done_ok.rate
joinpendingvds.idealstate.join_bucket.pending.average
logd.processed.lineslogd.processed.lines.count
max_query_latencyquery_latency.max
mean_query_latencyquery_latency.average
mergefailedvds.idealstate.merge_bucket.done_failed.rate
mergeokvds.idealstate.merge_bucket.done_ok.rate
mergependingvds.idealstate.merge_bucket.pending.average
peak_qpspeak_qps.max
queriesqueries.rate
query_latencycontent.proton.transport.query.latency.average
query_requestscontent.proton.transport.query.count.rate
search_connectionssearch_connections.average
sentinel.uptimesentinel.uptime.last
slobrok.heartbeats.failedslobrok.heartbeats.failed.count
splitfailedvds.idealstate.split_bucket.done_failed.rate
splitokvds.idealstate.split_bucket.done_ok.rate
splitpendingvds.idealstate.split_bucket.pending.average
totalhits_per_querytotalhits_per_query.average
visitvds.visitor.allthreads.created.sum.rate
visitorlifetimevds.visitor.allthreads.averagevisitorlifetime.sum.average
visitorqueuewaitvds.visitor.allthreads.averagequeuewait.sum.average

Other changes

Vespa will not any longer implicitly load the "search" components" in containers which load the "document-api" components. If your application depends on "search" functionality in a container specifying the <document-api> tag in services.xml, make sure this container also specifies the <search> tag.