A system tests suite is an invaluable tool both when developing and maintaining a complex Vespa application.
These are functional tests which are run against a deployment of the application package to verify,
and use its HTTP APIs to execute feed and query operations which are compared to expected outcomes.
Vespa provides two formalizations of this:
Java JUnit tests, for more advanced tests,
run as regular Java tests, with some extra configuration.
These two frameworks also includes an upgrade—or staging—test construct for scenarios where the application
is upgraded, and state in the backend depends on the old application configuration;
as well as a production verification test—basically a health check for production deployments.
For system and staging tests, the frameworks provide an easy way to perform HTTP request against a
designated test deployment, separating the tests from the deployment and configuration of the test clusters.
This document describes how each of these test categories can be run as part of an imagined
CI/CD system for safely deploying changes to a Vespa application in a continuous manner.
System tests are just functional tests that verify a deployed Vespa application behaves as expected
when fed and queried. Running a system test is as simple as making a separate deployment with the
application package to test, and then running the system test suite, or one or a few or those tests.
Note:
Each system test should be self-contained,
i.e., it should be able to run each test in isolation; or all tests, in any order.
To achieve this, system tests should generally start by clearing all documents from the cluster to test.
This is the case with our sample system tests, so take care to not run them against a production cluster.
For the most part, system tests must be updated due to changes in the application package.
Rarely, an upgrade of the Vespa version may also lead to changed functionality, but within
major versions, this should only be new features and bug fixes. In any case, it is a good idea
to always run system tests against a dedicated test deployment—both before upgrading the Vespa
platform, and the application package—before deploying the change to production.
Running system tests
The Vespa CLI makes it easy to set up a test deployment, and run system and staging tests.
To run a system test, first set up a test deployment:
$ vespa deploy --wait 600
Run the basic HTTP tests (prefer using this test suite for regular tests) - also see the
example application:
$ vespa test tests/system-test/feed-and-search-test.json
Example Java API tests (use for complex test cases) - also see the
example application:
The test config file used by the test runner in the maven-plugin defines the endpoints for each of the clusters in
services.xml as fields under a localEndpoints JSON object:
feed-service is the endpoint of the container cluster with <document-api>
in services.xml.
query-service is the endpoint of the container cluster with <search>
in services.xml.
Staging tests
The goal of staging (upgrade) tests is not to ensure the new deployment
satisfies its functional specifications, as that should be covered by
system tests; rather, it is to ensure the upgrade of the application
package and/or Vespa platform does not break the application, and is
compatible with the behavior expected by existing clients.
As an example, consider a change in how documents are indexed, e.g., adding new document processor.
A system test would test verify this new behavior by feeding a document, and then verifying the
document processor modified the document, or perhaps did something else.
A staging test, on the other hand, would feed the document before the document processor was added,
and querying for the document after the upgrade could give different results
from what the system test would expect.
Many such changes, which require additional action post-deployment, are also guarded by
validation overrides,
but the staging test is then a great way of figuring out what the exact consequences of the change are,
and how to deal with it.
As opposed to system tests, staging tests are not self-contained, as the state change during upgrade is
precisely what is tested. Instead, execution order of any staging tests that modify state, particularly
after upgrade, must be controlled. Indeed, some changes will require refeeding data, and this should then be
part of the staging test code. Finally, it is also good to verify the expected state prior to upgrade.
The clients of a Vespa application should be compatible with both the system and staging test expectations,
and this dictates the workflow when deploying a breaking change - steps:
The application code and system and staging tests are updated, so tests pass;
and clients are updated to reflect the updated test code.
The application is upgraded.
The staging setup code is updated to match the new application code.
Again, it is a good idea to always run staging tests before deployment of every change—be it a
change in the application package, or an upgrade of the Vespa platform.
Running staging tests
See system tests above for links to example applications.
Steps:
A dedicated deployment is made with the current setup (package and Vespa version).
staging setup code is run to put the test cluster in a particular state—typically
one that mimics the state in production clusters.
The deployment is then upgraded to the new setup (package and/or Vespa version).
staging test code is run to verify the cluster behaves as expected post-upgrade.
Example using JSON-tests:
# load old application code, deploy it, run setup$ vespa deploy --wait 600
$ vespa test tests/staging-setup
# make changes to the application, deploy it, run tests$ vespa deploy --wait 120
$ vespa test tests/staging-test
Example using Java tests (see system tests for test-config.json):
# load old application code, deploy it, run setup$ vespa deploy --wait 600
$ mvn test-D test.categories=staging-setup -D vespa.test.config=/path-to/test-config.json
# make changes to the application, deploy it, run tests$ vespa deploy --wait 120
$ mvn test-D test.categories=staging -D vespa.test.config=/path-to/test-config.json
Feature switches and bucket tests
With continuous deployment, it is not practical to hold off releasing a feature until it is done,
test it manually until convinced it works and then release it to production.
What to do instead?
The answer is feature switches: release new features to production as they are developed,
but include logic which keeps them deactivated until they are ready,
or until they have been verified in production with a subset of users.
Bucket tests is the practice of systematically testing new features or behavior for a controlled subset of users.
This is common practice when releasing new science models,
as they are difficult to verify in test, but can also be used for other features.
To test new behavior in Vespa, use a combination of
search chains and
rank profiles,
controlled by query profiles,
where one query profile corresponds to one bucket.
These features support inheritance to make it easy to express variation without repetition.
Sometimes a new feature requires
incompatible changes to a data field.
To be able to CD such changes, it is necessary to create a new field containing the new version of the data.
This costs extra resources but less than the alternative: standing up a new system copy with the new data.
New fields can be added and populated while the system is live.
One way to reduce the need for incompatible changes can be decreased
by making the semantics of the fields more precise.
E.g., if a field is defined as the "quality" of a document, where a higher number means higher quality,
a new algorithm which produces a different range and distribution will typically be an incompatible change.
However, if the field is defined more precisely as the average time spent on the document once it is clicked,
then a new algorithm which produces better estimates of this value will not be an incompatible change.
Using precise semantics also have the advantage of making it easier to understand
if the use of the data and its statistical properties are reasonable.