Testing with Java JUnit tests

This is the Vespa Testing reference for Vespa application system tests written in Java, as JUnit 5 unit test.

These tests verify the behaviour of a Vespa application by using its HTTP interfaces. To write tests without Java, see basic HTTP test reference.

See the testing guide for examples of how to run the tests.

Test suites

The testing documentation defines three test scenarios, comprised of four test code categories. The system test framework in com.yahoo.vespa:tenant-cd-api uses Java annotations to declare what category a JUnit test class belongs to. To run tests with Maven belonging to a specific category, a JUnit 5 test tag must be specified: 

$ mvn test -D test.categories=system -D vespa.test.config=/path-to/test-config.json
Category Annotation JUnit tag Description
System test @SystemTest system Independent, functional tests
Staging setup @StagingSetup staging-setup Set state before upgrade
Staging test @StagingTest staging Verify state after upgrade
Production test @ProductionTest production Verify domain specific metrics (deprecated, see below)

For an example including system and staging tests, check out the sample application test suite.

New applications should not write production tests in Java: use the declarative production test format, based on YAML or JSON, instead. The @ProductionTest category above is kept only for backward compatibility with applications that already use it — if a test package includes both a declarative production test and a Java @ProductionTest bundle, the declarative test is used, and the Java one is ignored.

TestNG

Combining Vespa JUnit 5 test suites with unit tests in TestNG is possible. You'll need to explicitly configure Maven's surefire plugin to enable integration for both frameworks. To execute the Vespa test suites specify -D test.categories=[tag], where [tag] is one of the values listed in Test suites. 

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit-platform</artifactId>
            <version>${surefire.vespa.tenant.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-testng</artifactId>
            <version>${surefire.vespa.tenant.version}</version>
        </dependency>
    </dependencies>
</plugin>