Basic HTTP testing

This is the Vespa Testing reference for basic HTTP tests, used to write Vespa application system tests.

These tests verify the behaviour of a Vespa application by using its HTTP interfaces. Basic HTTP tests are written in JSON; to write more advanced tests, see the Java testing reference.

See the testing guide for examples of how to run the tests.

Test suites

The testing documentation defines three test scenarios, comprised of four test code categories. For basic HTTP tests, the category of a test is defined by its placement in the application tests directory:

Each test is described by a JSON file, and may include other files using relative paths:

$ ls -1 tests/*/*

tests/production-test/metrics-test.json
tests/staging-setup/set-up-old-documents.json
tests/staging-test/verify-search-still-works.json
tests/system-test/data/document.json
tests/system-test/feed-and-search-test.json
tests/system-test/ranking-test.json

For an example with actual system and staging tests, check out a Vespa Cloud sample test suite. Since production tests are highly application specific, this suite has none, but such a test could be:

{
    "steps": [
        {
            "request": {
                "uri": "https://my.external.service/metrics/?query=customer-engagement"
            }
        }
    ]
}

Test file structure

Each .json file directly under any of the directories listed above describes one test. Each test consists of a series of steps, and each step specifies an HTTP request to run, and some assertions about the response to obtain. Some additional properties may also be specified on both the test and step levels. A full example, with // comments:

{
    "name": "my test",
    "defaults": {
        "cluster": "default",
        "parameters": {
            "timeout": "1.618s"
        }
    },
    "steps": [
        {
            "name": "clear existing documents",
            "request": {
                "method": "DELETE",
                "uri": "/document/v1/",
                "parameters": {
                    "cluster": "music",
                    "selection": "true"
                }
            }
        },
        {
            "name": "feed foo",
            "request": {
                "method": "POST",
                // should contain payload as expected by /document/v1/
                "body": "foo/body.json",
                // specify only the path and query for Vespa requests
                "uri": "/document/v1/test/music/docid/foo?timeout=8s",
                // JSON object file; merged with query from "uri"
                "parameters": "foo/parameters.json"
            }
            // no response spec: just assert code 200
        },
        {
            "name": "query for foo",
            "request": {
                // no "uri": defaults to "/search/"
                "parameters": {
                    "query": "artist: foo"
                }
            },
            "response": {
                "body": {
                    "root": {
                        "children": [
                            // assert "children" has a single element ...
                            {
                                // ... which has the field "fields" ...
                                "fields": {
                                    // ... where the field "artist" is "Foo Fighters" ...
                                    "artist": "Foo Fighters"
                                },
                                // ... and the field "relevance" close to 0.381862383599
                                "relevance": 0.381862383599
                            }
                        ]
                    }
                }
            }
        }
    ]
}

Test JSON specification

A full list of fields, with description:

JSON matching

All requests and responses must be in JSON format. The tests allow simple JSON verification, by describing what should be present in the actual response. This is done by specifying a JSON structure, a template, for each response, and requiring each field present in the template to match fields in the actual response. Unmatched fields result in test failure, with the following rules:

• Objects must contain all listed fields, and may also contain unlisted ones.
• Arrays must match element-by-element.
• Numbers must match within precision 1e-9.
• All other values must match exactly.

Note that the empty object { } matches any other object, and can be used to fill elements of an array that require no further validation.