Vespa provides metrics integration with CloudWatch, Datadog and Prometheus / Grafana, as well as a JSON HTTP API.
There are two main approaches to transfer metrics to an external system:
All pull-based solutions use Vespa's metrics API, which provides metrics in JSON format, either for the full system or for a single node. The polling frequency should be limited to max once every 30 seconds as more frequent polling would not give increased granularity but only lead to unnecessary load on your systems.
CloudWatch | Metrics can be pulled into CloudWatch from both Vespa Cloud and self-hosted Vespa. The recommended solution is to use an AWS lambda function, as described in Pulling Vespa metrics to Cloudwatch. |
---|---|
Datadog | Note: This method currently works for self-hosted Vespa only. The Vespa team has created a Datadog Agent integration to allow real-time monitoring of Vespa in Datadog. The Datadog Vespa integration is not packaged with the agent, but is included in Datadog's integrations-extras repository. Clone it and follow the steps in the README. |
Prometheus |
Vespa exposes metrics in a text based
format that can be
scraped by Prometheus.
For Vespa Cloud, append /prometheus/v1/values
to your endpoint URL. For self-hosted Vespa the URL is:
http://<container-host>:<port>/prometheus/v1/values, where
the port is the same as for searching, e.g. 8080. Metrics for each individual host
can also be retrieved at See the quick-start for a Prometheus / Grafana example. |
Note: This method currently works for self-hosted Vespa only.
This is presumably the most convenient way to monitor Vespa in CloudWatch. Steps / requirements:
<metrics> <consumer id="my-cloudwatch"> <metric-set id="default" /> <cloudwatch region="us-east-1" namespace="my-vespa-metrics"> <shared-credentials file="/path/to/credentials-file" /> </cloudwatch> </consumer> </metrics>This configuration sends the default set of Vespa metrics to the CloudWatch namespace
my-vespa-metrics
in the us-east-1
region.
Refer to the
metric list
for default
metric set.
Follow these steps to set up monitoring with Grafana for a Vespa instance. This guide builds on the quick start by adding three more Docker containers and connecting these in the Docker monitoring network:
Run the Quick Start:
Complete steps 1-7 (or 1-10), but skip the removal step. Clone repository:
$ git clone --depth 1 https://github.com/vespa-engine/sample-apps.git && \ cd sample-apps/examples/operations/album-recommendation-monitoring
Create a network and add the vespa container to it:
$ docker network create --driver bridge monitoring && \ docker network connect monitoring vespa
This creates the monitoring network and attaches the vespa container to it. Find details in docker-compose.yml.
Launch Prometheus:
$ docker run --detach --name sample-apps-prometheus --hostname prometheus \ --network monitoring \ --publish 9090:9090 \ --volume `pwd`/prometheus/prometheus-selfhosted.yml:/etc/prometheus/prometheus.yml \ prom/prometheus
Prometheus is a time-series database, which holds a series of values associated with a timestamp. Open Prometheus at http://localhost:9090/. One can easily find what data Prometheus has, the input box auto-completes, e.g. enter feed_operations_rate and click Execute. Also explore the Status dropdown.
Launch Grafana:
$ docker run --detach --name sample-apps-grafana \ --network monitoring \ --publish 3000:3000 \ --volume `pwd`/grafana/provisioning:/etc/grafana/provisioning \ grafana/grafana
This launches Grafana. Grafana is a visualisation tool that can be used to easily make representations of important metrics surrounding Vespa. Open http://localhost:3000/ and find the Grafana login screen - log in with admin/admin (skip changing password). From the list on the left, click Browse under Dashboards (the symbol with 4 blocks), then click the Vespa Detailed Monitoring Dashboard. The dashboard displays detailed Vespa metrics - empty for now.
Build the Random Data Feeder:
$ docker build album-recommendation-random-data --tag random-data-feeder
This builds the Random Data Feeder - it generates random sets of data and puts them into the Vespa instance. Also, it repeatedly runs queries, for Grafana visualisation. Compiling the Random Data Feeder takes a few minutes.
Run the Random Data Feeder:
$ docker run --detach --name sample-apps-random-data-feeder \ --network monitoring \ random-data-feeder
Check the updated Grafana metrics:
Graphs will now show up in Grafana and Prometheus - it might take a minute or two. The Grafana dashboard is fully customisable. Change the default modes of Grafana and Prometheus by editing the configuration files in album-recommendation-monitoring.
Remove containers and network:
$ docker rm -f vespa \ sample-apps-grafana \ sample-apps-prometheus
$ docker network rm monitoring