A Vespa system consists of one or more stateless and stateful clusters configured by an application package.
A Vespa system is configured and managed through an admin cluster as shown below.
All nodes of a Vespa system have the same software installed.
Which processes are started on each node and how they are configured
is determined by the admin cluster from the specification given in
services.xml in the application package.
Creating a multinode system from a sample application
To create a fully functional production ready multinode system from a single-node sample application,
follow these steps (also see next steps):
The following is a procedure to set up a multinode application on AWS EC2 instances.
Please run the procedure in
multinode-HA
first, to get familiar with the different Vespa concepts before running the AWS procedure below.
This procedure will use the name number of hosts, 10, and set up the same application.
Important:
Note the use of sudo.
The Vespa start scripts will modify the environment (directories, system limits), requiring root access -
refer to vespa-start-configserver
and vespa-start-services.
After the environment setup, Vespa is run as the vespa user.
Note:
The procedure below is a bare minimum, for educational purposes.
Make sure to use AWS instance types suitable for the application load,
and implement security mechanisms of choice.
Find AMI at CentOS AWS AMI Cloud Images -
this procedure is tested with CentOS Stream 8 us-east-1 x86_64 ami-0ee70e88eed976a1b
and vespa-8.30.50.
Use minimum t2.medium instances.
Let AWS create a security group for the nodes, or use an existing one.
Make sure to check for SSH traffic, for host login.
Launch 10 instances - the 3 first will be Vespa config server nodes, the 7 last Vespa nodes.
Write down private / public hostnames.
The private names are used in Vespa configuration, the public names for login to check status.
To find a hostname, click the instance
and copy hostname from Private IP DNS name (IPv4 only) and Public IPv4 DNS.
Create a table like:
type
Private IP DNS name (IPv4 only)
Public IPv4 DNS
configserver
ip-10-0-1-234.ec2.internal
ec2-3-231-33-190.compute-1.amazonaws.com
configserver
ip-10-0-1-154.ec2.internal
ec2-3-216-28-201.compute-1.amazonaws.com
configserver
ip-10-0-0-88.ec2.internal
ec2-34-230-33-42.compute-1.amazonaws.com
services
ip-10-0-1-95.ec2.internal
ec2-44-192-98-165.compute-1.amazonaws.com
services
ip-10-0-0-219.ec2.internal
ec2-3-88-143-47.compute-1.amazonaws.com
services
ip-10-0-0-28.ec2.internal
ec2-107-23-52-245.compute-1.amazonaws.com
services
ip-10-0-0-67.ec2.internal
ec2-54-198-251-100.compute-1.amazonaws.com
services
ip-10-0-1-84.ec2.internal
ec2-44-193-84-85.compute-1.amazonaws.com
services
ip-10-0-0-167.ec2.internal
ec2-54-224-15-163.compute-1.amazonaws.com
services
ip-10-0-1-41.ec2.internal
ec2-44-200-227-127.compute-1.amazonaws.com
Security group setup:
Click the Security Group for the nodes just provisioned (under the security tab),
then Edit inbound rules.
Add All TCP for port range 0-65535, specifying the name of the current Security Group as the Source.
This lets the hosts communicate with each other.
A successful config server start will log an entry like:
$ $VESPA_HOME/bin/vespa-logfmt | grep "Application config generation"
[2022-08-09 08:29:38.684] INFO : configserver
Container.com.yahoo.container.jdisc.ConfiguredApplication
Switching to the latest deployed set of configurations and components.
Application config generation: 0
{"log":[],"tenant":"default","url":"http://localhost:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default","message":"Session 2 for tenant 'default' prepared and activated.","configChangeActions":{"restart":[],"refeed":[],"reindex":[]}}
Vespa nodes setup
Start Vespa on the 7 hosts:
$ sudo systemctl start vespa
Validate the installation.
Use the
multinode-HA steps to check the health interfaces on all 10 nodes.
Note that in this guide, the ports are not mapped through a Docker container,
so the native Vespa ports should be used - e.g. for nodes 4 to 7 (see illustration below):
Remember to terminate the instances in the AWS console after use.
AWS ECS
The following is a procedure to set up a multinode application on
AWS ECS instances.
Please run the procedure in
multinode-HA
first, to get familiar with the different Vespa concepts before running the AWS procedure below.
This procedure will use the name number of host, 10, and set up the same application.
Running the EC2 procedure above can also be helpful,
this procedure has a similar structure.
Create a 10-node ECS cluster
Log in to AWS and the EC2 Container Service.
Click Clusters > Create Cluster > EC2 Linux + Networking > Next step, using the defaults and:
Cluster name
vespa
EC2 instance type
t2.medium
Number of instances
10
Key pair
Select or create your keypair
Security group inbound rules - port range
0 - 65535
Click Create, wait for the tasks to succeed, then View Cluster -
it should say Registered container instances: 10 in ACTIVE state.
Configure ECS instances
Click the ECS Instances tab - this should list 10 container instances.
Select the 3 first Container Instance checkboxes,
then Actions > View/Edit attributes.
Click Add attribute.
Set Name=type and Value=configserver,
click the green checkbox on the right, then Close.
Select the next 7 Container instance checkboxes,
then Actions > View/Edit attributes.
Click Add attribute.
Set Name=type and Value=services,
click the green checkbox on the right, then Close.
Write down private / public hostnames and create a table like in the EC2 procedure
The private names are used in Vespa configuration, the public names for login to check status.
To find a hostname, click ECS Instance > Instance ID
and copy hostname from Private IP DNS name (IPv4 only) and Public IPv4 DNS.
Start the config server task
Click Task Definitions > Create new Task Definition > EC2 > Next step.
Click Configure via JSON and replace the content with
(note the comma-separated hostnames of the config servers addresses):
Validate that the config servers started successfully -
use the same procedure as for EC2 instances,
checking /state/v1/health.
Do not continue before successfully validating this:
Click Save > Create.
Note the "command": [ "services" ].
See controlling which services to start
for details, this starts services only -
the start script starts both the configserver and services if given no arguments -
this is used for the config server above.
For these 7 nodes, services is given as an argument to the start script to only start Vespa services.
Choose Actions > Run task and configure:
Launch type
EC2
Cluster
vespa
Number of tasks
7
Placement templates
One Task Per Host
Click Run Task.
Validate startup.
This step is the same as for EC2 instances,
e.g. for nodes running a Vespa container the port is 8080:
Remember to delete the cluster in the AWS console after use.
Log collection
Logs are automatically collected from all nodes in real time to the admin node listed as adminserver.
To view log messages from the system,
run vespa-logfmt on this node.
Making changes to live systems
To change the system, deploy the changed application to the admin cluster.
The admin cluster will automatically change the participating nodes as necessary.
It is safe to do this while serving live query and write traffic.
In some cases the admin cluster will report that some processes must be restarted to make the change effective.
To avoid query or write traffic disruption,
such restarts must be done on one node at the time,
waiting until the node is fully up before restarting the next one.
Multiple proton processes
See multiple schemas
for an overview of how to map schemas to content clusters.
There is another way to distribute load over hosts,
by mapping multiple content clusters to the same hosts:
Observe that both clusters use node1.
This is a non-recommended configuration, as it runs multiple proton processes per node.
To reduce interference between the processes in this case, virtualize the host into more nodes.
One can use containers or VMs to do this:
Important:
Vespa's features for overload handling,
like feed-block,
requires that only one proton process is running on the node.
Autoscaling
A common question is, "Can AWS Auto Scaling be used?"
That is a difficult question to answer, here is a transcript from the Vespa Slack:
I have a question about deployment.
I set up cluster on two AWS auto-scaling groups (config & services) based on
multinode-systems.html#aws-ec2.
But if one of instances was replaced by auto-scaling group,
I need manually update hosts.xml file, zip it and deploy new version of the app.
I'm thinking about automation of this process by Cloudwatch & Lambda...
I wonder if there is some node-discovery mechanism which can e.g.
check instances tags and update hosts config based on it?
First, you see in aws-ec2 that there are two types of hosts,
configserver and services.
configserver setup / operations is documented at
configuration server operations.
This must be set up first.
This is backed by an Apache ZooKeeper cluster,
so should be 1 or 3 nodes large.
In our own clusters in Yahoo, we do not autoscale configserver clusters, there is no need - we use 3.
If that is too many, use 1. So this question is easy - do not autoscale configservers.
For the services nodes, observe that there are two kinds of nodes -
stateless containers and stateful content nodes -
see the overview.
In any way, you will want to manage these differently -
the stateless nodes are more easily replaced / increased / shrunk,
by changing services.xml and hosts.xml.
It is doable to build an autoscaling service for the stateless nodes,
but you need to make sure to use the right metrics for your autoscaling code,
and integrate the deploy-automation with the other deployments (say schema modifications).
A much harder problem is autoscaling the stateful nodes -
these are the nodes with the indexes and data. See elasticity -
adding a node + data redistribution can take hours,
and the node's load will increase during redistribution.
Building autoscaling here is very difficult to do safely and efficient.
Nothing of this is impossible,
and it is actually implemented at cloud.vespa.ai/autoscaling -
but it is a difficult feature to get right.
So, my recommendation is starting with a static set of hosts, like in
multinode-HA -
and in parallel try out cloud.vespa.ai/en/free-trial
with autoscaling experiments using your data and use cases.
Autoscaling can save money, but before going there,
it is wise to read docs.vespa.ai/en/performance/
and optimize resources using a static node set
(or use the sizing suggestions from the Vespa Cloud Console).
I.e., get the node resources right first,
then consider if autoscaling node count for your load patterns makes sense.
Next steps
Multinode-HA
is a high-availability multi-node template - use this as a basis for the final configuration.
The multinode
sample application is a useful for experimenting with node state transitions.
