Vespa Scaling Configuration Examples

This document has a set of example configurations for content clusters using flat or grouped data distribution. Data is distributed over nodes and groups using a Vespa's distribution algorithm. See Scaling Vespa for when to use grouped or flat data distribution. These examples illustrates common deployment patterns. In all examples, the number of stateless container nodes is fixed. The examples are services.xml deployed using Application Packages. See services.xml - 'content' element reference documentation.

Flat Distribution

Flat (single group) distribution with redundancy 3 and searchable-copies 1. Data is distributed and partitioned over 9 nodes and there are 3 copies of each document stored on 3 different nodes. Queries are dispatched in parallel to all nodes, and with searchable-copies 1, only one of the 3 copies of a document is indexed and active. This means less resource usage (memory, disk, and cpu). In case of a node failure, the remaining nodes will index (make ready) and activate the not ready (stored) copies to restore full search coverage.

<services version="1.0">
  <container id="stateless-container-cluster" version="1.0">  
    <search/>
    <document-api/>
    <nodes>
      <node hostalias="container0"/>
      <node hostalias="container1"/>
      <node hostalias="container2"/>
    </nodes>
  </container>

  <content id="my-content" version="1.0">
    <documents>
      <document type="my-document" mode="index">
    </documents>
    <redundancy>3</redundancy>
    <engine>
      <proton>
        <searchable-copies>1</searchable-copies>
      </proton>
     </engine>
    <nodes>
      <node hostalias="searcher0" distribution-key="0"/>
      <node hostalias="searcher1" distribution-key="1"/>
      <node hostalias="searcher2" distribution-key="2"/>
      <node hostalias="searcher3" distribution-key="3"/>
      <node hostalias="searcher4" distribution-key="4"/>
      <node hostalias="searcher5" distribution-key="5"/>
      <node hostalias="searcher6" distribution-key="6"/>
      <node hostalias="searcher7" distribution-key="7"/>
      <node hostalias="searcher8" distribution-key="8"/>
    </nodes>
  </content>
</services>

Grouped Distribution

When using grouped distribution in an indexed content cluster, the following restrictions apply:

  • There can only be a single level of leaf groups under the top group
  • Each leaf group must have the same number of nodes
  • The number of leaf groups must be a factor of the redundancy
  • The distribution partitions must be specified such that the redundancy per group is equal
  • The number of searchable-copies must be less than, or equal to, redundancy
  • The number of leaf groups must be a factor of searchable-copies
With a low number of nodes per group, it's important to remember that a node failure will cause the data to be re-distributed to the remaining nodes and their memory footprint and disk usage will grow when those nodes starts activating the documents originally activated on the failed node. E.g. with 2 nodes per group, the remaining healthy node will start activating all the content, which will cause a 2x memory and disk footprint compared with the ideal state.

The min-node-ratio-per-group controls the data distribution behavior inside a group in cases of node failures. This sets a lower bound on the ratio of nodes within groups that must be online and accepting feed and query traffic, before the entire group is automatically taken out of service from both feed and search/serving. Once number of nodes in the group have been restored, and ideal state has been achieved, the group will be automatically set in service.

9 nodes, 3 groups with 3 nodes per group

This example has 3 groups and each group index all of the documents over the 3 nodes in the group. With 3 groups there are 3 replicas in total of each document, and each replica is indexed and active. Losing a node does not reduce search coverage.

<services version="1.0">
  <container id="stateless-container-cluster" version="1.0">  
    <search/>
    <document-api/>
    <nodes>
      <node hostalias="container0"/>
      <node hostalias="container1"/>
      <node hostalias="container2"/>
    </nodes>
  </container>

  <content id="my-content" version="1.0">
    <documents>
      <document type="my-document" mode="index">
    </documents>
    <redundancy>3</redundancy>
     <engine>
      <proton>
        <searchable-copies>3</searchable-copies>
      </proton>
     </engine>
    <group name="top-group">
      <distribution partitions="*|*|*"/>
      <group name="group0" distribution-key="0">
        <node hostalias="searcher1" distribution-key="0"/>
        <node hostalias="searcher2" distribution-key="1"/>
        <node hostalias="searcher3" distribution-key="2"/>
      </group>
      <group name="group1" distribution-key="1">
        <node hostalias="searcher4" distribution-key="3"/>
        <node hostalias="searcher5" distribution-key="4"/>
        <node hostalias="searcher6" distribution-key="5"/>
      </group>
      <group name="group3" distribution-key="2">
        <node hostalias="searcher7" distribution-key="6"/>
        <node hostalias="searcher8" distribution-key="7"/>
        <node hostalias="searcher9" distribution-key="8"/>
      </group>
    </group>
  </content>
</services>

9 nodes, 9 groups with 1 node per group

This example has 9 groups and each group index all of the documents on a single node. With 9 groups there are 9 replicas in total of each document, and each replica is indexed and active. Losing a node does not reduce search coverage. With a single node, indexing throughput is limited by the single node performance, as all data needs to go all nodes.

<services version="1.0">
  <container id="stateless-container-cluster" version="1.0">  
    <search/>
    <document-api/>
    <nodes>
      <node hostalias="container0"/>
      <node hostalias="container1"/>
      <node hostalias="container2"/>
    </nodes>
  </container>

  <content id="my-content" version="1.0">
    <documents>
      <document type="my-document" mode="index">
    </documents>
    <redundancy>9</redundancy>
     <engine>
      <proton>
        <searchable-copies>9</searchable-copies>
      </proton>
     </engine>
    <group name="top-group">
      <distribution partitions="*|*|*|*|*|*|*|*|*"/>
      <group name="group0" distribution-key="0">
        <node hostalias="searcher1" distribution-key="0"/>
      </group>
      <group name="group1" distribution-key="1">
        <node hostalias="searcher2" distribution-key="1"/>
      </group>
      <group name="group2" distribution-key="2">
        <node hostalias="searcher3" distribution-key="2"/>
      </group>
      <group name="group3" distribution-key="3">
        <node hostalias="searcher4" distribution-key="3"/>
      </group>
      <group name="group4" distribution-key="4">
        <node hostalias="searcher5" distribution-key="4"/>
      </group>
      <group name="group5" distribution-key="5">
        <node hostalias="searcher6" distribution-key="5"/>
      </group>
      <group name="group6" distribution-key="6">
        <node hostalias="searcher7" distribution-key="6"/>
      </group>
      <group name="group7" distribution-key="7">
        <node hostalias="searcher8" distribution-key="7"/>
      </group>
      <group name="group8" distribution-key="8">
        <node hostalias="searcher9" distribution-key="8"/>
      </group>
    </group>
  </content>
</services>

Serving Availability Tuning

When using flat distribution, soft failing nodes is a challenge for serving with high availability and low latency. Soft failing nodes are nodes which answers health checks from cluster controllers and search container dispatch health checks, but still experiences issues which impacts serving latency (e.g. cpu frequency throttling due to thermal heating, memory corruptions and so forth). In a cluster with a flat distribution, the slowest node determines the latency, as the query request is dispatched to all content nodes in parallel. The probability of a soft failing node increases with the number of nodes used to distribute the data over.

Use adaptive coverage timeout to prevent slow soft failing nodes to impact availability. This allows the dispatcher to stop waiting for the slowest node(s). See also graceful search degradation.

Grouped distributions are less impacted by soft failing nodes in general, as queries are dispatched to one group at a time using a dispatch policy. The adaptive policy takes group latency into account when deciding which group the query request should be routed to.

Changing Group Configuration

Migrating from flat distribution to a hierarchy will temporarily reduce query coverage (likewise for changing from grouped config to another). When adding new group(s), it will take some time to populate the new nodes. During this transition period, queries hitting these nodes will return partial results. Reference documentation.

A possible workaround for some cases is increasing redundancy and let replicas re-distribute, before changing the group configuration.