Developing Cloud Config Model Plugins

The Cloud Config System (CCS) provides the ability to write custom plugins to the config model. This allows cloud config consumers to provide a custom syntax for their users to configure the system.

Why Create a Plugin

While creating a config model plugin is not strictly necessary for using CCS, it increases the usability of your platform. With a plugin, you can provide custom syntax that allows the users to configure the system at a higher level of abstraction.

What problem does it solve

Imagine that you are developing a platform that requires the user to configure a cluster of servers running some service that can potentially scale to hundreds of servers. A service may be composed of multiple processes running on the same server, and those processes might be containers running many components inside them. The amount of configuration parameters of those processes and components can be huge. Very often, the differences in the configurations might be small, but just enough that you need to duplicate information that you later on need to ensure that does not conflict.

Once you have this large amount of configuration parameters, the logical step is to create some sort of script that automatically generates a valid configuration based on a few parameters to reduce the amount of human errors. It is also common to create some form of validation program that can be run to ensure that the configuration options are valid and does not conflict. CCS model plugins allows you to take this even further.

In CCS, the configuration is represented as an application package, which contains various files that can represent the entire system configuration. However, the configuration is not directly given to the consumer processes and their components. First, the application package is provided as input to the config server, which builds a java object model of the system, also known as the config model. The config model is built from a set of model plugins that are able to produce a java object model based on the application package contents. The object model can then serve the appropriate configuration to a component within a service.

VespaAssemblySmaller.png

Benefits of writing plugins

The model plugins allows you to create abstractions on top of the low level configuration parameters that are given to the services. Here are some specific use cases the model plugins solve:

  • The application can be validated to ensure that there are no syntax errors in the configuration files.
  • The configuration files can be checked for conflicting configuration values
  • The plugin can allocate resources without the user needing to specify them (typically server ports)
  • The plugin may support any form of syntax as long as it can generate the object model, though using XML allows you to take advantage of many existing library features that i.e. make node and port allocation automatic or to allow the service to be automatically bootstrapped.
  • The plugin may provide a high level of abstraction that allows the user to change one configuration value, while in reality more than one of the configuration values served to the services may change.

Example

Vespa uses CCS with multiple config model plugins. In Vespa, you can set up a complete system with e.g this configuration:

<services version="1.0">

  <admin version="2.0">
    <adminserver hostalias="node1"/>
  </admin>

  <container id="stateless" version="1.0">
    <nodes>
      <node hostalias="node2"/>
    </nodes>
    <search/>
  </container>

  <content id="mycontent" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="myschema" mode="index"/>
    </documents>
    <nodes>
      <node distribution-key='0' hostalias='node3'/>
    </nodes>
  </content>

</services>
The Vespa model plugins ensure that the following is set up based on the above XML:

  • A config server cluster
  • A log server cluster receiving logs from all the nodes
  • A service location broker cluster (vespa-slobrok) which maintains the current observed state of all the nodes in the other clusters
  • A document processing cluster processing incoming data
  • A JDisc cluster set up to receive searches
  • A dispatch cluster doing scatter-gather
  • A cluster controller cluster which takes singular decisions about the current state of nodes in the content cluster
  • A distributor cluster managing content distribution
  • A search+content cluster storing and searching a partition

In addition it sets up the wiring between these clusters, the matching configuration of content and query processing etc. etc. , all told a number of instances of over hundred configuration types.

Going from the simple user-facing config shown above to this complete system specification is the task of the config model(s).

Technically the models are just Java objects that are instantiated in response to hitting a tag immediately below <services> in services.xml (so the above will create one admin, container and content model), which are embedded into the config server at runtime and answers incoming requests for config instances. During construction of the complete system model (consisting of multiple such config models), the models may create additional implicit config models and exchange information between themselves. They may also read other files of any format from the application package.

Building a Model

In this section, we will build a plugin for a simple echo service. First, we must have a config to serve. An echo server just needs to know which port it must listen to. To keep the initial example simple, we ignore bootstrap and which host the node should run on for now (see bootstrapping services for an extended example). The following config is created and stored in src/main/resources/configdefinitions in your plugin source tree:

namespace=echo
port int
See Using the Cloud config API and the Configuration File Reference for more information on config definition files. The service will be configured using:
<services version="1.0">
  <echo version="1.0">
    <port>1337</port>
  </echo>
</services>
To deal with your custom syntax, you first need to create a parser. The parser class must extend ConfigModelBuilder. The builder is required to specify which XML tags it can handle via the handlesElements method, and must hook the model into a subclass of ConfigModel object in the doBuild method. Its constructor must forward the class of the model in order for the superclass to instantiate the correct class and pass it to doBuild:
public class EchoModelBuilder extends ConfigModelBuilder<EchoModel> {
    public EchoModelBuilder() {
        super(EchoModel.class);
    }
    @Override
    public List<ConfigModelId> handlesElements() {
        return Arrays.asList(ConfigModelId.fromName("echo"));
    }
    @Override
    public void doBuild(EchoModel configModel, Element spec, ConfigModelContext modelContext) {
        int port = Integer.parseInt(XML.getValue(XML.getChild(spec, "port")));
        configModel.addServer(new EchoServer(configModel.getConfigProducer(), port));
    }
}
The config model represents the model that serves config when requested. The ConfigModel object does not itself serve config, but is used to proxy request to the appropriate config producers for that model. For the echo service, the model is very simple:
public class EchoModel extends ConfigModel {
    private final List<EchoServer> servers = new ArrayList<EchoServer>();
    public EchoModel(ConfigModelContext modelContext) {
        super(modelContext);
    }
    public void addServer(EchoServer server) {
        servers.add(server);
    }
    public int numServers() {
        return servers.size();
    }
}
The EchoServer is the actual object producing the config, and is a subclass of the AbstractConfigProducer class, which automatically hooks it into the parent node in the tree, and causes config requests to be relayed to this producer:
public class EchoServer extends AbstractConfigProducer implements EchoConfig.Producer {
    private final int port;
    public EchoServer(AbstractConfigProducer parent, int port) {
        super(parent, "server");
        this.port = port;
    }
    public void getConfig(EchoConfig.Builder builder) {
        builder.port(port);
    }
}
The EchoConfig class is the one generated from the config definition file we specified earlier.

Now we have a complete plugin that is able to serve the echo config to anyone who asks with the appropriate config id. The config id of a config producer is relative to its parent. In this example, the root producers has the id echo, while the server has the id echo/server.

Depending on other models

To create a modular system and facilitate code reuse, it is possible to depend on and use other models when building your own model. For instance, if we have a EchoProxyModel that handled a <echoproxy> tag, and which is dependent on the EchoModel being built first, we can add it as a constructor argument to signal the dependency and also allow us to access the EchoModel when building the EchoProxyModel.

public class EchoProxyModel extends ConfigModel {
    public EchoProxyModel(ConfigModelContext context, EchoModel echoModel) {
        // Store away model for use in builder.
    }
}

Unit Testing

Creating a unit test for your plugin is easy. Mock classes acting as the config model is available can can be used in your unit tests for top level testing:

public class EchoModelTest {
    @Test
    public void testEchoModel() {
        EchoModelBuilder builder = new EchoModelBuilder();
        assertThat(builder.handlesElements().size(), is(1));
        assertThat(builder.handlesElements().get(0).getName(), is("echo"));
        String xml = "<echo version="1.0">"
                   + "<port>1337</port>"
                   + "</echo>"
        TestDriver testDriver = new TestDriver().addBuilder(builder);
        TestRoot root = testDriver.buildModel(xml);
        EchoConfig config = root.getConfig(EchoConfig.class, "server");
        assertThat(config.port(), is(1337));

        EchoModel model = testDriver.getConfigModels(EchoModel.class).get(0);
        assertThat(model.numServers(), is(1));
    }
}
The ConfigModelTester is a helper class designed to make unit testing config model builders a breeze. Any builders required to parse your xml is added to the tester using the addBuilder method. To build the entire model, with all dependencies resolved, call buildModel with either services.xml or an ApplicationPackage as input. The result is a TestRoot object which can be used to inspect the entire model. The config retrieved from the TestRoot is the same as you would get from the config server in a production system.

The MockRoot class mocks the config model producer tree, and can be used to retrieve the config once the producers have been added to the tree. Before using it, however, the topology must be frozen by calling freezeModelTopology.

It is worth noting that on a running system, you can use the vespa-get-config tool to retrieve in the payload format as well:

$ vespa-get-config -n echo.echo -i echo/server -a path/to/echo.def

Bootstrapping Services

To ease the bootstrapping of services that must be run, we provide helper classes that help defining which hosts a service should run on, and which performs automatic start/stop of those services based on the config. We will now extend the echo service to support specifying clusters of echo servers that are automatically bootstrapped and run on multiple nodes.

In CCS, the term node is used about any machine capable of running the services. It can be an alias for physical host, a VM or (in the future) simply a set of parameters describing the resource demands, and CCS will simply acquire a node for you. In this example, we will use the traditional host alias specification.

First, the XML syntax must be changed in order to support the new requirements and for the helper classes to recognize them:

<services version="1.0">
  <echo version="1.0">
    <server hostalias="node0" port="1234" />
    <server hostalias="node0" port="1235" />
    <server hostalias="node1" port="1234" />
  </echo>
</services>
This syntax basically says "Run two echoservers on node0 using port 1337 and 1338 respectively. Run one echoserver on node1 using port 1337".

Now we need to change the plugin (only shows methods that we have changed):

public class EchoModelBuilder extends ConfigModelBuilder<EchoModel> {
    …
    @Override
    public void doBuild(EchoModel configModel, Element spec, ConfigModelContext modelContext) {
        configModel.setCluster(new EchoServerClusterBuilder().build(configModel.getConfigProducer(), spec));
    }
    …
}
public class EchoModel extends ConfigModel {
    private EchoServerCluster cluster;
    public EchoModel(ConfigModelContext modelContext) {
        super(modelContext);
    }
    public void setCluster(EchoServerCluster cluster) {
        this.cluster = cluster;
    }
}
public class EchoServerClusterBuilder extends DomConfigProducerBuilder<EchoServerCluster> {
    @Override
    protected EchoServerCluster doBuild(AbstractConfigProducer ancestor, Element producerSpec) {
        final Element repeatElement = XML.getChild(producerSpec, "repeat");
        final int repeat = repeatElement != null ? Integer.parseInt(XML.getValue(repeatElement)) : -1;

        final EchoServerCluster cluster = new EchoServerCluster(ancestor, "echocluster", repeat);
        for (Element server : XML.getChildren(producerSpec, "server")) {
            EchoServerBuilder builder = new EchoServerBuilder(cluster.numServers());
            cluster.addServer(builder.build(cluster, server));
        }
        return cluster;
    }
}
public class EchoServerCluster extends AbstractConfigProducer implements EchoConfig.Producer {
    private final List<EchoServer> echoServers = new ArrayList<EchoServer>();
    private final int repeat;
    public EchoServerCluster(AbstractConfigProducer parent, String subId, int repeat) {
        super(parent, subId);
        this.repeat = repeat;
    }
    public void addServer(EchoServer server) {
        echoServers.add(server);
    }
    public int numServers() {
        return echoServers.size();
    }
    @Override
    public void getConfig(EchoConfig.Builder builder) {
        if (repeat >= 0)
            builder.repeat(repeat);
    }
}
public class EchoServerBuilder extends DomConfigProducerBuilder<EchoServer> {
    private final int serverNumber;
    public EchoServerBuilder(int serverNumber) {
        this.serverNumber = serverNumber;
    }
    @Override
    protected EchoServer doBuild(AbstractConfigProducer parent, Element element) {
        int port = Integer.valueOf(element.getAttribute("port"));
        return new EchoServer(parent, "server." + serverNumber, port);
    }
}
public class EchoServer extends AbstractService implements EchoConfig.Producer {
    private final int port;
    public EchoServer(AbstractConfigProducer parent, String name, int port) {
        super(parent, name);
        this.port = port;
    }
    @Override
    public String getStartupCommand() {
        return "/mydir/bin/echoserver";
    }
    public void getConfig(EchoConfig.Builder builder) {
        builder.port(port);
    }
}

By using the helper classes, there is a minimal amount of extra code, but the gain is huge. Using CCS, all the echoservers will automatically start when Vespa is run on their respective nodes, and they will get their appropriate config id passed as an environment variable. See Using the Cloud config API for examples of how the echo server can subscribe and use the config.

In addition to host management, the DomConfigProducerBuilder class supports parsing config overrides as well.

Installing model plugins

To make use of the model plugins, they must be installed on the configserver host before the config server is started. The default folder for model plugins is $VESPA_HOME/lib/jars/config-models.

Having installed the model plugin, the configserver needs to be instructed to load the plugin. To load our example EchoModelBuilder, simply add the following to $VESPA_HOME/conf/configserver-app/config-models/echomodel.xml

<components>
  <component id="<package>.EchoModelBuilder" bundle="<bundlename>" />
</components>