File distribution

An application package might have components and other large files that should be distributed to the Vespa nodes. Vespa can distribute these files efficiently to the nodes at the time of deploying an application (when doing vespa-deploy prepare).

Files

Which files are distributed?

Distributed files are put in $VESPA_HOME/var/db/vespa/filedistribution on the nodes

When new components or files specified in config are distributed, the container gets a new file reference, waits for it to be available and switches to new config when all files are available.

Troubleshooting

vespa-status-filedistribution can be run to check if files have been distributed to all the hosts.

The following scenario has occasionally been overserved: Clients nodes (such as containers) request components from file distribution at reconfiguration, but the files are never delivered, such that a timeout occur and the node fail to reconfigure. Log messages starting by Could not install bundle followed by timeout information is found in the logs. This can be caused by too many file references (many historical versions of files) in the file distribution subsystem which for some reason have not been cleaned up - clean up manually:

$ zkctl rmr /vespa/filedistribution
$ rm $VESPA_HOME/var/db/vespa/filedistribution/*
The first command can be run on any config server node while config servers are running. The second should be run on all config server nodes.