• [+] expand all

Profiling the Search Container

This document describes how you can configure the Container to allow for profiling custom searchers in order to identify performance bottlenecks - be it lock contention or CPU intensive algorithms. It also describes how to enable remote debugging using the IntelliJ IDEA.

Install YourKit profiler on the Container

Yourkit is a good and simple tool for finding hotspots in Java code. It supports both sampling and tracing. Very often it is necessary to use both modes. Tracing is accurate as to how many times a method is invoked and from where. That can be used to analyze if you are actually not computing the same thing from multiple places and overall doing more than you need. However it will hide effects of cache miss and especially cost of atomic operations and synchronization costs.

Assume there is an installation in a data center that you would like to profile, preferably with a nice UI running on your local desktop. All this is just a few steps away:

  • Install yourkit
  • Modify services.xml:
      <jvm options="-agentlib:yjpagent -Dvespa.freezedetector.disable=true" />
    Read more about jvm tuning. Disabling the freezedetector stops the container from shutting down during profiling.
  • Re-deploy the application:
    $ vespa-deploy prepare appdir && vespa-deploy activate
  • restart Vespa on the node that runs the Container
Browse $VESPA_HOME/logs/vespa.log for errors. You are now ready to perform profiling; you just need to install the UI on your desktop.

Install YourKit UI on the Desktop

The server is ready for profiling, now install the YourKit profiler on the desktop. Download the distribution that fits the OS you are running from YourKit. Follow the installation instructions, including setting the license server.

Note: By default the YourKit agent runs on port 10001. If Vespa is running on hosts not directly reachable from the desktop, setting up an SSH tunnel can work around:

$ ssh -L 1080:$hostname:10001 $hostname

Where $hostname is the node that is running the container with the YourKit agent profiler. All traffic to localhost (the desktop) port 1080 will be forwarded to the remote application running on port 10001.

Using Yourkit

You are now ready to profile your application. (You will need to put some realistic load against the container instance, please see the Vespa benchmarking guide) After having started the load simulation you can start the profiling session, open the YourKit application installed locally and chose Monitor Remote Applications => Connect to remote application. Enter localhost:1080 and press Connect. You should now see the profiling screen with Remote application "Server" (PID XXXXX) is being profiled at localhost:1080.

Remote debug with IntelliJ IDEA

To remote debug the container, first set up a remote debugging configuration in the IDEA:

  1. Run -> Edit Configurations...
  2. Click "+" to add a new configuration.
  3. Select the "Remote JVM Debug" option in the left-most pane.
  4. Set hostname to the host running the container, change the port if needed.

Next, set the container's jvm options to the value in "Command line arguments for remote JVM". Insert the value verbatim in services.xml:

<container id="default" version="1.0">
    <jvm options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" />

Re-deploy the application, then restart Vespa on the node that runs the container. Start debugging, check vespa.log for errors.