# Profiling

Guidelines when profiling:

• Define clearly what to profile.
• Find a load that represents what to profile. This is often the hardest part, as there is a lot of noise if stressing other components.
• Make sure that there are no other bottlenecks that blocks stressing the profiled component. It makes little sense to do cpu profiling if the network is the limitation.
• If possible, write special unit-tests like benchmark programs that stress exactly what to profile.
• If the system is multithreaded:
• Always profile single threaded first - that gives a baseline for doing the scaling tests. Verify one is utilizing as many cores as expected.
• Increase scaling gradually to at least 2x numcores or until throughput degrades.

Also see Using valgrind with Vespa.

## CPU profiling

### Set up

It is useful to have kernel debug info installed to get symbol info for the Linux kernel:

$sudo yum install kernel-debuginfo Its important to get somewhat same version of kernel-debuginfo as the kernel package. Then setup OProfile to point to the kernel debuginfo file and ensure that one get per-application profiles:$ sudo opcontrol --separate=kernel --vmlinux=$(rpm -ql kernel-debuginfo | grep vmlinux) OProfile is typically run like:$ sudo opcontrol --start
$sudo opcontrol --reset$ # put some load on the system
$sudo opcontrol --shutdown ### Profile java applications To profile java applications, start the java application with -agentlib:jvmti_oprofile. For Vespa, set jvmargs or set the services.agentlib variable to jvmti_oprofile - refer to setting vespa variables. jvmti_oprofile is installed when installing oprofile-jit with yum. Then restart services. ### Troubleshooting If there is no output from opreport -l, it is probably because the kernel is to old for the CPU. OProfile will not be able to do to event profiling. To fix, force OProfile to use the timer mode:$ sudo opcontrol --shutdown
$sudo opcontrol --deinit$ sudo modprobe oprofile timer=1

### Analysis

$opreport -l # gives whole system - all one needs to know.$ opreport -l "path of binary" #  i.e. "*vespa-proton-bin" -> gives a view of the binary only.

For java applications, its important to let the application run for a while to allow the JIT compiler to do its magic. Then run opcontrol --dump before inspecting OProfile reports.