Bundle troubleshooting

edit page

This document describes how to troubleshoot the most common errors when working with bundles:

• Could not create component
• Class not found
• Slow Container start
• Unresolved constraint
• Multiple implementations of the same class

For general information on OSGi bundles for Vespa, refer to the introduction on how to develop OSGi bundles. Richard S. Hall's presentation on using OSGi as a means to achieve modularity is recommended - it explains a lot about how OSGi works, and why.

Could not create component

The Container fails to start if it cannot load bundles. Example, using wrong bundle name in the multiple-bundles sample app:

 <services version="1.0">
     <container id="fibonacci" version="1.0">
-        <component id="com.mydomain.lib.FibonacciProducer" bundle="multiple-bundles-lib" />
+        <component id="com.mydomain.lib.FibonacciProducer" bundle="multiple-bundles-typo" />

Looking at what is actually deployed in multiple-bundles:

$ ls -1 target/*.jar
  target/multiple-bundles-1.0.0-deploy.jar
  target/multiple-bundles-1.0.0-without-dependencies.jar
  target/multiple-bundles-lib-1.0.1-deploy.jar

Error in log:

[2020-01-23 14:28:01.367] WARNING : qrserver         Container.com.yahoo.container.di.Container
	Failed to set up new component graph. Retaining previous component generation.
	exception=
	java.lang.IllegalArgumentException: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'.
        Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo.
        If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
	at com.yahoo.osgi.OsgiImpl.resolveFromClassPath(OsgiImpl.java:74)
	at com.yahoo.osgi.OsgiImpl.resolveClass(OsgiImpl.java:65)
	at com.yahoo.container.di.Container.addNodes(Container.java:228)
	at com.yahoo.container.di.Container.createComponentsGraph(Container.java:217)
	at com.yahoo.container.di.Container.getConfigAndCreateGraph(Container.java:160)
	at com.yahoo.container.di.Container.getNewComponentGraph(Container.java:84)
	at com.yahoo.container.core.config.HandlersConfigurerDi.getNewComponentGraph(HandlersConfigurerDi.java:145)
	at com.yahoo.container.jdisc.ConfiguredApplication.lambda$startReconfigurerThread$1(ConfiguredApplication.java:275)
	at java.base/java.lang.Thread.run(Thread.java:834)
	
[2020-01-23 14:28:01.367] ERROR   : qrserver         Container.com.yahoo.container.jdisc.ConfiguredApplication
	Reconfiguration failed, your application package must be fixed, unless this is a JNI reload issue: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'. Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo. If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
	exception=
	java.lang.IllegalArgumentException: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'. Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo. If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
	at com.yahoo.osgi.OsgiImpl.resolveFromClassPath(OsgiImpl.java:74)
	at com.yahoo.osgi.OsgiImpl.resolveClass(OsgiImpl.java:65)
	at com.yahoo.container.di.Container.addNodes(Container.java:228)
	at com.yahoo.container.di.Container.createComponentsGraph(Container.java:217)
	at com.yahoo.container.di.Container.getConfigAndCreateGraph(Container.java:160)
	at com.yahoo.container.di.Container.getNewComponentGraph(Container.java:84)
	at com.yahoo.container.core.config.HandlersConfigurerDi.getNewComponentGraph(HandlersConfigurerDi.java:145)
	at com.yahoo.container.jdisc.ConfiguredApplication.lambda$startReconfigurerThread$1(ConfiguredApplication.java:275)
	at java.base/java.lang.Thread.run(Thread.java:834)

Make sure that the jar files (i.e. bundles) are actually deployed with correct names per services.xml.

Class not found

All classes that are referred to in a user bundle must either be embedded in the bundle, or imported from another bundle by an Import-Package statement in the bundle manifest. When this rule has been breached, we get one of the most commonly seen exceptions when working with OSGi bundles:

...
exception=
    java.lang.NoClassDefFoundError: com/acme/utils/Helper
    ...
    java.lang.ClassNotFoundException: com.acme.utils.Helper not found by my_bundle [29]

For the bundle-plugin to automatically add an Import-Package statement to your bundle's manifest, that package must be exported from another bundle that is declared as a 'provided' scope dependency in pom.xml. If the dependency that contains the missing class is under your own control, make sure it's packaged as an OSGi bundle, and export the package from that bundle. If not, the simplest way to resolve the issue is to embed the dependency in your own bundle, by setting its scope to 'compile' instead of 'provided'.

If the strategy above does not resolve the case, it's most likely because the class in question is loaded by reflection, e.g. Class.forName("com.acme.utils.Helper"). This is quite common when working with libraries for pluggable frameworks, for which we have a separate troubleshooting doc.

Slow Container start

In the vespa log, a container startup looks like:

[2021-01-07 10:13:35.325] INFO    : container        Container.com.yahoo.container.core.config.ApplicationBundleLoader  Installed bundles: {[0]org.apache.felix.framework:6.0.3, [1]container-disc:7.335.22 ...
...
[2021-01-07 10:26:57.291] INFO    : container        Container.com.yahoo.container.jdisc.ConfiguredApplication  Switching to the latest deployed set of configurations and components. Application config generation: 1

The container is ready at the last log line - note the long startup time. To get more details on what the container is doing at startup, inspect the ComponentGraph debug log. Find the container service name (here: "container"), set debug logging and restart the container:

$ vespa-sentinel-cmd list
vespa-sentinel-cmd 'sentinel.ls' OK.
container state=RUNNING mode=AUTO pid=246585 exitstatus=0 id="default/container.0"

$ vespa-logctl container:com.yahoo.container.di.componentgraph.core debug=on

$ vespa-stop-services && vespa-start-services

# Find DEBUG log messages for component creation, like:

[2021-01-07 10:13:37.006] DEBUG   : container        Container.com.yahoo.container.di.componentgraph.core.ComponentGraph        Trying the fallback injector to create component of class com.yahoo.container.jdisc.messagebus.SessionCache to inject
 into component 'chain.mychain in MbusServer' of type 'com.yahoo.container.jdisc.messagebus.MbusServerProvider'.

[2021-01-07 10:14:14.082] DEBUG   : container        Container.com.yahoo.container.di.componentgraph.core.ComponentNode Constructing 'com.yahoo.search.query.profile.compiled.CompiledQueryProfileRegistry'
[2021-01-07 10:26:54.669] DEBUG   : container        Container.com.yahoo.container.di.componentgraph.core.ComponentNode Finished constructing 'com.yahoo.search.query.profile.compiled.CompiledQueryProfileRegistry'

In this particular example, query profile compilation takes a long time.

Unresolved constraint

If the bundle has an Import-Package for a package that is not available at runtime, the OSGi framework will report an unresolved constraint error. The symptom as seen in the log is (line breaks have been added for legibility):

org.osgi.framework.BundleException: Unresolved constraint in bundle my_bundle [29]:
Unable to resolve 29.0:
missing requirement [29.0] osgi.wiring.package; (osgi.wiring.package=com.acme.utils)
     at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3974)

This means that the missing class resides in a 'provided' dependency referred to from the bundle's pom.xml, either directly or transitively. In order to make the dependency available at runtime, there are two options:

• The easiest is to set the dependency as 'compile' scope (instead of 'provided') to embed it in your own bundle. This works fine in most cases, unless two of your dependencies need two different versions of the same library.
• Add the missing jar file to the components/ folder of the application package, along with your own bundles. The maven-dependency-plugin has a goal called 'copy-dependencies' to help with this.

If the missing jar is a transitive dependency, maven can help visualize the dependency graph of your project:

$ mvn dependency:tree

Multiple implementations of the same class

When two bundles interact via their public APIs, it is crucial that both bundles resolve each and every participating class to the same Class object. If not, we will get error messages like:

java.lang.LinkageError: loader constraint violation: when resolving field
"DATETIME" the class loader (instance of
org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the referring
class, javax/xml/datatype/DatatypeConstants, and the class loader (instance of
<bootloader>) for the field's resolved type, pe/DatatypeConstants, have different Class
objects for that type

or:

java.lang.LinkageError: loader constraint violation: loader (instance of <bootloader>)
previously initiated loading for a different type with name "javax/xml/namespace/QName"

or (less frequently):

java.lang.ClassCastException: com.acme.utils.Helper cannot be cast to  com.acme.utils.Helper

All these error messages indicate that multiple implementations of one or more classes are used at runtime. This could mean that either

• Two interacting user bundles embed the same Java package.
• A user bundle embeds a Java package that is exported from one of the JDisc bundles.

Usually, the "duplicate" package is pulled in by the user bundle transitively from a library dependency.

Multiple implementations example

Let's take a look at an example resolving the duplicate javax.xml.namespace.QName class from the error message above.

All 'javax.xml' packages in the JDK are exported by the JDisc core bundle. This means that they should be imported by user bundles, instead of embedded inside them. Hence, we should ensure that there are no classes from packages prefixed by 'javax.xml' in our bundle. To find out which library that pulls in the package, we can use the following procedure:

1. Extract the full component jar, including any embedded jars. One tool that does the job is rjar.
2. Search the folder where the jar was extracted for 'javax.xml' classes. For example:

   $ find . | grep "javax/xml/.*\.class"
   

   This yields output like:

   ...
   ./my_bundle-deploy.jar/dependencies/stax-api-1.0.1.jar/javax/xml/namespace/QName.class
   ...
   

3. Now we need to find out which libraries that pulled in the offending classes. From this example, it was stax-api-1.0.1. Usually, these libraries are not pulled in by our pom as direct dependencies, but rather transitively via another library that we are using. In order to find the direct dependency, use maven's dependency plugin from your application directory:

   $ mvn dependency:tree -Dverbose
   

   This will yield output like:

   [INFO] +- com.acme.utils:jersey-utils:1.0.0:compile
   [INFO] |  +- com.sun.jersey:jersey-json:jar:1.13:compile
   [INFO] |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
   [INFO] |  |    \- stax:stax-api:jar:1.0.1:compile
   

   We can now see that stax:stax-api:1.0.1 is pulled in transitively from our direct dependency com.acme.utils:jersey-utils.
4. Finally, to exclude stax:stax-api we must add the appropriate exclusion from our direct dependency com.acme.utils:jersey-utils, in pom.xml:

   <dependency>
     <groupId>com.acme.utils</groupId>
     <artifactId>jersey-utils</artifactId>
     <version>1.0.0</version>
     <exclusions>
       <exclusion>
         <groupId>stax</groupId>
         <artifactId>stax-api</artifactId>
       </exclusion>
     </exclusions>
   </dependency>
   

Multiple implementations example slf4j-api

This is similar to the previous example, but logging libraries are maybe the most common problem teams encounter. Here we will see the symptom, use dependency:tree and add an exclusion. The symptom:

java.lang.RuntimeException: An exception occurred while
constructing 'com.acme.utils.Helper in acme-utils'
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the
current class, org/slf4j/LoggerFactory, and the class loader (instance of
sun/misc/Launcher$AppClassLoader) for the method's defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at
org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:299)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:269)

Running the mvn dependency:tree in the previous example gives:

[INFO] +- com.yahoo.vespa:container-dev:jar:5.28.29:provided
[INFO] |  +- com.yahoo.vespa:jdisc_core:jar:5.28.29:provided
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.5:compile - scope updated from provided; omitted for duplicate)
...
[INFO] +- com.acme.utils:smartlib:jar:1.0.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.6.6:compile

We see that slf4j-api is no longer provided from container-dev, which it should. To fix this, we add an exclusion on the offender:

<dependency>
  <groupId>com.acme.utils</groupId>
  <artifactId>smartlib</artifactId>
  <version>1.0.0</version>
  <scope>compile</scope>
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
    </exclusion>

But it still does not work! And we can see why:

$ jar -tf mailsearch-docprocs-deploy.jar  | grep slf
dependencies/slf4j-api-1.7.5.jar

Something still pulls in slf4j... Other candidates:

[INFO] \- com.yahoo.vespa:application:jar:5.28.29:test
...
[INFO]    +- com.yahoo.vespa:zkfacade:jar:5.28.29:test
[INFO]    |  +- org.apache.curator:curator-recipes:jar:2.4.1:test
[INFO]    |  |  +- org.apache.curator:curator-framework:jar:2.4.1:test
[INFO]    |  |  |  +- org.apache.curator:curator-client:jar:2.4.1:test
[INFO]    |  |  |  |  +- (org.slf4j:slf4j-api:jar:1.6.4:test - omitted for conflict with 1.7.5)
...
[INFO]    |  +- (org.slf4j:slf4j-jdk14:jar:1.7.5:test - omitted for duplicate)

Added the right excludes for application and used mvn dependency:tree and verified that all references were gone, except the ones for container-dev. Still found:

$ jar -tf mailsearch-docprocs-deploy.jar  | grep slf
dependencies/slf4j-api-1.7.5.jar

One can make it work by managing this dependency explicitly - add this at POM top-level:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.5</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>
</dependencyManagement>