Bundle troubleshooting
This document describes how to troubleshoot the most common errors when working with bundles:
- Could not create component
- Class not found
- Slow Container start
- Unresolved constraint
- Multiple implementations of the same class
Could not create component
The Container fails to start if it cannot load bundles. Example, using wrong bundle name in the multiple-bundles sample app:
<services version="1.0">
<container id="fibonacci" version="1.0">
- <component id="com.mydomain.lib.FibonacciProducer" bundle="multiple-bundles-lib" />
+ <component id="com.mydomain.lib.FibonacciProducer" bundle="multiple-bundles-typo" />
Looking at what is actually deployed in multiple-bundles:
$ ls -1 target/*.jar target/multiple-bundles-1.0.0-deploy.jar target/multiple-bundles-1.0.0-without-dependencies.jar target/multiple-bundles-lib-1.0.1-deploy.jarError in log:
[2020-01-23 14:28:01.367] WARNING : qrserver Container.com.yahoo.container.di.Container
Failed to set up new component graph. Retaining previous component generation.
exception=
java.lang.IllegalArgumentException: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'.
Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo.
If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
at com.yahoo.osgi.OsgiImpl.resolveFromClassPath(OsgiImpl.java:74)
at com.yahoo.osgi.OsgiImpl.resolveClass(OsgiImpl.java:65)
at com.yahoo.container.di.Container.addNodes(Container.java:228)
at com.yahoo.container.di.Container.createComponentsGraph(Container.java:217)
at com.yahoo.container.di.Container.getConfigAndCreateGraph(Container.java:160)
at com.yahoo.container.di.Container.getNewComponentGraph(Container.java:84)
at com.yahoo.container.core.config.HandlersConfigurerDi.getNewComponentGraph(HandlersConfigurerDi.java:145)
at com.yahoo.container.jdisc.ConfiguredApplication.lambda$startReconfigurerThread$1(ConfiguredApplication.java:275)
at java.base/java.lang.Thread.run(Thread.java:834)
[2020-01-23 14:28:01.367] ERROR : qrserver Container.com.yahoo.container.jdisc.ConfiguredApplication
Reconfiguration failed, your application package must be fixed, unless this is a JNI reload issue: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'. Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo. If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
exception=
java.lang.IllegalArgumentException: Could not create a component with id 'com.mydomain.lib.FibonacciProducer'. Tried to load class directly, since no bundle was found for spec: multiple-bundles-typo. If a bundle with the same name is installed, there is a either a version mismatch or the installed bundle's version contains a qualifier string.
at com.yahoo.osgi.OsgiImpl.resolveFromClassPath(OsgiImpl.java:74)
at com.yahoo.osgi.OsgiImpl.resolveClass(OsgiImpl.java:65)
at com.yahoo.container.di.Container.addNodes(Container.java:228)
at com.yahoo.container.di.Container.createComponentsGraph(Container.java:217)
at com.yahoo.container.di.Container.getConfigAndCreateGraph(Container.java:160)
at com.yahoo.container.di.Container.getNewComponentGraph(Container.java:84)
at com.yahoo.container.core.config.HandlersConfigurerDi.getNewComponentGraph(HandlersConfigurerDi.java:145)
at com.yahoo.container.jdisc.ConfiguredApplication.lambda$startReconfigurerThread$1(ConfiguredApplication.java:275)
at java.base/java.lang.Thread.run(Thread.java:834)
Make sure that the jar files (i.e. bundles) are actually deployed with correct names per services.xml.
Class not found
All classes that are referred to in a user bundle must either be embedded in the bundle,
or imported from another bundle by an Import-Package
statement in the bundle
manifest. When this rule has been breached, we get one of the most commonly seen
exceptions when working with OSGi bundles:
... exception= java.lang.NoClassDefFoundError: com/acme/utils/Helper ... java.lang.ClassNotFoundException: com.acme.utils.Helper not found by my_bundle [29]
For the bundle-plugin to automatically add an Import-Package statement to your bundle's manifest, that package must be exported from another bundle that is declared as a 'provided' scope dependency in pom.xml. If the dependency that contains the missing class is under your own control, make sure it's packaged as an OSGi bundle, and export the package from that bundle. If not, the simplest way to resolve the issue is to embed the dependency in your own bundle, by setting its scope to 'compile' instead of 'provided'.
If the strategy above does not resolve the case, it's most likely because the class in
question is loaded by reflection,
e.g. Class.forName("com.acme.utils.Helper")
. This is quite common when working
with libraries for pluggable frameworks, for which we have a separate troubleshooting doc.
Slow Container start
In the vespa log, a container startup looks like:
[2021-01-07 10:13:35.325] INFO : container Container.com.yahoo.container.core.config.ApplicationBundleLoader Installed bundles: {[0]org.apache.felix.framework:6.0.3, [1]container-disc:7.335.22 ... ... [2021-01-07 10:26:57.291] INFO : container Container.com.yahoo.container.jdisc.ConfiguredApplication Switching to the latest deployed set of configurations and components. Application config generation: 1
The container is ready at the last log line - note the long startup time. To get more details on what the container is doing at startup, inspect the ComponentGraph debug log. Find the container service name (here: "container"), set debug logging and restart the container:
$ vespa-sentinel-cmd list vespa-sentinel-cmd 'sentinel.ls' OK. container state=RUNNING mode=AUTO pid=246585 exitstatus=0 id="default/container.0" $ vespa-logctl container:com.yahoo.container.di.componentgraph.core debug=on $ vespa-stop-services && vespa-start-services # Find DEBUG log messages for component creation, like: [2021-01-07 10:13:37.006] DEBUG : container Container.com.yahoo.container.di.componentgraph.core.ComponentGraph Trying the fallback injector to create component of class com.yahoo.container.jdisc.messagebus.SessionCache to inject into component 'chain.mychain in MbusServer' of type 'com.yahoo.container.jdisc.messagebus.MbusServerProvider'. [2021-01-07 10:14:14.082] DEBUG : container Container.com.yahoo.container.di.componentgraph.core.ComponentNode Constructing 'com.yahoo.search.query.profile.compiled.CompiledQueryProfileRegistry' [2021-01-07 10:26:54.669] DEBUG : container Container.com.yahoo.container.di.componentgraph.core.ComponentNode Finished constructing 'com.yahoo.search.query.profile.compiled.CompiledQueryProfileRegistry'
In this particular example, query profile compilation takes a long time.
Unresolved constraint
If the bundle has an Import-Package for a package that is not available at runtime, the OSGi framework will report an unresolved constraint error. The symptom as seen in the log is (line breaks have been added for legibility):
org.osgi.framework.BundleException: Unresolved constraint in bundle my_bundle [29]:
Unable to resolve 29.0:
missing requirement [29.0] osgi.wiring.package; (osgi.wiring.package=com.acme.utils)
at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3974)
This means that the missing class resides in a 'provided' dependency referred to from the bundle's pom.xml, either directly or transitively. In order to make the dependency available at runtime, there are two options:
- The easiest is to set the dependency as 'compile' scope (instead of 'provided') to embed it in your own bundle. This works fine in most cases, unless two of your dependencies need two different versions of the same library.
- Add the missing jar file to the
components/
folder of the application package, along with your own bundles. The maven-dependency-plugin has a goal called 'copy-dependencies' to help with this.
If the missing jar is a transitive dependency, maven can help visualize the dependency graph of your project:
$ mvn dependency:tree
Multiple implementations of the same class
When two bundles interact via their public APIs, it is crucial that both bundles resolve
each and every participating class to the same Class
object. If not, we will
get error messages like:
java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME" the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the referring class, javax/xml/datatype/DatatypeConstants, and the class loader (instance of <bootloader>) for the field's resolved type, pe/DatatypeConstants, have different Class objects for that typeor:
java.lang.LinkageError: loader constraint violation: loader (instance of <bootloader>)
previously initiated loading for a different type with name "javax/xml/namespace/QName"
or (less frequently):
java.lang.ClassCastException: com.acme.utils.Helper cannot be cast to com.acme.utils.Helper
All these error messages indicate that multiple implementations of one or more classes are used at runtime. This could mean that either
- Two interacting user bundles embed the same Java package.
- A user bundle embeds a Java package that is exported from one of the JDisc bundles.
Multiple implementations example
Let's take a look at an example resolving the duplicate javax.xml.namespace.QName class from the error message above.
All 'javax.xml' packages in the JDK are exported by the JDisc core bundle. This means that they should be imported by user bundles, instead of embedded inside them. Hence, we should ensure that there are no classes from packages prefixed by 'javax.xml' in our bundle. To find out which library that pulls in the package, we can use the following procedure:
- Extract the full component jar, including any embedded jars. One tool that does the job is rjar.
- Search the folder where the jar was extracted for 'javax.xml' classes. For example:
$ find . | grep "javax/xml/.*\.class"
This yields output like:... ./my_bundle-deploy.jar/dependencies/stax-api-1.0.1.jar/javax/xml/namespace/QName.class ...
- Now we need to find out which libraries that pulled in the offending classes. From this
example, it was
stax-api-1.0.1
. Usually, these libraries are not pulled in by our pom as direct dependencies, but rather transitively via another library that we are using. In order to find the direct dependency, use maven's dependency plugin from your application directory:$ mvn dependency:tree -Dverbose
This will yield output like:[INFO] +- com.acme.utils:jersey-utils:1.0.0:compile [INFO] | +- com.sun.jersey:jersey-json:jar:1.13:compile [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | \- stax:stax-api:jar:1.0.1:compile
We can now see thatstax:stax-api:1.0.1
is pulled in transitively from our direct dependencycom.acme.utils:jersey-utils
. -
Finally, to exclude
stax:stax-api
we must add the appropriateexclusion
from our direct dependencycom.acme.utils:jersey-utils
, in pom.xml:<dependency> <groupId>com.acme.utils</groupId> <artifactId>jersey-utils</artifactId> <version>1.0.0</version> <exclusions> <exclusion> <groupId>stax</groupId> <artifactId>stax-api</artifactId> </exclusion> </exclusions> </dependency>
Multiple implementations example slf4j-api
This is similar to the previous example, but logging libraries are maybe the most common problem teams encounter. Here we will see the symptom, use dependency:tree and add an exclusion. The symptom:
java.lang.RuntimeException: An exception occurred while
constructing 'com.acme.utils.Helper in acme-utils'
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the
current class, org/slf4j/LoggerFactory, and the class loader (instance of
sun/misc/Launcher$AppClassLoader) for the method's defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at
org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:299)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:269)
Running the mvn dependency:tree in the previous example gives:
[INFO] +- com.yahoo.vespa:container-dev:jar:5.28.29:provided [INFO] | +- com.yahoo.vespa:jdisc_core:jar:5.28.29:provided [INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.5:compile - scope updated from provided; omitted for duplicate) ... [INFO] +- com.acme.utils:smartlib:jar:1.0.0:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.6.6:compile
We see that slf4j-api is no longer provided from container-dev, which it should. To fix this, we add an exclusion on the offender:
<dependency> <groupId>com.acme.utils</groupId> <artifactId>smartlib</artifactId> <version>1.0.0</version> <scope>compile</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion>
But it still does not work! And we can see why:
$ jar -tf mailsearch-docprocs-deploy.jar | grep slf dependencies/slf4j-api-1.7.5.jar
Something still pulls in slf4j... Other candidates:
[INFO] \- com.yahoo.vespa:application:jar:5.28.29:test ... [INFO] +- com.yahoo.vespa:zkfacade:jar:5.28.29:test [INFO] | +- org.apache.curator:curator-recipes:jar:2.4.1:test [INFO] | | +- org.apache.curator:curator-framework:jar:2.4.1:test [INFO] | | | +- org.apache.curator:curator-client:jar:2.4.1:test [INFO] | | | | +- (org.slf4j:slf4j-api:jar:1.6.4:test - omitted for conflict with 1.7.5) ... [INFO] | +- (org.slf4j:slf4j-jdk14:jar:1.7.5:test - omitted for duplicate)
Added the right excludes for application and used mvn dependency:tree and verified that all references were gone, except the ones for container-dev. Still found:
$ jar -tf mailsearch-docprocs-deploy.jar | grep slf dependencies/slf4j-api-1.7.5.jar
One can make it work by managing this dependency explicitly - add this at POM top-level:
<dependencyManagement> <dependencies> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.5</version> <scope>provided</scope> </dependency> </dependencies> </dependencyManagement>