Long delays when starting Weblogic application server with APM 9.x Java Agent enabled.

Document ID : KB000006643
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

A noticeable delay is observed when starting up Weblogic application server with 9.x version of the APM Java agent. (This can also occur in other application servers such as Websphere.)

Environment:
Java Agent 9.6x and 9.7x.
Cause:

Seeing error in the agent logs:

<Error> <WebLogicServer> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1368492926910> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "648" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@127b230", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.

The resulting stack trace will show signs of Introscope deep inheritance classes:

com.wily.diagnos.personality.java.classfile.DGClass.isInstanceOf(DGClass.java:1134)
com.wily.introscope.agent.db.DeepInheritanceHelper.addInheritanceMatchRequest(DeepInheritanceHelper.java:139)
com.wily.diagnos.cmp.classmatcher.DGDeepInheritedClassMatcher.executeMatches(DGDeepInheritedClassMatcher.java:60)
com.wily.diagnos.cmp.classmatcher.DGDeepInheritedClassMatcher.IClassMatcher_matches(DGDeepInheritedClassMatcher.java:110)

Resolution:

By disabling deep inheritance in IntroscopeAgent.profile, this will resolve the issue but instrumentation of classes that are "remotely" inherited from a Class/Interface defined in pbd won´t happen. (See the APM Java Agent Guide for more information about deep inheritance). By disabling the property, only 2 levels of class inheritance are checked

introscope.autoprobe.deepinheritance.enabled=false

Any classes that are missing a result of the change can be traced explicitly with normal pbd statements.