Perodically Unable to start up EM Process via Script

Document ID : KB000074250
Last Modified Date : 20/03/2018
Show Technical Document Details
Issue:
Customer is observing the problem where EM fails to fully start and seems to hang when starting all EMs in their cluster. This happens several times a month.
Environment:
  This happened in an APM 10.5.x environment but could impact any release
Cause:
OSGI deadlock. A thread dump shows threads blocked or waiting The MOM startup process hangs or becomes non-responsive.

Deadlock thread stack:

"SpringOsgiExtenderThread-3" #29 prio=5 os_prio=0 tid=0x00007f0e48065800 nid=0x6134 in Object.wait() [0x00007f0ea486d000]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:502)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.lock(BundleLoader.java:1159)
 - locked <0x00000006a5712178> (a org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:396)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:385)
 at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:87)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at org.springframework.osgi.util.internal.MapBasedDictionary.keys(MapBasedDictionary.java:180)
 at org.eclipse.osgi.framework.internal.core.ServiceRegistrationImpl$Properties.<init>(ServiceRegistrationImpl.java:596)
 - locked <0x00000006803a54d0> (a org.springframework.osgi.util.internal.MapBasedDictionary)
 at org.eclipse.osgi.framework.internal.core.ServiceRegistrationImpl$Properties.<init>(ServiceRegistrationImpl.java:617)
 at org.eclipse.osgi.framework.internal.core.ServiceRegistrationImpl.createProperties(ServiceRegistrationImpl.java:282)
 at org.eclipse.osgi.framework.internal.core.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:113)
 - locked <0x00000006803afe70> (a java.lang.Object)
 - locked <0x00000006a572bd80> (a org.eclipse.osgi.framework.internal.core.ServiceRegistryImpl)

...


"main" #1 prio=5 os_prio=0 tid=0x00007f0ee000a800 nid=0x6109 waiting for monitor entry [0x00007f0ee6ceb000]
 java.lang.Thread.State: BLOCKED (on object monitor)
 at org.eclipse.osgi.framework.internal.core.Framework.getServiceReferences(Framework.java:1230)
 - waiting to lock <0x00000006a572bd80> (a org.eclipse.osgi.framework.internal.core.ServiceRegistryImpl)
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.getServiceReferences(BundleContextImpl.java:736)
 at org.osgi.util.tracker.ServiceTracker.getInitialReferences(ServiceTracker.java:351)
 at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:316)
 - locked <0x0000000682398dd0> (a org.osgi.util.tracker.ServiceTracker$Tracked)
 - locked <0x0000000682398880> (a org.osgi.util.tracker.ServiceTracker)
 at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:274)
 at com.wily.introscope.osgi.ABundleActivator.start(ABundleActivator.java:57)
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:1009)
 at java.security.AccessController.doPrivileged(Native Method)
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:1003)
 at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:984)
 at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:346)
 at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:265)
 at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:400)
 at org.eclipse.core.runtime.internal.adaptor.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:111)
 at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:427)
 at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:193)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findLocalClass(BundleLoader.java:368)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClassInternal(BundleLoader.java:444)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:397)
 at org.eclipse.osgi.framework.internal.core.DependentPolicy.loadClass(DependentPolicy.java:52)
 at org.eclipse.osgi.framework.internal.core.PolicyHandler.doBuddyClassLoading(PolicyHandler.java:138)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClassInternal(BundleLoader.java:469)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:397)
 at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:385)
 at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:87)

 
Resolution:
At the present time, there are only workarounds to resolve the issue. 
These include
1) Restarting the EM
There is no other workaround other than restart EM

2) Changing the EM startup script to monitor EM status after startup by monitoring the default EM watchdog process.

A future release may remove OSGI from APM.