Threads get stuck on "Transaction Structure Queue Execution" in WAS application monitor.

Document ID : KB000048088
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

Sometimes, the threads get stuck. Looking at a thread dump shows threads similar to following:


"Transaction Structure Queue Execution" daemon prio=10 tid=0x0b85dc00nid=0x17a062 in Object.wait() [0x6207a000]   java.lang.Thread.State: WAITING (on object monitor)        at java.lang.Object.wait(Native Method)        - waiting on <0x7692fc90> (a com.wily.util.adt.BlockingQueue)        at java.lang.Object.wait(Object.java:485)        atcom.wily.util.adt.BlockingQueue.interruptableDequeue(BlockingQueue.java:123)        - locked <0x7692fc90> (a com.wily.util.adt.BlockingQueue)        atcom.wily.util.task.AsynchExecutionQueue.doTask(AsynchExecutionQueue.java:200)        at com.wily.util.task.ATask$CoreTask.run(ATask.java:132)        at java.lang.Thread.run(Thread.java:662) "Signal Dispatcher" daemon prio=10 tid=0x08977000 nid=0x17a042 runnable[0x00000000]   java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x08969000 nid=0x17a041 in Object.wait()[0x633c3000]   java.lang.Thread.State: WAITING (on object monitor)        at java.lang.Object.wait(Native Method)        - waiting on <0x7647f848> (a java.lang.ref.ReferenceQueue$Lock)        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)        - locked <0x7647f848> (a java.lang.ref.ReferenceQueue$Lock)        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=10tid=0x61eb8400 nid=0x17ab7c waiting for monitor entry [0x5f032000]   java.lang.Thread.State: BLOCKED (on object monitor)        atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)        - waiting to lock <0x780472e8> (a weblogic.socket.PosixSocketMuxer$1)        atweblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)        atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Sometime later, the stuck threads are released and APM works normally.

Solution:

The agent works as designed:

  • In the 9.1 and above in "new agent mode" the agent holds InvocationData objects of all the instrumented methods in a given transaction.

  • The available memory needs to be increased accordingly if the following occurs:

    The transaction length is very large for one or more concurrent transactions.
    The transaction takes more time to complete.

  • After the transaction completes, the agent performs garbage collection on the objects.
    A possible resolution is to either increase the agent heap size or switch the agent to use old/legacy mode (applies to 9.1x).