Events are not not firing in CA Service Desk Manager

Document ID : KB000104148
Last Modified Date : 05/07/2018
Show Technical Document Details
Question:
Events are not being triggered in CA Service Desk Manager. 

See this in the stdlogs a lot:

07/03 21:46:09.09 MY_SERVER      animator_nxd         6208 ERROR        animator_nxd.c        1839 AHD03026:Retirado por outro usuário in animator ANI:4072862 (cr:786923) for atev:3487875 in method examine_lock_list

07/03 21:46:09.09 MY_SERVER      animator_nxd         6208 ERROR        animator_nxd.c        1839 AHD03026:Retirado por outro 

There has been a slow buildup over a week of backlogged Animator entries:

It appears that Animator entries are not being processed, as are not seeing:
  • Updated Events in CA Service Desk Manager
  • Not receiving Event driven Notifications from CA Service Desk Manager to CA PAM
  • The number of rows in the Animator table is not decreasing entries which should have fired

However, the Animator process is up, as seen by:
  • Writes "animator_nxd" process entries to the stdlogs
  • Is incremented with new rows when a New Event is added to a CA SDM ticket


Looking at individual Animator entries, they appear to cycle through a Locked state:

Line 19868: 07/03 22:00:09.80 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c        1412 Got index 95 of backlog (ANI:4072880)

Line 19869: 07/03 22:00:09.80 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c        1730 Constructing animator ANI:4072880

Line 19870: 07/03 22:00:09.80 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c         988 Registering animator ANI:4072880; 165 now registered

Line 20095: 07/03 22:00:09.99 MY_SERVER      animator_nxd         6208 WARNING      animator_nxd.c        2498 Animator ANI:4072880 (cr:786942) for atev:3487891 firing 3 days 11:14:06 late; should have fired at 06/30/2018 10:46:03

Line 20096: 07/03 22:00:09.99 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c        1106 Firing backlogged animator ANI:4072880 for atev:3487891

Line 20336: 07/03 22:00:10.21 MY_SERVER      animator_nxd         6208 ERROR        animator_nxd.c        1839 AHD03026:Retirado por outro usuário in animator ANI:4072880 (cr:786942) for atev:3487891 in method examine_lock_list

Line 20337: 07/03 22:00:10.21 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c        2444 Suspending animator ANI:4072880 (cr:786942) for atev:3487891 until 07/03/2018 22:02:10

Line 32948: 07/03 22:02:10.05 MY_SERVER      animator_nxd         6208 VERBOSE      animator_nxd.c        1106 Firing backlogged animator ANI:4072880 for atev:3487891

Line 33153: 07/03 22:02:10.23 MY_SERVER      animator_nxd         6208 ERROR        animator_nxd.c        1839 AHD03026:Retirado por outro usuário in animator ANI:4072880 (cr:786942) for atev:3487891 in method examine_lock_list
    

It was noted in pdm_status that the rpc_srvr process was in a "connecting" state.

The jstd.log showed that the heap space had run out.

 
Answer:
Multiple attempts were made to resolve the issue.

The root cause was suspected to be too many Locks against animator row entries.
A full recycle of the system cleared these locks and restored all functionality.
It is suspected that this was the key error:
07/03 22:02:10.23 MY_SERVER      animator_nxd         6208 ERROR        animator_nxd.c        1839 AHD03026:Retirado por outro usuário in animator ANI:4072880 (cr:786942) for atev:3487891 in method examine_lock_list

These points were already in place:
  • Java heap space increased to 1024
  • SDM agents raised from 6 to 24
  • Dedicated domsrvr for the animator
  • No System Locks were present

These tasks were tried, with no apparent effect:
  • The process rpc_srvr was recycled 
  • The process animator_nxd was recycled (this may have had some effect
The following tracing was put on to get more information on the animator failure on the server where animator runs:
  • pdm_logstat -n animator_nxd verbose
To turn off, enable:
  • pdm_logstat -n animator_nxd 
Additional Information: