Jobs fail to run on an agent with "INTERNAL ERROR" or "SUBERROR"

Document ID : KB000022828
Last Modified Date : 14/02/2018
Show Technical Document Details

Symptoms: 

Jobs are failing on a particular agent with any of the following;

  1. Workload Object in "SUBERROR" with "error creating thread issue"

  2. Event status changed to "Not active. INTERNAL ERROR, UNABLE TO CREATE NEW NATIVE THREAD"

  3. Error messages reported in the Agent logs as below;
--- 
CybFileMonCreateHandler.run[:227] - java.lang.OutOfMemoryError: null 
--- 
cybermation.plugins.filemon.wobhandler.CybWOBFiletrigDefineHandler 
java.lang.OutOfMemoryError 
--- 
Error erasing hash 23757D16C3ED9261EA3A0689980507D864487656 from jobmap 
--- 
cybermation.plugins.filemon.wobhandler.CybWOBFiletrigDefineHandler 
java.lang.OutOfMemoryError: unable to create new native thread 
--- 
JVMDBG001: malloc failed to allocate 100016 bytes, time: Tue Jul 20 13:04:02 2010 
**Out of memory, aborting** 
*** panic: JVMCL052: Cannot allocate memory in initialize Heap for heap segment 
--- 
Unable to alloc heap of requested size, perhaps the maxdata value is too 
small - see README.HTML for more information.**Out of memory, aborting***** 
panic: JVMST017: Cannot allocate memory in 
initializeMarkAndAllocBits(markbits1)[1] + IOT/Abort trap(coredump) sudo 
/usr/local/opt/esp/bin/cybAgent & 
---- 
java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore 
threads due to memory or resource constraints 
---

Cause: 

This normally happens when the Agent is either;

  1. Out of memory and unable to create new thread, or

  2. Could not allocate heap of requested size

When the Agent is out of memory or could not allocate heap of requested size, it may prevent the Agent to be started or from processing new jobs.

Resolution:

Restart the agent and force complete all jobs that did not complete.

If this occurs frequently, then it is recommended to increase the jvm heap size for the Agent.

The parameter that controls the memory available to the Agent is;

 oscomponent.jvm.x.options==-Xms<nnn>;-Xmx<nnn>       
 where; 
 -Xms implies minimum heap size
 -Xmx implies maximum heap size
 nnn is a multiple of 64MB

For example, to increase the heap size, add the parameter in the agentparm.txt file (located in agent installation directory);

 oscomponent.jvm.x.options=-Xms64m;-Xmx128m  

And, restart the agent.

The above example increases the maximum heap size to 128MB, default is 64MB.

Note: Increase Xmx as per the size of your workloads and resources available on the Agent box.

For AIX Agent:

In addition to the above, add two AIX environment variables in cybAgent script that starts the Agent;

export IBM_JAVA_MMAP_JAVA_HEAP=true        
export LDR_CNTRL=MAXDATA=0x80000000 

These are AIX environment variables meant to increase the amount of memory available to processes running on AIX. On AIX, the number of segments that a process can use for data is controlled by the LDR_CNTRL environment variable.

The default is 256MB. Setting the variable to;

LDR_CNTRL=MAXDATA=0x80000000  

would increase the process memory limit to 2GB. This is also known as AIX Big Memory model.