mon:thread: ACE has timed out

Document ID : KB000101638
Last Modified Date : 15/06/2018
Show Technical Document Details
Issue:

The iXp users repeatedly seeing the following message:
Warning: this data is more than five(5) minutes old.
If this message persists, please contact your iXp administrator

 
$IXP_HOME/log/ixpDaemon.log shows the following messages: 
mon:thread: ACE has timed out 
mon:thread: ACE has timed out 
........
......
...
gc:iac:1:0:0/0:0:lac:0/0:0:jtc:0/0:0 
gc:bef=472M:avg=665M:aft=493M:low=94M:59ms:1101ms++ 
gc:free=505459k:total=1536000k: 
main:thread exit request: ACE:1528824408481* 
main:detected time out:ACE 
main:restarting collector:ACE 
main:java.lang.OutOfMemoryError 
main:: 
main:Failed to create a thread: retVal -1073741830, errno 11 
........
......
...
mon:thread: ACE has timed out 
mon:thread: ACE has timed out 

 
This scenario has caused the server disk utilization to hit 100%.
Environment:
CA Workload Automation iXp - 11.3.5 + version supported platforms
Cause:
The resources like disk space and memory are crucial for the tomcat applications.
iXp is not able to create threads normally could be due to hitting the max allocation limits for the process.
Resolution:
  • Verify the Basic System Requirements per the iXp documentation
  • Ensure there is enough disk space available in the server
  • Stop the tomcat process
    # $CATALINA_HOME/bin/shutdown.sh
Using CATALINA_BASE:   /opt/CA/iXp/tomcat9
Using CATALINA_HOME:   /opt/CA/iXp/tomcat9
Using CATALINA_TMPDIR: /opt/CA/iXp/tomcat9/temp
Using JRE_HOME:        /opt/CA/iXp/jdk1.8/jre
Using CLASSPATH:       /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar
  • Increase the JAVA_OPTS max memory according to the load to meet the memory requirements for iXp.
# export IXP_HOME=/opt/CA/iXp/iXphome
# export JAVA_HOME=/opt/CA/iXp/jdk1.8/jre
# export JAVA_OPTS="-Dixp.home=${IXP_HOME} -Xms1500m -Xmx12000m"
# export CATALINA_HOME=/opt/CA/iXp/tomcat9
# export PATH=$JAVA_HOME/bin:$PATH

 
Important! If the tomcat startup scripts are being used in the environment ensure the changes for the variable JAVA_OPTS are made inside the script.
  • Start the tomcat process​
    # $CATALINA_HOME/bin/startup.sh
    Using CATALINA_BASE:   /opt/CA/iXp/tomcat9
    Using CATALINA_HOME:   /opt/CA/iXp/tomcat9
    Using CATALINA_TMPDIR: /opt/CA/iXp/tomcat9/temp
    Using JRE_HOME:        /opt/CA/iXp/jdk1.8/jre
    Using CLASSPATH:       /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar
    Tomcat started.
  • Verify the set limit appears in the process 
# ps -ef|grep ixp
root     16638     1 99 23:31 pts/0    00:00:06 /opt/CA/iXp/jdk1.8/jre/bin/java -Djava.util.logging.config.file=/opt/CA/iXp/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dixp.home=/opt/CA/iXp/iXphome -Xms1500m -Xmx12000m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dignore.endorsed.dirs= -classpath /opt/CA/iXp/tomcat9/bin/bootstrap.jar:/opt/CA/iXp/tomcat9/bin/tomcat-juli.jar -Dcatalina.base=/opt/CA/iXp/tomcat9 -Dcatalina.home=/opt/CA/iXp/tomcat9 -Djava.io.tmpdir=/opt/CA/iXp/tomcat9/temp org.apache.catalina.startup.Bootstrap start
#
Additional Information: