SPS running out of memory and restart during load

Document ID : KB000044038
Last Modified Date : 14/02/2018
Show Technical Document Details

Issue:

Randomly SPS is crashing due to memory issue. Java is crashing and we can see the following errors in the nohup.out

[ERROR] - Exception in thread "ajp-bio-8009-Acceptor-0"
[ERROR] - java.lang.OutOfMemoryError: unable to create new native thread
[ERROR] - at java.lang.Thread.start0(Native Method)
[ERROR] - at java.lang.Thread.start(Thread.java:640)
[ERROR] - at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
[ERROR] - at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
[ERROR] - at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
[ERROR] - at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Unknown Source)
[ERROR] - at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Unknown Source)
[ERROR] - at org.apache.tomcat.util.net.JIoEndpoint.processSocket(Unknown Source)
[ERROR] - at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(Unknown Source)
[ERROR] - at java.lang.Thread.run(Thread.java:662)

Environment:

SiteMinder Secure Proxy Server WebAgent, Version 12.51, Update HF-01, Label 977
SM Policy server version: 12.5 CR04
OS: Linux

Cause:

There is a max amount of memory that could be allocated to the JVM and we should not use too much memory on the server. You have to remember that the SPS is a 32bit process.

Resolution:

This is a tunning excercise and number can vary on the environment / usage of SPS. In this specific issue, we changed the JVM_MEM_OPTS in the provxy_server.sh (-mx)
Before you had JVM_MEM_OPTS="-ms256m -mx3348m"
After JVM_MEM_OPTS="-ms256m -mx2048m" settings, no more crashes