Policy Server crashing after upgrade when loading JVM

Document ID : KB000117859
Last Modified Date : 19/10/2018
Show Technical Document Details
Question:
We have upgraded a Policy Server to R12.7, and everything went ok. When we start the Policy Server, it works fine for some minutes, but suddenly we notice the users cannot login anymore after some time working fine. When we check the logs we see the Policy Server has restarted. This happens continuously, and the time since the Policy Server starts, and it crashes, is different every time.

The last lines of the smps.log shows that it is loading the JVM:

17/04/2018|11:52:33.617|11:52:33|1484|139722905077504|SmJVMSupport.cpp:274|GetJVMEnv|SmJVMSupport: JVM library loaded sucessfully| 
17/04/2018|11:52:33.618|11:52:33|1484|139722905077504|SmJVMSupport.cpp:278|LogMessage:INFO:[sm-JavaApi-01030] SmJVMSupport: Using the following JRE: /opt/myjdk/jre/| 
17/04/2018|11:52:33.618|11:52:33|1484|139722905077504|SmJVMSupport.cpp:283|LogMessage:INFO:[sm-JavaApi-01040] SmJVMSupport: Loaded the following JVM library: /opt/myjdk/jre//lib/amd64/server/libjvm.so| 
17/04/2018|11:52:33.618|11:52:33|1484|139722905077504|SmJVMSupport.cpp:310|GetJVMEnv|SmJVMSupport: Attempting to retrieve existing JVM instance| 
17/04/2018|11:52:33.618|11:52:33|1484|139722905077504|SmJVMSupport.cpp:323|GetJVMEnv|SmJVMSupport: Attempting to create new JVM instance| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #0: -server| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #1: -Xbootclasspath/p:/opt/CA/siteminder/bin/endorsed/xercesImpl.jar:/opt/CA/siteminder/bin/endorsed/xml-apis.jar:/opt/CA/siteminder/bin/endorsed/resolver.jar:/opt/CA/siteminder/bin/endorsed/serializer.jar| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #2: -XX:+AllowUserSignalHandlers| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #3: -Xrs| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #4: -Xms128m| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #5: -Xmx256m| 
17/04/2018|11:52:33.620|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #6: -DNETE_PS_ROOT=/opt/CA/siteminder| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #7: -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #8: -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #9: -Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #10: -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #11: -Djava.endorsed.dirs=/opt/CA/siteminder/bin/endorsed| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #12: -Djava.class.path=/opt/CA/siteminder/resources:/opt/CA/siteminder/config/properties:/opt/CA/siteminder/bin/jars/smbootstrap.jar:/opt/CA/siteminder/bin/jars:/| 
17/04/2018|11:52:33.621|11:52:33|1484|139722905077504|SmJVMSupport.cpp:424|GetJVMEnv|JVM Option #13: opt/CA/siteminder:/opt/CA/siteminder/config/properties:/opt/CA/sitemin|


We have verified the JDK was successfully installed and upgraded (we had 32-bit Java in R12.52 SP1, and now we have 64-bit Java for R12.7). We have re-installed Java, and used different versions, but the problem remains the same.

How can we solve this issue?
Environment:
Policy Server R12.7 (upgraded from R12.52 SP1)
Answer:
The problem is caused because during the upgrade process the JVMOptions.txt was modified manually to maintain the custom class path, and when saving the txt file line breaks were placed on the longest lines. This causes the crash on Java when loading a schema that requires Java.

To solve the issue you need to edit JVMOptions.txt and ensure there are no line breaks in the middle of the custom path.

For example, moving from the following contents:
-server
-Xbootclasspath/p:/opt/CA/siteminder/bin/endorsed/xercesImpl.jar:/opt/CA/siteminder/bin/endorsed/xml-apis.jar:/opt/CA/siteminder/bin/endorsed/resolver.jar:/opt/CA/siteminder/bin/endorsed/serializer.jar
-XX:+AllowUserSignalHandlers
-Xrs
-Xms128m
-Xmx256m
-DNETE_PS_ROOT=/opt/CA/siteminder
-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
-Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Djava.endorsed.dirs=/opt/CA/siteminder/bin/endorsed
-Djava.class.path=/opt/CA/siteminder/resources:/opt/CA/siteminder/config/properties:/opt/CA/siteminder/bin/jars/smbootstrap.jar:/opt/CA/siteminder/bin/jars:/
opt/CA/siteminder:/opt/CA/siteminder/config/properties:/opt/CA/sitemin
der/config
-Djava.util.logging.config.file=/opt/CA/siteminder/config/properties/logging.properties


To:
-server
-Xbootclasspath/p:/opt/CA/siteminder/bin/endorsed/xercesImpl.jar:/opt/CA/siteminder/bin/endorsed/xml-apis.jar:/opt/CA/siteminder/bin/endorsed/resolver.jar:/opt/CA/siteminder/bin/endorsed/serializer.jar
-XX:+AllowUserSignalHandlers
-Xrs
-Xms128m
-Xmx256m
-DNETE_PS_ROOT=/opt/CA/siteminder
-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
-Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Djava.endorsed.dirs=/opt/CA/siteminder/bin/endorsed
-Djava.class.path=/opt/CA/siteminder/resources:/opt/CA/siteminder/config/properties:/opt/CA/siteminder/bin/jars/smbootstrap.jar:/opt/CA/siteminder/bin/jars:/opt/CA/siteminder:/opt/CA/siteminder/config/properties:/opt/CA/siteminder/config
-Djava.util.logging.config.file=/opt/CA/siteminder/config/properties/logging.properties