Oracle error: Fatal Exception while getting database connection - some probes starting slowly or not at all

Document ID : KB000034251
Last Modified Date : 14/02/2018
Show Technical Document Details
After upgrading or installing CAUIM 8.0, the following symptoms may be observed by customers using Oracle 11gR2 as their backend database platform with Red Hat Linux 6.x on the primary hub.

1. ?Probes ace, nis_server, maintenance_mode, and sla_engine may appear to take a very long time to start.
2. Other probes may not start at all (and eventually turn red), including ace, discovery_server, fault_correlation_engine, maintenance_mode, mpse, nis_server, relationship_services, sla_engine, and ugs.

In the logs for the probes which are having trouble, the following error will be seen:

Sep 12 15:39:39:899 FATAL [main, com.nimsoft.probe.service.ace.DatabaseProvider] Exception while getting database connection:
Sep 12 15:39:39:903 FATAL [main, com.nimsoft.probe.service.ace.DatabaseProvider] org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: Connection reset)
??? at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
??? at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
??? at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
??? at com.nimsoft.probe.service.ace.DatabaseProvider.getConnection(DatabaseProvider.java:75)
??? at com.nimsoft.probe.service.ace.DBAccess.instance(DBAccess.java:149)
??? at com.nimsoft.probe.service.ace.ArchiveSync.<init>(ArchiveSync.java:47)
??? at com.nimsoft.probe.service.ace.ACEProbe.run(ACEProbe.java:136)
??? at com.nimsoft.probe.service.ace.ACEProbe.main(ACEProbe.java:42)
Caused by: java.sql.SQLRecoverableException: IO Error: Connection reset
??? at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:421)
??? at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:531)
??? at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:221)
??? at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
??? at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
??? at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
??? at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
??? at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
??? at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
??? ... 7 more
Caused by: java.net.SocketException: Connection reset
??? at java.net.SocketOutputStream.socketWrite(Unknown Source)
??? at java.net.SocketOutputStream.write(Unknown Source)
??? at oracle.net.ns.DataPacket.send(DataPacket.java:199)
??? at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:211)
??? at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:227)
??? at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
??? at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
??? at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
??? at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
??? at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
??? at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1179)
??? at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
??? at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
??? at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
??? at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:366)
??? at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:752)
??? at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:359)
??? ... 15 more

The resolution to this issue is as follows:

1. Check the Raw Config of the affected probe. (discovery_server is shown below, but this could affect any of the java-based probes.)

the java_opts configuration option in the startup/opt section is "-server -XX:ErrorFile=./hs_err_pid.log".



2. Change the java_opts configuration?option?to "?-Djava.security.egd=file:///dev/urandom" in the startup/opt section.?


3. Restart the entire Robot/hub and the issue should be resolved.

This will be fixed permanently in UIM 8.1.