End users report that the management server lost connection with database and is not available for end users. When DB is verified it is working normally. One of the sessions within DB has an error message that connection timed out and potential issue with proxy or network.
2018-01-01 11:55:52,130 [ContinueStageExecutionTask-137282] ERROR (org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144) - Cannot get a connection, pool error Timeout waiting for idle object
2018-01-01 11:55:52,131 [QuartzScheduler_rocReleasesScheduler-NON_CLUSTERED_MisfireHandler] ERROR (org.springframework.scheduling.quartz.LocalDataSourceJobStore:3940) - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'springNonTxDataSource.rocReleasesScheduler': java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
To correct, the Release Management services was stopped and restarted, then the application started functioning normally without making any changes or restart on DB.
One possible theory to what may cause the issue may be the database connection pool settings, which can be adjusted as follows:
In wrapperContext.xml file, located under:
<property name="maxTotal" value="50"/>
Increasing "maxTotal" to a higher value, say 100
This is the value for application to allow x number of connections to DB, where x is "100"
A possible cause is that the connection pool was exhausted due to the increase in number of deployments in the environment. It may even exhaust in future if there is a spike or gradual increase in number of deployments and the solution is to increase the pool from application end.
Release Automation uses database connection pool to manage connections with the database. When an action with in RA requests data from the database, it will go through a persistence layer which will then query the database. When it queries the database, it uses up all connection that is retrieved from the connection pool.
The timeout happens when the connection pool has been exhausted and cause the above error recorded in the logs "java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object"
The suggested solution is to increase the number of connections that can mitigate this problem, increasing the maxTotal value from 50 to 100 in wrapperContext.xml file. The suggested value is ideal and should not have any side effects as far as increased overhead on resources.