Catalog 14.1 stops responding with DB errors

Document ID : KB000097611
Last Modified Date : 22/05/2018
Show Technical Document Details
Issue:
Catalog 14.1 stops responding with DB errors.
The view.log shows errors like:

2018/05/21 17.15.47.688 ERROR [Rule Action Executor-205] [TransactionInterceptor] Application exception overridden by rollback exception
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connection ConnectionID:2231 ClientConnectionId: de861844-486f-4815-a9b9-bca1ce8880c6 is closed.
Error Code: 0
Call: SELECT t0.form_entity_id, t0.form_entity_type, t0.created, t0.created_by, t0.last_modified, t0.last_modified_by, t0.form_entity_name, t0.form_entity_parent_id, t0.form_entity_path, t0.form_entity_position, t0.is_system, t0.tenant_id FROM usm_form_entities t0, usm_form_entities t1 WHERE ((t0.form_entity_parent_id = t1.form_entity_id) AND ((CHARINDEX(?, t1.form_entity_path) <> ?) OR (t1.form_entity_id = ?))) ORDER BY t0.form_entity_position ASC
    bind => [3 parameters bound]
Query: ReadAllQuery(name="children" referenceClass=FDObjectJpaEntity sql="SELECT t0.form_entity_id, t0.form_entity_type, t0.created, t0.created_by, t0.last_modified, t0.last_modified_by, t0.form_entity_name, t0.form_entity_parent_id, t0.form_entity_path, t0.form_entity_position, t0.is_system, t0.tenant_id FROM usm_form_entities t0, usm_form_entities t1 WHERE ((t0.form_entity_parent_id = t1.form_entity_id) AND ((CHARINDEX(?, t1.form_entity_path) <> ?) OR (t1.form_entity_id = ?))) ORDER BY t0.form_entity_position ASC")
    at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:674)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
    at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:298)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:299)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:694)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2738)
    at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2691)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:495)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
    at org.eclipse.persistence.mappings.CollectionMapping.executeBatchQuery(CollectionMapping.java:974)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.extractResultFromBatchQuery(ForeignReferenceMapping.java:643)
    at org.eclipse.persistence.mappings.CollectionMapping.extractResultFromBatchQuery(CollectionMapping.java:955)
    at org.eclipse.persistence.internal.indirection.BatchValueHolder.instantiateForUnitOfWorkValueHolder(BatchValueHolder.java:81)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:160)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:252)
    at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:423)
    at org.eclipse.persistence.indirection.IndirectList$1.<init>(IndirectList.java:551)
    at org.eclipse.persistence.indirection.IndirectList.listIterator(IndirectList.java:550)
    at org.eclipse.persistence.indirection.IndirectList.iterator(IndirectList.java:514)
    at com.ca.usm.formsdesigner.base.dao.impl.FormDesignerDaoImpl.loadAllChildren(FormDesignerDaoImpl.java:479)
    at com.ca.usm.formsdesigner.base.dao.impl.FormDesignerDaoImpl.getFormAndChildren(FormDesignerDaoImpl.java:127)
    at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy63.getFormAndChildren(Unknown Source)
    at com.ca.share.formsdesigner.base.service.impl.FormExportServiceImpl.getForms(FormExportServiceImpl.java:82)
    at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy69.getForms(Unknown Source)
    at com.ca.usm.billing.ServiceManager.RequestItemForm.getFDFormOrderedIdLabelMap(RequestItemForm.java:629)
    at com.ca.usm.billing.ServiceManager.RequestItemForm.getFormFieldTypes(RequestItemForm.java:999)
    at com.ca.usm.billing.ServiceManager.RequestItemForm.modifyFDRequestFormValues(RequestItemForm.java:1054)
    at com.ca.usm.billing.ServiceManager.RequestItemForm.modifyRequestFormValues(RequestItemForm.java:1026)
    at com.ca.usm.billing.ContentEngine.CRequest.viewRequestFormNew(CRequest.java:7268)
    at com.ca.usm.billing.ContentEngine.CRequest.getRequestItems(CRequest.java:6239)
    at com.ca.usm.billing.ContentEngine.CRequest.editRequest(CRequest.java:2167)
    at sun.reflect.GeneratedMethodAccessor318.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.ca.usm.contentInterface.PluginMethod.invoke(DefaultContentEngine.java:172)
    at com.ca.usm.contentInterface.DefaultContentEngine.invokeMethod(DefaultContentEngine.java:98)
    at com.ca.usm.contentInterface.DefaultContentEngine.processRequest(DefaultContentEngine.java:57)
    at com.ca.usm.contentInterface.DefaultContentEngine.processRequest(DefaultContentEngine.java:17)
    at com.ca.usm.util.ViewManager.executeRequest(ViewManager.java:92)
    at com.ca.usm.util.ViewManager.doRequest(ViewManager.java:51)
    at com.ca.usm.util.ViewManager.processRequest(ViewManager.java:28)
    at com.ca.usm.producer.DocumentGenerator.processRequest(DocumentGenerator.java:563)
    at com.ca.usm.producer.DocumentGenerator.generateDocument(DocumentGenerator.java:270)
    at com.ca.usm.producer.DocumentGenerator.generateDocument(DocumentGenerator.java:241)
    at com.ca.usm.producer.DomProcessor.renderPage(DomProcessor.java:205)
    at com.ca.usm.producer.DomProcessor.service(DomProcessor.java:123)
    at com.ca.usm.producer.framework.WPFFramework.processRequest(WPFFramework.java:165)
    at com.ca.usm.producer.framework.WPFFramework.processRequest(WPFFramework.java:29)
    at com.ca.usm.billing.ServiceManager.Request.getFormatedResult(Request.java:5708)
    at com.ca.usm.billing.ServiceManager.Request.getFormattedRequest(Request.java:5679)
    at com.ca.usm.billing.ServiceManager.EmailRequest.getEmailMessage(EmailRequest.java:864)
    at com.ca.usm.soap.services.RequestServiceImplUtil.sendRequestEmail(RequestServiceImplUtil.java:844)
    at com.ca.usm.ruleEngine.action.RequestEmailAction.sendEmail(RequestEmailAction.java:486)
    at com.ca.usm.ruleEngine.action.RequestEmailAction.run(RequestEmailAction.java:114)
    at com.ca.usm.ruleEngine.receiver.RuleActionExecutor.executeAction(RuleActionExecutor.java:60)
    at com.ca.usm.ruleEngine.receiver.RuleActionExecutor.run(RuleActionExecutor.java:30)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLException: Connection ConnectionID:2231 ClientConnectionId: de861844-486f-4815-a9b9-bca1ce8880c6 is closed.
    at org.apache.commons.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:398)
    at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:279)
    at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1556)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1505)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseCall.prepareStatement(DatabaseCall.java:778)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
    ... 92 more
2018/05/21 17.15.47.688 ERROR [ajp-bio-8009-exec-3] [TransactionInterceptor] Application exception overridden by rollback exception
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connection ConnectionID:2230 ClientConnectionId: fa773232-130a-440c-8bd5-cfcb57655dbc is closed.
Error Code: 0
Environment:
Catalog 14.1. However this may reoccur after installing cp5
Cause:
This may occur again because cp5 replaces the %USM_HOME%\view\webapps\usm\WEB-INF\applicationcontext.xml file
Resolution:
Edit %USM_HOME%\view\webapps\usm\WEB-INF\applicationcontext.xml file and edit the section:

bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    <property name="driverClassName" value="${DBSource.driver}" />
    <property name="url" value="${DBSource.url}" />
    <property name="username" value="${DBSource.username}" />
    <property name="password" value="${DBSource.password}" />
    <property name="logAbandoned" value="true" />
    <property name="removeAbandoned" value="true" />
    <property name="maxWait" value="1800000" />

To change it to (including the new line at the end):

bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    <property name="driverClassName" value="${DBSource.driver}" />
    <property name="url" value="${DBSource.url}" />
    <property name="username" value="${DBSource.username}" />
    <property name="password" value="${DBSource.password}" />
    <property name="logAbandoned" value="false" />
    <property name="removeAbandoned" value="true" />
    <property name="maxWait" value="1800000" />
    <property name ="maxActive" value ="100" />