APM Command Center does not clean inactive agents.

Document ID : KB000046401
Last Modified Date : 14/02/2018
Show Technical Document Details

 Problem: 

 APM Command Center does not remove old Agents from the environment.

 

 Environment:  

 Introscope from 10.x onward.

  

 Cause: 

 APM Command Center has a job that removes old Agents from the database, but sometimes this database get corrupted and breaks:

 INFO 28859 [taskScheduler-1] --- c.c.a.a.c.m.ConfigServerHousekeeping     : Running ACC Configuration Server Housekeeping Task
 INFO 28859 [taskScheduler-1] --- c.c.a.a.c.m.AgentHousekeepingBean        : Deleting pje-jb-testes:JBoss:<NameOfTheAgent>
 INFO 28859 [taskScheduler-1] --- c.c.a.a.c.m.AgentHousekeepingBean        : Deleting pje-jb-testes:JBoss:<NameOfTheAgent>
 ERROR 28859 [taskScheduler-1] --- o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task.

org.springframework.orm.jpa.JpaSystemException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.h2.jdbc.JdbcSQLException: Referential integrity constraint violation: "FK_DATA_FILE_DIAG_REPORT_ID: PUBLIC.DATA_FILE FOREIGN KEY(DIAG_REPORT_ID) REFERENCES PUBLIC.DIAG_REPORT(ID) (1)"; SQL statement:
DELETE FROM DIAG_REPORT WHERE (ID = ?) [23503-187]
Error Code: 23503
Call: DELETE FROM DIAG_REPORT WHERE (ID = ?)
    bind => [1 parameter bound]
Query: DeleteObjectQuery(com.ca.apm.acc.configserver.core.domain.DiagReport@40298437); nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.h2.jdbc.JdbcSQLException: Referential integrity constraint violation: "FK_DATA_FILE_DIAG_REPORT_ID: PUBLIC.DATA_FILE FOREIGN KEY(DIAG_REPORT_ID) REFERENCES PUBLIC.DIAG_REPORT(ID) (1)"; SQL statement:
DELETE FROM DIAG_REPORT WHERE (ID = ?) [23503-187]
Error Code: 23503
Call: DELETE FROM DIAG_REPORT WHERE (ID = ?)
    bind => [1 parameter bound]
Query: DeleteObjectQuery(com.ca.apm.acc.configserver.core.domain.DiagReport@40298437)
    at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:415)
    at org.springframework.orm.jpa.DefaultJpaDialect.translateExceptionIfPossible(DefaultJpaDialect.java:122)
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:417)
    at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:59)
    at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:147)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodIntercceptor.invoke(CrudMethodMetadataPostProcessor.java:122)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy129.findByDiagReportId(Unknown Source)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean.doDeleteReport(AgentHousekeepingBean.java:158)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean.doDeleteOldAgents(AgentHousekeepingBean.java:198)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean.doDeleteOld(AgentHousekeepingBean.java:287)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean.doAgentHousekeeping(AgentHousekeepingBean.java:136)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean$$FastClassBySpringCGLIB$$e4dd5e6e.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at com.ca.apm.acc.configserver.maintenance.AgentHousekeepingBean$$EnhancerBySpringCGLIB$$7fe36aef.doAgentHousekeeping(<generated>)
    at com.ca.apm.acc.configserver.maintenance.ConfigServerHousekeeping.doHousekeeping(ConfigServerHousekeeping.java:47)
    at sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.h2.jdbc.JdbcSQLException: Referential integrity constraint violation: "FK_DATA_FILE_DIAG_REPORT_ID: PUBLIC.DATA_FILE FOREIGN KEY(DIAG_REPORT_ID) REFERENCES PUBLIC.DIAG_REPORT(ID) (1)"; SQL statement:
DELETE FROM DIAG_REPORT WHERE (ID = ?) [23503-187]
Error Code: 23503
Call: DELETE FROM DIAG_REPORT WHERE (ID = ?)
    bind => [1 parameter bound]
Query: DeleteObjectQuery(com.ca.apm.acc.configserver.core.domain.DiagReport@40298437)
    at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
    at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)
    at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:110)
    at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:74)
    at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:99)
    at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:90)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:415)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:393)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$DefaultMethodInvokingMethodInterceptor.invoke(RepositoryFactorySupport.java:506)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    ... 34 common frames omitted
Caused by: org.eclipse.persistence.exceptions.DatabaseException:
Internal Exception: org.h2.jdbc.JdbcSQLException: Referential integrity constraint violation: "FK_DATA_FILE_DIAG_REPORT_ID: PUBLIC.DATA_FILE FOREIGN KEY(DIAG_REPORT_ID) REFERENCES PUBLIC.DIAG_REPORT(ID) (1)"; SQL statement:
DELETE FROM DIAG_REPORT WHERE (ID = ?) [23503-187]
Error Code: 23503
Call: DELETE FROM DIAG_REPORT WHERE (ID = ?)
    bind => [1 parameter bound]
Query: DeleteObjectQuery(com.ca.apm.acc.configserver.core.domain.DiagReport@40298437)
    at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
    at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
    at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:298)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
    at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.deleteObject(DatasourceCallQueryMechanism.java:210)
    at org.eclipse.persistence.internal.queries.StatementQueryMechanism.deleteObject(StatementQueryMechanism.java:104)
    at org.eclipse.persistence.queries.DeleteObjectQuery.executeDatabaseQuery(DeleteObjectQuery.java:218)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:798)
    at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)
    at org.eclipse.persistence.queries.DeleteObjectQuery.executeInUnitOfWorkObjectLevelModifyQuery(DeleteObjectQuery.java:119)
    at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1737)
    at org.eclipse.persistence.internal.sessions.CommitManager.deleteAllObjects(CommitManager.java:336)
    at org.eclipse.persistence.internal.sessions.CommitManager.deleteAllObjects(CommitManager.java:290)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1444)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithPreBuiltChangeSet(UnitOfWorkImpl.java:1587)
    at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:452)
    at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
    ... 51 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLException: Referential integrity constraint violation: "FK_DATA_FILE_DIAG_REPORT_ID: PUBLIC.DATA_FILE FOREIGN KEY(DIAG_REPORT_ID) REFERENCES PUBLIC.DIAG_REPORT(ID) (1)"; SQL statement:
DELETE FROM DIAG_REPORT WHERE (ID = ?) [23503-187]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    at org.h2.message.DbException.get(DbException.java:179)
    at org.h2.message.DbException.get(DbException.java:155)
    at org.h2.constraint.ConstraintReferential.checkRow(ConstraintReferential.java:426)
    at org.h2.constraint.ConstraintReferential.checkRowRefTable(ConstraintReferential.java:443)
    at org.h2.constraint.ConstraintReferential.checkRow(ConstraintReferential.java:318)
    at org.h2.table.Table.fireConstraints(Table.java:920)
    at org.h2.table.Table.fireAfterRow(Table.java:938)
    at org.h2.command.dml.Delete.update(Delete.java:100)
    at org.h2.command.CommandContainer.update(CommandContainer.java:78)
    at org.h2.command.Command.executeUpdate(Command.java:254)
    at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:157)
    at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:143)
    at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
    ... 76 common frames omitted

 

 

 Workaround:

 To resolve, replace the Database files from the ACC folder and do the following:

 1- Stop APM Command Server service.

 2- Navigate through APMCommandServer/data folder and backup "acc_data_h2.mv.db" and "acc_data_h2.trace.db" files.

 3- Start APM Command Server service.

 You should not see the old Agents anymore.

 

 

 Additional Information:

 For more APM Command Center errors, see the following reference:

 https://docops.ca.com/ca-apm/10-2/en/administrating/ca-apm-command-center/appendix/api-error-messages