Vertica health check failed

Document ID : KB000117870
Last Modified Date : 19/10/2018
Show Technical Document Details
Issue:
The following error is seen int he DA's karaf.log:

WARN  | heduler_Worker-1  | VerticaHealthCheckImpl | tica.impl.VerticaHealthCheckImpl   83 | re.database.irep.vertica |       | Vertica has fallen behind by at least 60 minutes: 294
WARN  | heduler_Worker-1 | trInstancePartitionRetentionImpl | trInstancePartitionRetentionImpl   78 | re.database.irep.vertica |      
| *** Vertica not in valid state, aborting attribute retention task. ***
ERROR | heduler_Worker-1  | ExceptionLog | .ca.im.core.util.ExceptionLogger   99 | m.ca.im.common.core.util |      
|A NEW application exception occurred (Key=xxx) :
Date retention task Attribute instance partition retention task failed at xxx : Attribute instance partition job - vertica health check failed
java.lang.Exception: Attribute instance partition job - vertica health check failed
    at com.ca.im.dm.core.database.dao.item.vertica.impl.AttrInstancePartitionRetentionImpl.executeWithRetentionSettings(AttrInstancePartitionRetentionImpl.java:79)
    at sun.reflect.GeneratedMethodAccessor751.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_144]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)[99:org.springframework.osgi.core:1.2.1]
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)[99:org.springframework.osgi.core:1.2.1]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)[99:org.springframework.osgi.core:1.2.1]
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)[99:org.springframework.osgi.core:1.2.1]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)[99:org.springframework.osgi.core:1.2.1]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[95:org.apache.servicemix.bundles.spring-aop:3.2.11.RELEASE_1]
    at com.sun.proxy.$Proxy190.executeWithRetentionSettings(Unknown Source)[:]
    at com.ca.im.dm.core.aggregator.retention.impl.DataRetentionJob.execute(DataRetentionJob.java:72)[224:com.ca.im.data-manager.core.aggregator.retention.impl:3.5.0.RELEASE-191]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:206)[224:com.ca.im.data-manager.core.aggregator.retention.impl:3.5.0.RELEASE-191]
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)[224:com.ca.im.data-manager.core.aggregator.retention.impl:3.5.0.RELEASE-191]
Environment:
CAPM 3.x
Cause:
Mis-configured clocks on the DR nodes
Resolution:
To resolve do the following

1. Sync the clock on all DR nodes
2. Monitor for the error over a few days and if error persists stop and start the DA service

You should no longer see the error at this point