healthindex probe issues - probe is green, but does not appear to be running - has no port/pid

Document ID : KB000033560
Last Modified Date : 14/02/2018
Show Technical Document Details

Problem:
After restoring the environment from a server crash, we noticed that the health_index probe was not running properly. It showed up as green, but it had only a PID - no port. We also noticed the health_index queues backing up. After trying to redeploy the probe, it wouldn't obtain a port OR pid and the queues were no longer present in the hub.

We noticed the following errors in the health_index log files:
>>Error injecting constructor, com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'cache/health_index_v2' with mode=rw?
>>Cannot load database's configuration. The database seems to be corrupted.?
>>The process cannot access the file because another process has locked a portion of the file?


Based on the above, we seem to be looking at either a corruption or file access issue.?

Environment:
UIM - 8.3.1?
UMP - 8.3.1?
Health_index - 1.11?
Hub - 7.80
Controller - 7.80?
OS - Windows 2012 R2

Errors (from health_index.log):
66) Error injecting constructor, com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'cache/health_index_v2' with mode=rw?
at com.ca.analytics.health.db.HealthIndexDatabase.<init>(HealthIndexDatabase.java:21)?
at com.ca.analytics.health.HealthIndexModule.configure(HealthIndexModule.java:46)?
while locating com.ca.analytics.health.db.HealthIndexDatabase?
for parameter 0 at com.ca.analytics.health.db.dao.MetricConfigDao.<init>(MetricConfigDao.java:32)?
at com.ca.analytics.health.HealthIndexModule.configure(HealthIndexModule.java:49)?
while locating com.ca.analytics.health.db.dao.MetricConfigDao?
for parameter 4 at com.ca.analytics.health.config.HealthIndexCtdConfiguration.<init>(HealthIndexCtdConfiguration.java:43)?
at com.ca.analytics.health.config.HealthIndexCtdConfiguration.class(HealthIndexCtdConfiguration.java:43)?
while locating com.ca.analytics.health.config.HealthIndexCtdConfiguration?
while locating com.ca.analytics.dmc.config.CtdConfiguration?
for parameter 1 at com.ca.analytics.dme.config.DerivedMetricsConfig.<init>(DerivedMetricsConfig.java:28)?
at com.ca.analytics.dme.config.DerivedMetricsConfig.class(DerivedMetricsConfig.java:28)?
while locating com.ca.analytics.dme.config.DerivedMetricsConfig?
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'cache/health_index_v2' with mode=rw?
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:220)?
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:244)?
at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:56)?
at com.orientechnologies.orient.object.db.OObjectDatabaseTx.open(OObjectDatabaseTx.java:126)?

Caused by: com.orientechnologies.orient.core.exception.OSerializationException: Cannot load database's configuration. The database seems to be corrupted.?
at com.orientechnologies.orient.core.storage.impl.local.OStorageConfigurationSegment.load(OStorageConfigurationSegment.java:84)?
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:166)?
... 56 more?

Caused by: java.io.IOException: The process cannot access the file because another process has locked a portion of the file?
at java.io.RandomAccessFile.read0(Native Method)?
at java.io.RandomAccessFile.read(Unknown Source)

Resolution:
1. Please ensure that any AntiVirus software has an exception configured for the Nimsoft directory (best practice).?
2. Once this is confirmed, delete the probe in IM (right click + delete) and then delete the health index directory.?
3. Redeploy the probe and monitor it for improvement. Be sure to check if the queues are created in the hub GUI.?

This should clear out any corruption and allow the probe to come online and create the health_index queues.

keywords: healthindex probe issue issues probe green not running no port PID pid port/pid