EM failed to start. org.rocksdb.RocksDBException: IO error: lock data/metadata/LOCK: Resource temporarily unavailable

Document ID : KB000126467
Last Modified Date : 11/02/2019
Show Technical Document Details
Issue:
EM failed to start. org.rocksdb.RocksDBException: IO error: lock data/metadata/LOCK: Resource temporarily unavailable


 
Environment:
Applies to any APM 10.5.0, 10.5.1
Cause:
The issue is due transaction log corruption due to unclean shutdown of the Introscope EM (kill, lack of space, EM crash).

This error indicated that the metadata for the SmartStor database on this EM has either corrupted or being locked for some other reasons (for eg. concurrent access), and is preventing the EM to start up successfully.
 
Resolution:
To address the file corruption root cause, please do the following:

1. Back up the current <EM_home>\data\metadata directory
2. Delete everything inside the <EM_Home>\data\metadata directory
3. Start the EM

Deleting the metadata db on the EM should have no significant impact, as the main agents SmartStor data are stored on the collectors instead.

Note: It may be due to concurrent access of the metadata files by other existing EM process, and rebooting the server has seemed to released the lock