The MOM reports clock skew errors for a collector, but the system clocks are in sync.

Document ID : KB000004029
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

 The Enterprise Manager log for the MOM is reporting clock skew errors for a collector like the following:

5/13/13 11:13:17.402 AM CEST [WARN] [Collector 10.10.10.10@5001
[Manager.Cluster] Collector clock is too far skewed from MOM. Collector clock 
is skewed from MOM clock by 7,735 ms. The maximum allowed skew is 3,000 ms. 
Please change the system clock on the collector EM.

 But when the system clock on the offending EM is checked, it shows the same time as the MOM clock.

Environment:
All APM releases
Resolution:

 If the last update time of the IntroscopeEnterpriseManager.log file on the collector does not match the timestamp of the last entry, check if there is a genuine discrepancy in the time the JVM reports in the log compared to the system time.

A simple script can be run to put a timestamp into the log file approximately once a second in parallel with the normal EM logging, as follows:

$ while true 
> do 
> date >> IntroscopeEnterpriseManager.log 
> sleep 1 
> done

The output produced may show that the time reported by the JVM is not in sync with the time reported by the date command (which uses the system clock).

Wed May 29 13:06:45 METDST 2013 
Wed May 29 13:06:46 METDST 2013 
Wed May 29 13:06:47 METDST 2013 
5/29/13 01:06:30.192 PM CEST [INFO] [PO Route Down Executor] 
[Manager.Cluster] MOM Introscope Enterprise Manager disconnected 
Wed May 29 13:06:48 METDST 2013 
Wed May 29 13:06:49 METDST 2013

In the instance reported, the Java VM version in use was "Java HotSpot(TM) Server VM 1.6.0.06" from Hewlett-Packard Co.

Research showed that time sync problems have been reported on this Java platform in HP IA64 environments.

To correct this, use the JVM option -XX:+UseGetTimeOfDay

This option can be defined by including it in the value defined for the property lax.nl.java.option.additional in the IntroscopeEnetrpriseManager.lax file.