Trace View shows incorrect cross-process calls order (downstream agent before upstream agent) compared to Sequence View

Document ID : KB000008071
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

Using cross-process/cross-jvm transaction tracing to correlate transactions across multiple JVMs/agents. The Trace View shows downstream agent method calls first when it should show upstream method calls first like the Sequence View does.

Environment:
APM 9.x, 10.x
Cause:

The Trace View will display traces in the order of the agent server timestamp.

If click on each method in the Trace View it will show the Timestamp for the method and it should then be seen that the downstream trace timestamp is earlier than the upstream trace timestamp. That is caused by the system clock on the downstream agent server being behind that of the upstream agent server. The result is that the downstream method appears before the upstream method in the Trace View and is working as designed. 

The Sequence View uses a different mechanism to establish the caller/callee order and was designed specifically to overcome ordering problems in the Trace View due to unsynchronised agent server system clocks.

Resolution:

Either ensure that all agent servers have synchronised system clocks or always use the Sequence View to view reliable call order information.

Additional Information:

Using Trace and Analyze Events > How to Trace Cross-Process Transaction > Cross-Process Traces Time Alignment

Usually traces display in an order that is based on the system clock where they originated