What are names of the headers used for cross correlation traces?

Document ID : KB000047559
Last Modified Date : 14/02/2018
Show Technical Document Details

 Introduction: 

   This is an under-documented area that first came up in the Community post. The Agent sends a set of information as part of protocol headers when a cross-process call is made by the application. This information serves to do transaction trace correlation, knowing who the caller-callee information in cross-process traces, trace propagation flags among others.

 Question: 

  What are the actual names of the headers are for cross correlation between JVM's? I see the header decorator in the agent profile talks about the x-wily-info parameter to enable correlation between Introscope and CEM. Is this the same GUID that is being used between JVM's or is there something different? 

 

 Environment:  All APM 10.x Releases 

 

 Answer: 

  A user transaction traverses multiple JVM processes/threads. For each JVM process/thread, APM creates a transaction trace "wedding cake". These sets of "wedding cakes" all share the same GUID called as "CorProcessData"  When looking at the properties of a Transaction Trace, all the ones that start with the prefix "Cor" are used to stitch Transaction Traces together. All the other ones are either not related or more cosmetic.

 In the transaction trace, the GUID is called CorProcessData. The HTTP header that is transmitted over the wire is called CorID and includes a lot more, e.g. the sequence number.

Format: CorID = <GUID>,<sequenceID>,<propagation flag>,<dep propagation flag>, <outgoingCallerNodeID>,<outgoingCallerNodeType>,<optionalParams>

  • GUID (CorCrossProcessData): The GUID that defines which traces are related to each other. Defines the set of traces that are part of the same user transaction.

  • sequenceId: obsolete field
  • propagation flag: The flag is used to instruct the downstream agents whether they have to also send a trace or not. e.g. a filter session is active on a upstream agent and traces were matched in upstream agent, hence we could set this flag to 1 to trigger trace from downstream agents automatically
  • dep propagation flag: The flag is used for SPM dependency map (now obsolete)
  • outgoingCallerNodeID: fully qualified name of the component/node that made the outgoing cross process call
  • outgoingCallerNodeType: type of the  component/node that made the outgoing cross process call (type = 2, indicated client node)
  • optionalParams: encoded set of key/value pairs coming from a hashmap. Some of the key fields include 
    • TxnTraceID: TxnTraceId and CallerTxnTraceId are other GUIDs that help to identify the caller-callee relationship between the wedding cake traces. All these guids are sent on the cross-process correlation headers  
    • CallerTxnTraceId: see the note for TxnTraceID
    • CallerTimestamp: When the client made a outgoing client socket call, it denotes the timestamp of the component

Example:

GET /MathProxy/rest/hello/mathcomplex?operation=mean&values=1,2,3,4,5,6 HTTP/1.1

CorID: B63D31070A82B2D560A6C783CD3209CA,1:1,0,0,,,AgAABClIQgAAAAFGAAAAAQAAABFqYXZhLnV0aWwuSGFzaE1hcAAAAAlIQgAAAAJGAAAAAgAAABBqYXZhLmxhbmcuU3RyaW5nABFBcHBNYXBDYWxsZXJBZ2VudEhCAAAAA0UAAAACAApNYXRoQ2xpZW50SEIAAAAERQAAAAIAEEFwcE1hcENhbGxlclR5cGVIQgAAAAVFAAAAAgAKV2ViU2VydmljZUhCAAAABkUAAAACAA9DYWxsZXJUaW1lc3RhbXBIQgAAAAdFAAAAAgANMTQ3NjIzMTI0NjA4N0hCAAAACEUAAAACABZBcHBNYXBDYWxsZXJNZXRob2ROYW1lSEIAAAAJRQAAAAIAI1dlYlNlcnZpY2UgYXQgaHR0cF8vLzEyNy4wLjAuMV84MDEwSEIAAAAKRQAAAAIAEEFwcE1hcENhbGxlckhvc3RIQgAAAAtFAAAAAgANYXBtLWltbWVyc2lvbkhCAAAADEUAAAACABFVcHN0cmVhbUdVSURDYWNoZUhCAAAADUYAAAADAAAAE2phdmEudXRpbC5BcnJheUxpc3QAAAAKSEIAAAAORQAAAAIAIEI2M0QyN0ZDMEE4MkIyRDU2MEE2Qzc4Mzc2RERDRkQzSEIAAAAPRQAAAAIAIEI2M0QyODhDMEE4MkIyRDU2MEE2Qzc4MzdGRjc0RTk3SEIAAAAQRQAAAAIAIEI2M0QyNzk4MEE4MkIyRDU2MEE2Qzc4MzhCMDVDNDdESEIAAAARRQAAAAIAIEI2M0QyNTkwMEE4MkIyRDU2MEE2Qzc4MzU1Qzg5RkRBSEIAAAASRQAAAAIAIEI2M0QyODBGMEE4MkIyRDU2MEE2Qzc4M0NBNTM3ODRBSEIAAAATRQAAAAIAIEI2M0QyNTk1MEE4MkIyRDU2MEE2Qzc4MzRCRUNBQUM5SEIAAAAURQAAAAIAIEI2M0QyNDJCMEE4MkIyRDU2MEE2Qzc4M0QwMzA3M0JDSEIAAAAVRQAAAAIAIEI2M0QyNzVDMEE4MkIyRDU2MEE2Qzc4MzMyQUU4RTM0SEIAAAAWRQAAAAIAIEI2M0NEREY3MEE4MkIyRDU2MEE2Qzc4MzkyQjQyRUM5SEIAAAAXRQAAAAIAIEI2M0NEQjI3MEE4MkIyRDU2MEE2Qzc4MzcxOTNCMDNGSEIAAAAYRQAAAAIAE0FwcE1hcENhbGxlclByb2Nlc3NIQgAAABlFAAAAAgAGVG9tY2F0SEIAAAAaRQAAAAIAClR4blRyYWNlSWRIQgAAABtFAAAAAgAkQjVDODgyREEwQTgyQjJENTYwQTZDNzgzOTY3QzNENTA0MjI5SEIAAAAcRQAAAAIADkFwcE1hcEFwcE5hbWVzSEIAAAAdRQAAAAMAAAACSEIAAAAeRQAAAAIACk1hdGhDbGllbnRIQAAAAB4=

Host: 127.0.0.1:8010

Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.5 (Java/1.7.0_101)

"B63D31070A82B2D560A6C783CD3209CA" is the CorProcessData, 1;1 the sequence number.