CA Access Gateway (Formerly SPS) custom error page Issue when Dynatrace Agent is installed

Document ID : KB000097146
Last Modified Date : 18/05/2018
Show Technical Document Details
Issue:
Dynatrace Agent installed on CA Access Gateway to monitor performance.
When enabling the Custom error page on the CA Access Gateway, it was observed that Tomcat is Evaluating the Exception properly and Redirecting to the correct Error page, however for some reason, the 500 Status generated by Tomcat for the Error page is not being committed.

Use case as Follows:
1) Error page Enabled on CA Access Gateway per the below Guide
https://docops.ca.com/ca-single-sign-on/12-8/en/using/ca-access-gateway-administrative-ui/custom-error-pages-configuration

2)  http_socket_timeout was set to 2 min which means CA Access Gateway will wait for a Response from backend for a period of 2 min. If not request received, Tomcat will issue a Timeout
     Exception along with 500 Status 

[Noodle.java][Noodle::doGet][][][][][][][][][][][][][][][][java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method)]
[Thread performing the required IO interupted. might be the connection time out.] 
[Thread performing the required IO interupted. might be the connection time out.] 
[ErrorPageImpl.java][ErrorPageImpl::displayMessage][][][][][][][][][][][][][][][][ after calling generateHTML, message: <HTML><HEAD><TITLE>Secure Proxy Server - Error Report</TITLE><style>body { font-family: verdana, Georgia, Arial, "Times New Roman", Times, serif; }</style></HEAD><BODY><H1><center>Secure Proxy Server - Error Report</center></H1><H2>Error Details</H2><TABLE><TR VALIGN="TOP"><TD><H3>Request URI</H3></TD><TD>:</TD><TD>/YourResourceHere</TD></TR><TR VALIGN="TOP"><TD><H3>Error Type</H3></TD><TD>:</TD><TD>SPS Exception</TD></TR><TR VALIGN="TOP"><TD><H3>Error Code</H3></TD><TD>:</TD><TD>Noodle_InteruptedIOException</TD></TR><TR VALIGN="TOP"><TD><H3>Error Message</H3></TD><TD>:</TD><TD>Thread performing the required IO interupted. might be the connection time out.</TD></TR></TABLE></BODY></HTML>] 

3) The issue is the 500 Generated by Tomcat is not being set properly, rather, from the mod_jk trace, it was observed that the message returned to client (browser) was Status 200 instead which is leading to blank page displayed on the browser side instead of the error message

[debug] jk_ajp_common.c (739): (ajp13) status = 200
 
Resolution:
Debugging the issue lead to Dynatrace Agent causing issue with the status commit.
Below is a snippet from Debugging a Transaction:

30-Apr-2018 4:27:28 PM org.apache.coyote.Response setCommitted 
- dynatrace called flush() and attempted to commit: 

INFO: *** defect *** stack is: at org.apache.coyote.Response.setCommitted(Response.java:226) 
at org.apache.coyote.ajp.AbstractAjpProcessor.prepareResponse(AbstractAjpProcessor.java:1027) 
at org.apache.coyote.ajp.AbstractAjpProcessor.action(AbstractAjpProcessor.java:321) 
at org.apache.coyote.Response.action(Response.java:175) 
at org.apache.coyote.Response.sendHeaders(Response.java:373) 
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:349) 
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:308) 
at org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112) 
at com.dynatrace.diagnostics.agent.introspection.uem.impl.AgentInjectingWriter.close(null:-1) 
at com.dynatrace.diagnostics.agent.introspection.uem.impl.AutoHtmlInjectorWriter.close(null:-1) 

- When the call to flush happened, the commit flag (Tomcat side) for this response transaction becomes TRUE. 
*** What that means is no other parts of the code can change or modify the status, response message, and anything about the response. 
*** If any attempt to change any part of the response, tomcat will throw an illegal state exception 

- This dynatrace flush happened BEFORE CA Access Gateway can attempt to change the status response from 200 to 500 due to the IOException captured, due to the connection timeout: 

30-Apr-2018 4:27:28 PM org.apache.catalina.connector.Response setStatus 
WARNING: *** de355930 *** (1) Catalina does NOT set status in setStatus!!! 

- Tomcat finally got the recycle called which resets everything including the commit flag, but the status in the previous response was sent as 200 instead of the intended 500. 

30-Apr-2018 4:27:28 PM org.apache.coyote.Response recycle 
INFO: *** defect *** stack is: at org.apache.coyote.Response.recycle(Response.java:537) 
at org.apache.coyote.ajp.AbstractAjpProcessor.recycle(AbstractAjpProcessor.java:629) 

Possible Solution if Dynatrace to be kept:
1) attempt to REVERSE the call to dynatrace and have the application call dyantrace LAST.  the ordering is important. 
2) Or, customer suppress the flush called happening in dyntrace or somewhere in its application . CA Access Gateway should be the one and only one that should call the response flush