issue when NetScalar route traffic to Layer7 over SSL

Document ID : KB000092792
Last Modified Date : 23/04/2018
Show Technical Document Details
Issue:
SSL communication between API Gateway and Netscaler fails with SSL “Alert (Level: Fatal, Description: Record Overflow)”
The over flow alert is coming from remote device local port 20498 (NetScaler) during setup of SSL connection to APIMGateway_device port 8443, results in RST from NetScaler
The reassembled certificates is 7519 bytes this is resulting in NetScaler Fatal Rrecord Overflow
 
      3 0.000786       netscaler_device       20498       APIMGateway_device         8443             TLSv1.2  256    Client Hello
      4 0.000823       APIMGateway_device         8443        netscaler_device       20498            TCP      54     8443 → 20498 [ACK] Seq=1 Ack=203 Win=15544 Len=0
      5 0.002053       APIMGateway_device         8443        netscaler_device       20498            TCP      2814   8443 → 20498 [ACK] Seq=1 Ack=203 Win=15544 Len=2760 [TCP segment of a reassembled PDU]
      6 0.002111       APIMGateway_device         8443        netscaler_device       20498            TCP      2814   8443 → 20498 [ACK] Seq=2761 Ack=203 Win=15544 Len=2760 [TCP segment of a reassembled PDU]
      7 0.002128       APIMGateway_device         8443        netscaler_device       20498            TCP      2814   8443 → 20498 [ACK] Seq=5521 Ack=203 Win=15544 Len=2760 [TCP segment of a reassembled PDU]
      8 0.002138       APIMGateway_device         8443        netscaler_device       20498            TCP      2814   8443 → 20498 [ACK] Seq=8281 Ack=203 Win=15544 Len=2760 [TCP segment of a reassembled PDU]
      9 0.002148       APIMGateway_device         8443        netscaler_device       20498            TCP      2814   8443 → 20498 [ACK] Seq=11041 Ack=203 Win=15544 Len=2760 [TCP segment of a reassembled PDU]
     10 0.002302       netscaler_device       20498       APIMGateway_device         8443             TCP      60     20498 → 8443 [ACK] Seq=203 Ack=2761 Win=32928 Len=0
     11 0.002330       APIMGateway_device         8443        netscaler_device       20498            TLSv1.2  2814   Server Hello, Certificate [TCP segment of a reassembled PDU]
     12 0.002472       netscaler_device       20498       APIMGateway_device         8443             TCP      60     20498 → 8443 [ACK] Seq=203 Ack=6901 Win=28788 Len=0
     13 0.002488       netscaler_device       20498       APIMGateway_device         8443             TCP      60     20498 → 8443 [ACK] Seq=203 Ack=11041 Win=24648 Len=0
     14 0.002584       APIMGateway_device         8443        netscaler_device       20498            TLSv1.2  834    Encrypted Handshake Message
     15 0.002732       netscaler_device       20498       APIMGateway_device         8443             TCP      60     20498 → 8443 [ACK] Seq=203 Ack=15181 Win=20508 Len=0
     16 0.002754       netscaler_device       20498       APIMGateway_device         8443             TLSv1.2  61     Alert (Level: Fatal, Description: Record Overflow)
 
Environment:
API Gateway 9.2 
Nertscaler 
TLS 1.2


 
Cause:
NetScaler: Client Hello with 16 different Algorithms
APIM:  Server Hello back is a LARGE 7519 bytes work of certificates (all certificates in the Private Key Properties)
NetScaler:  chokes on the certificate size with  “Fatal Error: record_overflow(22)”
NetScaler problem, however we can provide a workaround on APIM side to reduce the size of the certificate

 
Resolution:
From APIM side: You can reduce the size being sent by not including non-CA certificates Policy Manager -> Tasks -> Certificates, keys and secrets -> Manage Certificates Select properties of the trusted certificate, click Options and uncheck “Signing Client Certificate” - inbound SSL