CA API Gateway (formerly layer 7) fails to communicate with SiteMinder policy server (in FIPS mode) via the SiteMinder agent SDK.

Document ID : KB000046059
Last Modified Date : 14/02/2018
Show Technical Document Details

Customer Issue: 

CA API Gateway (formerly layer 7) fails to communicate with SiteMinder policy server (in FIPS mode) via the SiteMinder agent SDK.

smps log:-

[696/6668][Thu Aug 11 2016 22:28:06][CServer.cpp:1974][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3153

[696/6668][Thu Aug 11 2016 22:28:06][CServer.cpp:1983][ERROR][sm-Tunnel-00040] Handshake error: Bad version number in hello message

[696/6668][Thu Aug 11 2016 22:28:06][CServer.cpp:2147][ERROR][sm-Server-01070] Failed handshake with 10.136.134.101:38167

Agent sdk log:-

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the agent name: pwauslssgapp01_smagentapi

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the ACONAME name: ACO_pwauslssgapp_smagentapi

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the HCONAME name: HC_TAI_wau_primary

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the FIPSMODE name: ONLY

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the AGENT_HOST_NAME : pwauslssgapp01

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 141 - Here is the P_ACTION name: GET

2016-08-11T22:28:06.618-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 143 - Here is the path to the Host Configuration Object: /opt/SecureSpan/Gateway/node/default/etc/conf/xyzsmsession/SmHost.conf

2016-08-11T22:28:06.621-0500 SEVERE 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 147 - GetConfig method returned error. Check agentName & SmHost.conf file path is correct

2016-08-11T22:28:06.621-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: Line 153 - Making the first connection to the policy server 10.136.152.14

2016-08-11T22:28:06.621-0500 FINER 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: The return code from the init method is: -1

2016-08-11T22:28:06.621-0500 SEVERE 593 com.xyz.customassertion.xyzsmsession.xyzSmAgent: FAILED_CONNECT

 

Environment:

Policy server version: - 12.51 CR5

OS: - Win 2008 r2

Agent SDK on red hat Linux

 

Cause: 

API gateway by default has a file called “siteminder-env.sh”. This file has the siteminder details which is utilized by api gateway while communicating with policyserver via agent sdk.

This file by default can be found at (/opt/SecureSpan/Gateway/runtime/etc/profile.d/siteminder-env.sh)

Customer was missing below environment variable entry related to policy server FIPS mode in the siteminder-env.sh file, which was causing it to communicate in non fips mode.

------------------------------------------

CA_SM_PS_FIPS140=ONLY

CAPKIHOME=${CAROOT}/CAPKI

export CAROOT LD_LIBRARY_PATH CAPKIHOME CA_SM_PS_FIPS140

-------------------------------------------

Resolution:

1. Applied JAVA JCE patches on the SiteMinder Agent SDK machine.

2. Updated the “siteminder-env.sh” file with FIPS only environment variable entry as below and restarted API gateway.

 ------------------------------------------

CA_SM_PS_FIPS140=ONLY

CAPKIHOME=${CAROOT}/CAPKI

export CAROOT LD_LIBRARY_PATH CAPKIHOME CA_SM_PS_FIPS140

-------------------------------------------

 

Additional Information:

Refer CA API Gateway documentation, where you can find more information how to integrate it with SiteMinder:- https://docops.ca.com/ca-api-gateway/8-3/en/configure-security/tasks-menu-security-options/manage-ca-single-sign-on-configurations/working-with-ca-single-sign-on