SiteMinder Policy Server Throwing Handshake Errors from WAOP

Document ID : KB000109079
Last Modified Date : 01/08/2018
Show Technical Document Details
Issue:
The Web Agent Option Pack (WAOP) has been installed on our JBoss servers, but some of them are causing handshake failures on one of the policy servers.  In the Policy Server's smps.log we see the following when the WAOP is attempting to initialize:

[18169/4116695920][Mon Apr 23 2018 10:43:23][CServer.cpp:2056][ERROR][sm-Tunnel-00100] Handshake error: Bad hostname in hello message [18169/4116695920][Mon Apr 23 2018 10:43:23][CServer.cpp:2207][ERROR][sm-Server-01070] Failed handshake with xx.xx.xx.xx:33010
Cause:
The Policy Server that is showing handshake errors was installed with a different encryption key than the other Policy Servers using the same policy store.
Resolution:
Once the policy server encryption key was updated to the correct value using 'smreg -key', the handshake failures went away and WAOP successfully initialized against the Policy Server that was formerly throwing errors.
 
Additional Information:
Please note that the passwords in the Policy Server Management Console need to be updated after resetting the Policy Server encryption key as this key is used to encrypt those passwords.  If this step is missed the Policy Server will fail to start as it will not be able to connect to the policy store.

Also, in this case the sensitive data in the policy store was encrypted with the correct encryption key, so exporting the data in clear text and then importing it back after resetting the encryption key was not necessary in this instance.  Had the policy store sensitive data been encrypted with a different key this step would be necessary.