To speed up the SSH connection initialization the best solution is to ensure that the 8550 query packet is 'REJECTED' instead of 'DROPPED'. When a packet is 'REJECTED' the sender is actually notified and will not continue to retry the connection. This will speed up the initialization because CA PAM will no longer hold off the SSH session to wait for a response.
In lab testing the initialization time went from ~5 seconds to <1 second.
Note: this should not be done if SFA is being used on the target device.