To be FIPS compliant, CAPKI5 has a restriction to load the library libcaopenssl_crypto.dll at a specific memory address. This is restriction a from OpenSSL.
If a process (boplgin/webengine) obtains more than 250MB of heap memory before calling etpki_lib_init (a function in CAPKI5 that SDM needs to initialize CAPKI5), the call will fail and so the program crashes.
A solution for this is available as a testfix (defect DE44381 for SDM 17.1 release).
Raise a case with CA Support to obtain appropriate solution for the release/platform of SDM 17.1 that is in use.
NOTE: Make sure CAPKI5 is installed properly first. If CAPKI5 is not installed properly then, the above solution will not make any difference. Here is an article on the same: https://comm.support.ca.com/kb/service-desk-configuration-fails-with-the-following-error-pdmsql-4468-error-encryptcpp-136-unable-to-initialise-encryption-operation/KB000041327