To troubleshoot the issue, we setup a wireshark capture from the SDC, as well as enabled debugging on the SDM and SDC logs. During the discovery of a target device, we noticed the SDC is receiving the requests from the SDM. The SDC is sending out the ICMP/SNMP requests to the target device. The target device replies back to the requests, and then we get the following error message recorded in the sdcLog.log file:
SdmConnIcmpMultiClient::run() sendMsg returned error_code=2, sdMgrIP=xx.xx.xx.xx:6844, msgUID=581b591d-0082-1000-00ba-005056a0297b-1
The SDM is having trouble receiving the replies back from the SDC. The wireshark packet capture shows we are holding the secure tunnel open, so it does not appear to be a firewall related issue.
Sustaining Engineering looked up the error code and said it is defined as follows:
SDM_MSG_RC_INVALID_TRANSPORT // The messaging transport infrastructure was invalid or not ready
The root cause is the "Accept Endpoint" on the SDM side is getting corrupt and not able to accept responses from the SDC.