How to troubleshoot SMS OTP Delivery Error: An error has occurred while sending the Security Code from SMS Service. Please try again later

Document ID : KB000032779
Last Modified Date : 14/02/2018
Show Technical Document Details

Summary:

Secure Cloud 1.5x could report the following error when it’s trying to deliver a Security Code over Text Message (SMS OTP)

Error: An error has occurred while sending the Security Code from SMS Service. Please try again later

0.png

As a Secure Cloud Service Provider, how do we troubleshoot the problem before we contact CA Secure Cloud Product Support?

 

Instructions:

Secure Cloud 1.5x delivers SMS OTP via Arcot Common Data Service which is running on the Advanced Authentication server. Hence when we troubleshoot SMS OTP Delivery problem, we focus on the involved components Advanced Authentication Server and the corresponding settings

The following are the troubleshooting steps:

  1. Confirm 'Security Code' is enabled on Configure Credential Types: Credential Type

    1.png

     

  2. Confirm Security Code is enable on the current Advanced Authentication Flow. 

    For example, the following is a typical setting for ArcotID OTP with Risk flow which can trigger Security Code in a RiskMinder Advised Increased Authentication scenario.

    2-1.png

    2-2.png

     

  3. Confirm ‘Security Code over SMS’ is Enabled and configured properly
    3.png

    Note:
    OOTB Secure Cloud supports Clickatell as the SMS Provider. By default it send the SMS OTP by a HTTP POST request.
    If we uses Clickatell for SMS delivery, we need to ensure the Advanced Authentication server machine (normally it's also the SiteMinder Policy Server machine) can directly connect to api.clickatell.com port 80 and 443.
    If we uses other SMS delivery service, we need to ensure the Advanced Authentication server machine can send HTTP POST request to the SMS delivery service

     

  4. Check the settings for other alternative OTP delivery approach works.
    For example, if Security Code over Email is enabled on ArcotID OTP with Risk Flow, we can check if Email OTP can be sent to the end user.
    This can be done by using 'Forgot my PIN' function on a new browser whereon the end user didn’t pass Risk evaluation before. 
    This will trigger a Increased Authentication scenario. 

    • Visit 'Forgot my PIN' link on a browser on a new machine 
      4-1.png
      4-2.png

    • RiskMinder triggers a Increased Authentication scenario. 
      4-3.png

    • By selecting ‘Receive Security Code over Email’, the end user will receive an Email OTP.
      image_thumb7

      Note:
      If the end user can not receive such Email OTP, that means the issue could be a general OTP issue rather than a SMS OTP specific issue. We may have to contact CA Support to troubleshoot a SMS OTP specific issue.

  5. Verify the end user mobile can receive SMS messages from other sources.

  6. Check if the issue only happen on certain user’s Mobile number.

    Note:
    A KNOWN ISSUE:  For customers in certain countries which have a phone number which was  issued by one phone carrier but which was later moved to another carrier may not be able to receive SMS messages from the Clickatell SMS delivery service . 

     

  7. Ensure the end user mobile number containing the country code if the SMS Provider is Clickatell, i.e +61432100000.

  8. On Advance Authentication server, enable adjust logger settings in /opt/CA/AdvancedAuth/Tomcat/lib/log4j.properties

    log4j.logger.com.ca=ALL

    log4j.logger.com.arcot=ALL

    log4j.logger.com.arcot.integrations.toksvr.client.SimpleTSClientImpl=INFO

    log4j.appender.LOGHANDLE.File=/opt/CA/AdvancedAuth/Tomcat/logs/cm-aads.log


    Note: Restart the Advance Authentication servers after the change

     

  9. Reproduce the issue and check the logs:

    a. On Advance Authentication Server, we need to review /opt/CA/AdvancedAuth/Tomcat/logs/cm-aads.log

    Search SMSSender in cm-aads.log to find details about SMS delivery probem, i.e. the SMSSender can not connect to Clickatell 

    Typical log snippet:
    2015-10-08 13:46:16,718 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:90) – Received Clickatel Integation URL: http://api.clickatell.com/http/sendmsg?

    2015-10-08 13:46:16,756 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:114) – Message received: [Security Code for TEST001 is 96543]
    2015-10-08 13:46:16,756 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:115) – OTPData for Clickatell:: [user=theuser&password=thepassword&api_id=0123210&to=0123443210&from=54321&mo=1&text=Security+Code+for+TEST001+is+96543]
    2015-10-08 13:46:17,537 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:126) – Processing URL response
    2015-10-08 13:46:17,538 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:138) – strReturn::ID: 3aa6f965640efedf9d6d57ce24e61498
    2015-10-08 13:46:17,538 [http-bio-9090-exec-5] DEBUG SMSSender,(http-bio-9090-exec-5:159) – SMS sent

    b. On SiteMinder SecureProxy Server, please review /opt/CA/secure-proxy/proxy-engine/logs/cm-aa.log
    Search ProvideOTPAndDeliver in cm-aa.log to find additional details about how ProvideOTPAndDeliver remotely invoke the Common Data Service on Advance Authentication Server.

    Typical log snippet:
    2015-10-01 21:17:49,165 [ajp-bio-8009-exec-8] DEBUG ProvideOTPAndDeliver,(ajp-bio-8009-exec-8:98) – OTP successfully generated for user TEST001
    2015-10-01 21:17:49,165 [ajp-bio-8009-exec-8] DEBUG ProvideOTPAndDeliver,(ajp-bio-8009-exec-8:103) – otp delivery channel is sms

    2015-10-01 21:17:50,251 [ajp-bio-8009-exec-8] DEBUG ProvideOTPAndDeliver,(ajp-bio-8009-exec-8:244) – SMS OTP sent


     

    Note:
    By checking these logs, we will know if the SiteMinder SecureProxy Server correctly sent SMS OTP request to the Arcot Common Data Service running on Advanced Authentication Server, also if the Arcot Common Data Service correctly sent the SMS OTP request to SMS Delivery Gateway Server. 
    This will help us to narrow the issue scope.
     

     

  10. Use tcpdump to identify network or Load Balancer problem, tcpdump can be used to capture the http traffic between the Advanced Authentication server and the SMS delivery server, command line example:
    tcpdump -s 0 -i eth1 -A host api.clickatell.com and tcp port http

    A typical scenario that we ran on Advanced Authentication server machine:

    [root@yongcm~]# tcpdump -s 0 -i eth1 -A host api.clickatell.com and tcp port http
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
    00:31:49.724243 IP cm151b.10986 > api.clickatell.com.http: Flags [S], seq 2643595182, win 5840, options [mss 1460,sackOK,TS val 955958669 ecr 0,nop,wscale 7], length 0
    E..<.#@.@.y..#.}….*..P…………b……….
    8………..
    00:31:49.743188 IP api.clickatell.com.http > cm151b.10986: Flags [S.], seq 3563331, ack 2643595183, win 65535, options [mss 1436,nop,wscale 5,sackOK,TS val 1258856570 ecr 955958669], length 0
    E..<..@.8……..#.}.P*..6_C…………………..
    K..z8…
    00:31:49.743215 IP cm151b.10986 > api.clickatell.com.http: Flags [.], ack 1, win 46, options [nop,nop,TS val 955958688 ecr 1258856570], length 0
    E..4.$@.@.y..#.}….*..P…..6_D….\……
    8…K..z
    00:31:49.744291 IP cm151b.10986 > api.clickatell.com.http: Flags [P.], seq 1:285, ack 1, win 46, options [nop,nop,TS val 955958689 ecr 1258856570], length 284
    E..P.%@.@.x..#.}….*..P…..6_D………..
    8…K..zPOST /http/sendmsg? HTTP/1.1
    Content-Type: application/x-www-form-urlencoded
    Cache-Control: no-cache
    Pragma: no-cache
    User-Agent: Java/1.7.0_67
    Host: api.clickatell.com
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection: keep-alive
    Content-Length: 126
    00:31:49.744319 IP cm151b.10986 > api.clickatell.com.http: Flags [P.], seq 285:411, ack 1, win 46, options [nop,nop,TS val 955958689 ecr 1258856570], length 126
    E….&@.@.x..#.}….*..P…..6_D…..%…..
    8…K..zuser=xxxxxxx&password=xxxxxxxx&api_id=xxxxxxx&to=614300xxxxx&from=xxxxx&mo=1&text=Demo+Env0+Security+Code+for+TEST001+is+93957
    00:31:49.765072 IP api.clickatell.com.http > cm151b.10986: Flags [.], ack 411, win 2087, options [nop,nop,TS val 1258856572 ecr 955958689], length 0
    E..4.&@.8……..#.}.P*..6_D…I…’Sd…..
    K..|8…
    00:31:50.553150 IP api.clickatell.com.http > cm151b.10986: Flags [P.], seq 1:229, ack 411, win 2091, options [nop,nop,TS val 1258856661 ecr 955958689], length 228
    E…..@.8……..#.}.P*..6_D…I…+…….
    K…8…
    HTTP/1.1 200 OK
    Date: Mon, 19 Oct 2015 07:31:50 GMT
    Server: Apache
    Keep-Alive: timeout=10, max=50
    Connection: Keep-Alive
    Transfer-Encoding: chunked
    Content-Type: text/html
    24
    ID: 6cd6f6dccfc89d0c8935636faa8e94ec
    0

    We can see the tcpdump shows the HTTP POST request sent to the SMS delivery gateway.

    Note: 
    In some environment we uses a third-party SMS delivery service different to host api.clickatell.com, we will need to check if the third-party SMS delivery service can correctly handle the HTTP POST request sent from Arcot Common Data Service. 

    If there is Load Balancer settling at front of the SMS delivery gateway, we may check the Load Banalcer logs to see if the HTTP POST request is correctly forward to the SMS delivery gateway and received proper response.