I am setting up Keyring and certificates for SSL, how can we debug SSL handshake errors?

Document ID : KB000013028
Last Modified Date : 10/05/2018
Show Technical Document Details
Question:

I am setting up Keyring and certificates for SSL, how can we debug SSL handshake errors?

Answer:

The best tool for debugging SSL handshake failures related to Keyring/Certificate setup issues is the ACF2 OMVS SECTRACE.

The OMVS SECTRACE is set from the z/OS console prior to the server starting(Keyring and certificates are loaded during server initialization):

      SECTRACE SET,ID=mytrace,TYPE=OMVS,SFUNC=RDATALIB,END 

After re-creating the problem the SECTRACE can be deleted/disabled as  follows.                                                              

      SECTRACE DELETE,ID=mytrace                                           

UNIX System Service (OMVS) SECTRACES will be defaulted to the console

Sample OMVS SECTRACE Showing the Retrieval of Keyring and Certificates:

KEYRING LDAPR15Ring

  and certificates:

CERTAUTH certificate with label: MVSSYS8 CA                                              

PERSONAL certificate with label: LDAPServer

Notes:

  1. Look for R_datalib calls
  2. The first call is a DataGetFirst which shows the Ring Name and the first certificate, expected rc 0/0:0.
  3. The next calls will be a DataGetNext for each additional certificate in the Keyring until there are no more certificates and the DataGetNext gets a 8/8:44 indicating no more certificates(end of file), expected RCs 0/0:0 for all certificates except the last 8/8:44.

Things to verify:

  1. Correct Ring Name for Keyring
  2. Correct Usage for CERTAUTH and Personal Certificates
  3. Correct labels for all certificates
  4. If there is a certificate specified in the Keyring as DEFAULT, the certificate should be flagged as *Default*
  5. The Personal certificate should have a Private Key as denoted by ‘Priv Key len’ and ‘type=

CAS2205I REQUEST=R_datalib       ,EXIT=PRE ,RC=N/A                    
CAS2206I USER=LDAPR15 ,UID=0         ,GROUP=OMVSGRP ,GID=1            
CAS2206I Function=DataGetFirst  ,Userid=                                
CAS2206I Ring Name=LDAPR15Ring                                        
CAS2206I Usage=Certauth len=029A     ptr=0C434760                      
CAS2206I Label=MVSSYS8 CA                                              
CAS2206I Subj DN len=0000004E ptr=0C42EAA8                            
CAS2206I Certificate Owner CERTAUTH                                    
CAS2206I Recd id len=00000030 Val=00.CN=MVSSYS8CA.OU=Auditing Department
CAS2205I REQUEST=R_datalib       ,EXIT=POST,RC=0/0:0    

CAS2205I REQUEST=R_datalib       ,EXIT=PRE ,RC=N/A                    
CAS2206I USER=LDAPR15 ,UID=0         ,GROUP=OMVSGRP ,GID=1            
CAS2206I Function=DataGetNext   ,Userid=                              
CAS2206I Usage=Personal len=027F     ptr=0C434760                      
CAS2206I Priv Key  len=0000027B ptr=0BC69160 type=PKCS1 Private key    
CAS2206I Label=LDAPServer                              *Default*       
CAS2206I Subj DN len=00000033 ptr=0C42EAA8                              
CAS2206I Certificate Owner LDAPR15                                    
CAS2206I Recd id len=00000030 Val=01.CN=MVSSYS8CA.OU=Auditing Department
CAS2205I REQUEST=R_datalib       ,EXIT=POST,RC=0/0:0                   

CAS2205I REQUEST=R_datalib       ,EXIT=PRE ,RC=N/A                  
CAS2206I USER=LDAPR15 ,UID=0         ,GROUP=OMVSGRP ,GID=1          
CAS2206I Function=DataGetNext   ,Userid=                            
CAS2205I REQUEST=R_datalib       ,EXIT=POST,RC=8/8:44      

Debugging Tips

  1. If you get non-zero RCs on the DataGetFirst calls run the ACFRPTRV report and check for RFAC violations for resource IRR.DIGTCERT.LISTRING
  2. If you do not see the CAS2206I message withPriv Key len’ and ‘type= run the ACFRPTRV report and check for RFAC violations for resource IRR.DIGTCERT.LIST or RRDA violations for <ringOwner>.<ringName>.LST
  3. If all certificates of the Keyring are not returned, verify that all certificates are not expired and that the certificates are flagged as TRUSTed by doing CHKCERTs of each certificate in the keyring.
  4. The handshake failure could be caused but not having the complete signing chain of CERTAUTH certificates CONNECTed to the KEYRING. To verify the complete signing chain do a CHKCERT of the PERSONAL certificate with the CHAIN parameter, for example ‘CHKCERT personal.cert CHAIN’ look for the ‘Chain Information’ and the message ‘Chain is COMPLETE’ and ‘Chain contains common ring - keyring.ring’. The CHKCERT will return the complete signing CHAIN of CERTAUTH certificates, verify that all certificates are CONNECTed to the Keyring.
  5. For further assistance contact CA and provide the following.
  • Server log showing the error messages related to the KEYRING/certificates.
  • To verify KEYRING setup: LIST of the KEYRING that is being used. 
    From TSO, ACF:                  
         ACF                                                                     
         SET PROFILE(USER) DIV(KEYRING)                                          
         LIST ringname       
  • To verify the certificate status, keys and signing chain: the LIST of the KEYRING will show all of the certificates in the KEYRING based on CERTDATA record names. Please do a TSO, ACF CHKCERT for  ALL of the certificates in the KEYRING.
    From TSO, ACF:                  
        ACF                                                                     
        CHKCERT certdata.recordname   
  • To verify the server task has the authority to access the Keyring and the private key of the PERSONAL certificate, the run the ACFRPTRV report against the SMF active at the time of the errors.
  • To verify that the Keyring, certificates and PERSONAL certificate Private Key are being returned to the server: Prior to the region/started task that the KEYRING/CERTIFICATES are being used for, start the ACF2 OMVS SECTRACE. This can be done from the console as follows.
          SECTRACE SET,ID=mytrace,TYPE=OMVS,SFUNC=RDATALIB,END 

            After re-creating the problem the SECTRACE can be deleted/disabled as follows.                                    
                           
            SECTRACE DELETE,ID=mytrace

            UNIX System Service (OMVS) SECTRACES will be defaulted to the console