What can be done when PAM is inaccessible

Document ID : KB000100506
Last Modified Date : 11/07/2018
Show Technical Document Details
Introduction:
In some instances PAM can become inaccessible.  This document explains a tool that may be used to possibly bring the PAM instance back to life.
Instructions:
From time to time a PAM instance may become inaccessibly.  PAM respond to a ping, but logging does not work, not even using the config user.  When this happens Support might be able to resolve the issue if we can get into PAM using ssh.  This requires that the ssh debug patch was installed, and Remote Debugging enabled, on the config diagnostics page.  When the ssh debug patch was provided it most likely was provided along with a remove patch and a ppk file.  If the ssh debug patch was installed and enabled then Support can try to connect to the system, using putty and the ppk file provided along with the patch.  This would be done in a webex session, during which the support engineer would have to type in the userid(root) and the passphrase to login to PAM.  The passphrase is one that cannot be share outside of Support, in order to protect CA's Intellectual Property.  If access is established it may be possible to bring the PAM instance back to life.  In the case where an ssh session cannot be established there are a few other things that may be tried:

For a physical appliance:  try recovering the PAM instance from the secondary ssd.  This is an option because PAM copies the entire contents of the primary ssd to a secondary ssd whenever an upgrade is performed that requires a reboot.  It is also possible to manually perform such a backup,  Bear in mind that only one such copy is available at a time.  If a manual backup was installed after a patch was successfully installed it will not be possible to go back to the state prior to the installation of that patch.  The recovery can be done from the GUI, which would not be possible in the situation being discussed here.  In this case it would be necessary to make a serial connection to the PAM console port, and reboot the PAM appliance.  The boot can be interrupted, and the recovery performed, as discussed here:  https://communities.ca.com/thread/241808792-tech-tip-recover-x304l-from-backup-ssd.

In the case of a PAM VM the best option would be to revert to the most recent snapshot of the PAM VM.  If one was not taken, or it was taken so long ago as to useless, then the remaining option would be to deploy and license a new PAM VM.  You then can try restoring the most recent database backup and cfg file.  Look into setting up scheduled backups(found here:  https://docops.ca.com/ca-privileged-access-manager/3-2/EN/administrating/maintenance/configuration-and-database-backups/schedule-a-backup-of-the-database).

Recovering a PAM AMI would be similar.  It would depend on snapshots of the AWS volume having been taken from the AWS console.  The "bad" volume would then be detached, the backup attached and a boot performed.

Regardless of the method used there may be additional work necessary to get PAM back in a completely functional state.  Configuration changes may have been made since the time the backup was performed, and accounts may now be out of sync.  Please contact Support if you need additional assistance.