CA PAM 4.3.0 is not Running new Processes

Document ID : KB000127253
Last Modified Date : 19/02/2019
Show Technical Document Details
Issue:
  • Existing CA PAM 4.3.03 Processes originating from CA Service Catalog are "stuck." 
  • New CA PAM Processes are Queued instead of Running. 
  • There is a very large backlog of queries - over 30,000.
  • There is one CA PAM node only. 
  • A newly created test CA PAM Process with only a single simple instruction (check hostname) goes into a Queued State, with no error message in the CA PAM interface.
  • There are thousands of messages of different types, including errors, in the CA PAM logs.
  • Here is a sample from the c2o.log:

2019-02-12 16:04:09,113 ERROR [com.optinuity.c2o.transport.Resolver] [       Timer-14] Error retrieving transport URL of master of Domain Orchestrator

2019-02-12 16:04:09,113 ERROR [com.optinuity.c2o.transport.Resolver] [       Timer-14] Error retrieving transport URL of domain master - 
com.optinuity.c2o.util.C2OException: Error retrieving transport URL of master of Domain Orchestrator
    at com.optinuity.c2o.transport.Resolver.getDomainTransportURL(Resolver.java:283)
    at com.optinuity.c2o.transport.Resolver.getDomainTransportURL(Resolver.java:216)
    at com.optinuity.c2o.c2oserver.ServerManager.performConfigTimerOperation(ServerManager.java:3544)
    at com.optinuity.c2o.c2oserver.ServerManager.access$1100(ServerManager.java:247)
    at com.optinuity.c2o.c2oserver.ServerManager$9.run(ServerManager.java:2130)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

2019-02-12 16:04:09,113 ERROR [com.optinuity.c2o.c2oserver.ServerManager] [       Timer-14] Exception occurred while sending the HEARTBEAT - 
com.optinuity.c2o.util.C2OException: Error retrieving transport URL of domain master : Error retrieving transport URL of master of Domain Orchestrator
    at com.optinuity.c2o.transport.Resolver.getDomainTransportURL(Resolver.java:287)




NOTE
This symptom and resolution was provided for a specific client issue.
It may not be representative for all sites. Please exercise due diligence in applying findings.

 
Environment:
CA Process Automation 4.3 versions below CA PAM 4.3.03.
CA Process Automation is formerly known as CA IT Process Automation (CA ITPAM).
Resolution:
The root cause is a known issue in versions of CA PAM 4.3 below CA PAM 4.3.03.

The CA PAM Scheduler was not clearing entries from the ActiveMQ table.

This caused a data pile-up, which not only prevented the old CA PAM Processes from Running, but prevented new ones from Running.

Resolution

1) Upgrade CA PAM to 4.3.03, which contains a Scheduler patch, and reboot system.

2) If necessary, remove old entries from the ActiveMQ table if they are not processed in a timely manner. These processes will be lost.
The recommended way to do this is by "Archiving" these entries via the CA PAM interface.
If there are a lot of queries, this may require multiple passes to get all of the States effectively.

3) ONLY IF you cannot remove the stuck Processes via the CA PAM interface, these can be removed by direct database removal.

CAUTION: As these steps will result in data loss, and need care to exercise, the steps and tables are not being provided in this general purpose knowledge document.
There are related knowledge documents below though that may assist.

If you suspect that you have this issue, please log an issue with CA Support, so that we may assist with the review of the data.
It would also help to rule out similar but different issues, such as some processes are stuck but others run freely.
Please backup your database.

Please log an issue with the following:

A) Confirmation that the system has been upgraded to CA PAM 4.3.03 and rebooted.
NOTE: This is a mandatory step. CA Support cannot assist if this is not completed.

B) Include the CA PAM logs, in particular the c2o.logs, view.logs, and db.logs
Note that these logs may have additional numbers in their names as additional log files are generated, so sort by date and attach the most recent.
Please also include screenshots and a summary to help state the nature of the issue.

C) It will be necessary to arrange a time between your site Database Administrators, your site CA PAM Administrator, CA Support and CA Engineering.

So please allow time for these meetings to be arranged at a mutually overlapping timezones, and for the review of the initial log and summary information.

CAUTION: Direct database manipulation will result in permanent removal - or inadvertent loss or corruption - of data.
Additional Information:
How to properly restart the CA Performance Management environment
https://comm.support.ca.com/kb/how-to-properly-restart-the-ca-performance-management-environment/KB000032043

Can not delete a running process instance in PAM. 
https://comm.support.ca.com/kb/can-not-delete-a-running-process-instance-in-pam/kb000044010 

CA Process Automation is unable to start ActiveMQ after numerous retries
https://comm.support.ca.com/kb/ca-process-automation-unable-to-start-activemq-after-numerous-retries/kb000115989

Some scenarios that require clearing the ActiveMQ persistence store.
https://comm.support.ca.com/kb/some-scenarios-that-require-clearing-the-activemq-persistence-store/KB000012112