CA Process Automation Scheduled Processes Not Executing

Document ID : KB000097239
Last Modified Date : 21/05/2018
Show Technical Document Details
Issue:
CA Process Automation 4.3 SP01 We had to reboot our CA Process Automation servers (including database) yesterday at the urging of our Server Engineering team. After rebooting the db server, load balancer and all domain orchestrators the on-demand processes worked fine. However, when we started work today, we noticed none of the scheduled processes for the evening and morning have executed. I tried deactivating/activating several schedules, but that did not appear to help.
Cause:
This can happen when more than one of the domain orchestrators is reporting that it is the domain master. 

There are a couple of different ways to tell if more than one orchestrator is the domain master.
First by running this URL in a browser, once for each server in the cluster:

http:/server_name_node1:8080/itpam/genericNoSecurity?method=getDomainMasterNode

http:/server_name_node2:8080/itpam/genericNoSecurity?method=getDomainMasterNode

http:/server_name_node3:8080/itpam/genericNoSecurity?method=getDomainMasterNode

Check the results from each, looking specifically at this section:

<TransportID>domainMasterNode</TransportID> 
<Hostname>server_name_node1</Hostname> 
<IPAddress>10.10.10.10</IPAddress> 

All results should show the same hostname and IP address for that host, indicating that only one of the servers is recognized as the master node.

The second check is in the c2o.log for each of the nodes. 
We are looking specifically for the term "ArchiverManager" to appear in the logs of more than one of the servers.
The domain master node is the only node that should be running the archiving process, and if we see that it is running on more than one, that means there is confusion in the communication process.
Resolution:
To resolve the issue in the case where there is more than one domain master node, all orchestrators must be stopped.
Each one then has to be brought up one at a time.
Start the first node, and wait for that to become available, and verify that you are able to log into the server URL directly - not using the load balancer URL, but the direct URL as: http://server_name_node1:8080/itpam

Once you can log in, the next node can be started. Again, wait for that to be up and available on the direct URL of http://server_name_node2:8080/itpam, and continue.