Clarity PPM: Testing for Active Nodes in the Cluster

Document ID : KB000051927
Last Modified Date : 30/08/2018
Show Technical Document Details
Issue:

The following are symptoms of communication issues between Clarity nodes:

1. <server_url>/niku/nu#action:security.logs does not show all servers in the cluster

2. The command: admin tower
does not list all nodes/services

3. CSA server dropdown does not show all nodes in the cluster

4. Jobs will error out with: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

5. Processes not starting or completing

 

 

 

Cause:
The communication between the Clarity servers are not working.
 
Resolution:

Resolution 1: JGroups Test

1. For affected environment, please have the infrastructure team run the following command on all nodes on the PPM cluster:
admin general enable-discovery

Test if issue persists. If not, go to the next step.

2. Test multicast communication using Jgroups:

Jgroups is the open source component that handles multicast communication between Clarity servers in a Cluster. The following procedure assumes you have at least two clarity servers and the corresponding network configuration have been made to allow multicast traffic.

On the machine running NSA (SENDER) execute the following:

For UNIX:

export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar

For Windows:
set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar

Then execute:

java org.jgroups.tests.McastSenderTest -mcast_addr the_multicast_address -port the_beacon port -use_all_interfaces
 
e.g. 

java org.jgroups.tests.McastSenderTest -mcast_addr 2XX.xx.xx.x -port 9090 -use_all_interfaces -ttl 32


B. On the application servers (RECEIVER):

For UNIX:

export CLASSPATH=$NIKU_HOME/lib/jgroups-all.jar
 
For Windows:

set CLASSPATH=%NIKU_HOME%/lib/jgroups-all.jar
 
Then execute:

java org.jgroups.tests.McastReceiverTest -mcast_addr the_multicast_address -port the_beacon port -use_all_interfaces

e.g.

java org.jgroups.tests.McastReceiverTest -mcast_addr 2xx.xx.xx.x -port 9090 -use_all_interfaces


The same multicast address must be used on both the SENDER and on the RECEIVER.
You should use the same multicast address/port specified in the CSA.
If this fails, try using a different address/port.

Once the sender and receiver are running, messages from SENDER to RECEIVER
should been visible in the terminal windows.

The same message send from the SENDER window should IMMEDIATELY appear in the
RECEIVER window.

This test should be done on each application server to see if it can RECEIVE messages from the SENDER.

Test that messages are being sent by typing a test string, e.g. hello,world

If no information is being passed to the RECEIVER, then the environment is not
passing multicast traffic.

Contact the site network team to troubleshoot further as there could be more than one of the root causes:
1. Lack of IGMP Snooping enabled on the switch in between the app servers
2. A server patch/change

Resolution 2: JDBC Ping

Prior to Clarity 15.4.1, multicast messaging at the router layer was required so that Clarity cluster services could communicate.

To reduce network congestion, Clarity uses the Internet Group Management Protocol (IGMP) v3 protocol used by IPv4 systems to report IP multicast group membership to neighboring multicast routers. The problem encountered with this method is that some networks cannot support multicast due to provisioning, router, and cost constraints. Hybrid cloud providers also encounter this limitation as it restricts multicast.

For information on how to enable JDBC Ping, please refer to the following documentation:

https://docops.ca.com/ca-ppm/15-5-1/en/administration/csa-ca-ppm-system-administration/csa-configure-jdbc-ping-as-an-alternative-to-multicast

JDBC Ping works in Clarity 15.3+ and will be used going forward.

Resolution 3: Use a single-server system
If multicast is not working, another option or recommendation is to go from a 2-node system to a 1 server system. This will no longer be require multicast.