The Clarity application is composed of several complimentary services (called a cluster) that can be distributed over a network. In a distributed network Clarity services need a way to communicate with each other to perform their individual functions in a coordinated manner. This is accomplished by an application called a beacon that sends multicast messages over a network. The services must exist in the same sub-network in order for the multicast messages to reach each beacon. Varying aspects of the cluster, sub-network, multicast and beacon are managed in the Clarity System Administrator (CSA). Refer to the diagram below for a more detailed explanation.
In order to improve application throughput and performance Clarity services can be distributed across several servers. This workload-sharing approach is called a Clarity Cluster. All services in a Clarity Cluster are considered "managed services" if they can be controlled though the CSA.
A sub-network (subnet) is an identifiably separate part of an organization's network. All the services in a Clarity cluster should be located within the same subnet so the services can communicate with each other. Co-locating the services within the same subnet restricts multicast transmissions thus preventing the spread of unwanted traffic across the network. It is possible to link together separate subnets using "subnet bridges" in case the services cannot be co-located. This configuration is more challenging to setup and requires expertise in using hardware switching.
A service is a distinct software function made available to other applications or services. By default, all Clarity application services are considered "managed" when using Orion/Tomcat. This is because when services are running under Orion/Tomcat they can be monitored by the Clarity beacon All beacons are registered in the CSA where they can "managed". If you are using WebSphere or WebLogic then the control of the App service and the CSA is not managed.
Note: Unless the database is dedicated to Clarity use then it may be better to leave it unmanaged by Clarity.
A beacon is a monitoring application installed on each server in a cluster to transmit and receive multicast messages which are, in turn, used to manage various Clarity services. The beacon sends a discovery message every five seconds telling anyone listening in the cluster that it exists. Each beacon monitors the Clarity services on each server in the cluster.
The following information must be entered into the CSA in order to use the beacon:
Beacon Multicast Address - The multicast address used by the beacon and CSA for service discovery. Data addressed to this IP address is forwarded to all members of the multicast group.
Note: The goal of Remote Method Invocation (RMI) is to provide services to remote clients. RMI is a distributed systems technology that allows one Java Virtual Machine (JVM) to invoke object methods that will be run on another JVM located elsewhere on a network.
Beacon MulticastPort (Also called the RMI port - see details below.) - Each Clarity cluster should have a unique address and port. To make each cluster unique change the last two numbers of the multicast address to match the last two or three numbers in the ip address of the server where the CSA is installed. For example, if the server ip address is 192.168.32.4, change the multicast address to 188.8.131.52 or 184.108.40.206. This should help ensure unique multi-cast addresses for each Clarity cluster on the network.
Each server in the cluster must use these same settings. A typical multicast port number to use is 9091.
Beacon ClientPort - The Beacon Client Port is the server port used by the Beacon service.
Multicast is used for starting up Clarity from scratch by sending a single message, using a single address, to an entire group of recipient services. This is called "Bootstrapping". Multicast also makes it possible for services to be "discovered" by other beacons existing within the cluster. Multicast also works with an application function called Cache Management. Cache Management enables expired session data to be simultaneously "flushed" from the various distributed services via a multicast transmission.
Multicast addresses used in Clarity to discover Beacon services in cluster must be in the Class D address range -- that is, addresses between 220.127.116.11 and 18.104.22.168. Every server in a cluster must have the same address value.
Note 1: Multicast uses a technology called JGroups which allows Clarity to maintain a list of services that are "listening" on the network.
Note 2: If you are running a different cluster in the same subnet e.g. Production, Testing, and Development, ensure that each cluster is using a separate port number or possible conflicts can arise. It is also a good practice to use unique multicast addresses for each cluster.
The CSA's beacon listens for discovery messages from other servers in the cluster. The beacon automatically discovers any server that uses the same CSA password, multicast address, and multicast port. Once a server is discovered the service information is registered and stored in the CSA. The Beacon services must always be running on each server, including the CSA server, or the CSA cannot manage the cluster services.
All distributed services may be stopped, started, and monitored through Clarity's CSA application.
The CSA's beacon sends one message (a Unicast) per registered beacon every 10 seconds to determine if the beacon is still alive.
- List of Servers
When CSA receives a beacon discovery message, it verifies the Beacon's password against its own, and if successful, adds it to its list of servers. The Servers page lists only servers that can be accessed in the cluster.
In the Field: Common Issues
Q1: While installing Clarity in a Sun Solaris environment I am prompted for "root" access to start the beacon. As a company policy, we don't grant root access. How do we change the installation setup so that it (beacon service) runs under a non-root account?
A: If you are operating on a port <1024 then you can only start beacon services as the root user. If you are operating on a port >1024 then you can change the root user field in CSA System Properties. Your user ID should grant to the entire "Clarity" directory structure.
- Log into the CSA.
- Click on Properties.
- Click on System.
- In the Root User field, change the username to a user other than root.
Q2: I just installed Clarity but the CSA isn't listing any servers in the Cluster, why?
A: Run 'tower' command-line to check the servers listed in the tower on all application servers. If you still do not see any servers listed then:
Clarity 15.x: Tower Command = admin tower
- Restart beacon on all application servers to list servers in the tower.
- Type "refresh" in the "tower" prompt - to list servers in the tower.
If servers are still not listed then check if the CSA passwords are the same on all servers.
- Go to Q6 below.
If servers are still not listed then check if multicast is enabled.
- Type "trace on" in the "tower" prompt.
If messages are sent and received then multicast is enabled. If not, please check with your IT department to enable multicast. Without multicast, servers will not be listed in the CSA.
- Verify port 7500 is not being used. Multicast uses port 7500 for diagnostic purposes. If another application is using port 7500 then beacon will not run.
Q3: Why is one of my servers not listed in the CSA?
A: It is quite possible that either the unlisted server's beacon is down or the unlisted server is located in a different subnet. To restart the beacon go to Q7. If Q7 does not show results check with your IT department to determine if the unlisted server is on the same subnet as the other servers.
Q4: Per company policy, one of servers must be installed in a separate subnet and as a result I can't see the server in the CSA. What can I do?
A: You can use a multicast "bridge". A bridge will forward the multicast transmissions to the server in the separate subnet. Check with your IT department for details on how to proceed.
Q5: I just upgraded my version of Clarity. Why can't I see any of the servers listed in the CSA?
A: The code for Beacon has changed in 7.5.3 FP02 and later. Other servers cannot be recognized by Beacon (which is now running under the new code) until the upgrade has been applied to the other server as well. Because Beacon cannot see the other server until after the upgrade has occurred, the Distribute option in the CSA will not work. The extra step of installing the upgrade to each server will only be necessary the first time that you install a version of Clarity that includes the new Beacon Code So, the first time you apply an upgrade to a version of Clarity that includes the new Beacon code you will need to run the full upgrade on each of your individual servers in order for Beacon to recognize the server.
(Ref. Knowledge Base Article 7805/TEC439276)
Q6: Why can't I see any of the servers listed in the CSA after I changed the CSA password?
A: The CSA password must be the same on all servers, even the servers not actually running the CSA.
- Change the CSA Password on each server whether they are running the CSA or not to match the system that was already changed. Change the CSA password by running the "password" command.
Clarity 15.x Password Command = admin password
- Restart the Beacon Service on all servers. Once the Beacon Service is restarted, the new password will be recognized.
(Ref. Knowledge Base Article 7752/TEC439232)
Q7: The Beacon service is not running/fails to start on the server. How can I get it started?
A: Redeploy the beacon and check its status:
- From the app server command prompt "service add beacon" then "service deploy beacon"
- Enter "service status beacon". If it has a [+] next to it, it's ok. Otherwise, run "service start beacon", and then "service status beacon"
- If the beacon is still not running, verify the path and verify that execute privileges are correctly setup for the beacon executable
Q8: Every time I start Beacon a random UDP port is used. Why is this?
A: User Datagram Protocol (UDP) is used to send short messages (called datagrams) over an IP network and uses UDP ports to create "channels" to send those messages.
By default, the UDP bind port is set to 0. The Clarity multicasting code (called JGroups) then looks for a port number that is automatically allocated from a predefined range (called Ephemeral ports) in the operating system. For every piece of Clarity code that utilizes JGroups it will go grab a UDP port. This will make it look like "random" ports are being opened every time you start the beacon. You can limit this at an OS level by throttling your Ephemeral ports to narrow down which ones will be opened. On shutdown and restart JGroups will attempt to set the bind port to 0 (zero) and the process starts over again.
This only causes a problem if you are trying to firewall your app server with a software firewall. At the time of this writing this functionality is under review.
(Ref. Knowledge Base Article 8847/TEC440167a)