Exchanger issues when a remote node is down or stopped.

Document ID : KB000085597
Last Modified Date : 14/04/2018
Show Technical Document Details
Issue:
Affects Release version(s): 5

Error Message :
In the universe.log:

#####
uxech - connection refused
#####

and/or
#####
uxech - connection timeout
#####

Patch level detected:Dollar Universe 5.6
Product Version: Dollar.Universe 5.6.0 FX25010

Description :Exchanger problems on local node when a remote node is down or stopped.
Internode sessions not working correctly.
Remote event not detected
Environment:
OS: All
Cause:
Cause type:
Configuration
Root Cause: If a remote node is not reachable (down for maintenance or network issue), it's a good practice to un-register it (or delete it from node list) so that the Exchanger won't try to send objects/jobs.
Resolution:
We recommend to use the following tool which provide uxblacklist command.
It will avoid such situation when a node is down or not reachable:

http://www.orsypforum.com/index.php?s=1589b587847e1bff296bdbcc7dce7a60&app=downloads&showfile=37

Description

If a node is not reachable (down for maintenance or network issue), it's a good practice to un-register it so that the Exchanger won't try to send objects/jobs.
uxblacklist is a tool to remove the hostname associated to a node. It will then act like a blacklist.

Note: this is only for Dollar Universe version 5. This is not needed in Dollar Universe version 6.


Effect

After adding the node to the blacklist, the remote commands, distribution and exchanges from the local node to this nodes won't be possible.
This means that the node has to be removed from the blacklist afterward.
The main benefit: This makes the Exchanger more resource efficient and data corruptions are less likely to occur.


App Content

There are 2 items in the current uxblacklist package:
The uxblacklist command line: This will add/remove a node to the blacklist. This command will impact only 1 node
An example of implementation that will automatically scan all the nodes to add/remove them from the blacklist
Usage and implementation: uxblacklist

The uxblacklist command works like any other dollar universe commands.

Select the uxblacklist file according to your operating system (unix or windows). Paste this file with the other dollar universe executables: in the "exec" directory.

Make sure you have set the environment before calling uxblacklist: uxsetenv in the "mgr" directory.

The uxblacklist command has the following parameters:
uxblacklist add node= ($U nodename) : adds xxx before the hostname corresponding with the $u nodename to force a connection error. It works if the node is not local. It adds a line in the sck file if it cannot find the node.
uxblacklist add hostname= (hostname or IP address) : adds xxx before the hostname.
uxblacklist dlt node= ($U nodename) : removes xxx before the hostname to have the connection back. It works if the node is not local. It removes the line if there was no hostname
uxblacklist dlt hostname=(hostname or IP address) : removes xxx before the hostname.
It technically adds or removes "xxx" before the hostname in the uxsrsrv.sck file.
Usage and implementation: auto_blacklist_node.pl

uxblacklist can be called on demand and directly, but this is an example of implementation that would automate the use of uxblacklist on a server.

The script will
... scan all nodes from the list of nodes
... check if the node is reachabe of not
... Remove or add the node from the blacklist

One possible implementation of this script: you can call this script periodically in an uproc. Then you might want to deploy this uproc on all servers that should have the uxblacklist. This is typically from nodes that do frequent deployment / distribution of objects.

Note: auto_blacklist_node.pl is a perl script. To use this script you have to have perl on your system.

Fix Status: Released

Fix Version(s):
Component: Application.Server
Version: Dollar.Universe 5.6.0 FX25010
Additional Information:
Workaround :
N/A