UMP wasp probe stops working

Document ID : KB000094740
Last Modified Date : 04/05/2018
Show Technical Document Details
Issue:
After running fine for about a week or two, UMP shows the following error after attempting to log in:

an-unexpected-system-error-occurred com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.exception.SystemException: java.lang.NullPointerException

We see the following in the UMP portal.log when this problem occurs:

27 Mar 2018 17:38:14,402 ERROR [PortalInitAction:111] (80) Session error, Unable to open a client session for 128.244.251.35:48040: Connection refused: connect

The discovery_server on the primary hub is using port 48040.

Restarting wasp probe clears the error and allows users to log in again.
Environment:
UIM / UMP 8.51
Cause:
Suspect a resource issue with the discovery_server and/or udm_manager probes which eventually prevents the secondary UMP from accessing the discovery_server on the primary hub.
Resolution:
A.  Clear out all stale data with these steps.

1.  Deactivate the UMP wasp on all UMP robots
2.  Delete the following 2 temporary directories:

<installPath>\Nimsoft\probes\service\wasp\work
C:\Windows\temp\liferay (if the directory exists)

If the robot is owned by a user other than local system, the liferay directory may be located somewhere under the robot logon user's %TEMP% directory.

3.  Activate the wasp probe on all UMP robots starting with the primary UMP robot.
4.  Before accessing the UMP from your web browser clear the browser cache from the beginning of time.

  B.  On the primary hub, make the following configuration changes:

udm_manager:
1. In the <startup><opt> section, increase the java_mem_max to 4g and java_mem_init to 2g.
2. Modify/Add the following key values in the <setup> section:

schema_connection_timeout_minutes = 30
schema_transact_retries = 5000

3. Modify/Add the following key values in the <setup><datomic> section:

memory_index_max = 512m
object_cache_max = 1024m
heartbeat_interval_msec = 30000

4.  Cold start (deactivate, then activate) the probe

discovery_server:

1.  In the <startup><opt> section, increase the java_mem_max to 8g and java_mem_init to 6g.
2.  Cold start the probe

C:  On your database server, recommended executing a daily job to rebuild the indexes on the following set of tables:

CM_COMPUTER_SYSTEM
CM_DEVICE CM_COMPUTER_SYSTEM_ATTR
CM_DEVICE_ATTRIBUTE
CM_CONFIGURATION_ITEM
CM_CONFIGURATION_ITEM_METRIC
CM_CONFIGURATION_ITEM_DEFINITION
CM_CONFIGURATION_ITEM_METRIC_DEFINITION
CM_NIMBUS_ROBOT
CM_DEVICE
CM_COMPUTER_SYSTEM_ORIGIN
CM_CONFIGURATION_ITEM_ATTRIBUTE
CM_RELATIONSHIP_CI_CI
CM_RELATIONSHIP_CI_CS
CM_RELATIONSHIP_CS_CI
CM_DISCOVERY_NETWORK
S_QOS_DATA
NAS_TRANSACTION_SUMMARY
NAS_ALARMS