How to optimize distsrv probe performance when distributing superpackages

Document ID : KB000033940
Last Modified Date : 14/02/2018
Show Technical Document Details
Background
There are two main options to improve distsrv performance. One option focuses on performance tuning for the CA UIM environment, especially the hub, and the other focuses on the distsrv probe configuration itself.

Option 1:

Generally, prior to hub version 7.x, you can have the following queue bulk_size settings set anywhere from 100-999 with no issues. Below are some example settings used MSP environments with a lot of message traffic. These settings can be modified using Raw Configure mode for the hub probe under the postroute section. Note that they should not be set ?blindly? and it is highly recommended that any hub performance tuning should be done with assistance from CA UIM (NMS) Support personnel.

Sample bulk_size settings

Hub bulk_size settings (local ATTACH queues)
  • data_engine??????????????? ? 999
  • alarm_enrichment?????? ? 100
  • nas????????????????????????????? ??? 20
  • audit??????????????????????????? ?????? 1 (is the default but it can be set from 100-999 if it?s not ????????????????????????????????????????????? draining well)
?
Probe *bulk_size configuration parameters (set via Raw Configure for each probe mentioned below:
  • hub_bulk_size (data_engine)? ? ????????????????????? ??????????? 500 (minimum 300)
  • bulk_read_size (nas)?????? ??????? ??????????????????????? ??????????? 2000 (minimum 300)
  • message-receiver-bulk-size (qos_processor) 2000 (Recommended by development)
?
Sample bulk_size settings for GET queues configured on the primary hub/relay, configured for busy remote hubs:
  • <remote_busy_hub1_qos>???? 999
  • <remote_busy_hub2_qos>???? 999
  • <remote_busy_hub1/relay1_alarms>?? 50
  • <remote_busy_hub2/relay2_alarms>?? 50
?
Tip: As a rule of thumb and in practice for very busy remote hubs double the message receive rate and add 20% more to calculate the best bulk_size number but in hub v5.82 or lower, if it?s close to 800-900, you can just set it to 999. Note that if you find that the hub is not processing messages fast enough and therefore the data_engine queue is not draining fast enough, as of 7.x or higher, you can set the bulk_size higher than 999, e.g., to 2000, but you should work with Support to assess the message volume for the primary hub as well as any remote hubs first.
?
Option II:

To some degree, distsrv 'slowness' may be due to performance 'limitation' in the current design. File transfer of the distsrv works in a sequential synchronous transfer mode, where the files that need to be transmitted get divided up into chunks which are then sent and received/ack'd one by one, and chunk n+1 does not get put on the bus until chunk n has been ack'd.

To try and achieve higher throughput, the following distsrv configuration parameters should be considered.

<setup>
block_size = <number of bytes>
max_inst = <number of concurrent distributions>
forward_to_remote_hubs = <yes/no>
remote_max_inst = <number of distributions to forward during one session to the remote distribution server>
accept_remote_distributions = <yes/no>
use_local_archive = <yes/no>
</setup>
?
- block_size
Files are transferred in chunks of the configured size. The default setting for this one is quite low out of the box, so you can try increasing it, e.g., from 32k to 512k.
?
- max_inst
A thread is started for each concurrent installation. This setting determines how many concurrent threads will be started. The default setting is 10, but if the discovery server is running on a server with plenty of network bandwidth, you can try increasing it.
- forward_to_remote_hubs
This setting determines if distribution jobs are passed on to secondary distribution servers. Turn this on to offload the main distribution server and let the distribution servers on the robots' hub do the task.
?
- remote_max_inst
Determines how many jobs to transfer to secondary distribution servers (on remote hubs) in one session. You should only need to increase this if the secondary distribution server finishes so quickly that their queues empty while there are still jobs on the primary distribution server waiting to be performed. Note also that this may in some situations need to be reduced if the primary distribution server is too congested.
?
- accept_remote_distributions
This option must be on, on the secondary distribution servers (it IS on by default).
?
- use_local_archive
This setting determines if the secondary distribution servers will request the packages from the primary or use packages in their own local archive. If you want to turn this on, you should also forward packages and licenses from the primary to the secondary distribution servers.
Note that all of the above settings can be modified via Raw Configure. To enter Raw Configure mode, select the distsrv probe and hold down the SHIFT key and then rt-click and Select Raw Configure?

Here are some of the settings that are exposed through the GUI:

User-added image

User-added image


Note that when you?re editing the parameters using Raw Configure mode, you will find all of the configuration settings under the <setup> section which is displayed by default when you open the probe in that mode.
?
For more information please refer to the distsrv help documentation at:
http://docs.nimsoft.com/prodhelp/en_US/Probes/Catalog/distsrv/5.3/distsrv-5.3.pdf



Keywords: distsrv probe performance tuning slow slowness distribution jobs delayed delay delays optimize Distribution package superpackage hub robot hubs robots seems too slow long