Enabling multi-threaded inserts for data_engine to improve performance and resolve queueing

Document ID : KB000034014
Last Modified Date : 14/02/2018
Show Technical Document Details

Introduction:

In a busy environment, data_engine can be set to use multiple threads for inserting QoS data which will improve the performance and allow the queue to drain faster.

Instructions:

To enable multi-threaded inserts, add the following two parameters to the data_engine using Raw Configure:

thread_count_insert = 4 
queue_limit_total = 100000

thread_count_insert should be set to a number that is less than or equal to twice the number of CPU cores available on the primary hub where data_engine is located.   As an example, if your primary hub has 8 cores you should not set thread_count_insert higher than 16.  In general, setting this value to equal the number of cores on the primary hub is a good starting point.


queue_limit_total defines how many messages can be queued internally in data_engine (waiting for insert) before a data_engine goes back to the hub for more messages.  This value is used to throttle data after enabling bulk commits via thread_count_insert.  100000 is the recommended value but this can be adjusted if necessary.  Setting it to 50000, for example, will cause data_engine to commit data to the database more frequently than 100000.

These parameters are applicable to data_engine running against any supported database platform.