Slow LDAP query after load balancer

Document ID : KB000043337
Last Modified Date : 14/02/2018
Show Technical Document Details

Issue:

Clients reporting slowness in application access 

Environment:

Windows policy server Version: 12.5; Update: 00.03; Build: 875; CR: 03;

Analysis: Policy server profiler logs, they show transactions taking time to process

TIME/Number of TX

      1 sec 500 ms - 1 sec 600 ms   1715

      1 sec 600 ms - 1 sec 700 ms   1315

      1 sec 700 ms - 1 sec 800 ms   1399

      1 sec 800 ms - 1 sec 900 ms   1013

      1 sec 900 ms - 2 secs   981

      2 secs  - 3 secs 6020

      3 secs  - 4 secs 2522

      4 secs  - 5 secs 1089

      5 secs  - 6 secs 417

      6 secs  - 7 secs 170

      7 secs  - 8 secs 62

      8 secs  - 9 secs 30

      9 secs  - 10 secs       4

      10 secs  - 11 secs      1

      11 secs  - 12 secs      2

Break down of where time spent, each transaction performs multiple LDAP search, each search can take anywhere from anywhere from: 500ms to 5 seconds

LDAP Search/Number of Transactions

900 ms - 1 sec 4584

1 sec - 1 sec 100 ms 4385

1 sec 100 ms - 1 sec 200 ms 3077

1 sec 200 ms - 1 sec 300 ms 3046

1 sec 300 ms - 1 sec 400 ms 1935

1 sec 400 ms - 1 sec 500 ms 1465

1 sec 500 ms - 1 sec 600 ms 1371

1 sec 600 ms - 1 sec 700 ms 1127

1 sec 700 ms - 1 sec 800 ms 1015

1 sec 800 ms - 1 sec 900 ms 566

1 sec 900 ms - 2 secs 596

2 secs - 3 secs 2101

3 secs - 4 secs 288

4 secs - 5 secs 73

We do not see waiting for the handle during transaction processing, policy server sends multiple searches over the handle at the same time, however the examples below looks like we are buffering the results before continuing the transaction - this set of transactions all take the exact same amount of time, four different threads using the same handle - results are buffered

[2256][5312][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]

[2256][6096][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]

[2256][5952][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]

[2256][5964][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds

 

Resolution:  

In logs provided we only see one LDAP bank to process requests from 20 threads.

Need to expand the bandwidth for Policy Server processing LDAP request to more than one BANK (load balance)

Information on how to create addition independent connections to the user store – this will allow each bank to failover one at a time

Unlike ODBC connection model (connection pooling) LDAP implementation is one LDAP bank using one connection to search and another to BIND. 

Ideation you can vote on the add connections pooling for LDAP user store: https://communities.ca.com/ideas/235718429

To handle load the User Directory object can be load balanced, which creates multiple LDAP Banks to handle the incoming load: 

If there is only one physical LDAP server the need to create multiple DNS names or HOST entries the policy server local host file may be needed to handle load – IDEALLY a future release of SiteMinder will do connection pooling to the LDAP server.

Below configuration creates 6 LDAP Banks to the same hosts (Primary 10.130.158.108  Failover 10.130.158.109)

NOTE: If the same HOSTNAME or IP address is repeated 6 times SiteMinder will mark all connections bad to the same name if error occurs on a single request   (A B, C D, E F, G H, I J, K L)

      SiterMInder UserDir Object: CA_Dir001andDir002_LB:

            CAldap001-1.mysite.com:2001 CAldap002-1.mysite.com:2001,

            CAldap001-2.mysite.com:2001 CAldap002-2.mysite.com:2001,

            CAldap001-3.mysite.com:2001 CAldap002-3.mysite.com:2001,

            CAldap001-4.mysite.com:2001 CAldap002-4.mysite.com:2001

CAldap001-5.mysite.com:2001 CAldap002-5.mysite.com:2001

CAldap001-6.mysite.com:2001 CAldap002-6.mysite.com:2001

      /etc/hosts OR C:\Windows\System32\drivers\etc\hosts

            10.130.158.108    CAldap001-1.mysite.com

            10.130.158.108    CAldap001-2.mysite.com

            10.130.158.108    CAldap001-3.mysite.com

            10.130.158.108    CAldap001-4.mysite.com

            10.130.158.108    CAldap001-5.mysite.com

            10.130.158.108    CAldap001-6.mysite.com

            10.130.158.109    CAldap002-1.mysite.com

            10.130.158.109    CAldap002-2.mysite.com

            10.130.158.109    CAldap002-3.mysite.com

            10.130.158.109    CAldap002-4.mysite.com

            10.130.158.109    CAldap002-5.mysite.com

            10.130.158.109    CAldap002-6.mysite.com