Login Response times are high on SP Policy servers in Production

Document ID : KB000006872
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

Need to understand code changes in CR6 would decrease performance by twice 200 ms increasing to over 400ms response time

Environment:
Solaris Policy server upgraded from R12.52 SP1 to R12.52 SP1 CR6 - software upgrade not other changes
Cause:

CA introduce thread throttling in CR6 (common code all platforms) – starts 8 threads (default), PS will increase thread count when request is added to the queue and all other threads are busy

Before the change when a new request was added to the queue the reactor thread woke up all worker threads causing higher CPU and large amounts of misses (thread that did not find any request in queue to process).  This scenario effect Windows and Linux policy server more than Solaris.   Solaris threading architecture is different to Windows/Linux.

SMPS logs in CR6 provide improved performance status

Average throughput – 1.833/sec

Average response times – 100.57msWait time in queue = 0.028ms

Formula to increase thread count:  Max threads and busy threads are the same and current normal queue depth is not 0 a new thread is added

[13704/47][Mon May 15 2017 09:25:01][CServer.cpp:4787][INFO][sm-Server-02030] Thread pool: Msgs=493860 Throughput=1.836/sec Response Time=100.818ms Wait Time In Queue=0.028ms Max HP Msg=1 Max NP Msg=8 Current Depth=0 Max Depth=8 Current High Depth=0 Current Norm Depth=0 Current Threads=9 Max Threads=9 Busy Threads=1

[13704/47][Mon May 15 2017 09:30:01][CServer.cpp:4787][INFO][sm-Server-02030] Thread pool: Msgs=495369 Throughput=1.840/sec Response Time=101.071ms Wait Time In Queue=0.028ms Max HP Msg=1 Max NP Msg=8 Current Depth=0 Max Depth=8 Current High Depth=0 Current Norm Depth=0 Current Threads=9 Max Threads=9 Busy Threads=3

[13704/47][Mon May 15 2017 09:35:01][CServer.cpp:4787][INFO][sm-Server-02030] Thread pool: Msgs=496865 Throughput=1.843/sec Response Time=101.324ms Wait Time In Queue=0.028ms Max HP Msg=1 Max NP Msg=8 Current Depth=0 Max Depth=8 Current High Depth=0 Current Norm Depth=0 Current Threads=9 Max Threads=9 Busy Threads=1

 Based on the data provided the Policy serve is handling the load – it added only one thread 9 since startup.

Resolution:

Two Solution:

Leave as is:  When he load and response times continue to increase new threads would be added when formula is reached

OR

To get previous behavior to the upgrade of R12.52 SP1 CR6 do the following on each policy server

1.      Stop Policy server process

2.      Add entry to the registry (Solaris sm.registry)

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\PolicyServer=10113

.

Thread Pool Size=     0x80;                 REG_DWORD

Thread Pool Initial Size=             0x80;                  REG_DWORD

 

How it works:

The registry values:

               "Thread Pool Size"                           Sets the MAX number of threads.  Default is now 20.

                "Thread Pool Initial Size"               Sets the initial number of threads to startup. Default is 8, the old original default max.

 

If the "Thread Pool Size" is less than the "Thread Pool Initial Size"  then the initial value is set equal to the "Thread Pool Size".