Under normal conditions, a DevTest agent has negligible effects on an environment.
However, the DevTest agent was not designed for load testing.
The DevTest agent is a design testing tool, not a performance testing tool.
The JVM where the agent is installed has many ways of dealing with an increased load like using threads. Any java agent, including the DevTest agent, does not have this luxury.
Therefore, expectations and testing procedures will need to be adjusted.
Indications of load test issues:
- VSE errors.
- Errors on the application side due to dropped transactions or connections.
- The JVM where the agent is installed can freeze or crash.
- Errors or Warnings in the agent log including:
- Message Flooding – Too many transactions are being processed.
Default value: 200 transactions per second.
- CPU Thrashing – The CPU is too high for a period of time.
Default values: > 80% for 5 seconds.
- GC Thrashing – The java garbage collector is having trouble freeing up enough memory to continue processing.
Default values: GC uses 30% CPU for 3 seconds.
- Heap size approaching limit.
Memory: > 95% of heap or > 90% of permgen for 10 seconds.
How to reduce an agent bottleneck:
- Understand the test:
Exactly what are you trying to accomplish?
Will this load test accomplish it?
- Reduce the load:
Will a less stressful load test accomplish the same thing?
- Transaction capture levels should be set to count:
Transactions should never be captured from a Virtual Service because it doubles the agent overhead and this transaction has already been captured.
Best Practice: Leave capture levels at “Count” unless you specifically need to capture transactions.
- Increase the memory for the application’s JVM during the load test:
This will trigger GC collection less often – however, too much memory will cause additional issues.
- Adjust properties using ATK or rules.xml.
Note: Adjustments in ATK take effect immediately while changes in rules.xml require an agent restart.
These property values should be adjusted based on the hardware to handle the load. Message Flooding messages are meant for information only and do not indicate an overload situation.
Properties to adjust:lisa.agent.stats.alarm.threshold.messages.per.interval, lisa.agent.stats.alarm.threshold.transactions.per.interval
CPU Thrashing and GC Thrashing
These messages indicate the hardware and/or JVM configuration is at or approaching the physical limits.
An overloaded VSE, even on a different physical box, can also cause these messages.
CPU property values to adjust (not recommended): lisa.agent.stats.alarm.threshold.cpu.num.intervals, lisa.agent.stats.alarm.threshold.cpu
GC property values to adjust (not recommended): lisa.agent.stats.alarm.threshold.gc.num.intervals, lisa.agent.stats.alarm.threshold.heap, lisa.agent.stats.alarm.threshold.permgen
- Turn off Capture/Pathfinder using the Portal or ATK. This setting will need to be verified before the start of each test.
By default the agent captures a large amount of transaction level data for debugging. During a load test, this information is not needed.
- Run the load test on a faster box,
However, doing so may invalidate the load test.
- Eliminate transactions that produce large payloads.
The larger the payload the more resources required for the agent to process.
- Reduce agent logging:
See DevTest_Home/rules.xml.sample for example.
Note: The default agent logging level is “Warning”, setting the level higher will have only a small effect.
- Reduce the non-load transactions:
Do not run transactions that are not part of the load test.
- Virtual Service Environment (VSE):
Make sure the VSE has been optimized for load testing.
An non-optimized or overloaded VSE will cause the agent to wait on the replies, resulting in CPU and GC thrashing for the agent JVM.
The VSE and all deployed Virtual Services (VS) run under one JVM, therefore reduce the number of VSs running.
Eliminate all Exceptions and Errors in all VSs, even the one not used in the load testing.
Every environment and load test is different. In the end, it may not be possible to eliminate enough load from the agent and still perform the required test.
Since the DevTest agent was not designed for load testing, this type of testing is inherently unsupportable.
However, support will offer a best effort and can offer general guidelines for your unique environment.