How to use AppDynamics with LoadRunner for Load Testing

While AppDynamics is most commonly used by IT Operations, Developers, and DevOps teams in production environments, to troubleshoot those difficult to recreate and see problems a growing trend of pre-production usage occurs. A large number of our users have been using AppDynamics for years within performance engineering practices. The key is being proactive, avoiding performance issues before going live in production. The following benefits are typically associated with integration of AppDynamics with load testing processes and technologies:

  • Gaining deep application-level visibility to answer the ‘Why?’ vs. simply determining ‘What?’
  • Rapidly pinpointing bottlenecks in their environment
  • Comparing results across releases and performance tests
  • Improving performance by measuring the end-to-end performance from the end-user perspective
  • Increasing scalability of the application by optimizing software and avoiding unnecessary hardware investments

While this blog centers on LoadRunner specifically, it is important to note that these use-cases may be applied using any load testing solution and methodology, not just LoadRunner. Here are the highlights of this technical whitepaper with deeper details.

The simplest and fastest way to get LoadRunner transactions in AppDynamics is ‘painting’ your LoadRunner requests. This method involves adding a custom header that contains the LR transaction name in the request so that AppDynamics can display the transactions exactly as they appear in LoadRunner.

  • Deep code-level visibility, in the context of a LoadRunner Transaction, end-to-end, inclusive of the end user experience.
  • Application container-level data across languages, for example, on Java this would include Heap size, standard and custom JMX metrics.
  • Deep database visibility, in the context of the transaction, including details SQL breakdowns
  • Extended visibility into application infrastructure (CPU, Disk, Network, and Memory).

AppDynamics provides these benefits automatically without the need to configure multiple HP tools (Example SiteScope, Diagnostics, and others). Once your tests are run, you can use custom time ranges which correspond to an important event. For performance engineering, these typically map to timeframes which correspond to a load test or an application build.

The Compare Releases functionality allows for comparing the performance of the application, inclusive of various KPIs, and trending metrics between any two timeframes.  This comparison can be made at the application-level or specific to individual elements. For performance engineers, being able to quickly compare results between two load tests, and identify deltas leads to rapid root-cause analysis, saving time and effort in every load testing cycle.


The KPI view shows the top stats about the test, such as volume, average performance, the transaction scorecard and any events which occurred.


Transaction snapshot comparisons allow for direct comparison of transaction snapshots related to specific transactions between both load tests. For reference, a snapshot is a deep-level capture of a single occurrence of a transaction that allows one to see code-level details of that transaction’s execution. By comparing snapshots, users can identify differences in code execution paths and identify slowness down to the individually-executed methods.

AppDynamics further extends application visibility from the code-level into the database, providing deep-level visibility in the SQL execution and detailed database performance metrics.  Acting as a bridge between the application team and the DBAs.  It provides a common viewpoint while showing data in context that is relevant for both parties.

Detailed breakdown of a single SQL statement, its execution over time, clients making the request, and the ability to pull an explain plan.  This enables a DBA to rapidly pinpoint the root cause for a poorly-performing SQL operation.

Database Time Range Comparison enables comparison of the performance of the database across two different time frames or the performance of two difference databases, such as Perf and Prod, for a similar timeframe.  This provides a similar functionality for databases as Compare Releases does for deep application visibility.

The Scalability and Correlation Analysis functions enables users to performance X-Y graphing comparisons and stacked graph comparisons of key information. This is useful when analyzing during a live load test or after a load test has completed to identify related behavioral patterns, such as inverse or direct relationships, across one or more different measurements, such as CPU vs. Load or Response Time.


Scalability Analysis comes with several pre-built views for specific comparative analysis. CPU vs. Load comparisons may be made at the overall application level, or against individual elements such as a single application instance or transaction. Here we are using an X-Y graph to plot the data points against each other to see a direct relationship.


Correlation Analysis functions selecting any two measurements for comparison. All measurements collected by AppDynamics are available within this view.  Both stacked charts and X-Y graphs are available.  In this example, the Errors per Min vs. Volume are plotted, showing a direct correlation between the increase in Volume and Error Rate.

AppDynamics has some features and functions which not only help production deployments but also add new visibility and analytics for those doing performance engineering. The whitepaper goes into far more details. In summary, the following key benefits are available:

  • AppDynamics can easily be configured to show LoadRunner transactions in the way that LoadRunner users see them today.
  • Flexible dashboards are not only reusable but enable historical and real-time analysis. This benefits users with a high degree of visibility while reducing configuration effort, saving time and manpower.
  • Automatic Baselining allows rapid comparative-analysis against a specific ‘master’ performance profile. This results in more realistic load testing analysis and higher-quality results by measuring against actuals vs. ‘guesstimates’ for expected performance.
  • Built-in comparison functionality between two different load tests facilitates rapid analysis and reduced effort to answer questions including ‘what changed?’ or ‘did performance improve between releases?’
  • Deep Code-level visibility provides performance engineers the ability to quick triage bottlenecks in the application, minimizing the need for developers during the analysis phase.  This keeps developers focused on building new functionality while reducing the time that performance engineers spend pinpointing problems.
  • Deep Database visibility offers contextualized database performance breakdowns to further reduce root-cause analysis times and provide details in the same ‘language’ that DBA’s use.
  • Real-time alerting during load tests enables notification of breached SLAs or deviations from expected results. This saves performance engineers time and allows for more rapid analysis of issues by surfacing errors more quickly.
  • Automatically-configured, out-of-the-box metrics reduce configuration time and deployment effort, as well as ongoing maintenance.

Interested in reading more? Check out the full whitepaper on LoadRunner + AppDynamics, written by my colleague David Hoenig.

Why Testing in Production isn’t as stupid as it sounds

One of my colleagues this week was consolidating the results from our recent Application Performance Management survey, and one interesting finding was that 40% of customers have at least one release cycle a month. Out of those respondents, one third experience a Severity-1 incident each month as well. That’s a pretty compelling pair of statistics, and they might explain the continued frustration and conflict between development and operations teams. It’s also perhaps the reason why this DevOps underground movement can no longer be ignored (even by Gartner). There is no doubt development organizations have become agile, but does deploying this frequent change make the business more or less agile? For example, if one in three releases creates a Severity-1 incident, then surely agile development becomes a risk to the business. We’re at the point where Operations either has to start managing change better or simply restrict the amount of change that can occur.

So why are Sev 1 incidents so common? Based on my experiences and customer interaction, I’d strongly argue that testing in development isn’t enough. At the very least, it’s certainly not an insurance policy for deploying an application in production. When a Formula 1 team designs a car in a wind tunnel and tests it on a simulator pre-season, they don’t assume that the performance they see in test will mirror the results they see in a race. Yet, that is pretty much what happens today in the application development lifecycle. Development teams build and test their apps in pre-production before handing it off to operations for deployment in production, and they assume everything will work just fine. This is probably the worst assumption IT has made over the last decade, because development and production environments differ significantly. It’s also a lame excuse for any development team to use when a production issue occurs: “Well it ran fine in test so you must have deployed it wrong.” Yes people make mistakes occasionally, but if one in every three releases has an issue, deployment error may not be the sole reason. If development never get to see how their baby runs in production, they’ll never learn how to build robust, scalable, and high-performance applications.