How to use AppDynamics with LoadRunner for Load Testing

While AppDynamics is most commonly used by IT Operations, Developers, and DevOps teams in production environments, to troubleshoot those difficult to recreate and see problems a growing trend of pre-production usage occurs. A large number of our users have been using AppDynamics for years within performance engineering practices. The key is being proactive, avoiding performance issues before going live in production. The following benefits are typically associated with integration of AppDynamics with load testing processes and technologies:

  • Gaining deep application-level visibility to answer the ‘Why?’ vs. simply determining ‘What?’
  • Rapidly pinpointing bottlenecks in their environment
  • Comparing results across releases and performance tests
  • Improving performance by measuring the end-to-end performance from the end-user perspective
  • Increasing scalability of the application by optimizing software and avoiding unnecessary hardware investments

While this blog centers on LoadRunner specifically, it is important to note that these use-cases may be applied using any load testing solution and methodology, not just LoadRunner. Here are the highlights of this technical whitepaper with deeper details.

The simplest and fastest way to get LoadRunner transactions in AppDynamics is ‘painting’ your LoadRunner requests. This method involves adding a custom header that contains the LR transaction name in the request so that AppDynamics can display the transactions exactly as they appear in LoadRunner.

  • Deep code-level visibility, in the context of a LoadRunner Transaction, end-to-end, inclusive of the end user experience.
  • Application container-level data across languages, for example, on Java this would include Heap size, standard and custom JMX metrics.
  • Deep database visibility, in the context of the transaction, including details SQL breakdowns
  • Extended visibility into application infrastructure (CPU, Disk, Network, and Memory).

AppDynamics provides these benefits automatically without the need to configure multiple HP tools (Example SiteScope, Diagnostics, and others). Once your tests are run, you can use custom time ranges which correspond to an important event. For performance engineering, these typically map to timeframes which correspond to a load test or an application build.

The Compare Releases functionality allows for comparing the performance of the application, inclusive of various KPIs, and trending metrics between any two timeframes.  This comparison can be made at the application-level or specific to individual elements. For performance engineers, being able to quickly compare results between two load tests, and identify deltas leads to rapid root-cause analysis, saving time and effort in every load testing cycle.

cid:image010.png@01D0EF1C.FCCDE8A0

The KPI view shows the top stats about the test, such as volume, average performance, the transaction scorecard and any events which occurred.

cid:image014.png@01D0EF1C.FCCDE8A0

Transaction snapshot comparisons allow for direct comparison of transaction snapshots related to specific transactions between both load tests. For reference, a snapshot is a deep-level capture of a single occurrence of a transaction that allows one to see code-level details of that transaction’s execution. By comparing snapshots, users can identify differences in code execution paths and identify slowness down to the individually-executed methods.

AppDynamics further extends application visibility from the code-level into the database, providing deep-level visibility in the SQL execution and detailed database performance metrics.  Acting as a bridge between the application team and the DBAs.  It provides a common viewpoint while showing data in context that is relevant for both parties.

Detailed breakdown of a single SQL statement, its execution over time, clients making the request, and the ability to pull an explain plan.  This enables a DBA to rapidly pinpoint the root cause for a poorly-performing SQL operation.

Database Time Range Comparison enables comparison of the performance of the database across two different time frames or the performance of two difference databases, such as Perf and Prod, for a similar timeframe.  This provides a similar functionality for databases as Compare Releases does for deep application visibility.

The Scalability and Correlation Analysis functions enables users to performance X-Y graphing comparisons and stacked graph comparisons of key information. This is useful when analyzing during a live load test or after a load test has completed to identify related behavioral patterns, such as inverse or direct relationships, across one or more different measurements, such as CPU vs. Load or Response Time.

cid:image017.png@01D0EF1C.FCCDE8A0

Scalability Analysis comes with several pre-built views for specific comparative analysis. CPU vs. Load comparisons may be made at the overall application level, or against individual elements such as a single application instance or transaction. Here we are using an X-Y graph to plot the data points against each other to see a direct relationship.

cid:image019.png@01D0EF1C.FCCDE8A0

Correlation Analysis functions selecting any two measurements for comparison. All measurements collected by AppDynamics are available within this view.  Both stacked charts and X-Y graphs are available.  In this example, the Errors per Min vs. Volume are plotted, showing a direct correlation between the increase in Volume and Error Rate.

AppDynamics has some features and functions which not only help production deployments but also add new visibility and analytics for those doing performance engineering. The whitepaper goes into far more details. In summary, the following key benefits are available:

  • AppDynamics can easily be configured to show LoadRunner transactions in the way that LoadRunner users see them today.
  • Flexible dashboards are not only reusable but enable historical and real-time analysis. This benefits users with a high degree of visibility while reducing configuration effort, saving time and manpower.
  • Automatic Baselining allows rapid comparative-analysis against a specific ‘master’ performance profile. This results in more realistic load testing analysis and higher-quality results by measuring against actuals vs. ‘guesstimates’ for expected performance.
  • Built-in comparison functionality between two different load tests facilitates rapid analysis and reduced effort to answer questions including ‘what changed?’ or ‘did performance improve between releases?’
  • Deep Code-level visibility provides performance engineers the ability to quick triage bottlenecks in the application, minimizing the need for developers during the analysis phase.  This keeps developers focused on building new functionality while reducing the time that performance engineers spend pinpointing problems.
  • Deep Database visibility offers contextualized database performance breakdowns to further reduce root-cause analysis times and provide details in the same ‘language’ that DBA’s use.
  • Real-time alerting during load tests enables notification of breached SLAs or deviations from expected results. This saves performance engineers time and allows for more rapid analysis of issues by surfacing errors more quickly.
  • Automatically-configured, out-of-the-box metrics reduce configuration time and deployment effort, as well as ongoing maintenance.

Interested in reading more? Check out the full whitepaper on LoadRunner + AppDynamics, written by my colleague David Hoenig.

Performance testing tools explained: The client side

In my last post I showed different approaches for load testing the server side. In this post I will highlight some tools I use to monitor the performance of the client side.

In modern Javascript-intensive web applications, users spend more time waiting on client-side rendering than the server-side processing. The reality is that as much effort that goes into optimizing the server-side, more effort should be made to optimize the client-side. End user monitoring has never been so important.

Why does performance matter?

Just to recap a few statistics about the business impact of performance at major internet companies:

Performance Impact

As you iterate through the software development cycle it is important to measure application performance on both the server side and the client side and understand the impact of every release. Here are some tools you can use to test the performance of the entire end user experience:

Google PageSpeed Insights

Google PageSpeed Insights provides actionable advice for improving the performance of client-side web applications. PageSpeed Insights analyzes the content of a web page, then generates suggestions to make that page faster. Reducing page load times can reduce bounce rates and increase conversion rates. The service is available as part of the Google Chrome Developer Tools as an extension, a web service, and extensions for Apache and Nginx.

Use the Google PageSpeed Insight API to integrate client-side optimizations into your continuous integration setup.

curl "https://www.googleapis.com/pagespeedonline/v1/runPagespeed?url=http://dustinwhittle.com/&key=xxx"

WBench

WBench is a tool that uses the HTML5 navigation timing API to benchmark end user load times for websites.

1) Install WBench:

gem install wbench

2) Run WBench:

wbench http://dustinwhittle.com/

WebPageTest.org

WebPageTest.org enables anyone to test the client-side performance on a range of browsers from anywhere in the world for free. This service is great and worth every penny I didn’t pay for it. Not only does it provide a range of mobile/desktop browsers and locations, but it also shows a waterfall timeline and a video of the rendering.

Screen Shot 2013-10-23 at 3.52.50 PM

AppDynamics

With AppDynamics Pro you get in-depth performance metrics to evaluate the scalability and performance of your application. Use the AppDynamics Pro Metrics Browser to track end user experience times and errors over the duration of the load tests:

With AppDynamics Pro End User Experience Dashboard you get visibility into both the server-side and the client-side:

Use AppDynamics Pro to compare multiple application releases to see the change in performance and stability:

release

In my next post in this series I will cover load testing tools for native mobile applications. Get started with AppDynamics Pro today for in-depth application performance management.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Performance testing tools explained: the server side

The performance of your application affects your business more than you might think. Top engineering organizations think of performance not as a nice-to-have, but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately, most engineering teams do not regularly test the performance and scalability of their infrastructure. In my last post on performance testing I highlighted a long list of tools that can be used for load testing. In this post we will walk through three performance testing tools: Siege, MultiMechanize, and Bees with Machine Guns. I will show simple examples to get started performance testing your web applications regardless of the language.

Why performance matters?

A few statistics about the business impact of performance at major internet companies:

Performance Impact

As you iterate through the software development cycle it is important to measure application performance and understand the impact of every release. As your production infrastructure evolves you should also track the impact of package and operating system upgrades. Here are some tools you can use to load test your production applications:

Apache Bench

Apache bench is simple tool for load testing applications provided by default with the Apache httpd server. A simple example to load test example.com with 10 concurrent users for 10 seconds.

Install Apache Bench:

apt-get install apache2-utils

Apache bench a web server with 10 conncurrent connections for 10 seconds:

ab -c 10 -t 10 -k http://example.com/

Benchmarking example.com (be patient)
Finished 286 requests

Server Software:        nginx
Server Hostname:        example.com
Server Port:            80

Document Path:          /
Document Length:        6642 bytes

Concurrency Level:      10
Time taken for tests:   10.042 seconds
Complete requests:      286
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      2080364 bytes
HTML transferred:       1899612 bytes
Requests per second:    28.48 [#/sec] (mean)
Time per request:       351.133 [ms] (mean)
Time per request:       35.113 [ms] (mean, across all concurrent requests)
Transfer rate:          202.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        9   20  14.2     17     148
Processing:   117  325  47.8    323     574
Waiting:      112  317  46.3    314     561
Total:        140  346  46.0    341     589

Percentage of the requests served within a certain time (ms)
  50%    341
  66%    356
  75%    366
  80%    372
  90%    388
  95%    408
  98%    463
  99%    507
 100%    589 (longest request)

 

Siege + Sproxy

Personally, I prefer Siege to Apache Bench for simple load testing as it is a bit more flexible.

Install Siege:

apt-get install siege

Siege a web server with 10 conncurrent connections for 10 seconds:

siege -c 10 -b -t 10S http://example.com/

** SIEGE 2.72
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:              263 hits
Availability:              100.00 %
Elapsed time:              9.36 secs
Data transferred:          0.35 MB
Response time:             0.35 secs
Transaction rate:          28.10 trans/sec
Throughput:                0.04 MB/sec
Concurrency:               9.82
Successful transactions:   263
Failed transactions:       0
Longest transaction:       0.54
Shortest transaction:      0.19

 

More often than not you want to load test an entire site and not just a single endpoint. A common approach is to crawl the entire application to discover all urls and then load test a sample of those urls. The makers of Siege also make sproxy which in conjunction with wget enables you to crawl an entire site through a proxy and record all of the urls accessed. It makes for an easy way to compile a list of every possible url in your appplication.

 

1) Enable sproxy and specify to output all the urls to a file urls.txt:

sproxy -o ./urls.txt

2) Use wget with sproxy to crawl all the urls of example.com:

wget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -e robots=on -e "http_proxy = http://127.0.0.1:9001" "http://example.com/"

3) Sort and de-duplicate the list of urls from our application:

sort -u -o urls.txt urls.txt

4) Siege the list of urls with 100 concurrent users for 3 minutes:

siege -v -c 100 -i -t 3M -f urls.txt

** SIEGE 2.72
** Preparing 100 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:              263- hits
Availability:              100.00 %
Elapsed time:              90.36 secs
Data transferred:          3.51 MB
Response time:             0.35 secs
Transaction rate:          88.10 trans/sec
Throughput:                0.28 MB/sec
Concurrency:               9.82
Successful transactions:   2630
Failed transactions:       0
Longest transaction:       0.54
Shortest transaction:      0.19

 

Multi-Mechanize

When testing web applications sometimes you need to write test scripts that simulate virtual user activity against a site/service/api. Multi-Mechanize is an open source framework for performance and load testing. It runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. Multi-Mechanize is most commonly used for web performance and scalability testing, but can be used to generate workload against any remote API accessible from Python. Test output reports are saved as HTML or JMeter-compatible XML.

 

1) Install Multi-Mechanize:

pip install multi-mechanize

2) Bootstrapping a new multi mechanize project is easy:

multimech-newproject demo

import mechanize
import time

class Transaction(object):
    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)

        start_timer = time.time()
        resp = br.open('http://www.example.com/')
        resp.read()
        latency = time.time() - start_timer

        self.custom_timers['homepage'] = latency

        assert (resp.code == 200)
        assert ('Example' in resp.get_data())

3) Run the multi-mechanize project and review the outputted reports

multimech-run demo

multimech

 

Bees with Machine Guns

In the real world you need to test your production infrastructure with realistic traffic. In order to generate the amount of load that realistically represents production, you need to use more than one machine. The Chicago Tribune has invested in helping the world solve this problem by creating Bees with Machine Guns. Not only does it have an epic name, but it is also incredibly useful for load testing using many cloud instances via Amazon Web Services. Bees with Machine Guns is a utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).

1) Install Bees with Machine Guns:

pip install beeswithmachineguns

 

2) Configure Amazon Web Services credentials in ~/.boto:

[Credentials]

aws_access_key_id=xxx
aws_secret_access_key=xxx

[Boto]

ec2_region_name = us-west-2
ec2_region_endpoint = ec2.us-west-2.amazonaws.com

 

3) Create 2 EC2 instances using the default security group in the us-west-2b availabily zone using the ami-bc05898c image and login using the ec2-user user name.

bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k aws-us-west-2 -l ec2-user

Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine guns...
.
.
.
.
Bee i-3828400c is ready for the attack.
Bee i-3928400d is ready for the attack.
The swarm has assembled 2 bees.

 

4) Check if the ec2 instances are ready for battle

bees report

Read 2 bees from the roster.
Bee i-3828400c: running @ 54.212.22.176
Bee i-3928400d: running @ 50.112.6.191

 

5) Attack a url if the ec2 instances are ready for battle

bees attack -n 100000 -c 1000 -u http://example.com/

Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 125 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 1 is out of ammo.
Bee 0 is out of ammo.
Offensive complete.
     Complete requests:   100000
     Requests per second: 1067.110000 [#/sec] (mean)
     Time per request:    278.348000 [ms] (mean)
     50% response time:   47.500000 [ms] (mean)
     90% response time:   114.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

6) Spin down all the EC2 instances

bees down

 

AppDynamics

With AppDynamics Pro you get in-depth performance metrics to evaluate the scalability and performance of your application. Use the AppDynamics Pro Metrics Browser to track key response times and errors over the duration of the load tests:

metrics browser

Use the AppDynamics Scalability Analysis Report to evaluate the performance of your application against load tests:

Screen Shot 2013-10-15 at 5.44.52 PM

Use AppDynamics Pro to compare multiple application releases to see the change in performance and stability:

release

Get started with AppDynamics Pro today for in-depth application performance management.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Tools of the trade for performance and load testing

Your application is fast and scalable, right? How do you know? How often do you run performance or load tests? In this post I will give an overview of the tools of the trade for performance and load testing web applications.

Open-Source performance testing tools

These tools allow you to load test your application for free. My preferred tool is Bees with Machine Guns — not just because of the epic name, but primarily because it uses Amazon’s EC2 to generate high levels of concurrency with ease.

  • Bees with Machine Guns – A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).
  • MultiMechanize – Multi-Mechanize is an open source framework for performance and load testing. It runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. Multi-Mechanize is most commonly used for web performance and scalability testing, but can be used to generate workload against any remote API accessible from Python.
  • Siege – Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”
  • HttpPerf – Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus of httperf is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httpperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility to new workload generators and performance measurements.
  • Apache Bench – AB is a tool for benchmarking your Apache HTTP server. It is designed to give you an impression of how Apache performs.
  • JMeter – Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, databases and queries, FTP servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

Performance testing tools as a service

Through these services you can build, execute, and analyze performance tests.

  • Apica Load Test – Cloud-based load testing for web and mobile applications
  • Blitz.io – Blitz allows you to continuously monitor your app 24×7 from around the world. You can emulate a single user or hundreds of users all day, everyday and be notified immediately if anything goes wrong.
  • Soasta – Build, execute, and analyze performance tests on a single, powerful, intuitive platform.
  • Blazemeter – BlazeMeter is a self- service performance & load testing cloud, 100% JMeter-compatible. Easily run tests of 30k, 50k, 80k or more concurrent users, on demand.

Performance testing on the client side

The best place to get started is at Google with the Web Performance best practices.

  • Google PageSpeed Insights – PageSpeed Insights analyzes the content of a web page, then generates suggestions to make that page faster. Reducing page load times can reduce bounce rates and increase conversion rates.
  • Google ngx_pagespeed – ngx_pagespeed speeds up your site and reduces page load time. This open-source nginx server module automatically applies web performance best practices to pages, and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.
  • Google mod_pagespeed – mod_pagespeed speeds up your site and reduces page load time. This open-source Apache HTTP server module automatically applies web performance best practices to pages, and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.

Web acceleration services

Through a simple DNS change, your website’s traffic is routed through these services and your content is optimized and cached globally for better performance. This is an easy way to improve performance with minimum efforts.

  • Yottaa – All-in-one web optimization solution delivers speed, scale, security and actionable insight for any website.
  • Cloudflare – Offers free and commercial, cloud-based services to help secure and accelerate websites.
  • Torbit – Torbit helps you accurately measure your website’s performance and quantify how speed impacts your revenue.
  • Incapsula – Incapsula offers state-of-the-art security and performance to websites of all sizes.

Recommended reading

A great book to learn about capacity planning and preparing for production traffic is The Art of Capacity Planning by John Allspaw. John Allspaw wrote the book from his experience scaling Flickr.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

AppDynamics and Apica: A Partnership Made for Performance

AppDynamics and Apica: A Partnership Made for PerformanceAt AppDynamics we’re always looking to partner with vendors that can significantly enhance the visibility we provide our customers when it comes to managing application performance in production. One area we see synergies is in synthetic monitoring and load testing, specifically for the next generation of cloud and mobile applications. So when Apica, the performance and load testing company for cloud, web and mobile applications came to us to join forces, we were stoked.

On June 26th, Apica and AppDynamics announced a partnership to provide DevOps an integrated monitoring solution to gain 10x visibility into application availability and performance, with the power to identify the root cause of slow downs and outages in as few as 3-clicks.

We’ve seen how development agility can be directly proportional to production fragility. Reproducing complex, distributed production architectures in test is tough, because data volumes, computing resource and user behavior always differ. This is why many organizations are starting to test application performance in production, so they can stress test out-of-hours and pro-actively identify severity-1 incidents before end users and the business is impacted.

Monitoring visibility across geo-locations, browsers and mobile devices

Giving DevOps the visibility they need to see how their applications perform, in production, under load, across geo-locations, different browsers and mobile devices helps DevOps be pro-active in managing application performance. This is where Apica’s synthetic monitoring and load testing products helps organizations understand how an application is performing from an end user, location, browser or device perspective.

“Users expect a fast and reliable web, cloud, and mobile experience. Every second delay can cost businesses valuable customers and revenue,” says Sven Hammar, CEO of Apica. “Together with AppDynamics, we’re providing users with best-of-breed solutions to ensure uptime and availability for revenue-critical applications. They’ll have the most complete understanding available of the metrics that are powering or causing problems for their applications so they can take measures to improve performance.”

AppDynamics integrates with the following three Apica products:

Apica WebPerformance – Verifies the performance and availability of your mission-critical applications from over 80 different locations world-wide.

Apica ProxySniffer – Automates creation of load test scripts and scenarios by recording real end-user traffic

Apica LoadTest – A cloud-based load testing tool for your mission-critical applications

You can access the Apica portal via any web browser or mobile device. Thru this new partnership AppDynamics real-time monitoring data is now seamlessly available in Apica portal, so DevOps can now drill-down a level further to understand the root cause of slowdowns and availability issues.

Apica Web Performance

Apica WebPerformance

Like AppDynamics, Apica has similar synergies to application performance management (APM):

Complete Lifecycle Visibility – We have customers that deploy our solutions in both production and pre-production environments. For Apica, scripts used for load testing thru LoadTest can also be applied to WebPerformance as well to test application availability and performance in production.

Real-time monitoring – understanding the true end user experience 24/7; specifically around business transactions and the application infrastructure, so pro-active alert notifications can be sent to DevOps when service degradation is identified.

Built for the Cloud – Apica LoadTest is cloud-ready so organizations can automate load tests and find the optimal configuration for their cloud deployments.

Below is a short clip of the direct integration of AppDynamics with Apica allowing DevOps to monitor all of the business transactions and system resources in real-time. There is also contextual business transaction drill-down that takes a user from the Apica portal into the AppDynamics user interface so the root cause of performance issues can rapidly found.


You can start your free trial with Apica’s performance and load testing solution by clicking here. For more details on monitoring your applications with Apica and AppDynamics, please visit Apica’s AppDynamics partner page.

 

Why Testing in Production isn’t as stupid as it sounds

One of my colleagues this week was consolidating the results from our recent Application Performance Management survey, and one interesting finding was that 40% of customers have at least one release cycle a month. Out of those respondents, one third experience a Severity-1 incident each month as well. That’s a pretty compelling pair of statistics, and they might explain the continued frustration and conflict between development and operations teams. It’s also perhaps the reason why this DevOps underground movement can no longer be ignored (even by Gartner). There is no doubt development organizations have become agile, but does deploying this frequent change make the business more or less agile? For example, if one in three releases creates a Severity-1 incident, then surely agile development becomes a risk to the business. We’re at the point where Operations either has to start managing change better or simply restrict the amount of change that can occur.

So why are Sev 1 incidents so common? Based on my experiences and customer interaction, I’d strongly argue that testing in development isn’t enough. At the very least, it’s certainly not an insurance policy for deploying an application in production. When a Formula 1 team designs a car in a wind tunnel and tests it on a simulator pre-season, they don’t assume that the performance they see in test will mirror the results they see in a race. Yet, that is pretty much what happens today in the application development lifecycle. Development teams build and test their apps in pre-production before handing it off to operations for deployment in production, and they assume everything will work just fine. This is probably the worst assumption IT has made over the last decade, because development and production environments differ significantly. It’s also a lame excuse for any development team to use when a production issue occurs: “Well it ran fine in test so you must have deployed it wrong.” Yes people make mistakes occasionally, but if one in every three releases has an issue, deployment error may not be the sole reason. If development never get to see how their baby runs in production, they’ll never learn how to build robust, scalable, and high-performance applications.