Continuous Performance Testing in a DevOps World

The benefits of adopting a DevOps approach are widely known. By unifying developer and operations groups and emphasizing monitoring and automation, companies are increasing the speed and frequency of deployments and recovering faster from failures. The results of firms that successfully implement DevOps can be eye-opening. According to the 2017 State of DevOps report produced by Puppet and DORA (DevOps Research and Assessment), high-performing DevOps organizations reported 46 times more frequent code deployments and 440 times faster lead times from commit to deploy. Results like these are inspiring more companies to adopt DevOps. A separate survey by Forrester Research found 50% of companies implemented or expanded DevOps initiatives last year. Another 27% are planning to adopt DevOps by the fall of 2018.

Historically, however, the failure of DevOps initiatives is high. In 2015, Ian Head, a research director at Gartner, famously predicted that “90% of I&O organizations attempting to use DevOps without specifically addressing their cultural foundations will fail.”

In this four-part blog series, I argue that continuous performance testing holds the key to unlocking organizational transformation and DevOps success, and I lay out a methodology for creating an effective performance engineering program. This adaptable methodology is one that I have developed over years of trial and error working as a performance engineering lead and architect. The approach I recommend supports not just CI/CD, traditionally understood as continuous integration/continuous delivery, but also leads directly to a culture of continuous improvement in process, borrowing from Lean-Agile and Six Sigma concepts.

A NASA Mindset

To be good at DevOps, the latter needs to embrace the former. Ops needs to get on the Agile train. And, once ops is onboard, everyone will discover the importance of striving for Lean-Agile. Lean-Agile is rooted in Lean Manufacturing and focuses on eliminating waste. By reducing waste, you will increase the speed at which you get things done. Good CI/CD and DevOps not only continuously improves code and code quality, it enables the continuous improvement of the processes and automation servicing the software development life cycle.

Think about how much time systems in QA, UAT, and other lower environments sit idle. If you are like many organizations there is an immensity of wasted compute resources that can be converted into productive time by implementing and automating continuous testing. Simply decreasing idle time is not enough, however. Highly optimized processes for gathering metric data are vital if you are to be successful. To have really good metric data and telemetry, you need to approach performance testing like NASA does.

On a NASA mission, the failure of even a small component can be catastrophic. Long before an initial launch occurs, components are modeled with equations, assembled into composite systems, working their way up to increasingly complex systems, all with rigor in testing to ensure all variables are known. By the time a system is ready for launch, engineers fully understand all the environmental variables that lead to component failure and have optimized the processes for achieving the desired outcome.

In performance testing of software systems, variations or deviations in metrics during component tests must likewise be completely understood. They must have a positive or negative correlation coefficient to other metrics. If a metric deviation exists with a neutral coefficient, meaning the deviation is uncorrelated to another variable or cannot be explained, you cannot predict its behavior. In the modern software-defined world, when companies implement application performance monitoring absent a well-defined strategy, this is an all-too-common problem faced by DevOps. While AI and ML promise to rescue us, it’s still vital that teams understand why metrics deviate and strive to deeply understand the relationship between the variables that cause those deviations.

Organizations need to test mission-critical code with NASA-like rigor. You may be thinking that such meticulousness would lead to impossible bottlenecks. In fact, the opposite happens. By break-testing individual services continuously rather than trying to performance-test the entire system, you build confidence in every components’ tolerances and their impact on parent and child dependencies. Organizations will eliminate waste and achieve “leanness” by continuously running a multitude of small, repeatable, concurrent tests. Coupled with precision monitoring and pipeline automation serving as the basis for amplified feedback loops, this approach supercharges the CI/CD pipeline, and DevOps is unchained.

Hobbled by Old Habits

During the days of monolithic applications, the approach of discreetly testing individual components would have been hard, if not impossible. Today, I work with many organizations that are investing heavily in decomposing legacy apps into microservices, yet they still struggle to shift the mindset of their teams toward more effective component-level and scale-model testing.

Instead of testing individual services of an application with fast, repeatable performance tests and leveraging mocks for dependencies, many organizations run performance tests against the entire system as if it were still monolithic.

Performance teams spend an incredible amount of time setting up, configuring, and stressing an application at production-level load for 24, 36, or 72 hours to prove it’s stable, sometimes requiring developers and technical leads to bail out of the next sprint cycle to help out.

When these large-scale tests do break, it’s often hard to pinpoint the failure because they cannot be replicated consistently. Teams end up spending inordinate hours—days and sometimes weeks—troubleshooting issues and re-tuning parameters to keep an app from blowing up so they can release it to production.

Three Steps to Continuous Testing

Three things need to happen for developers and operations engineers to break their old habits and achieve the impressive DevOps results mentioned earlier.

First, basic DevOps principles should be in place. If QA and performance teams are still siloed, they should be reorganized and rolled up under operations. Operations team members should then be embedded with development teams. Ops engineers should take an active partnering role in amplifying the feedback loops to developers by writing stories in the backlog for performance issues and by participating in scrums and sprint retros. These ops engineers should become the automation experts at isolating, replicating and describing the environmental variables causing issues, and ensuring that test harnesses are continuously improving. In this way, the pipeline becomes more efficient, giving developers, tech leads, QA engineers, and product managers direct insight into what is happening in every stage leading up to production.

Second, if your tests are large, you need to start breaking them up. The goal is to componentize the tests and run as many tests as you can in a half-hour to an hour. This should be done at the API layer so that different services are tested at the same time but independently of one another. Each test should have an underpinning goal and should provide an answer to a specific what-if scenario.

Third, you want to replace downstream services with mocks wherever possible. This allows you to more easily test what-if scenarios for dependent services without relying on them to be up or stable.

As you continuously repeat the runs of smaller tests and begin to provide real-time feedback to developers, you should get to the point where you are able to form a hypothesis about how to improve your code and then quickly make the improvements. And, as you get into a more Lean-Agile state, you will be equipped to multiply the number of hypotheses that you are trying to derive answers for at any given time.

In today’s blog post, I’ve provided an overview of an approach to performance testing that enables effective DevOps, borrowing from Lean-Agile and Six-Sigma. In my next blog,  “The Importance of Application Decomposition in Performance Testing,” I’ll lay out the basis for how to properly instrument your applications so you can begin collecting high-quality metric data.

How Continuous Integration Works, and The Big Benefit No One Talks About

In a digital world that moves as fast as ours, programmers are applying new, creative ways of thinking to the software development process in a non-stop push for ever-faster turnaround times. In DevOps, Continuous Integration (CI) is increasingly the integration method of choice, in large part because of the speed at which it enables the release of new features, bug fixes, and product updates.

CI dictates that every time a developer pushes code to an application, an automated process grabs the current code from a shared repository, integrates the new build, and tests the software for problems. This approach leads to faster results and ensures software is tested on a regular basis, which enables further DevOps automation processes such as delivery, deployment, and experimentation.

How CI Works

With CI, the software code is contained in a shared repository, accessible by developers so they can “check out” code to individual workstations. When ready, they push the code back into the repository to commit changes. From there, the automated CI server takes over, automatically building the system and running unit and integration tests to ensure the code did not break another part of the software. If the build is unsuccessful, the server pinpoints where in the testing process the code failed, letting the team address it at the earliest point of failure.

This CI process occurs many times per day, meaning the system constantly builds and tests new code. The updated software can then be released manually, or DevOps teams can further automate the project by electing to have the system deploy and deliver the product.

Since developers don’t have to backtrack to find where code breaks, DevOps teams save big in terms of time and resources. And because the process is continuous, programmers never work on out-of-date products, or try to hurriedly push changes to keep up with demand or internal deadlines.

CI allows developers to automate long-running processes and use parallel containers or virtual machines to run tests and builds. Because these processes are automated, programmers can work on other tasks while everything runs in the background. And since the code is only merged once the build passes testing, the chances of breaking master code are greatly reduced.

The (not so) Secret Benefit of CI

Sure, CI saves time and reduces costs, but so does every other noteworthy innovation in technology or business processes these days. There’s another major reason CI is so successful that isn’t talked about as much because it’s more difficult to quantify than productivity and cost: team morale.

If you talk to any development team, they’ll tell you that almost nothing in the world is as frustrating as building a process, integrating it with the code, and then watching the software break. Not only have hours of work been wasted, but team members know that more hours lie in front of them trying to comb back through the process to pinpoint where it failed. As any business leader knows, an unhappy team results in an inferior and/or more costly product. As frustration mounts, work slows down. Then as a deadline approaches, changes are frantically pushed through, increasing the probability of a flaw in the master branch or a bug being deployed with the product.

The transparency of CI can be a big boost to the confidence level within DevOps. Suddenly, as developers work, they can see exactly where problems arise, which allows for a much faster response and repair. And if the build passes, team members can feel good about a job well done.

Takeaways for CIOs

The continuous integration approach to DevOps increases transparency, builds automation into processes, decreases costs by maximizing developers’ time, and creates repeatable processes upon proven successes. On top of all that, it relieves pressure from programmers and helps teams gain confidence.

Though there are variations among details of different platforms and approaches, the key tenets of CI hold true among development teams:

  • Maintain a single source repository with easy access for developers
  • Automate the build and testing processes
  • Make sure every build occurs on an integration container or VM
  • Utilize a production environment for testing
  • Make the testing results and processes transparent and visible to teams


If you want more speed and more smiles out of your development team, consider applying a continuous integration approach to your DevOps processes. Make sure to consult both with your team and a CI service provider to determine what makes sense for your organization and ensures a smooth implementation. Then sit back and watch the code fly.


The Role of APM in Continuous Integration and Continuous Release

Today’s software defined and driven business requires fast changes in business models, this permeates successful businesses. What this means is that almost every company is learning how to make small rapid changes and adjustments within their business, especially within software systems. The result of this is that IT is feeling immense pressure to evolve, the net result is a major uptick of private cloud (especially Apprenda, Pivotal Cloud Foundry, RedHat OpenShift) and public clouds (especially Amazon and Microsoft) within our install base of large enterprises. These new platforms enable faster development, testing, and releasing of software. Enterprise customers are getting much smarter, by using automation to drive the software lifecycle including continuous integration, and even experimenting and beginning to use continuous release processes. This is no longer a startup scenario, it’s a strategy for our large enterprise customers. These companies trying to build agile development and operations teams, by changing people, process, and technology to enable a DevOps feedback loop:

Wikipedia describes the continuous delivery loop as follows:


Within this loop, APM can be part of both build, unit, regression, and load tests to drive the automated acceptance testing before new code is pushed to production. In fact, at the recent AppSphere 2015 user conference we had several great talks on this subject.

The first talk was put on by our customer The Container Store in this presentation August Azzarello digs into “How The Container Store uses AppDynamics in the Development Lifecycle”. in their environment they began first with APM in production. The goal was to improve software quality before production, hence they began the expansion by installing APM within test, and integrating it into functional and performance test suites. They also enabled alerting from dev/test so that developers and testers understood when they had performance deviations and issues before production. Key features they leveraged in pre-production included dashboards such as this one:


Comparative analysis view such as this one:

and being able to understand between releases if they were improving or degrading:

The Container Store is using open source tools including Selenium for functional testing, and for performance testing. In the video August also explains some of the major benefits they saw from implementing APM in pre-production. Here are some best practices outlined by August in his discussion:

  • Monitor everything

  • Test continuously

  • Performance test early in development life cycle

  • Empower development & QA team members

Some benefits the container store has seen according to August are:

  • Set performance expectations before production deployments – ~40% improvement since we started using AppDynamics in test

  • Fine tune alert and Business Transactions policies prior to production deployments

  • Identify testing requirements, and testing gaps

  • Decrease performance test result reporting from 5 hours to 20 minutes

One of my favorite talks (which is why I selected it for the deep dive track at AppSphere) was the talk given by one of our Senior Sales Engineers Steve Sturtevant he came to AppDynamics from PayPal 2 years ago. In his talk titled “DevOps and Agile: AppDynamics in Continuous Integration Environments” Steve explains the integration work he did, but more importantly in dynamic environments can you scale and auto implement monitoring? Do you quantify the impact of change? Steve gives a good demo with integration between Puppet, Jenkins, ServiceNow, and AppDynamics in his examples with automation throughout the lifecycle.

One of our partners Electric Cloud specializes in continuous delivery, and embraces an open ecosystem of technology providers and open source software. They’ve put together some great resources for people looking to explore the rest of the feedback loop.