Tips for Modern Software Testing

Developers often dream of starting a new code project from scratch, but most have to make do with a legacy project. The best they can hope for is a rewrite.

I was lucky. In my previous role as a development manager at a large financial institution, I had the pleasure to work on a greenfield project, one of the company’s first truly agile efforts. How did we start? The first step was to form scrum teams and fill the backlog of work. One of the problems that arose early on involved software testing.

Many developers on our team were big fans of test-driven development (TDD), a software development process where you write tests before writing the code. The goal is to write a failing test first, although this test is really more of a specification before you start to flesh out the code. The major problem with TDD is that it focuses primarily on developers rather than embracing the entire organization. Enter BDD.

BDD: A Better Approach

Behavior Driven Development, or BDD, is a modern software testing method of getting testers, developers and the business to work together. The idea is to take things up a level and start talking about what your business and customers really want from a software product.

Once you manage to get these guys talking and collaborating with each other, you need a common language. This is often achieved with the famous Given-When-Then method of writing user stories. We used on our project, in fact.

However, there’s a risk that this formula gets abused, and that business leaders start writing stories with Given-When-Then syntax, only without the collaboration. This simply does not work.

Why? In my experience, the business often didn’t understand the terminology and would use it in a way that made no sense at all. More importantly, their approach to Given-When-Then was all wrong. For this method to work, the business needs to collaborate with developers and testers to write user stories.

Automating Your Tests

Once your team has a shared understanding of user behavior—and has articulated this in a common language—it’s time to start automating tests to prove the success of your business definition.

Unfortunately, many companies have a bad strategy when it comes to software testing: too many end-to-end tests and not enough unit tests. The problem here is that end-to-end tests take a long time to run and can be very difficult to maintain. A better approach is to run end-to-end tests and manual tests to cover only the business specification of the user story.

You want to avoid looking at edge cases and bad data in heavy end-to-end (e2e) tests, which are complicated, time-consuming and expensive to run. It’s smarter to do this in other parts of the test package, mainly unit, component, integration and contract tests.

In a unit test, for instance, you only have to exercise a tiny piece of code to check for proper operation. The idea here is to devise a test that achieves a small goal. But too often companies find themselves in a situation with a million end-to-end tests, and a test suite that takes weeks and weeks to run. The end result? They’re not getting any value from their tests.

Your ratio of UI and manual tests to unit tests should resemble the pyramid below. The higher the spot on the pyramid, the lower the number of tests:

Due to the complexity of maintaining end-to-end tests, some companies have ditched them altogether in favor of synthetic testing in production, something that AppDynamics is uniquely positioned to provide with our browser synthetic monitoring. For example, most organizations use Selenium to write their end-to-end tests, and we use Selenium browser automation on the backend for synthetic testing.

The Virtues of Contract Testing

For microservices testing, contract tests can provide the greatest value. When a consumer binds to a producer, a contract forms between them. Since components in a microservices architecture have many contracts, this area of testing becomes critical. In the old days of the monolithic application, code would compile together. Although there would still be implied contracts between various parts of the code, these contracts would be enforced at compile time. Since this safety net isn’t available in a microservices environment, you absolutely need reliable contract tests in place.

Each consumer forms a different contract based on how it uses a service. Here’s a quick  example I’ve borrowed from Martin Fowler:

  • A service exposes a resource with three fields: identifier, name and age.

  • Three consumers, each coupled to different parts of the resource, have adopted the service.

  • Consumer A couples only to the identifier and name fields. The corresponding contract test suite asserts that the resource response contains those fields, but makes no assertion concerning the age field.

  • Consumer B uses the identifier and age fields, but not the name field. Consumer C uses all three fields. The contract test suite for Consumer B makes no assertion for the name field. But for Consumer C, the test suit asserts that the resource response contains all three fields.

It’s important in microservices that these contracts be satisfied, even though downstream services are likely to change over time.

With contract testing, each consuming service runs a suite of unit tests that check whether the inputs and outputs are valid for their purposes. Those tests are then run by the producer as part of its build pipeline should any changes occur to that service.

In my next blog, I’ll cover the practical aspects of setting up a successful build pipeline, which covers contract and microservice testing.

Cloud Migration Tips #3: Plan to Fail

Planning to deploy or migrate an application to a cloud environment is a big deal. In my last post we discussed the value of using real business and IT requirements to drive the justification of using a cloud architecture. We also explored the importance of using monitoring information to understand your before and after picture of application performance and overall success.

In this post I am going to dive deeper into the planning phase. You can’t expect to throw a half assed plan in place and just deal with problems as they pop up during an application migration. That will almost certainly result in frustration for the end users, IT staff, and the business who relies upon the application.

In reality, at least 90% of your total project time should be dedicated to planning and at most 10% to the actual implementation. The big question is “What are the most important aspects of the planning phase?”. Here’s my cloud migration planning top 10 list:

  1.  Application Portfolio Rationalization  – Let’s face reality for a moment… If you’re in a large enterprise you have multiple apps that perform a very similar business function at some level. Application Portfolio Rationalization is a method of discovering the overlap between your application and consolidating where it makes sense. It’s like spring cleaning for your IT department. You need to get your house in order before you decide to start moving applications or you will just waste a lot of time and money moving duplicate business functionality across your portfolio.
  2. Business Justification and Goal Identification – If there is one thing I try to make clear in every blog post it is the fact that you need to justify your activities using business logic. If there is no business driver for a change then why make the change? Even very techie-like activities can be related back to business drivers.
    Example… Techie Activity: Quarterly server patching Business Driver: Failure to patch exposes the business to risk of being hacked which could cause brand damage and loss of revenue.
    I included goal identification with business justification because your goals should align with the business drivers responsible for the change.
  3. Current State Architecture Assessment (App and Infra) – This task sounds simple but is really difficult for most companies. Current State Architecture Assessment is all about documenting the actual deployed application components, infrastructure components, and application dependencies. Why is this so difficult? Most enterprises have implemented a CMDB to try and document this information but the CMDB is typically manually populated and updated. What happens in reality is that over time the CMDB is neglected when application and infrastructure changes occur. In order to solve this problem some companies have turned to Automated discovery and dependency mapping tools. These tools are usually agentless so they login to each server and scan for processes, network connections, etc… at pre-defined intervals and create a very detailed mapping that includes all persistent connections to and from each host regardless of whether or not they are application related. The periodic scans also miss the short lived services calls between applications unless the scan happens to be at approximately the same time of the transient application call. An agent based APM tool covers all the gaps associated with these other methods.


    How well do you know the current architecture and dependencies of your application?

  4. Current State Performance Assessment – Traditional monitoring metrics (CPU, Memory, Disk I/O, Network I/O, etc…) will help you size your cloud environment but tell you nothing about the actual application performance. The important performance metrics encompass end user response time, business transaction response time, external service response time, error and exception rates, transaction throughput, with baselines for each. This is also a good time to make sure there are no glaring performance issues that you are going to promote into your cloud environment. It’s better to fix any known issues before you migrate as the extra complexity of the cloud can amplify your application problems.

    Screen Shot 2012-08-14 at 1.59.33 PM

    You don’t want to carry application problems into your new environment.

  5. Architectural Change Impact Assessment – Now that you know what your real application and infrastructure components are, you need to assess the impact of the difference between traditional and cloud architectures. Are there components that wont work well (or at all) in a cloud architecture? Are code changes required to take advantage of the dynamic features available in your cloud of choice? You need to have a very good understanding of how your application works today and how you want it to work after migration and plan accordingly.
  6. Problem Resolution Planning – Problem resolution planning is about making a commitment to your monitoring tools and strategy as a core layer of your overall application architecture. The number of potential points of failure increases dramatically from traditional to cloud environments due to increased virtualization and dynamic scaling. In highly distributed applications you need monitoring tools that will tell you exactly where problems are occurring or you will spend too much time isolating the problem location. Make monitoring a part of your application deployment and support culture!!!
  7. Process re-alignment – Just hearing the word “process” makes me cringe and have flashbacks to the giant, bloated , slow moving enterprise environments that I used to call my workplace. The unfortunate truth is that we really do need solid processes if we want to maintain order and have any chance of managing a large environment in a sustainable fashion. Many of the traditional IT development and operations processes need to be modified when we migrate to the cloud so you can’t overlook this task.
  8. Application re-development – The fruits of your Architectural Change Impact Assessment work will probably precipitate some level of development work within your application. Maybe only minor tweaks are required, maybe significant portions of your code need to change, maybe this application should never have been selected as a cloud migration candidate. If you need to change the application code you need to test it all over again and measure the performance.
  9. Application Functional and Performance Testing – No surprises here, after the code has been modified to function as intended with your cloud deployment it needs to be tested. APM tools really speed up the testing process since they show you the root of your performance problems down to the line of code level. If you rely only upon the output of your application testing suite your developers will spend hours trying to figure out what code to change instead of minutes fixing the problematic code.

    Call Graph

    Know exactly where the problem is whether it is in your code or an external service.

  10. Training (New Processes and Technology) – With all of those new and/or modified processes and new software required to support your cloud application training is imperative. Never forget the “people” part of “people, process, technology”.

There’s a lot more that really goes into planning a cloud migration but these major tasks are the big ones in my book. Give these 10 tasks the attention they deserve and the odds will shift in your favor for a successful cloud migration. Next week we’ll talk about important work that should happen after your application gets migrated.