Cost of Performance Issues [INFOGRAPHIC]

We all know performance issues — application crashes, stalls, slow downs, etc. — can hurt our business and reputation. However, how can we put an exact dollar amount to these issues, and are other companies experiencing the same problems? Surely Fortune 500 companies can’t afford to have performance issues, right?

Wrong. Everyone experiences issues, it’s about limiting these problems and resolving them as quickly as possible to curb the impact to your bottom line.

99% uptime, or more commonly known as “two nines” in the IT world, still means you’re down over three and a half days per year. If one of those days happened to be Cyber Monday or another peak period, Amazon, Walmart, or Best Buy would definitely be able to see the actual cost and consequences of performance issues.

So we decided to do some research and create this infographic around the prevalence of performance issues and how they impact the bottom line. What we found might surprise you:

Don’t wait to be reactive with your performance issues, check out a FREE trial of AppDynamics today!

Beyond DevOps … APM as a Collaboration Engine

In the beginning there was a simply acronym: MTTI (mean time to innocence). Weary after years of costly and time-consuming warroom battles, IT organisations turned to AppDynamics to give an objective application-level view of production incidents. As a result, application issues are swiftly pinpointed and fixed, accelerating time to repair by up to 90%.

In fact, gravitation towards fact-based constructive issue management spawned a whole new movement – DevOps – with the goal of ingraining this maturity and cooperative spirit into IT organisations from the ground up. The movement was discussed by Jim in a previous blog post. Of course, AppDynamics (or at least, easily accessible fact-based information about application behaviour in production) is a necessary prerequisite to this.

Looking back, before DevOps or even MTTI were topical buzzwords, this basic ability to foster communication between teams proved to be an invaluable benefit to the more drab and well-worn business realities of offshoring and outsourcing.

This blog reviews 3 real-life examples of this:

  • Managing external offshore development organisations
  • Facilitating near shore development teams
  • Bringing external developments in-house

Managing external offshore development organisations

Some months ago, I did some work with a then prospect who had started a self-service trial of AppDynamics.

When I spoke to them, they were delighted with the visibility that AppDynamics provided out of the box for their .NET application, a SaaS Learning Management System.

Digging into what had sparked their interest in AppDynamics, they told me they had commissioned an outside development firm to rewrite their flagship application, which was somewhat dated and not architecturally fit to support some newer services the business wanted to offer to customers.

The good news was the new version of the app was live, and supporting around 10% of their existing customer base. The bad news?  This 10% used the same hardware footprint as the remaining 90% on the old system. Extrapolating this hardware requirement for the entire user base would not only require a new datacentre, but also entirely break the business model for the application from an operational cost perspective (not the first time that hardware savings alone could pay for AppDynamics!)

For months prior to trying AppDynamics, the external developers had been under huge pressure to optimize the application footprint (and some pretty lacklustre performance too). Armed with only windows performance counters and intuition, weeks had been spent optimizing slow database queries, which only turned out to be 5% of the errant response times at a transaction level.

Having put AppDynamics in place, the prospect

  • Easily found specific application bottlenecks, allowing them to focus developers on high-impact remediation
  • Could verify the developers had made the required improvements with each new release

Clearly, huge benefits at a technical level.

At a higher level, this helped lead to a more constructive relationship between the development shop and their customer – moving things away from the edge of litigation, constant finger-pointing, and blame shifting.

Facilitating near shore development teams

Another group I have worked with recently are responsible for a settlement system within a large global investment bank based in London. The system is developed in-house, and typical with most financial services institutions, the actual development team itself is located ‘near-shore’ in Eastern Europe to cut costs. The development processes are Agile, with new releases every few weeks.

Inevitably, with new releases can come new production issues and – of course – the best people to deal with these during the “bedding in” period are the developers themselves.

Another thing that is very common in the financial services industry is regulation, and this poses a problem in this scenario. Nobody is permitted to directly access the production systems from outside the UK due to data privacy regulations.

This means hands-on troubleshooting must be left to the on-shore architect staff who are not only expensive, but are not as well-equipped as the developers themselves to dig in to the issues in new code.

Enter AppDynamics. Our agents deployed in production made all the performance data readily available to anyone with the appropriate credentials, but – critically – having access to this does not expose ANY business data from the production system. Now, the near-shore development team can look directly at the non-functional behaviour of their code in production, eliminating the time spent gathering sufficient log data to enable reproduction of issues in test environments.  Bingo, the business case for the AppDynamics purchase is made!

There is an interesting side note to this, which applies much more widely too. Many customers have observed an “organic” improvement in service levels once AppDynamics is installed in production. For the first time, developers can see how their code is actually working in the wild. Developer pride kicks in and suddenly non-functional stories are added to development backlogs to fix latent issues that get observed, which would have previously have gone unnoticed.

Bringing external developments in-house

Of course, as we all know the only constant in life is change, so no outsource is a one-way journey. As a result, I have come across several organisations that are now working on projects which were previously outsourced. Of course, once these customers have completed the initial challenge of recruiting a new development team they then need to get their arms around the existing codebase. Usually handover workshops can help with this, but in many cases these systems have been out- and in- sourced several times, with many changes of personnel along the way. There is only so much you can distill onto a whiteboard in a brain dump session, however long and well-intentioned.

It is here where the high-level visibility that AppDynamics provides can be invaluable. Out of the box, AppDynamics instruments previously unseen systems, automatically detecting and following transactions and draws up flow-maps. The end-to-end visibility of the entire system greatly eases the process. In fact, this system overview (and the ability to view how it changes over time) has proved invaluable for many customers for a number of reasons beyond whole-scale in (or out) sourcing, such as onboarding new team members, verifying compliance with architectural governance of externally developed code changes and so forth.

Conclusion

In summary, AppDynamics does not have to be all about troubleshooting and MTTI.  Nor even necessarily about DevOps and brave new worlds. The easily configured deep insight that we provide into the dynamic behaviour of your applications has many uses – and business cases – beyond the traditional MTTI/MTTR domain.  APM is, after all, just one use-case (albeit an important one) for our Application Intelligence Platform.

Take five minutes to get complete visibility and control into the performance of your production applications with AppDynamics Pro today.

The APPrentice

Screen Shot 2013-05-28 at 3.04.27 PMIn this week’s episode, Donald Trump enlists Team ROI and Team Overhead to solve a Severity1 incident on the “Trump Towers Website”. Team Overhead used “Dynoscope” and took 3 weeks to solve the incident, while Team ROI took 15 minutes by using AppDynamics.

 

Data Stinks, Information Rocks!

I’ve got many years of performance geekery under my belt and I’ve learned many lessons during that time. One of the most important lessons is paying close attention to the distinction between data and information. Let’s take a look at how the dictionary defines each term:

Data – facts and statistics collected together for reference or analysis.
Information – facts provided or learned about something or someone.

What do these definitions reveal to us about data and information and how does it apply to monitoring tools? Let’s explore that together. I’ll provide specific examples along the way to illustrate my points.

The Problems with Data

Data is fundamental to problem solving, but I don’t want to have to dig through a bunch of data while my business critical, mission critical, revenue generating, etc… applications are down. To me, data is just like this picture…

SteamingPile

Covis Software GmbH Speeds up CRM Application by 33x with AppDynamics

I received an X-Ray from Covis Software GmbH in Germany who provides CRM solutions. They’ve been managing their .NET application performance with AppDynamics Pro for several months now in production. In the below X-Ray (as documented by the customer), Covis were able to improve the performance of a mission-critical business transaction from over 10 seconds to around 300 milliseconds, representing a 33x improvement. The business impact of such change meant average call time for some CRM agents dropped by almost 10 seconds.

If you would like to get started with AppDynamics you can download AppDynamics Lite (our free version) or you can take a free 30-day trial of AppDynamics Pro.

App Man.

4 Lessons that Operations Can Learn From Batman

In honor of the summer blockbuster The Dark Knight Rises, let’s take a look at one of the most celebrated heroes in history—Batman. (Okay, granted, he’s not real. But that doesn’t mean he can’t help us understand some best practices regarding application uptime and availability, right? Look, just bear with me here.)

1. Always have the right tools.

Batman knows his limitations: he doesn’t have superpowers (unlike my esteemed colleague, App Man). He’s just an ordinary guy with a weird-looking suit and five tons of cash. So what does he do? He buys batarangs, batcopters, bat anti-shark repellant, etc. He knows that in order to give himself an edge against the forces of darkness, he needs serious gadgets.

Similarly, Application Operations teams need to recognize that they shouldn’t wade into a production outage or firefight without some kind of application management solution. It’s amazing how many times we talk to teams that are simply using log files to manage a production application. Log files are not effective tools in a true firefight, and they’re no way to get to root cause quickly when your end users experience poor performance. Get an APM solution that equips you with 10x visibility and code-level detail in production, at less than 2% overhead. Otherwise, your office may start to feel like the rough & tumble streets of Gotham.

2. The bad guys always come back.

How many times has Batman stuck the Joker in Arkham Asylum, only see him escape and start inventing new death traps? But the Dark Knight doesn’t grumble—well, actually, he does, but let’s not worry about that right now—he just suits up and goes back on patrol.

Similarly, poor performing business transactions, memory leaks, and slow SQL calls are never going to disappear entirely, no matter how many iterations your developers perform. Agile release cycles and complex code mean that no environment will be completely free of evil. It’s important to have the right web application monitoring solution to put your rogues gallery on ice, time and time again.

3. War Room sessions are a waste of time.

“It’s the network.” “No, it’s the database.” “No, it’s the application.” How much time does your team spend pointing fingers and establishing innocence? And in the meantime, how many more issues are bound to terrorize your end users?

Batman doesn’t sit in a fire circle and hold hands with his colleagues. He doesn’t parley with the police and he doesn’t make long speeches. He gets the job done. The right application performance management tool can do the same for your own attempts at application heroism: less on the fingerpointing, more on the Mean-Time-to-Resolution.

4. Never stop until you find root cause.

One of the things that people forget about Batman is that he’s not just a superhero: he’s also the World’s Greatest Detective. He uses his brains as often as his fists. And when it comes to a mystery, he’s not content to just a get a vague sense of the story, or poke around the edges of a crime scene. Rather, he wants to find the killer and the smoking gun.

In the world of application performance, that means getting to code-level detail in a production environment. It does not mean using a tool to monitor infrastructure or a network monitor that skims the surface of the application layer.  Those tools can get close, but “close” only works in horseshoes and hand grenades. What’s necessary is to find the true culprit, the method and class that’s causing a production outage or performance issue. And if you have the right tool for the job, you won’t be patrolling all night in order to do it. Rather, you can solve the mystery in literally minutes.

One lesson you might not want to take from the Caped Crusader is to cough up a huge P.O. for your APM solution. Not everyone can be a bazillionaire like Bruce Wayne. And it’s possible to arm yourself for battle without making your IT budget as big and bloated as the Penguin.

Don’t believe me? Check out the pricing for AppDynamics Pro for yourself—then send up a signal to take our 30-Day Trial. When you get a taste of what we can do, you may find yourself swinging from the rooftops.

Storm Clouds in 2012? – Results of AppDynamics APM Survey

We recently finished conducting our annual Application Performance Management survey. Over 250 IT professionals participated, and they shared insights such as:
– Many Ops and Dev teams are anticipating growth in their applications by 20% or more
– Over 50% are planning to move to the cloud, and are architecting brand-new applications to be cloud-ready
– Most teams are using log files to monitor application performance, rather than an Application Performance Management (APM) tool.

We’ll release the full report soon, but here’s an infographic that summarizes some of the main findings:

AppDynamics Inforgraphic - Storm Clouds in 2012

Embed this image on your site:

What I found personally surprising was the heavy reliance on log files. When you’re troubleshooting distributed architectures, time is of the essence–and there’s no way to cut your MTTR down when you’re relying on log files to identify root cause.

In fact, there’s only one guy who ever made using a log file look cool:

And I think we can all agree that’s a pretty unique use case.

We’ll have the full survey results available soon.

 

 

Online media company gets proactive with application monitoring in production

On Wednesday I delivered a keynote at WJAX in Munich. Everything went really well, but I was a little shocked at the response I got when I asked the audience “How many of you monitor the performance of your apps in production?” As I scanned the audience, I counted 9 out of ~950 developers had put their hands up, meaning about 1% had visibility of how their applications actually performed in production. I know what you’re thinking: “But isn’t application performance in production the responsibility of Operations?”  Well, it is and it isn’t. Most organizations think that when an application has an issue, it’s related to the infrastructure it runs on. That’s like saying when a car crashes, it’s because a part failed on the car whereas in actual fact most accidents are caused by the driver. Yes, hardware fails occasionally, but application logic and configuration drives how infrastructure resource is used, which is why most issues today occur when new code is deployed in production.

Planning for Failure with Application Performance Management (APM)

If you are running mission-critical applications and do not have a strategy to deal with failure, you are putting your whole organization at risk. You may think that your application cannot fail, but at some point everything fails.  It may not be the software running your application that fails – it could be the hardware, the network, or even a natural disaster in your area that causes your application to go down. In case of such failure, no matter how rare, your customers will still expect the same level of service, not to mention preservation of their data.  Without a failover strategy and a tested backup infrastructure you will be out of service for an unknown period of time, which will lead to angry customers and loss of revenue.

Most of you have some failover strategy in place. I’m sure many of you have spent large amounts of time and money ensuring that your application is resilient to failure because you understand your app’s importance. But you may still be missing one key component, without which you are still at risk. That component is monitoring, or more specifically, Application Performance Management (APM). While mission-critical applications rely on an APM system to help monitor application performance and health, APM is often forgotten on the failover systems. APM needs to goes hand-in-hand with any failure testing plan to ensure that your company’s strategy will work in case of a real emergency.

AppDynamics helps Insurance Customer avoid Production Outage

Last week I published my winning Customer X-Ray of the Quarter, which showed how AppDynamics was able to help a media customer solve a production issue that had plagued their application for over two years. This week I’m posting the runner-up X-Ray entry. This one describes how AppDynamics was able to help an Insurance customer avoid a production outage by spotting a major bottleneck as their application was migrated from dev to pre-production during performance testing. All of the X-Rays you see published in this blog were written by customers, so the stories you read are real, factual, and credible.