The Fourth Digital Wave: The Age of Application Intelligence

This post originally appeared on APM Digest

Welcome to the fourth era of digital.

The first three periods or ages or phases — call them what you like — were each defined clearly by transformative events.

First, the dawn of the personal computer age in April 1977 with the debut of the Apple II (and validated in August 1981 with the introduction of the IBM PC).

Next, the beginning of the Internet age when the Netscape browser was released in 1994, which redefined forever the way we connect.

Then, on June 29, 2007 — ushered in again by Steve Jobs and Apple — the mobility era began with the unveiling of the first iPhone, which ushered in a “Mobile First” mindset for the masses.

And now we’re in the fourth era. This time there’s been no single, monumental event or technology to mark its beginning, though mobility and the cloud are the primary enabling technologies. What’s happening instead is that a number of technologies are coalescing and achieving, even as we speak, a critical mass that will make this age as transformative or more so than any of the previous three.

This is the age of multi-device mobility, the cloud, seamless computing from one device to another, a growing ecosystem of connected devices (watches, cars, thermostats), instant and ubiquitous communication, the blurring of the lines and hours between work and not-work. It’s a transformation that may have started with the smartphone, but has now engulfed everything about the way we use technology for, well everything.

Organizations that master the ability to collect, understand and act upon knowledge derived from user experiences, application behaviours, and infrastructure use from across this connected ecosystem will outcompete those that don’t, and win in this fourth era of Digital: The Age of Application Intelligence.

A Tectonic Technological Shift

There’s really no precedent for the speed of what has become a tectonic technological shift. In her much-anticipated Internet Trends 2014, KPCB’s Mary Meeker characterizes a tech market that saw 20 percent growth for smartphones, 52 percent for tablets, and 82 percent for mobile data in 2013. She predicts 10x growth in mobile Internet units in this decade — from the one billion-plus desktop Internet units/users to more than 10 billion for the mobile Internet.

Seemingly overnight, we have new models for hardware and software development, new models of behavior, and unforgiving expectations from consumers — for more apps, more functionality, more entertainment, more speed — driven by mobility, but extending to all online experiences regardless of interaction preference.

This is good, and it’s a great time to be in the thick of the enabling technology platforms — if you’re functioning with a model designed for this fourth era of digital.

On the other hand, it’s a pretty challenging time if you’re dealing with technology that matured early in the 2000s. Think huge, monolithic apps, sprawling private data centers with proprietary consoles for every piece of your infrastructure supported by “engagements” — a very loaded term — when a literal or figurative truckload of consultants, engineers, and programmers would descend on an enterprise and spend several months and multiple man-years engrossed in a single project only to emerge at the end with a big, bloated, largely rigid “deliverable.”

And if the applications themselves were large and unwieldy and slow to adapt, the Application Performance Management systems were (and legacy systems still are) similarly complex, difficult to adapt, and slow to process the limited amount of data they collected. The notion of “real time” was not even a consideration.

It wasn’t that long ago, but it’s hard to imagine trying to do business like that today. And in fact, you really can’t do business that way today. Some of the legacy APM platforms are trying to make the transition. But it’s a difficult maneuver that requires the kind of wholesale reinvention that few entrenched enterprises are willing to attempt, or that those brave enough to try can accomplish successfully.

The recent challenge faced by OpTier is a case in point. It’s always a bit alarming to see a player leave the arena, even a competitor. But it’s not likely to be the last such story we’ll hear.

Whether you’re building the applications themselves or the platforms to optimize their performance and business value, today everything is about speed, agility, and creating exceptional end-user experiences.

If you’re providing the applications, that means you have to be able to iterate quickly — often multiple times per day — and deliver the features and functionality your customers want, whether they’re outside or inside your enterprise. And of course, your apps have to be continuously available and meet your customers’ expectations for speed and performance, whatever OS or device they’re using. And you have to do this in an environment that is distributed, heterogeneous, complex, and ever-changing.

To pull this off requires a level of application intelligence designed specifically to succeed with these challenges in these environments.

Delivering Real APM Value

Specifically, for an APM platform to deliver real value for the application and the enterprise, it has to satisfy a number of key requirements, including:

  • Fast setup: minutes or hours vs. days or weeks, without need for a professional services ‘engagement.’
  • Self-learning, auto-configuring: Your apps and infrastructure change frequently; your APM platform needs to automatically detect and learn those changes and configure itself in real time, without manual intervention; there’s simply no time or resources for that.
  • Detect, diagnose, and respond: If there’s a problem, a slowdown, an outage, your APM platform should be the first to know about it, and whenever possible, should fix it before you know about it; or if it requires a bigger intervention, give you the data you need to solve it quickly.
  • Deliver actionable intelligence in real-time: In the old days, APM was about speed and availability and not much else. In today’s software-enabled enterprises, the APM platform not only has to measure, monitor, and manage system health, it has to be able to tell you, in real time, what impact performance is having on the business. It’s a focus far beyond availability and throughput, on the business transaction for the end user.
  • Provide end-to-end transaction visibility: Your applications may be running on your premises, in the cloud, or both; you need to be able to see what’s happening everywhere, through one pane of glass, because you can only manage, fix, and optimize performance that you can see.
  • Be insanely fast: When you do a release, you need to know immediately what’s working, what’s not, and how to fix things in a hurry, live, in production.

And it has to be stingy with the overhead, be able to scale itself and your applications up or down in response to changing demand, make the most of your resources and infrastructure, and many more things.

That’s a far cry from the big, heavy, slow systems and processes of a few short years ago. And characteristics like these don’t just apply to APM — it’s the way of all technology development today, from VR gaming headset hardware to massive e-commerce systems. Fail fast and recover (smarter next time). Design from the outside-in. Iterate quickly. Respond in real time. Innovate faster than the competition, in technology and marketing. Create user experiences that drive success.

In Internet Trends, Mary Meeker says that “New companies — with new data from new device types — [are] doing things in new ways and growing super fast.” And she describes the rapid growth of “uploadable/sharable/findable real-time data.” These are ideas that describe much of what is driving this new, fourth era of digital.

The old adage is true now more than ever: Change is the one constant you can count on. Those organizations who can adapt continuously are the ones that will thrive and win.

This post originally appeared on APM Digest

What APM Vendors can learn from building Supercars

P1McLaren this year will launch their P1 Supercar, which will turn the average driver into a track day hero. What’s significant about this particular car is that it relies on modern day technology and innovation to transform a drivers ability to accelerate, corner and stop faster than any other car on the planet–because it has:

  1. 903bhp on tap derived from a combined V8 Twin Turbo and KERS setup, meaning it has a better power/weight ratio than a Bugatti Veyron
  2. Active aerodynamics & DRS to control the airflow so it remains stable under acceleration and braking without incurring drag
  3. Traction control and brake steer to minimize slip and increase traction in and out of corners
  4. 600Kg of downforce at 150mph so it can corner on rails up to 2G
  5. Lightness–everything exists for a purpose so there is less weight to transfer under braking and acceleration

You don’t have to be Lewis Hamilton or Michael Schumacher to drive it fast. The P1 creates enormous amounts of mechanical grip, traction, acceleration and feedback so the driver feels “confident” in their ability to accelerate, corner and stop, without losing control and killing themselves. I’ve been lucky enough to sit in the drivers seat of a McLaren MP4-12C and it’s a special experience – you have a driving wheel, some dials and some pedals – that’s really it, with no bells or whistles that you normally get in a Mercedes or Porsche. It’s “Focused” and “Pure” so the driver has complete visibility to drive as fast as possible, which is ultimately the whole purpose of the car.

How does this relate to Application Performance Monitoring (APM)?

Well, how many APM solutions today allow a novice user to solve complex application performance problems? Erm, not many. You need to be an uber geek with most because they’ve been written for developers by developers. Death by drill-down is a common symptom because novice APM users have no idea how to interpret metrics or what to look for. It would be like McLaren putting their F1 wheel with a thousand buttons in the new P1 road car for us novice drivers to play with.

It’s actually a lot worse than that though, because many APM vendors sell these things called “suites” that are enormously complex to install, configure and use. Imagine if you paid $1.4m and McLaren delivered you a P1 in 5 pieces and you had to assemble the engine, gearbox, chassis, suspension and brakes yourself? You’d have no choice but to pay McLaren for engineers to assemble it for with your own configuration. This is pretty much how most vendors have sold APM over the past decade–hence why they have hundreds of consultants. The majority of customers have spent more time and effort maintaining APM than using it to solve performance issues in their business. It’s kinda like buying a supercar and not driving it.

Fortunately, a few vendors like AppDynamics have succeeded in delivering APM through a single product that combines End User Monitoring, Application Discovery and Mapping, Transaction Profiling, Deep Diagnostics and Analytics. You download it, install it and you solve your performance issues in minutes–it just works out-of-the-box. What’s even great is that you can lease the APM solution through annual subscriptions instead of buying it outright with expensive perpetual licenses and annual maintenance.

If you want an APM solution that lets you manage application performance, then make sure it does just that for you. If you don’t get value from an APM solution in the first 20 minutes, then put it in the trash can because that’s 20 minutes of your time you’ve wasted not managing application performance. Sign up for a free trial of AppDynamics and find out how easy APM can be. If these vendors built their solutions like car manufacturers build supercars, then the world would be a faster place (no pun intended).

Appman.

The APPrentice

Screen Shot 2013-05-28 at 3.04.27 PMIn this week’s episode, Donald Trump enlists Team ROI and Team Overhead to solve a Severity1 incident on the “Trump Towers Website”. Team Overhead used “Dynoscope” and took 3 weeks to solve the incident, while Team ROI took 15 minutes by using AppDynamics.

 

Finding the Root Cause of Application Performance Issues in Production

The most enjoyable part of my job at AppDynamics is to witness and evangelize customer success. What’s slightly strange is that for this to happen, an application has to slow down or crash.

It’s a bittersweet feeling when End Users, Operations, Developers and many Businesses suffer application performance pain. Outages cost the business money, but sometimes they cost people their jobs–which is truly unfortunate. However, when people solve performance issues, they become overnight heroes with a great sense of achievement, pride, and obviously relief.

To explain the complexity of managing application performance, imagine your application is 100 haystacks that represent tiers, and somewhere a needle is hurting your end user experience. It’s your job to find the needle as quickly as possible! The problem is, each haystack has over half a million pieces of hay, and they each represent lines of code in your application. It’s therefore no surprise that organizations can take days or weeks to find the root cause of performance issues in large, complex, distributed production environments.

End User Experience Monitoring, Application Mapping and Transaction profiling will help you identify unhappy users, slow business transactions, and problematic haystacks (tiers) in your application, but they won’t find needles. To do this, you’ll need x-ray visibility inside haystacks to see which pieces of hay (lines of code) are holding the needle (root cause) that is hurting your end users. This X-Ray visibility is known as “Deep Diagnostics” in application monitoring terms, and it represents the difference between isolating performance issues and resolving them.

For example, AppDynamics has great End User Monitoring, Business Transaction Monitoring, Application Flow Maps and very cool analytics all integrated into a single product. They all look and sound great (honestly they do), but they only identify and isolate performance issues to an application tier. This is largely what Business Transaction Management (BTM) and Network Performance Management (NPM) solutions do today. They’ll tell you what and where a business transaction slows down, but they won’t tell you the root cause so you can resolve the issues.

Why Deep Diagnostics for Production Monitoring Matters

A key reason why AppDynamics has become very successful in just a few years is because our Deep Diagnostics, behavioral learning, and analytics technology is 18 months ahead of the nearest vendor. A bold claim? Perhaps, but it’s backed up by bold customer case studies such as Edmunds.com and Karavel, who compared us against some of the top vendors in the application performance management (APM) market in 2011. Yes, End User Monitoring, Application Mapping and Transaction Profiling are important–but these capabilities will only help you isolate performance pain, not resolve it.

AppDynamics has the ability to instantly show the complete code execution and timing of slow user requests or business transactions for any Java or .NET application, in production, with incredibly small overhead and no configuration. We basically give customers a metal detector and X-Ray vision to help them find needles in haystacks. Locating the exact line of code responsible for a performance issue means Operations and Developers solve business pain faster, and this is a key reason why AppDynamics technology is disrupting the market.

Below is a small collection of needles that customers found using AppDynamics in production. The simple fact is that complete code visibility allows customers to troubleshoot in minutes as opposed to days and weeks. Monitoring with blind spots and configuring instrumentation are a thing of the past with AppDynamics.

Needle #1 – Slow SQL Statement

Industry: Education
Pain: Key Business Transaction with 5 sec response times
Root Cause: Slow JDBC query with full-table scan

Needle #2 – Slice of Death in Cassandra

Industry: SaaS Provider
Pain: Key Business Transaction with 2.5 sec response times
Root Cause: Slow Thrift query in Cassandra

Needle #3 – Slow & Chatty Web Service Calls

Industry: Media
Pain: Several Business Transactions with 2.5 min response times
Root Cause: Excessive Web Service Invocation (5+ per trx)

Needle #4 -Extreme XML processing

Industry: Retail/E-Commerce
Pain: Key Business Transaction with 17 sec response times
Root Cause: XML serialization over the wire.

Needle #5 – Mail Server Connectivity

Industry: Retail/E-Commerce
Pain: Key Business Transaction with 20 sec response times
Root Cause: Slow Mail Server Connectivity

 Needle #6 – Slow ResultSet Iteration

Industry: Retail/E-Commerce
Pain: Several Business Transactions with 30+ sec response times
Root Cause: Querying too much data

Needle #7 – Slow Security 3rd Party Framework

Industry: Education
Pain: All Business Transactions with > 3 sec response times
Root Cause: Slow 3rd party code

Needle #8 – Excessive SQL Queries

Industry: Education
Pain: Key Business Transactions with 2 min response times
Root Cause: Thousands of SQL queries per transaction

Needle #9 – Commit Happy

Industry: Retail/E-Commerce
Pain: Several Business Transactions with 25+ sec response times
Root Cause: Unnecessary use of commits and transaction management.

Needle #10 – Locking under Concurrency

Industry: Retail/E-Commerce
Pain: Several Business Transactions with 5+ sec response times
Root Cause: Non-Thread safe cache forces locking for read/write consistency

 Needle #11 – Slow 3rd Party Search Service

Industry: SaaS Provider
Pain: Key Business Transaction with 2+ min response times
Root Cause: Slow 3rd Party code

 Needle #12 – Connection Pool Exhaustion

Industry: Financial Services
Pain: Several Business Transactions with 7+ sec response times
Root Cause: DB Connection Pool Exhaustion caused by excessive connection pool invocation & queries

Needle #13 – Excessive Cache Usage

Industry: Retail/E-Commerce
Pain: Several Business Transactions with 50+ sec response times
Root Cause: Cache Sizing & Configuration

If you want to manage and troubleshoot application performance in production, you should seriously consider AppDynamics. We’re the fastest growing on-premise and SaaS based APM vendor in the market right now. You can download our free product AppDynamics Lite or take a free 30-day trial of AppDynamics Pro – our commercial product.

Now go find those needles that are hurting your end users!

App Man.

APM Market Disruptors – AppDynamics vs New Relic

Last week a performance engineer called Ben Bramley published a blog entitled “APM Market Disruptors – AppDynamics and New Relic“. The purpose of his article was to provide an overview of AppDynamics and New Relic, whilst also summarizing the key approaches each vendor/solution has taken to simplify and disrupt the APM marketplace.

Firstly, we’re thrilled to be recognized by a blogger, who in this case, had previous hands on experience with Application Performance Management (APM) products like OpTier, CA Wily, HP and dynaTrace. Secondly, whilst it was obviously good (and slightly nerving at times) to read our features and capabilities compared with another vendor (and the APM market in general), it was actually nice to see our freemium and SaaS based go-to-market strategy being recognized as well. I guess these things were actually the main reason why a blogger could access, compare and contrast two next generation APM solutions in the first place. It’s not like IBM, CA or Compuware would make their APM solution available to the masses for evaluation, let alone welcome an independent opinion.

You can read Ben’s blog article in full here.

App Man.

Update: HP does in fact offer their solution (HP Diagnostics v9) via trial, but you’ve got to download and install 4GB of their software.  In the time it takes to do this you could already be up and running with AppDynamics Lite.

AppDynamics recognized by Forrester in APM market overview

Interest in the Application Performance Management (APM) category is very high right now.   To stay one step ahead of their clients, the Industry Analysts who cover the category and write research to advise their clients have been very busy.  In December alone, there were six different analyst reports being researched by the major analyst firms.

Forrester published the results of their research in the 2nd week of December with the report: Market Overview: Application Performance Management, Q4 2011.  Forrester clients can access the report at www.forrester.com. In this report, Forrester provides very sound advice on why APM exists and what it should do for clients. Forrester has created their own “Reference Model” for APM and evaluated the vendor landscape against those criteria.

Raison d’etre for APM

Forrester VP and Principal Analyst, JP Garbani, gives readers very pragmatic advice on the raison d’etre for APM.  Simply put, APM’s job is to:

1) Alert IT to application performance and availability issues before a full-scale outage occurs

2) Isolate or pinpoint the problem source

3) Provide deep-diagnostics to enable IT to determine the root cause

For several years now, JP Garbani has been on the forefront of proclaiming that modern APM solutions should enable IT organizations to manage apps not by gauging the heath of their servers or servlets, but instead by assessing what the customer or end-user cares about most – whether their Business Transaction completes quickly and doesn’t make them wait.  He states that this has become even more critical as applications have gotten more distributed and complex.

France’s #1 Travel Site Karavel Selects AppDynamics for APM over Compuware Dynatrace & CA Wily

AppDynamics vs CA Wily vs DynaTrace2011 was an amazing year for AppDynamics. We experienced tremendous growth and success, largely down to the many customers around the world who believed in our vision, technology, and ability to help Dev and Ops teams better manage application performance in production. The Application Performance Management (APM) market isn’t an easy market to succeed in, with well over 30 vendors competing against each other. In just three years we’ve managed to take on the big players like Compuware DynaTrace, CA Wily, HP and IBM to change the industry perception that APM is expensive to own and difficult to deploy/use.

We feel APM should be for everyone. It should be affordable, it should be easy to deploy, and easy to use. APM should not be a luxury that only an elite group of enterprises can afford. Today, we have customers who monitor applications with 5 nodes, 50 nodes, 500 nodes and 5,000 nodes. Application performance impacts organizations of all sizes; that’s why we wanted our APM solution to be accessible to the masses over the web via our free download and SaaS trial. We wanted to be transparent with our buyers and demonstrate that they can evaluate and use our solution all by themselves with no account manager or technical consultant by their side. We really wanted prospects to see for themselves that APM can be simple to deploy and easy to use.

A major validation of this market disruption was when a customer called Karavel in France was looking for an APM solution and evaluated CA Wily, Compuware dynaTrace and AppDynamics. Karavel requested a trial, downloaded our software and we sent them a trial license key for 30 days. The whole AppDynamics install, deployment and evaluation was solely conducted by the customer on their own. This might not sound that impressive, but this is what the software buying experience should be all about: the customer and the solution. If the customer can’t install, deploy and evaluate an APM solution on their own, how will they manage this process when it comes to a production deployment? Software should sell itself these days–if it requires an army of people to sell it, it probably requires an army of people to implement it as well.

You can read the full Karavel press release here:
http://www.appdynamics.com/press/press-release-01-03-12.php

Full case study is available here also:
http://www.appdynamics.com/documents/roi_studies/AppDynamics_ROI_Karavel.pdf

Remember, software like APM doesn’t have to be complex and expensive. With the internet these days, there is no excuse why a prospect can’t download or evaluate solutions online in just a few hours.

App Man.

Storm Clouds in 2012? – Results of AppDynamics APM Survey

We recently finished conducting our annual Application Performance Management survey. Over 250 IT professionals participated, and they shared insights such as:
– Many Ops and Dev teams are anticipating growth in their applications by 20% or more
– Over 50% are planning to move to the cloud, and are architecting brand-new applications to be cloud-ready
– Most teams are using log files to monitor application performance, rather than an Application Performance Management (APM) tool.

We’ll release the full report soon, but here’s an infographic that summarizes some of the main findings:

AppDynamics Inforgraphic - Storm Clouds in 2012

Embed this image on your site:

What I found personally surprising was the heavy reliance on log files. When you’re troubleshooting distributed architectures, time is of the essence–and there’s no way to cut your MTTR down when you’re relying on log files to identify root cause.

In fact, there’s only one guy who ever made using a log file look cool:

And I think we can all agree that’s a pretty unique use case.

We’ll have the full survey results available soon.

 

 

Just how complex can a Login Transaction be? Answer: Very!

People in our industry always talk about IT complexity and cost. Cost is pretty easy to calculate, because IT budgets are allocated and audited every year. Complexity is very different–we know it exists, but we can’t really see or measure it. Complexity is often when our brain tries to understand something and stalls in the process, trying to make sense of information that has never been seen before.

Well, this happened to a few of us in AppDynamics last week. A customer was kind enough to share how a single login business transaction flowed across their entire infrastructure. You might be thinking: “How can a login transaction be complex? That’s just a simple call to an LDAP or SiteMinder tier”–which is pretty much what we all thought it was. However, the screenshot that graced us was one of shock, beauty and amazement. In fact, I’m looking at it right now before I scrub the customer details, and I’m still thinking “Hmmmm, this is bonkers.”

Without delaying further, here is that very screenshot showing the Login Business Transaction:

Scary huh? What you see is the flow and timing of a Customer Login business transaction as it executes across a well governed, regulated, SOA environment consisting of many services (denoted by the Java Tiers). The Customer Login transaction begins at the Java node to the right marked “START” and propagates across the entire SOA environment using a combination of sync/async JMS messages, HTTP and RMI communication to notify other Services that a customer is now active and logged in. You can also see many services writing to a database as a result of this transaction. These invocations are simply auditing the customer login to satisfy the legal regulations that this organization has to comply with. So if you ever wonder what impact Governance and Legislation has on IT, this is a perfect example of the complexity storm it creates. What’s interesting is that the Logout business transaction for this application was just as complex!

The screenshot above unfortunately reflects the enormous complexity that many IT departments have to deal with everyday, especially when a user complains that their business transaction is slow. The problem for 95% of IT departments is they don’t have this type of visibility in production. They can feel pain, but they can’t see it. A slow business transaction may take 25 seconds to complete and touch many infrastructure tiers along the way. Unless IT sees this end to end journey they’ll always struggle to troubleshoot and manage it.

The good news is you’re 30 minutes away from getting this visibility in production by evaluating a next generation application monitoring solution like AppDynamics Pro. AppDynamics will auto-discover your business transactions, map their specific flows across your infrastructure, and give you a latency breakdown across and inside every tier the business transaction touches.

To manage and master IT complexity you have to visualize and see it.  Seeing how your business actually runs across IT is completely different to guessing how your business runs across IT. Next time a user complains that their business transaction is slow, what will you do? Bury your head in a log file, or visualize how that business transaction executed using an application performance monitoring solution like AppDynamics?

Isn’t it about time you mapped your app?

App Man.