Apdex is Fatally Flawed

September 10 2013
 

Apdex can lead you to believe your app is working fine when users are really frustrated. Let's review a better way to manage your application performance.


I’ve been looking into a lot of different statistical methods and algorithms lately and one particularly interesting model is Apdex. If you haven’t heard of it yet, Apdex has been adopted by many companies that sell monitoring software. The latest Apdex specification (v1.1) was released on January 22, 2007 and can be read by clicking here.

Here is the purpose of Apdex as quoted from the specification document referenced above: “Apdex is a numerical measure of user satisfaction with the performance of enterprise applications, intended to reflect the effectiveness of IT investments in contributing to business objectives.” While this is a noble goal, Apdex can lead you to believe your application is working fine when users are really frustrated. If Apdex is all you have to analyze your application performance then it is better than nothing but I’m going to show you the drawbacks and a better way to manage your application performance.

The Formula

The core fatal flaw in the Apdex specification is the formula used to derive your Apdex index. Apdex is an index from 0-1 derived using the formula shown below.

Apdex Formula

At the heart of Apdex is a static threshold (defined as T). The T threshold is set as a measure of the ideal response time for the application. This T threshold is set for an entire application. If you’ve been around application performance for a while you should immediately realize that all application functions perform at very different levels comparatively. For example, my “Search for Flights” function response time will be much greater than my “View Cart” response time. These 2 distinctly different functionalities should not be subjected to the same base threshold (T) for analysis and alerting purposes.

Another thing that application performance veterans should pick up on here is that static thresholds stink. You either set them too high or too low and have to keep adjusting them to a level you are comfortable with. Static thresholds are notorious for causing alert storms (set too low) or for missing problems (set too high). This manual adjustment philosophy leads to a lack of consistency and ultimately makes historical Apdex charts useless if there have been any manual modification of “T”.

So let’s take a look at the Apdex formula now that we understand “T” and it’s multiple drawbacks:

  • Satisfied count = the number of transactions with a response time between 0 and T
  • Tolerating count = the number of transactions with a response time between T and F (F is defined next)
  • F = 4 times T (if T=3 seconds than F=12 seconds)
  • Frustrated count (not shown in the formula but represented in Total samples) = the number of transactions with a response time greater than F

When my colleague Ian Withrow looked at the formula above he had some interesting thoughts on Apdex: “When was the last time you ‘tolerated’ a site loading 4 times slower than you expected it to? If T=1 I’ll tolerate 4 seconds but if T=3 no way am I waiting 12 seconds on a fast connection… which brings me to another point. The notion of a universal threshold for satisfied/tolerating is bunk even for the same page (let alone different pages). My tolerance for how a website loads at work on our fast connection is paper thin. It better be 1-2 seconds or I’m reloading/going elsewhere. On the train to work where I know signal is spotty on my iPhone I’ll load something and then wait for a few seconds.”

Getting back to the Apdex formula itself; assuming every transaction completes in less than the static threshold (T) the Apdex will have a value of 1.00 which is perfect, except the real world is never perfect so let’s explore a more plausible scenario.

An Example

You run an e-Commerce website. Your customers perform 15 clicks while shopping  for every 1 click of the checkout functionality. During your last agile code release the response time of the checkout function went from 4 seconds to over 40 seconds due to an issue with your production configuration. Users are abandoning your website and shopping elsewhere since their checkouts are lost in oblivion. What does Apdex say about this scenario? Let’s see…

T = 4 seconds
Total samples = 1500
Satisfied count = 1300 (assuming most were under 4 seconds with a few between 4 and 16 seconds (F=4T))
Tolerating count = 100 (the assumed number of transactions between 4 and 16 seconds)

Apdex = .90

Apdex Index Table

According to the Apdex specification anything from a .85 – .93 is considered “Good”. We know there is a horrible issue with checkout that is crushing our revenue stream but Apdex doesn’t care. Apdex is not a good way to tell if your application is servicing your business properly. Instead it is a high level reporting mechanism that shows overall application trends via an index. The usefulness of this index is questionable based upon the fact that Apdex is completely capable of manipulation based upon manually changing the T threshold. Apdex, like most other forms of analytics must be understood for what it is and applied to the appropriate problem.

A Better Way – Dynamic Baselines and Analytics

In my experience (and I have a lot of enterprise production monitoring experience) static thresholds and high level overviews of application health have limited value when you want your application to be fast and stable regardless of the functionality being used. The highest value methodology I have seen in a production environment is automatic baselining with alerts based upon deviation from normal behavior. With this method you never have to set a static threshold unless you have a really good reason to (like ensuring you don’t breach a static SLA). The monitoring system automatically figures out normal response time, compares each transactions actual response time to the baseline and classifies each transaction as normal, slow, or very slow. Alerts are based upon how far each transaction has deviated from normal behavior instead of an index value based upon a rigid 4 times the global static threshold like with Apdex.

baselines

Load and response time charts. Baselines are represented by dashed lines.

business baseline

Automatic baseline of business metrics (books sold) . Baseline represented by dashed line.

If you want to make yourself look great to your bosses (momentarily) then set your T threshold to 10 seconds and let the Apdex algorithm make your application look awesome. If you want to be a hero to the business then show them the problems with Apdex and take a free trial of AppDynamics Pro today. AppDynamics will show you how every individual business transaction is performing and can even tell you if your application changes are making or costing you money.

Jim Hirschauer
Jim Hirschauer is a Technology Evangelist for AppDynamics. He has an extensive background working in highly available, business critical, large enterprise IT operations environments. Jim has been interested in application performance testing and monitoring since he was a Systems Administrator working in a retail bank. His passion for performance analysis led him down a path where he would design, implement and manage the cloud computing monitoring architecture for a top 10 investment bank. During his tenure at the investment bank, Jim created new processes and procedures that increased overall code release quality and dramatically improved end user experience.

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form