Product

When Is Research Not Research?

By | | 3 min read


Summary
  We all know that analysts and independent research firms that publish market insight reports and predictions for the future vary widely in validity and credibility.

 

We all know that analysts and independent research firms that publish market insight reports and predictions for the future vary widely in validity and credibility. There are huge variances in how vendors are ranked and, at AppDynamics, we tend to take these reports with a pinch of salt, content as we are with our Gartner MQ position and phenomenal growth.

Nevertheless, when a report this week was released by Research in Action, a small Independent Research and Consulting organization in Germany, we felt obliged to comment. We support ‘fair and factual’ based research into the APM market as there are a number of vendors with differing solutions, but upon analysis, we are dismayed by the research conducted by Research In Action and worry that it is misleading to any business interested in APM solutions or the market as a whole.

In the report, vendors were evaluated based on two dimensions, each broken into sub-criteria. A number of the results jump out as being inaccurate, for example:

  1. Market Share and Growth. Research in Action state that: “There are more than 400 active software and SaaS vendors generating around $ 4,6 B in annual revenue”. This number is significantly higher than published numbers from IDC ($3b) and Gartner ($2.6b). Gartner also concludes that the market share growth for AppDynamics was 246.5%. Equally, IDC’s APM market share 2014 report tells a similar story, showing that AppDynamics is growing faster than any of the APM vendors listed – in sharp contrast to Research in Action’s baseless assertion that “AppDynamics has recently come to a more or less abrupt halt.”
  2. Company Viability and Execution Capabilities. Along with our market share growth, we have consistently achieved strong financial growth. In fiscal year 2015 we doubled our bookings to over $150 Million and in 2014 we experienced a 175% increase in bookings, clearly demonstrating our company viability and execution. The author also includes vendors in the research which are clearly not APM vendors. Micro Focus is a mainframe company who does not offer any APM products, yet they are included with oddly large revenues in the market. Similarly network performance monitoring vendor Infovista who focuses on carrier network instrumentation is included.
  3. Forecasted revenue and market share percentage. The report has a table showing forecasted revenue and market share percentage for vendors analyzed. AppDynamics has not provided any figures or information for Research In Action to base these figures upon making their validity questionable, especially based on the market share growth numbers by Gartner and IDC explained in (1).

As a former analyst myself – I could go on to criticise the dubious and unscientific survey methodology which compares a 900 participant IT buyers telephone survey in Q1 and a 700 IT buyers online survey in Q2 2015, and the vague, subjective conclusions that are drawn from subsets of this audience. However my conclusion can be aptly summed up in this interesting blogpost by Aneel Lakhani, former Gartner research director which describes the various types of analysts and the content they produce.

“Many analyst firms are pay­-for-­play. Many will write white papers and ‘advertorials’ that are underwritten by vendors. I find these (after too much experience) highly suspect and particularly worthless.”

Despite AppDynamics being labeled as a “Market Leader” in the paper, we cannot recommend it to our readers as a good source of information if you are evaluating the APM marketplace. Check out the Gartner Magic Quadrant instead.