AppD on AppD: Scaling Our Custom Dashboards Platform

This blog post is a developer’s perspective on how using our own AppDynamics software has helped us find and fix performance-related issues – and how other developers can do the same. 

One of the most challenging aspects of developing cloud-based platforms is scalability. As we innovate and build new features, it is essential for developers to ensure these new platform features are scalable and do not impact the performance of our applications, especially as any impact on performance is likely to impact all tenants on the cluster.

This is a challenge that we experienced firsthand. My team here at Appdynamics is responsible for building custom dashboards, one of the most powerful features of AppDynamics. It allows you to group relevant metrics into one central dashboard as well as build sophisticated dashboards with drill down capabilities.

As we added more capabilities and support for the different types of metrics, we realized that scaling our dashboards was becoming a challenge. Many of our customers were exploring different dashboard capabilities and features, which was truly stressing our endpoints. This is when our practice of using AppDynamics internally – “AppD on AppD” – helped give us insight and visibility into the issues we faced while developing new features on the platform.

Our AppD on AppD Approach

Recently, we made some changes on the backend of a REST endpoint that returns metrics for different types of widgets on custom dashboards. Our team’s internal process is to ensure every feature we implement is stress-tested under a real type of load profile, so we made sure to test this as well. Below is what we discovered.

On local dev environments, the feature worked fine and the data returned by the API took a few milliseconds. Next, we deployed our changes to our performance environment, which is similar to an actual production instance with large amounts of data. We immediately noticed that under stress, the UI which uses the endpoint was very slow and the average response time of the endpoint was considerably high. Luckily, since AppDynamics was monitoring the performance environment, it was easy for us to dig into the issue.

We configured a business transaction to monitor the REST endpoint with the slow response time and within a few minutes, we collected transaction snapshots which gave us valuable behind-the-scenes information of the REST API calls. There were two things that caught our attention:

Resolved Issue #1

A particular method which was being called repeatedly was taking more time to execute than before. The method itself took around 100-150 milliseconds, but if it was called 100 times in a single transaction, it would take around 15,000 milliseconds to execute, which is roughly 15 seconds.

The image below shows the total time it took for all the calls of this method in this single transaction.

Here is a code snippet of that method:

After a few minutes of looking at the full implementation of this method, we found that String.replace could be a potential problem here, which happens to be slower than StringUtils.replace. As a result, we made a minor change and modified the code to use StringUtil implementation. Here is some information on StringUtils.replace vs String.replace.

Resolved Issue #2

Another issue we noticed was that there were too many database calls for a single request.

We made a few optimizations here, including caching the data and bulking-up the queries. After applying these fixes, we measured the performance again and saw the average response time for the endpoint improve drastically.

Lastly, we created our own custom dashboard to measure performance of our specific endpoint, showing us different metrics like average response time, errors, and calls per minute along with thresholds and baselines. We also created a scheduled report that sends our team a dashboard snapshot everyday, allowing us to easily spot outliers and proactively address issues. We also could have created alerts, which notify you when certain conditions are met or exceeded based on your configuration. Scheduled reports and alerts make it easy to spot outliers and proactively address issues.

Without AppDynamics, it would have been difficult for a developer to quickly pinpoint these issues within a large codebase. But with AppDynamics, not only does it become easier to find issues in production, but developers can proactively ensure that features are robust and scalable before deploying them to production. This reduces the time spent on performance-related issues, and instead, gives developers more time to innovate and write code.

Learn more about our business-centric dashboards or get a demo from our sales team today.

7 Steps to Great APM Dashboards

One of the most highly rated sessions at AppSphere 2016 and again this May at AppD Summit Europe was AppDynamics Senior Sales Engineer Andy Jackson’s dashboard top tips presentation. If you’re an AppDynamics customer, this is a not-to-be missed opportunity to learn how to create dashboards that inform and encourage action. Thought-provoking, educational, and entertaining, Andy delivers a compelling tutorial with real actionable takeaways.

Eliminating “Noise” in the Dashboard

Sharing best practices learnt from supporting multiple enterprise customers, Andy at first explains the inherent challenges with dashboards, namely if it just shows red, it only tells part of the story. He highlights the importance of colors and “noise” — is everything shown in the dashboard relevant, and/or is something missing? If a dashboard shows all problems, all it does is tell you there’s a problem with the application.

There are many pitfalls to avoid, and this session provides best practice guidance, based on processes learnt and dashboards created for enterprise customers.

Dashboard “Dos and Don’ts”

He starts with a commonly created AppDynamics dashboard featuring key customer journeys and major steps.

Screen Shot 2017-05-17 at 5.32.51 PM.png

Screen Shot 2017-05-17 at 5.34.30 PM.png

Andy then asks if something is red, why is it red? We then should go find the problem. Colleagues won’t know if the issue is page loading or Business Transaction response time, so a dashboard needs labels for others to make sense of the data. He explains why graphs are a pet peeve, as they often have no point of reference and are crying out to show the baselining capabilities of AppDynamics.

Andy also dismisses dashboards that just look pretty, but are functionally useless. How much information do you want? How do you want to view it — on a wall or on a desktop, for example?

7 Steps to Take

It’s best to start with a plan, and Andy referenced a series of key steps towards building great dashboards that can be summarized as follows:

1. Identify the audience

Who is the actual audience? Why do you want a dashboard? Prove you need it. What can a dashboard do that isn’t better managed via an SMS alert, an email, or a support ticket raised.

2. Focus on goals

What is the desired outcome? Do we want to dive into the problem or keep senior managers happy?

3. Identify use cases and environments

Will the dashboard be used interactively or just look pretty on a wall? For example, this has an impact on dashboards that need a mouse to double click, but are expected to be wall-mounted.

4. Build the end state vision

Have all stakeholders agreed on the criteria? Do they know what they want to see — e.g., see red for an issue and double click to dive into details, or business metrics to keep a broader, more business-oriented audience happy? Andy suggests that it’s better to start by drawing a draft dashboard.

5. Use “traditional” design tools

Visualize the end state dashboard using paper and pen to draw candidate dashboards and solicit feedback — is this what senior managers are envisaging? It’s much easier updating a sketch than revising a fully built-out dashboard after the fact.

6. Check alignment before moving forward

Review the plan with the customer, and how you intend to build it. Sell the plan back to the customer — can you achieve all your goals?

7. Dashboards are dynamic, living things

Is there any feedback generated by the dashboard? Take a continuous improvement approach — the perfect dashboard is never done. It will change over time and be added to. No updates suggests the dashboard isn’t being used or truly engaged with.

Ideally, customers should follow the “Plan”, “Do”, “Check”, and “Act” stages.

Real-World Examples

The topic is brought to life by showing examples of good and bad dashboards, including one created for a real customer to track performance on Black Friday, with respective Business Transactions.

If you’re interested in how leading retailer Dixons Carphone Warehouse used AppDynamics to make Black Friday successful for them, read this Forrester report. Graphs are shown to be useful if a dynamic baseline is established so we know if a status is good or bad. An example of how to set this up is shown below:

Screen Shot 2017-05-17 at 5.46.28 PM.png

The importance of events is then demonstrated by showing exactly what took place that triggered a drop in traffic, for example.

Andy took the audience step-by-step through his favored approach to building a simple yet impactful dashboard with the help of a few encoder and widget sites. An example of one he created is shown below:

Screen Shot 2017-05-17 at 5.44.48 PM.png

Best explained by watching the presentation, he then shows the specific steps involved in creating an impactful dashboard, such as how an event widget can be very helpful when added to a graph, creating context. While the new 4.3 release does not include extensive new dashboard features, it does have a few real time-savers such as the ability to undo delete.

The session ended with an interactive Q & A, again dispensing valuable guidance on how to create dashboards that really matter.

Don’t Delay Your Dashboard Transformation

There’s much more covered by Andy in his deck, and if you’re interested in building great dashboards (who isn’t?) then it’s well worth watching his presentation — along with the other track sessions from AppD Summit Europe — and it takes less than an hour to watch.

 

 

Accessing your performance data with the AppDynamics REST API

App Man previously covered how to go mobile with the AppDynamics REST API and Dojo highlighting the AppDynamics Gauges project. In this post, we will cover how to get the most out of AppDynamics by leveraging the REST API.

Custom Dashboards

AppDynamics comes with very powerful and sexy custom dashboards that anyone can build. With our custom dashboards you can combine and graph any metric and easily come up with an ops- or business- focused dashboard for your executive team.

DevOps Dashboard:

DevOps Dashboard

Management Dashboard:

Management Dashboard

Get started building a custom dashboard.

Sometimes a custom dashboard is not enough and you really need the data underneath. If our amazing HTML5 custom dashboards are not enough, at the end of the day your data is yours. The goal of this post is to provide an introduction on how to access your performance data.

AppDynamics is extensible

AppDynamics Pro allows you to track any custom metric through the machine agent. This means you can easily track any application metrics via AppDynamics and use the REST api to mashup the data (think of it as a replacement for the popular StatsD and Graphite combination). The REST API allows you to access metrics and mashup your data with your internal tools.

How to access performance data via REST API

From within the AppDynamics metrics browser you can easily access the url for the API for a particular metric by right clicking as shown here:

REST_URL

Access data via the command line

Once you have the API endpoint needed for the metrics you want you can easily access it via curl on the command line. Our REST API uses HTTP basic authentication. If you have installed the Controller on a single-tenant platform, your default account is:

Username: username@customer1
Password: xxxx

If you are using the SaaS Controller, your username is your AppDynamics user name in your AppDynamics account:

username@accountname

The credentials are the same you use to log in to the controller. Below are common examples:

Retrieve a list of applications available


curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications?output=JSON'

[
    {
    "description": "",
    "id": 7,
    "name": "AppDynamics"
  },
    {
    "description": "",
    "id": 5,
    "name": "Knp Ipsum"
  }
]
Retrieve the average response time of a specific application


curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications/AppDynamics/metric-data?metric-path=Overall%20Application%20Performance%7CAverage%20Response%20Time%20(ms)&time-range-type=BEFORE_NOW&duration-in-mins=60&output=JSON'

[{
  "frequency": "ONE_MIN",
  "metricPath": "Overall Application Performance|Average Response Time (ms)",
  "metricValues": [  {
    "current": 41,
    "max": 131,
    "min": 9,
    "startTimeInMillis": 1375298040000,
    "value": 50
  }]
}]
Retrieve number of requests per minute for a specific application


curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications/AppDynamics/metric-data?metric-path=Overall%20Application%20Performance%7CCalls%20per%20Minute&time-range-type=BEFORE_NOW&duration-in-mins=15&output=JSON'

[{
  "frequency": "ONE_MIN",
  "metricPath": "Overall Application Performance|Calls per Minute",
  "metricValues": [  {
    "current": 2,
    "max": 0,
    "min": 0,
    "startTimeInMillis": 1375300860000,
    "value": 2
  }]
}]
Getting started with the REST API via SDKs

The easiest way to get started working with the REST API is to use the SDK available for Java and Python. The SDKs provide a lightweight wrapper around our REST API and are available via AppDynamics X. The AppDynamics REST API offers XML and JSON for interoperability.

Accessing the REST API via the Python SDK


from appd.cmdline import parse_argv
from appd.request import AppDynamicsClient


args = parse_argv()
c = AppDynamicsClient(args.url, args.username, args.password, args.account, args.verbose)
for app in c.get_applications():
print app.name, app.id

The best example is for fetching metrics for business transactions. Checkout the Python SDK for more examples.

How to track a custom metric with machine agent

The machine agent can act as a REST gateway to capture custom metrics into the AppDynamics metrics browser. In order to enable the machine agent API, start the machine agent with the following options:


sudo java -Dmetric.http.listener=true -Dmetric.http.listener.port=8293 -jar /opt/appdynamics/machine-agent/machineagent.jar

Once the machine agent is listening, tracking custom metrics is as easy as making a post request to:


http://localhost:8293/machineagent/metrics?name=foo|bar&value=42&type=average

Going even farther

Two weeks ago we announced AppDynamics X where we released many integrations that leverage the machine agent to track custom metrics from Nginx, Apache, Cassandra, and more. The real power of AppDynamics is not the user interface, but the stats and business intelligence platform under the hood.

As always, please feel free to comment if you think I have missed something or if you have a request for content for an upcoming post.