AppD on AppD: Scaling Our Custom Dashboards Platform

This blog post is a developer’s perspective on how using our own AppDynamics software has helped us find and fix performance-related issues – and how other developers can do the same. 

One of the most challenging aspects of developing cloud-based platforms is scalability. As we innovate and build new features, it is essential for developers to ensure these new platform features are scalable and do not impact the performance of our applications, especially as any impact on performance is likely to impact all tenants on the cluster.

This is a challenge that we experienced firsthand. My team here at Appdynamics is responsible for building custom dashboards, one of the most powerful features of AppDynamics. It allows you to group relevant metrics into one central dashboard as well as build sophisticated dashboards with drill down capabilities.

As we added more capabilities and support for the different types of metrics, we realized that scaling our dashboards was becoming a challenge. Many of our customers were exploring different dashboard capabilities and features, which was truly stressing our endpoints. This is when our practice of using AppDynamics internally – “AppD on AppD” – helped give us insight and visibility into the issues we faced while developing new features on the platform.

Our AppD on AppD Approach

Recently, we made some changes on the backend of a REST endpoint that returns metrics for different types of widgets on custom dashboards. Our team’s internal process is to ensure every feature we implement is stress-tested under a real type of load profile, so we made sure to test this as well. Below is what we discovered.

On local dev environments, the feature worked fine and the data returned by the API took a few milliseconds. Next, we deployed our changes to our performance environment, which is similar to an actual production instance with large amounts of data. We immediately noticed that under stress, the UI which uses the endpoint was very slow and the average response time of the endpoint was considerably high. Luckily, since AppDynamics was monitoring the performance environment, it was easy for us to dig into the issue.

We configured a business transaction to monitor the REST endpoint with the slow response time and within a few minutes, we collected transaction snapshots which gave us valuable behind-the-scenes information of the REST API calls. There were two things that caught our attention:

Resolved Issue #1

A particular method which was being called repeatedly was taking more time to execute than before. The method itself took around 100-150 milliseconds, but if it was called 100 times in a single transaction, it would take around 15,000 milliseconds to execute, which is roughly 15 seconds.

The image below shows the total time it took for all the calls of this method in this single transaction.

Here is a code snippet of that method:

After a few minutes of looking at the full implementation of this method, we found that String.replace could be a potential problem here, which happens to be slower than StringUtils.replace. As a result, we made a minor change and modified the code to use StringUtil implementation. Here is some information on StringUtils.replace vs String.replace.

Resolved Issue #2

Another issue we noticed was that there were too many database calls for a single request.

We made a few optimizations here, including caching the data and bulking-up the queries. After applying these fixes, we measured the performance again and saw the average response time for the endpoint improve drastically.

Lastly, we created our own custom dashboard to measure performance of our specific endpoint, showing us different metrics like average response time, errors, and calls per minute along with thresholds and baselines. We also created a scheduled report that sends our team a dashboard snapshot everyday, allowing us to easily spot outliers and proactively address issues. We also could have created alerts, which notify you when certain conditions are met or exceeded based on your configuration. Scheduled reports and alerts make it easy to spot outliers and proactively address issues.

Without AppDynamics, it would have been difficult for a developer to quickly pinpoint these issues within a large codebase. But with AppDynamics, not only does it become easier to find issues in production, but developers can proactively ensure that features are robust and scalable before deploying them to production. This reduces the time spent on performance-related issues, and instead, gives developers more time to innovate and write code.

Learn more about our business-centric dashboards or get a demo from our sales team today.

7 Steps to Great APM Dashboards

One of the most highly rated sessions at AppSphere 2016 and again this May at AppD Summit Europe was AppDynamics Senior Sales Engineer Andy Jackson’s dashboard top tips presentation. If you’re an AppDynamics customer, this is a not-to-be missed opportunity to learn how to create dashboards that inform and encourage action. Thought-provoking, educational, and entertaining, Andy delivers a compelling tutorial with real actionable takeaways.

Eliminating “Noise” in the Dashboard

Sharing best practices learnt from supporting multiple enterprise customers, Andy at first explains the inherent challenges with dashboards, namely if it just shows red, it only tells part of the story. He highlights the importance of colors and “noise” — is everything shown in the dashboard relevant, and/or is something missing? If a dashboard shows all problems, all it does is tell you there’s a problem with the application.

There are many pitfalls to avoid, and this session provides best practice guidance, based on processes learnt and dashboards created for enterprise customers.

Dashboard “Dos and Don’ts”

He starts with a commonly created AppDynamics dashboard featuring key customer journeys and major steps.

Screen Shot 2017-05-17 at 5.32.51 PM.png

Screen Shot 2017-05-17 at 5.34.30 PM.png

Andy then asks if something is red, why is it red? We then should go find the problem. Colleagues won’t know if the issue is page loading or Business Transaction response time, so a dashboard needs labels for others to make sense of the data. He explains why graphs are a pet peeve, as they often have no point of reference and are crying out to show the baselining capabilities of AppDynamics.

Andy also dismisses dashboards that just look pretty, but are functionally useless. How much information do you want? How do you want to view it — on a wall or on a desktop, for example?

7 Steps to Take

It’s best to start with a plan, and Andy referenced a series of key steps towards building great dashboards that can be summarized as follows:

1. Identify the audience

Who is the actual audience? Why do you want a dashboard? Prove you need it. What can a dashboard do that isn’t better managed via an SMS alert, an email, or a support ticket raised.

2. Focus on goals

What is the desired outcome? Do we want to dive into the problem or keep senior managers happy?

3. Identify use cases and environments

Will the dashboard be used interactively or just look pretty on a wall? For example, this has an impact on dashboards that need a mouse to double click, but are expected to be wall-mounted.

4. Build the end state vision

Have all stakeholders agreed on the criteria? Do they know what they want to see — e.g., see red for an issue and double click to dive into details, or business metrics to keep a broader, more business-oriented audience happy? Andy suggests that it’s better to start by drawing a draft dashboard.

5. Use “traditional” design tools

Visualize the end state dashboard using paper and pen to draw candidate dashboards and solicit feedback — is this what senior managers are envisaging? It’s much easier updating a sketch than revising a fully built-out dashboard after the fact.

6. Check alignment before moving forward

Review the plan with the customer, and how you intend to build it. Sell the plan back to the customer — can you achieve all your goals?

7. Dashboards are dynamic, living things

Is there any feedback generated by the dashboard? Take a continuous improvement approach — the perfect dashboard is never done. It will change over time and be added to. No updates suggests the dashboard isn’t being used or truly engaged with.

Ideally, customers should follow the “Plan”, “Do”, “Check”, and “Act” stages.

Real-World Examples

The topic is brought to life by showing examples of good and bad dashboards, including one created for a real customer to track performance on Black Friday, with respective Business Transactions.

If you’re interested in how leading retailer Dixons Carphone Warehouse used AppDynamics to make Black Friday successful for them, read this Forrester report. Graphs are shown to be useful if a dynamic baseline is established so we know if a status is good or bad. An example of how to set this up is shown below:

Screen Shot 2017-05-17 at 5.46.28 PM.png

The importance of events is then demonstrated by showing exactly what took place that triggered a drop in traffic, for example.

Andy took the audience step-by-step through his favored approach to building a simple yet impactful dashboard with the help of a few encoder and widget sites. An example of one he created is shown below:

Screen Shot 2017-05-17 at 5.44.48 PM.png

Best explained by watching the presentation, he then shows the specific steps involved in creating an impactful dashboard, such as how an event widget can be very helpful when added to a graph, creating context. While the new 4.3 release does not include extensive new dashboard features, it does have a few real time-savers such as the ability to undo delete.

The session ended with an interactive Q & A, again dispensing valuable guidance on how to create dashboards that really matter.

Don’t Delay Your Dashboard Transformation

There’s much more covered by Andy in his deck, and if you’re interested in building great dashboards (who isn’t?) then it’s well worth watching his presentation — along with the other track sessions from AppD Summit Europe — and it takes less than an hour to watch.