Design For Success – Design, Implement, Refine, Succeed

When an organization gets to the point where they realize they need an Application Performance Management (APM) solution, things are usually on fire and getting more unmanageable by the day. For any organization, the purchase of an APM toolset is a large budget item and the pressure is getting these new tools into production and delivering value today, not tomorrow. In these circumstances, it is often difficult to make the time to really think through the process and do the implementation correctly. I know all too well how this feels, because I have been in this position with APM solutions on several occasions. I have deployed almost every major APM solution in this exact situation – executives breathing down my neck to both fix problems and make sure that the money spent delivers value quickly. An alternate title for this post may be “Don’t Make the Same Mistakes I Did” or even “My Pain, Your Gain”.

Tactical Deployment vs. Strategic Enablement

When it comes to enabling agent-based APM solutions in your environment, it’s typically a relatively easy task. Put agent binaries on the machines you want to monitor, modify your application startup parameters to use the agent binaries, restart the application, and presto, you have an application monitored by APM – or so you can tell your bosses. However, the reality is, if you don’t think through how the agents are organized within the solution, and you don’t plan for how you will automate the deployment and enablement of those agents, you will get nowhere near the value you could get from the money spent on your solution. It is very hard to put the brakes on and wait to do things right, but even with the smallest level of effort ahead of time, you can reap 10 times that effort in lost productivity and greater visibility down the line. This small amount of effort is what enables your APM solution rather than merely deploying a product.

A Tale of Two Deployments

I currently spend my time in the field helping AppDynamics enterprise customers get the most value from their APM investment. However, these stories come from my past experience deploying APM solutions at large Fortune 500 companies, pre-AppDynamics.

At one of these companies, we had just made a significant investment in an APM solution, and the pressure was on to deploy the solution overnight. I thought it was really as simple as I described earlier, so I went about making plans to get this deployment done. The mindset was purely tactical (get agents out, turn agents on, PROFIT). When it came to questions like how the application should organized within the solution (what is a tier, what is a node, how should tiers be grouped together) it seemed as simple as just grouping tiers owned by one group together into a single bucket, one “Application” in AppDynamics terms. So this is what I did. In a matter of a few days, after some herculean efforts to get change requests in, planning for application restarts, getting change management exceptions, etc, all of the agents were up and reporting in to the APM solution. However, when I started looking at the visibility I gained from this solution I was perplexed because calls that should be flowing from one tier to another tier simply weren’t showing up as I expected. As it turns out, data that flows from one tier to another tier that was part of another “Application” were showing as a call to an external system rather than a call flowing through the other tier as I expected. Consequently, a significant reconfiguration effort had to made in order to get this data flow corrected. Even a little time spent initially understanding how to design the hierarchy in the APM tool would have avoided this significant headache, nevermind the credibility lost by deploying something that didn’t deliver the value promised.

At another company, I took the time to make a bit of a sketch of how I thought the data should flow from tier to tier. One meeting with the application team who owned the monitored application was sufficient to whiteboard this high level picture, and from here determine what the proper agent configuration should be. With this knowledge, I then developed scripts that did the work of pushing the agent binaries out to the servers and configuring the relevant properties to enable them. In a few days longer than was required to simply push the agents manually and ask questions later, I had the agents deployed, and a fully enabled APM solution.

Lessons Learned and Applying Lessons to AppDynamics Deployments

The most important thing to consider when deploying any APM solution is how the tool enables you to understand true application performance. One key element of this is the application hierarchy inside of your APM tool. A good way to think of this in AppDynamics terms is as follows:

  1. Application – This is the top level of the hierarchical configuration. All services that talk to each other should be in the same AppDynamics Application. For most customers, from very small to the biggest customers using AppDynamics, the starting point should be a single Application. Only when it is absolutely known that a monitored process communicates with no other monitored processes should they be broken out into a separate application.
  2. Tier – A tier should combine all runtime elements that implement similar services. If there are several runtimes that are load balanced, this is a good indication that they belong in the same tier. If two runtimes implement the same services, they should be grouped together.
  3. Node – This is the lowest level element in the AppDynamics monitoring hierarchy and the easiest to understand. This is a single monitored run time, whether it be a JVM, a .NET CLR, or a PHP server.

Take the time to briefly sketch out a high-level flow map, and this will get you thinking in the right way about how to lay these things out. If you need assistance, reach out to AppDynamics field enablement teams to help you understand how this should be laid out. Additional information can be found in the AppDynamics Center of Excellence Best Practices Guides.

Lastly, take the time to automate your agent rollout and enablement. Building an APM solution should be an iterative process, where you can easily deploy and remove agents, and reconfigure them as needed. Doing this small amount of pre work will save you much of the pain I experienced in my earliest APM deployments.

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics Pro today.

Intelligent Alerting for Complex Applications – PagerDuty & AppDynamics

Screen Shot 2013-04-16 at 2.39.00 PMToday AppDynamics announced integration with PagerDuty, a SaaS-based provider of IT alerting and incident management software that is changing the way IT teams are notified, and how they manage incidents in their mission-critical applications.  By combining AppDynamics’ granular visibility of applications with PagerDuty’s reliable alerting capabilities, customers can make sure the right people are proactively notified when business impact occurs, so IT teams can get their apps back up and running as quickly as possible.

You’ll need a PagerDuty and AppDynamics license to get started – if you don’t already have one, you can sign up for free trials of PagerDuty and AppDynamics online.  Once you complete this simple installation, you’ll start receiving incidents in PagerDuty created by AppDynamics out-of-the-box policies.

Once an incident is filed it will have the following list view:

incident

When the ‘Details’ link is clicked, you’ll see the details for this particular incident including the Incident Log:

incident_details

If you are interested in learning more about the event itself, simply click ‘View message’ and all of the AppDynamics event details are displayed showing which policy was breached, violation value, severity, etc. :

incident_message

Let’s walk through some examples of how our customers are using this integration today.

Say Goodbye to Irrelevant Notifications

Is your work email address included in some sort of group email alias at work and you get several, maybe even dozens, of notifications a day that aren’t particularly relevant to your responsibilities or are intended for other people on your team?  I know I do.  Imagine a world where your team only receives messages when the notifications have to do with their individual role and only get sent to people that are actually on call.  With AppDynamics & PagerDuty you can now build in alerting logic that routes specific alerts to specific teams and only sends messages to the people that are actually on-call.  App response time way above the normal value?  Send an alert to the app support engineer that is on call, not all of his colleagues.  Not having to sift through a bunch of irrelevant alerts means that when one does come through you can be sure it requires YOUR attention right away.

on_call_schedules

Automatic Escalations

If you are only sending a notification and assigning an incident to one person, what happens if that person is out of the office or doesn’t have access to the internet / phone to respond to the alert?  Well, the good thing about the power of PagerDuty is that you can build in automatic escalations.  So, if you have a trigger in AppDynamics to fire off a PagerDuty alert when a node is down, and the infrastructure manager isn’t available, you can automatically escalate and re-assign / alert a backup employee or admin.

escalation_policy

The Sky is Falling!  Oh Wait – We’re Just Conducting Maintenance…

Another potentially annoying situation for IT teams are all of the alerts that get fired off during a maintenance window.  PagerDuty has the concept of a maintenance window so your team doesn’t get a bunch of doomsday messages during maintenance.  You can even setup a maintenance window with one click if you prefer to go that route.

maintenance_window

Either way, no new incidents will be created during this time period… meaning your team will be spared having to open, read, and file the alerts and update / close out the newly-created incidents in the system.

We’re confident this integration of the leading application performance management solution with the leading IT incident management solution will save your team time and make them more productive.  Check out the AppDynamics and PagerDuty integration today!

Deploying APM in the Enterprise Part 6: Spread the APM Love

Welcome back to my series on Deploying APM in the Enterprise. In Part 5: Alerts – Storm of the Century … Every Week! we explored just how important it is to your business to implement alerts the right way.

This week we’ll take a look at one of my favorite aspects of deploying APM properly … showing off (just a little bit) and helping others find their inner rockstar!

How do I help others find their inner rockstar and why would I want to do that?

When it comes right down to it you need to show results. You most likely played a big part in convincing someone to spend money on APM software, or had a big enough problem with an application that someone decided to help you out by purchasing APM software, or you just really want to show your company how good you are at your job. Whatever your motivation, a major key to success lies with getting solid results across your organization. If you are the only person using a tool successfully the odds are stacked against you when it comes time to renew your subscription or maintenance for that tool.