Improving GIS performance with AppDynamics

Twenty years ago I started my professional career working in a niche field of IT called Geographical Information Systems (GIS) – basically mapping systems. This is an area where Big Data was prevalent, long before the term was officially coined, GIS covered a range of concepts from satellite imagery and aerial photography processing to geo-demographic data analysis, address cleansing and geocoding to routing analysis, spatial algorithms and finally to publishing maps & associated data.

Google and Microsoft Bing have recently commoditised a lot of the publishing aspects of background mapping data, but the need for analysing and publishing an organisation’s own geographic data is as important today is it has ever been.

Historically, Application Performance Management (APM) and GIS have never made good bedfellows mainly due to the effort required in deploying APM.  At a recent conference I spoke to a GIS developer who stated they had no interest in APM because “they work in GIS.” Upon further questioning, this GIS developer acknowledged processing and accessing these large datasets is complicated and it’s slow and that’s “just how it is.”. So I challenged this and decided to demonstrate AppDynamics on GeoServer,  an open source mapping server written in Java that allows users to share and edit geospatial data.

Within a few minutes of installation, AppDynamics automatically detected key transactions such as publishing a map layer.

01B-WMS_Service

Taking this a step further, we looked at how AppDynamics can help a system administrator understand which parts of their service is accessed most and which components are the slowest (in this case average response times for publishing mapping datasets).

02-Layer_Performance

In the race to prove innocence, the challenge is to be able to determine whether any slow service calls were due to the end user or consumer of a service rather than the service itself — to understand if users are doing something you did not expect. For example, they could be pulling back too much data from the service (this applies to any system dealing with data).

In the case of mapping systems, AppDynamics can capture the bounding box representing the coordinates used to outline and return an area of interest, if the area of interest retrieved is too large and that leads to a slow transaction, then it’s likely a system administrator is allowing users to pull back too much data in one go:

04-HTTP_Params

If an administrator does see a slow down in the service they can drill into the service to isolate where performance problems occur. Problems in this type of application can manifest themselves in coordinate conversion, vector data reading (roads, rivers & buildings), database queries, grid handlers (reading image files such as satellite and aerial images), writing to an image file: 

03-WMS_at_Method_Level

System administrators and owners can improve their application or data processing times by identifying bottlenecks. The following example shows how an address lookup service was sped up from processing 60 calls per minute taking a second each, to more than 700 calls per minute taking less than 50ms.

Geocode Response Time

Geocode SQL

AppDynamics for Databases identified the cause of the slowness as the spatial query used to query the postcode, without an index it went from querying 1,684,930 rows…

MySQL 1a

To just 5,976: –

MySQL 2a

Tuning this query led to a tenfold performance improvement!

“Software is eating the world” [Marc Andreessen] and ensuring your application or service is performing is key. This message applies across all sectors, not just banking and ecommerce, “typical” APM customers, but everyone who offers a service or process. Our customers & consumers expect Google-like response times for all online services and we need to deliver.

Using AppDynamics, you can halve or better your data processing times, free up resources, achieve a faster time to market or execution and ensure your consumers in the wider world get that up to date map when they need it.

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics Pro today.

Why BSM Fails to Provide Timely Business Insight

Business Service Management (BSM) projects have always had a reputation for over promising and under delivering. Most people know BSM as the alleged “manager of managers,” or “single source of truth.” According to the latest ITIL definition, BSM is described as:  “The management of business services delivered to business customers.” Like much of ITIL this description is rather ambiguous.

Wikipedia however, currently describes BSM’s purpose as facilitating a “cultural change from one which is very technology-focused to a position which understands and focuses on business objectives and benefits.” Nearly every organization I talk to highlights being technology-focussed as one of their biggest challenges, as well as having a desire for greater alignment to business goals. BSM should therefore be the answer everyone is looking for… it’s just a shame BSM has always been such a challenge to deliver.

Some years ago I worked as a consultant for a small startup which provided BSM software and services. I got to work with many large organizations who all had one common goal: “to make sense of how well IT was supporting their business.” It was a tremendous learning experience for me where I frequently witnessed just how little most organizations really understand the impact major IT events had on their business. For example, I remember working with a major European telco who would have an exec review meeting on the 15th calendar day in the month, to review the previous months’ IT performance. The meeting was held on this date because it was taking 4 people 2 weeks to collate all the information and crunch them into a “mega-spreadsheet.” That’s 40 man days effort to report on the previous 30 day period!

As organizations continue to collect an increasing amount of data from a growing list of data sources, more and more organizations I talk to are looking for solutions to manage this type of “information-fogginess,” but are skeptical about undertaking large scale BSM projects due to the implementation timescale and overall cost.

Implementing BSM:

I’m sure the person who first coined the term “scope creep” must have been involved in implementing BSM, as most BSM projects have a nasty habit of growing arms and legs during the implementation phase. I dread to think how many BSM projects have actually provided a return on their substantial investments.

BSM has always been a heavily services-led undertaking as it is attempting to uniquely model and report on an organization. No two organisations are structured in quite the same way; each having its own unique IT architecture, operating model, tools, challenges and business goals. This is why BSM projects almost always begin with a team of consultants conducting lots of interviews.

Let’s look at cost of implementation for a typical deployment such as the European Telco example I described earlier. This type of project could easily expect 100 – 200 man days of professional services in order to deliver. Factoring in software license costs, training, support & maintenance costs, the project needs to deliver a pretty substantial return in order to justify the spend:

Cost of BSM implementation:

Professional services

(100-200 days @ $1800 per day)

$180,000 – $360,000

Software license

$200,000 -$500,000

Annual support and maintenance

$40,000  – $100,000

Training

$25,000

TOTAL

$445k – $985k

Now if we compare to the pre-existing cost of manually producing the monthly analysis:

Existing manual process costs:

Days per month creating reports

10

Number of people

4

Total man days effort per year

480 days

Average annual salary

$45,000

Total working days per year

225

Annual cost to generate reports

$96,000

Even with our most conservative estimates it would take almost 5 years before this organization would see a return on their investment by which time things will probably have changed sufficiently to require a bunch of additional service days in order to update the BSM implementation. This high cost of implementation is one reason why there is such a reluctance to take the leap of faith needed to implement such technologies.

The most successful BSM implementations I am aware of have typically been the smaller projects, which are primarily focused around data visualization; but with powerful open-source reporting tools such as graphite, graphiti or plotly available for free, I wonder if BSM still has a place even with these small projects today?

What does success look like?

Fundamentally, BSM is about mapping business services to their supporting IT components. However, modern IT environments have become highly distributed, with SOA architectures that have data dispersed across numerous cloud environments and it is just not feasible to map basic 1:1 relationships between business and IT functions any more. This growing complexity can only increase the amount of time and money it takes to complete a traditional BSM implementation. A simplified, more achievable approach is needed in order to fulfil the need to provide meaningful business insight from today’s complex IT environments.

In 2011 Netscape founder Mark Andreessen famously described how today’s businesses depend so heavily on applications when he wrote “software is eating the world”. These applications are built with the purpose of supporting whatever the individual business goals are. It seems logical then that organizations should look into the heart of these applications to get a true understanding of how well the business is functioning.

In a previous post I described how this can be achieved using AppDynamics Real-time Business Metrics (RtBM) to enable multiple parts of an IT organisations to access business metrics from within these applications. By instrumenting the key information points within your application code and gathering business metrics in real time such as the number of orders being placed, the amount of revenue per transaction, and more, AppDynamics can enable everyone in your organization to focus on the success or failure of the most important business metrics.

These are very similar goals to those of a traditional BSM project, however in stark contrast to every BSM project I have ever heard of; AppDynamics can be deployed in under an hour, without the need for any costly services as detailed in a previous blog post introducing Real-time Business Metrics.

Instead of interviewing dozens of people, architecting and building complex dependency models, gathering data and analyzing it all to make sense of what is happening, AppDynamics Real time Business Metrics focuses on the key metrics which matter to your business, providing focus and a common measurement for success across IT and The Business.

So before you embark on a long and costly BSM project to understand what is happening in your business, why not download a free trial of AppDynamics and see for yourself; there is an easier way!

DevOps Scares Me – Part 2

What is DevOps?

In the first post of this series, my colleague Jim Hirschauer defined what DevOps means and how it impacts organizations. He concluded DevOps is defined as “A software development method that stresses communication, collaboration and integration between software developers and information technology professionals with the goal of automating as much as possible different operational processes.”

I like to think DevOps can be explained simply as operations working together with engineers to get things done faster in an automated and repeatable way.

DevOps Cycle

From developer to operations – one and the same?

As a developer I have always dabbled lightly in operations. I always wanted to focus on making my code great and let an operations team worry about setting up the production infrastructure. It used to be easy! I could just ftp my files to production, and voila! My app was live and it was time for a beer. Real applications are much more complex. As I evolved my skillset I started to do more and expand my operations knowledge.

Infrastructure Automation

When I was young you actually had to build a server from scratch, buy power and connectivity in a data center, and manually plug a machine into the network. After wearing the operations hat for a few years I have learned many operations tasks are mundane, manual, and often have to be done at two in the morning once something has gone wrong. DevOps is predicated on the idea that all elements of technology infrastructure can be controlled through code. With the rise of the cloud it can all be done in real-time via a web service.

Growing pains

When you are responsible for large distributed applications the operations complexity grows quickly.

  • How do you provision virtual machines?
  • How do you configure network devices and servers?
  • How do you deploy applications?
  • How do you collect and aggregate logs?
  • How do you monitor services?
  • How do you monitor network performance?
  • How do you monitor application performance?
  • How do you alert and remediate when there are problems?

Combining the power of developers and operations

The focus on the developer/operations collaboration enables a new approach to managing the complexity of real world operations. I believe the operations complexity breaks down into a few main categories: infrastructure automation, configuration management, deployment automation, log management, performance management, and monitoring. Below are some tools I have used to help solve these tasks.

Infrastructure Automation

Infrastructure automation solves the problem of having to be physically present in a data center to provision hardware and make network changes. The benefits of using cloud services is that costs scale linearly with demand and you can provision automatically as needed without having to pay for hardware up front.

Configuration Management

Configuration management solves the problem of having to manually install and configure packages once the hardware is in place. The benefit of using configuration automation solutions is that servers are deployed exactly the same way every time. If you need to make a change across ten thousand servers you only need to make the change in one place.

There are other vendor-specific DevOps tools as well:

Deployment Automation

Deployment automation solves the problem of deploying an application with an automated and repeatable process.

Log Management

Log management solves the problem of aggregating, storing, and analyzing all logs in one place.

Performance Management

Performance management is about ensuring your network and application are performing as expected and providing intelligence when you encounter problems.

Monitoring

Monitoring and alerting are a crucial piece to managing operations and making sure people are notified when infrastructure and related services go down.

In my next post of this series we will dive into each of these categories and explore the best tools available in the devops space. As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Click here for DevOps Scares Me Part 3.

Crabs and IT Operations – Different but the Same

I recently had the luxury of sitting on a beach for a while to unwind, de-stress, and recharge my batteries. Even though I really shouldn’t have been thinking about work, the IT performance geek inside of me is always lurking. He can’t be switched off so I allow him to enter the picture, run away with my thoughts for a while, and then lock him away until the next time he breaks out into the wild. This blog post is about one interesting observation the IT performance geek made while I was watching crabs go about their business on the beach.

Background

On my first day of vacation I saw a bunch of holes in the beach with mounds of sand piled up outside of each hole. I thought this to be quite interesting and wondered what sort of creature was the excavating genius. After some time passed by I noticed some suspiciously crabby eyes peering out of the holes at me. If I made any moves the crabs would dart back deep into their burrows and let some time pass until they peered outside again.

Later that day the tide came in and washed away the sand piles covering up the holes that the crabs had made. I wondered what happened to them while the tide was up. Did they stay under the sand and water? Did they go somewhere else? Did they rebuild their holes every single day after the tide subsided? I needed to pay attention during the rest of my vacation to figure this out.

Tides of Change

IT Operations Automation

The tides brought serious problems to the crab burrows. My inner IT performance geek immediately likened the tidal damage to the problems brought about by an onslaught of user activity to a web application. As the tide built the sand pile outside of the burrow washed away. In my head I was imagining a swell of user generated workload eroding away business critical application performance. Only after enough pounding did the sand pile completely give way and the crab burrow collapsed under the stress of the ocean.

This is how I’ve seen most IT performance problems develop as well. It’s usually not an immediate collapse (system crash), but a degradation over a period of time that leads to a failure of the application.

Where Have All the Good Crabs Gone?

How did the crabs respond to all of this? At first I didn’t know where the crabs were while their burrows were under water. After some observation I realized that some had retreated to the tropical woods at the edge of the beach and that others had found shelter on, in, and around a rock wall that ran from the beach into the water. This isn’t really relevant to my story but I thought you’d be interested in finding out. The interesting part to me is what happened when the tide subsided. As the day wore on and the tide became compliant with crab low tide regulations I noticed a massive crab undertaking. Most of the crabs returned at nearly the same time and started re-digging their burrows.

I was amazed at how much sand these little crustaceans could move with a single scoop of their crabby claws. These crabs knew exactly how to fix the problem that the tide had burdened them with and within 30 minutes their burrows were completely restored.

SandPiles

Crabs and IT Operations

This entire process reminded me of my days in IT operations when an alert would fire, we would figure out what the problem was, and soon after problem isolation we would have service restored. That is all fine and good but here is the big problem. The crabs did the same thing every single day of my vacation. They built a burrow, the tide washed it away, and they rebuilt their burrow when the tide subsided. Day after day after day forever.

Many of us did the same thing in IT operations. After the problem was fixed and service restored we congratulated each other and enjoyed our burrows instead of taking the extra time to figure out automated detection and remediation strategies for problems we had seen before. As IT practitioners we usually have more on our plates than we can possibly handle. We live in an age of “do more with less”. The problem is that if we don’t take the extra time to automate our detection and remediation of application issues we will endlessly repeat the cycle of the crabs.

Crabs can’t use complex tools, but thankfully we can. AppDynamics application runbook automation is the cure for the crab cycle. It detects problems and remediates them automatically (you choose if you want to authorize the action or not). We’ve written a couple of other posts that describe the functionality in detail, so please read through them and get familiar with this amazing set of features…

Don’t be an “Also-Ran” – Application Runbook Automation for World Class IT

Application Runbook Automation – A Detailed Walk Through

If you’re tired of the crab cycle do yourself a favor and fix it for good. Click here to get started with your free trial of AppDynamics Pro and see how powerful, easy to use software can make your life better today.

DevOps Scares Me – Part 1

Call of Duty: DevOpsDevOps is scary stuff for us pure Ops folks that thought they left coding behind a long, long time ago. Most of us Ops people can hack out some basic (or maybe even advanced) shell scripts in Perl, ksh, bash, csh, etc… But the term DevOps alone makes me cringe and think I might really need to know how to write code for real (which I don’t enjoy, that’s why I’m an ops guy in the first place).

So here’s my plan. I’m going to do a bunch of research, play with relevant tools (what fun is IT without tools?), and document everything I discover here in a series of blog posts. My goal is to educate myself and others so that we operations types can get more comfortable with DevOps. By breaking down this concept and figuring out what it really means, hopefully we can figure out how to transition pure Ops guys into this new IT management paradigm.

What is DevOps

Here we go, I’m probably about to open up Pandoras Box by trying to define what DevOps means but to me that is the foundation of everything else I will discuss in this series. I started my research by asking Google “what is devops”. Naturally, Wikipedia was the first result so that is where we will begin. The first sentence on Wikipedia defines DevOps as “a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.” Hmmm… This is not a great start for us Ops folks who don’t really want anything to do with programming.

Reading further on down the page I see something more interesting to me … “The goal is to automate as much as possible different operational processes.” Now that is an idea I can stand behind. I have always been a fan of automating whatever repetitive processes that I can (usually by way of shell scripts).

Taking Comfort

My next stop on this DevOps train lead me to a very interesting blog post by the folks at the agile admin. In it they discuss the definition and history of DevOps. Here are some of the nuggets that were of particular interest to me:

  • “Effectively, you can define DevOps as system administrators participating in an agile development process alongside developers and using many of the same agile techniques for their systems work.”
  • “It’s a misconception that DevOps is coming from the development side of the house – DevOps, and its antecedents in agile operations, are largely being initiated out of operations teams.”
  • “The point is that all the participants in creating a product or system should collaborate from the beginning – business folks of various stripes, developers of various stripes, and operations folks of various stripes, and all this includes security, network, and whoever else.”

Wow, that’s a lot more comforting to my fragile psyche. The idea that DevOps is being largely initiated out of the operations side of the house makes me feel like I misunderstood the whole concept right from the start.

For even more perspective I read a great article on O’Reilly Radar from Mike Loukides. In it he explains the origins of dev and ops and shows how operations has been changing over the years to include much more automation of tasks and configurations. He also explains how there is no expectation of all knowing developer/operations super humans but instead that operations staff needs to work closely or even be in the same group as the development team.

When it comes right down to it there are developers and there are operations staff. The two groups have worked too far apart for far too long. The DevOps movement is an attempt to bring these worlds together so that they can achieve the effectiveness and efficiency that the business deserves. I really do feel a lot better about DevOps now that I have done more research into the basic meaning and I hope this helps some of you who were feeling intimidated like I was. In my next post I plan to break down common operations tasks and talk about the tools that are available to help automate those tasks and their associated processes.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Click here for DevOps Scares Me Part 2.

AppDynamics & Splunk – Better Together

AppD & Splunk LogoA few months ago I saw an interesting partnership announcement from Foursquare and OpenTable.  Users can now make OpenTable reservations at participating restaurants from directly within the Foursquare mobile app.  My first thought was, “What the hell took you guys so long?” That integration makes sense on so many levels, I’m surprised it hadn’t already been done.

So when AppDynamics recently announced a partnership with Splunk, I viewed that as another no-brainer.  Two companies with complementary solutions making it easier for customers to use their products together – makes sense right?  It does to me, and I’m not alone.

I’ve been demoing a prototype of the integration for a few months now at different events across the country, and at the conclusion of each walk-through I’d get some variation of the same question, “How do I get my hands on this?”  Well, I’m glad to say the wait is over – the integration is available today as an App download on Splunkbase.  You’ll need a Splunk and AppDynamics license to get started – if you don’t already have one, you can sign up for free trials of Splunk and AppDynamics online.

QCon: Enough with the theory, already—Let’s see the code

San Francisco’s QCon was expecting a smaller crowd, but ended up bursting at the seams: the event sold out weeks ahead of time and in many sessions it was standing room only.

Targeted at architects and operations folks as much as developers, QCon was heavy on the hot topics of the day: SOA, agile, and DevOps. But if there was a consistent trend throughout the three days, it was “No more theory. Show us the practice.”

At Jesper Boeg’s talk for example—“Raising the Bar: Using Kanban and Lean to Super Optimize Your Agile Implementation”—the talk was peppered with some good sound bites (“If it hurts, do it more often and bring the pain forward”). But it also stressed the meat: Boeg demonstrated a “deployment pipeline” that represented an automated implementation of the build, deploy, test, and release process—a way to find and eliminate bottlenecks in agile delivery.

Similarly, John Allspaw started high in his talk—sharing his ideas on the areas of ownership and specialization between Ops and Dev, a typical DevOps presentation—but backs up the theory with code-level discussions of how logging, metrics, and monitoring works at Etsy.  (His blog entry on the subject and complete Qcon slides can be found on his blog, Kitchen Soap.)

Adrian Cockroft, who is leading a trailblazing public cloud deployment of production-level applications at Netflix, also wrapped theory around juicy substance. He “showed the code” and the screenshots of his company’s app scaling and monitoring tools (you can find his complete slide presentation here).

Not everyone took the time to drill down, though. Tweets from QCon attendees showed that the natives got restless in talks that stayed too high level:

“OK, just because you can draw a block diagram out of something doesn’t mean it makes sense.”

“Ok, we get it. Your company is very interesting, now get to the nerd stuff.”

“These sessions are high-level narratives. Show me the code, guys! Devil’s in the details.”

At the same time, they would shower plaudits and congratulations on speakers who gave them what they wanted: something new to learn.

When the Twitter stream started to compare QCon’s activities with an event happening concurrently in the city, Cloud Expo, the nature of the attendees was draw into sharp relief:

“At #cloudexpo people used laptops during sessions to check email… At #qconsf they are writing code.”

When it comes to agile, SOA, DevOps, and other problems of the day, people are ready for answers.

If Your App Had a Facebook Status It’d Be, “It’s Complicated”

AppDynamics is founded on a set of deeply held beliefs regarding our industry and how it’s changed over the last several years.  But it’s never good to let deeply held beliefs stay unchallenged.  So every now and then, we do a reality check.

Our most recent reality check was during our webinar presentation of AppDynamics 3.0. We attracted hundreds of IT ops and dev professionals who wanted to learn both about our solution as well as the specific features of our new release—so we took the opportunity to poll them and ask them a few questions. First, we asked if they operated in a SOA environment:

AppDynamics believes that applications are increasingly moving to SOA, turning monolithic web architectures from the early 2000s into obsolete antiques.  As you can see, that belief was confirmed; the vast majority of our webinar attendees have already entered that world.

Then we asked if they followed an agile development approach:

Again, the vast majority of attendees have embraced agile—in fact, nearly 50% release new features or capabilities at least monthly!  Only 8% report that they follow the traditional, waterfall approach to development. With those kinds of tumultuous deadlines, AppDynamics remains impressed that these hardy souls were actually able to take enough time out of their schedule to watch our presentation.

Finally, we wanted to know the punch line: what’s the effect of all this change on their ability to manage application performance?

Over half were really feeling the crunch, and only a scant few had escaped unscathed.

It’s not that AppDynamics enjoys the pain of others. (We don’t. Really.)  But having our fundamental beliefs confirmed—that the world of applications has changed, and application management solutions need to change with them—simply lets us know that we exist for the right reasons.

Take the example of one of our most recent customers, TiVo.  Operating in a highly distributed environment, TiVo has hundreds of individual Java and proprietary applications, designed to work together to deliver service to its customers.

“We used to spend hours troubleshooting issues,” Richard Rothschild, Senior Director of IT, told us.  “If a service was running slowly and we didn’t know the cause, finding that root cause was like looking for a needle in a haystack.

He continued, “We used to spend up to 6 hours on root cause. AppDynamics brought that time down to ten minutes.  We’ve already seen a big improvement in the reliability and uptime of our services—anything that simplifies our job in this complex environment makes us feel much more confident about taking on new business projects.”

It’s complicated out there, and with the advent of cloud and virtual environments, it’s not getting any easier. But we went into this business in order to simplify application performance management and support application teams in their quest for both performance and availability.  So far, it looks like we chose the right reasons to exist.

DevOps Days – Visibility & Aligning Goals

This summer, I had a chance to sit on a panel at DevOps Days with a few industry colleagues to discuss the role of monitoring, testing, and performance in solving DevOps issues. It was a lively discussion on the teams, tools, and techniques that play a role in managing application and network performance.