Big Data Monitoring

The term “Big Data” is quite possibly one of the most difficult IT-related terms to pin down ever. There are so many potential types of, and applications for Big Data that it can be a bit daunting to consider all of the possibilities. Thankfully, for IT operations staff, Big Data is mostly a bunch of new technologies that are being used together to solve some sort of business problem. In this blog post I’m going to focus on what IT Operations teams need to know about big data technology and support.

Big Data Repositories

At the heart of any big data architecture is going to be some sort of NoSQL data repository. If you’re not very familiar with the various types of NoSQL databases that are out there today I recommend reading this article on the MongoDB website. These repositories are designed to run in a distributed/clustered manner so they they can process incoming queries as fast as possible on extremely large data sets.

MongoDB Request Diagram

Source: MongoDB

An important concept to understand when discussing big data repositories is the concept of sharding. Sharding is when you take a large database and break it down into smaller sets of data which are distributed across server instances. This is done to improve performance as your database can be highly distributed and the amount of data to query is less than the same database without sharding. It also allows you to keep scaling horizontally and that is usually much easier than having to scale vertically. If you want more details on sharding you can reference this Wikipedia page.

Application Performance Considerations

Monitoring the performance of big data repositories is just as important as monitoring the performance of any other type of database. Applications that want to use the data stored in these repositories will submit queries in much the same way as traditional applications querying relational databases like Oracle, SQL Server, Sybase, DB2, MySQL, PostgreSQL, etc… Let’s take a look at more information from the MongoDB website. In their documentation there is a section on monitoring MongoDB that states “Monitoring is a critical component of all database administration.” This is a simple statement that is overlooked all too often when deploying new technology in most organizations. Monitoring is usually only considered once major problems start to crop up and by that time there has already been impact to the users and the business.

RedisDashboard

Dashboard showing Redis key metrics.

One thing that we can’t forget is just how important it is to monitor not only the big data repository, but to also monitor the applications that are querying the repository. After all, those applications are the direct clients that could be responsible for creating a performance issue and that certainly rely on the repository to perform well when queried. The application viewpoint is where you will first discover if there is a problem with the data repository that is actually impacting the performance and/or functionality of the app itself.

Monitoring Examples

So now that we have built a quick foundation of big data knowledge, how do we monitor them in the real world?

End to end flow – As we already discussed, you need to understand if your big data applications are being impacted by the performance of your big data repositories. You do that by tracking all of the application transactions across all of the application tiers and analyzing their response times. Given this information it’s easy to identify exactly which components are experiencing problems at any given time.

FS Transaction View

Code level details – When you’ve identified that there is a performance problem in your big data application you need to understand what portion of the code is responsible for the problems. The only way to do this is by using a tool that provides deep code diagnostics and is capable of showing you the call stack of your problematic transactions.

Cassandra_Call_Stack

Back end processing – Tracing transactions from the end user, through the application tier, and into the backend repository is required to properly identify and isolate performance problems. Identification of poor performing backend tiers (big data repositories, relational databases, etc…) is easy if you have the proper tools in place to provide the view of your transactions.

Backend_Detection

AppDynamics detects and measures the response time of all backend calls.

Big data metrics – Each big data technology has it’s own set of relevant KPIs just like any other technology used in the enterprise. The important part is to understand what is normal behavior for each metric while performance is good and then identify when KPIs are deviating from normal. This combined with the end to end transaction tracking will tell you if there is a problem, where the problem is, and possibly the root cause. AppDynamics currently has monitoring extensions for HBase, MongoDB, Redis, Hadoop, Cassandra, CouchBase, and CouchDB. You can find all AppDynamics platform extensions by clicking here.

HadoopDashboard

Hadoop KPI Dashboard 1

HadoopDashboard2

Hadoop KPI Dashboard 2

Big data deep dive – Sometimes KPIs aren’t enough to help solve your big data performance issues. That’s when you need to pull out the big guns and use a deep dive tool to assist with troubleshooting. Deep dive tools will be very detailed and very specific to the big data repository type that you are using/monitoring. In the screen shots below you can see details of AppDynamics monitoring for MongoDB.

MongoDB Monitoring 1

 MongoDB Monitoring 2

MongoDB Monitoring 3

MongoDB Monitoring 4

If your company is using big data technology, it’s IT operations’ responsibility to deploy and support a cohesive performance monitoring strategy for the inevitable performance degradation that will cause business impact. See what AppDynamics has to offer by signing up for our free trial today.

Data Clouds Part II: My Big Data Dashboard

In my previous blog, I wrote at length about the complexities of running a data cloud in production. This logical data set, spread across many nodes, requires a whole new set of tools and methodologies to run and maintain. Today we’ll look at one of the biggest challenges in managing a data cloud – monitoring.

Database monitoring used to be easy in the days before data clouds. Datasets were stored in a single large database, and there were hundreds of off-the-shelf products available to monitor the performance of that database. When problems occurred, one had simply to open up the monitoring tool and look at a set of graphs and metrics to diagnose the problem.

There are no off-the-shelf tools for monitoring a data cloud, however. There’s no easy way to get a comprehensive view of your entire data cloud, let alone diagnose problems and monitor performance. Database monitoring solutions simply don’t cut it in this kind of environment. So how do we monitor the performance of our data cloud? I’ll tell you what I did.

It just so happens I work at AppDynamics, one of the most powerful application monitoring tools on the market. We monitor all parts of your application including the data layer, with visibility into both Relational and NoSQL systems like Cassandra. With AppDynamics I was able to create a dashboard that gives me a single pane-of-glass view into the performance of my data cloud.

Big Data Dashboard

My Big Data Dashboard

This dashboard is now used in several departments at AppDynamics including Operations, QA, Performance and development teams to see how our data cloud is running. All key metrics about all of our replicas are graphed side by side on one screen. This is the dream of anyone running big data systems in production!

Of course, not all problems are system wide. More often than not you need to drill into one replica or replica set to find a problem. To do that, I simply double click on any part of my big data dashboard to focus on a single replica, change the time range, and add more metrics.

Data clouds are difficult to run, and there aren’t any database monitoring tools fit to monitor them yet. But instead of sitting around waiting for data monitoring tools to catch up with our needs, I’ve built my own Big Data Dashboard with monitoring tool designed for applications.

Of course the fun doesn’t stop here…I still need to find a way to set up alerts and do performance tuning for my data cloud. Stay tuned for more blogs in this series to see how I do it!