Q&A: Add Apache Web Server to your Unified Monitoring Toolkit

Last month Amod Gupta and I presented in a webinar titled “Add Apache Web Server to your Unified Monitoring Toolkit”.

I kicked off the webinar by providing an overview of AppDynamics Unified Monitoring solution and an overview of Apache Server Monitoring solution.

Screen Shot 2015-09-09 at 11.23.17 PM.png

Figure 1. Application Flow Map with Business Transaction starting from an Apache Server

Amod followed me to discuss how the Apache Web Server Agent, a key part of AppDynamics Unified Monitoring Solution, lets enterprises:

  • Discover and check the health of your Apache web server(s)
  • Alert in the case of any anomalies
  • Perform a root cause analysis on issues quickly

Here is the presentation that was used during the webinar.

http://www.slideshare.net/appdynamics/add-apache-web-server-to-your-unified-monitoring-toolkit

We had a very interactive sessions after the prepared content was presented. We answered many questions asked during this Q&A, but many of these questions remained unanswered. We responded to all the unanswered questions to the individuals who asked the questions.

Here are the top unanswered questions and answers from the webinar. You can listen to the questions and answered that were covered during the webinar at the on-demand recording of the webinar.

Q: What are the costs for the apache module?

A: Current pricing for Apache Server Monitoring solution is $3,300 per OS instance (Pricing as of September 08, 2015)

You could get volume discount in case you need to purchase more than 10 licenses. In addition, first time customers purchasing 1-10 units can get starter discount.  You can find latest pricing and more details at http://www.appdynamics.com/pricing/ .

As you may know, AppDynamics offers a freemium pricing model with complete access to AppDynamics product for free for 15 days. After 15 days of free trial period, customers can either purchase the products or keep evaluating the product forever under lite plan.

Q: If we multiple instances of Apache Server running on the server, do we create multiple proxy for each? Or will one proxy support multiple apache instances?

A: One proxy will handle all the instances. Our proxy has built in support for multi-tenancy.

Q: What about reverse-proxy, does it capture any performance metrics?

A: Yes it captures the same performance metrics.

Q: How does the mod_appdynamics work in case of security agents like siteminder agent or Oracle Access Manager agent?

A: It treats calls going out to siteminder as exit calls and shows you the time spent on mod_sm  which translates as the time spent by a request in the authentication module.

Q: At what priority level is the agent loaded? and does it determine the time taken between modules…. example Webserver->Siteminder agent->webserver->mod_ajp->webserver?

A: It shows you the time spent by a request in each module and it shows you the time spent by request in apache. It doesn’t show the time spent between one module and another.

Q: Do we have separate agent and machine agent for Apache web server like java agent?

A: Yes, that’s correct.

Q: Where can we find documentation for apache web server configuration?

A: Documentation for Instrumenting Apache Web Server monitoring can be found at https://docs.appdynamics.com/display/PRO41/Instrument+Web+Servers . Additional details on the Apache Web Server monitoring solution can be found at https://docs.appdynamics.com/display/PRO41/APM+for+Web+Servers .

Q: Do we require only one license for multiple web server agents running on a single box?

A: Yes, that’s correct.

Q: Are you referring to the Apache HTTP Server that comes with the RedHat Linux? We don’t use the default Apache that comes with the OS instead we installed our own Apache at a separate location on the server. Will AppDynamics detect this apache server?

A: Yes, it will detect and monitor your Apache Server. It doesn’t have to be the web server that comes bundled with any OS. It doesn’t even have to be on RedHat Linux.

Q: Do you support an environment where F5 load balancer is load balancing in front of apache server?

A: Yes, in this case, we’d start monitoring the requests when they hit apache server. We also have an extension for F5 Load Balancer that can help you monitor the metrics from F5 and then you can correlate them with rest of your application environment metrics in AppDynamics.

Q: Is there any business transaction detection we need to configure for the web server agent? Any special instrumentation that’s needed, or should we expect things to work out of the box? I’m using apache 2.4.

A: You should expect things to work out of the box. You can configure Business Transactions naming based on URL segments but by default you should expect to see automatic business transactions being detected and named using the last two segments of the URL.

Q: How much overhead we should expect while having AppDynamics agent on webserver?

A: AppDynamics agent does not add significant overhead. Overhead depends heavily on the hardware and load on the box.

 

To learn more about the Apache Web Server Monitoring solution, please watch the webinar at the on-demand recording of the webinar .

Apache Web Server Performance Monitoring

 

We are very excited to announce that from version 4.1 onwards, AppDynamics will support first class monitoring of Apache Web Servers. Apache Web Server has been the most popular web server for some time and is often the gateway to highly distributed applications. Needless to say, monitoring Apache’s availability and performance is critical to gauging the health of a business application.

What does it do?

AppDynamics Web Server Agent lets you

  • Discovers web servers in your application topology
  • Check the health of your Apache web server(s)
  • Alert in the case of any anomalies
  • Perform a root cause analysis quickly

AppFlowmap1.png

Figure 1: Application Flow Map with Web Server

The figure above shows a partial screenshot of an Application Flow Map in AppDynamics. Notice the Apache web server which was discovered automatically and its KPIs – load and average response time.

But how do I do these things?

Bird’s eye view

AppDynamics Application Flow Map will automatically show you all the web servers in the flow map along with their KPIs (as shown in the picture above). The Application Flow Map is constructed using all the traffic that is coming in and flowing between different tiers of your distributed application. Sometimes, you may want to focus only on your web servers and the Tier dashboard allows you to do just that. The Tier Flow Map represents the aggregate traffic flowing through all your web servers, however, if you wanted to focus on one particular web server instance, you could accomplish that by going to the node level dashboard. The gif below shows the different level of details you can view while cycling through these views.

output_llRs3b.gif

Figure 2: Drilling down from Application Flow Map to Node Dashboard

Outliers and baselining

The web server agent automatically captures three KPIs – load, response time, and errors. AppDynamics also calculates baselines for these KPIs so that you can look instantly look at the outliers (shown below).

KPI.png

Figure 3: Key Performance Indicators

What about Business Transactions?

AppDynamics automatically categorizes all the incoming traffic into Business Transactions (BT) and it’s no different for the web server agent. Your BTs now start at the web server and you get all the goodness that accompanies it like – transaction scorecards, transaction analysis, snapshots etc. More information about BTs can be found here.

For instance, the gif below shows how to troubleshoot the anomaly seen in Figure 3 above around 4:32 PM by going from the dashboard to BT snapshot in a few clicks.

troubleshoot.gif

Figure 4: Troubleshooting

So, web servers have BT snapshots?

AppDynamics captures snapshots for all BTs flowing through your application. More information about BT snapshots can be found here. Since these BTs now start at the web server tier, the same snapshots will have a web server segment. Valuable troubleshooting information is included in these snapshots. Specifically for Apache, these snapshots will include the Apache modules where most time is being spent. For instance, Figure 5 shows an Apache segment in a snapshot. Apache’s mod_proxy is taking up most time (222 ms) in this snapshot in the Content Handler stage of Http request processing in Apache. Clicking on the drill down button in the bottom half will take you to the next tier where this http request is routed (Shipping Portal).

Snapshots.png

Figure 5: Snapshots with Apache Modules and Exit calls

Alright, I am sold. How do I get my hands on this?

Apache Web server agent is now generally available. You can download the installer package from AppDynamics Download Zone.

An example of how Node.js is faster than PHP

I wanted to see what all the Node.js hype was about so I decided to run some head to head load tests using Ghost (Node.js) and WordPress (php). The results were incredible with Ghost soundly trouncing WordPress. It was like watching a starship racing an airplane (well, what I imagine that would be like anyway).

There is a new blogging platform that was recently made available to the general public called Ghost. What’s interesting about Ghost is that it is built on the Node.js platform. If you’re not familiar with Node.js you should read my blog post about it here. If your not familiar with Ghost you can read about this kickstarter project here.

The Setup

To provide a little background, Ghost is just a blogging platform and nothing more while WordPress is a full up CMS. I wanted to make this comparison as fair as possible so I limited my load testing scripts to executing against only the blog pages. I also wanted to test the “out of the box” experience so I did not make configuration changes to either platform (besides hooking them both up to MySQL). I spun up a single 64-bit RHEL m1.large (reference server sizing image below for specs) instance on Amazon EC2 to host both blogging platforms.

Amazon Server Sizes

I wanted to test the most common configurations so I used NginX to front end Ghost and used Apache to front end WordPress. Both platforms shared the same local MySQL backend database instance (Ghost comes with SQLite by default but I wanted to make sure I provided a level playing field on the back end).

I had both Ghost (listening on port 80) and WordPress (listening on port 8080) running at the same time but only applied load to one blogging platform at any given time.

That brings me to the load generation portion of this little experiment. I spun up another EC2 instance (64-bit ubuntu, size m1.medium – reference server sizing image above for specs) in the same availability zone in an attempt to minimize network impact on test results. I asked my colleague @dustinwhittle to recommend a load test configuration and he referred me to his blog post about load test tools and recommended I used a combination of Siege and Sproxy.

After I had the blogging platforms installed and tested as working I added an 8 part blog series in plain text (no images) to each site and removed any pre-existing blogs. In WordPress I left the standard URL pattern in place and did NOT implement permalinks so that I would not slow things down by using that feature. I also did not turn on any caching technology for WordPress as I was trying to measure the out of the box experience. Basically I didn’t attempt any sort of tuning at all on either platform.

The other major configuration to note was that I used the AppDynamics machine agent to collect and chart OS metrics during these load tests.

The Tests

In order to use Siege to test many concurrent connections against many URLs I had to create a list of the URLs in a text file. For this I used Sproxy. Reference slides 20-23 of the following presentation for the details on using Sproxy https://speakerdeck.com/dustinwhittle/agile-performance-testing-checklist

I ran Sproxy against both Ghost and WordPress and ended up with my list of URLs. I modified each of these files to include the exact same list of blog posts so that the load tests would be as similar as possible. You can see the contents of each file below.

Ghost Load Test URLS

Wordpress Load Test URLS

So now I was ready to fire up Siege and start hitting each blog with load. Siege is a nice tool that allows you to manipulate some key parameters. The ones I played with the most were the number of concurrent connections (-c) and the delay (-d in seconds) between batches of requests. Here is the command for your reference… siege -v -c 100 -i -t 10M -f urls.txt -d 1

The Results

In a word, staggering! I ran siege for 10 minutes with 100 concurrent connections and a 1 second delay between batches of web requests. The results are shown below…

Ghost Performance Under Heavy Load

Siege load test results for Ghost with Nginx under heavy load.

Wordpress Siege Results

Siege load test results for WordPress and Apache under heavy load.

As you can see from the output shown above, Ghost with Nginx outperformed WordPress with Apache by about 678% when looking at total transactional throughput over a 10 minute test. Impressively, the longest transaction response time for Ghost was 2.62 seconds compared the an abysmal 33.41 seconds for WordPress. I repeated these test runs multiple times and got very similar results so I am not going to show the rest of the test results since they are redundant. My goal here was not to run an exhaustive analysis of performance at varying loads but instead to create a substantial load and to see how each platform handled it.

Some other interesting data points to note. During the load test, Ghost ran with only 1 process and Nginx had a total of 2 processes. WordPress and Apache on the other hand spawned a total of ~110 httpd processes which makes sense since Siege was throwing 100 concurrent connections at it. The interesting part is in the CPU data during the load tests. I have plotted Average, Min, and Max CPU utilization on the charts below. You can clearly see that Ghost CPU consumption was about 40% while WordPress consumption was about 70%.

Ghost Heavy Load - AppDynamics

CPU consumption during Ghost load test showing Average, Min, and Max values.

Wordpress Heavy Load - AppDynamics

CPU consumption during WordPress load test showing Average, Min, and Max values.

Now don’t think that I have forgot about normal loading patterns. How do things look with a moderate load as compared to the super high load that I placed on these platforms with the 100 concurrent connections test? To find out I dropped the number of concurrent connections to 10 and set the delay between batches of connections to 5 seconds. The results are shown below and are still incredibly impressive for Ghost. WordPress was outperformed in every way possible. Ghost had higher throughput and most importantly the slowest transaction response time was .18 seconds compared to 2.72 seconds for WordPress. From a CPU perspective Ghost only consumed ~4% on average during this test while WordPress consumed ~30% on average.

Siege load test results for Ghost with Nginx under light load.

Siege load test results for Ghost with Nginx under light load.

Siege load test results for WordPress with Apache under light load.

Siege load test results for WordPress with Apache under light load.

Update on 10/18/2013 – It’s not fair!!! Apples and Oranges!!!

There have been some who say I’m comparing apples to oranges. To this I say, you’re damn right! In this post I set out to compare the common combinations of Nginx + Ghost and Apache + WordPress. I set out to compare these in their most basic forms, no tuning, no caching, just what you get out of the box. But I understand the outcry and I decided to level the playing field. Some people thought that Apache was a bottleneck so I decided to use Apache as the front end web server for Ghost and to re-run my load tests. I ran multiple tests again but they were all very consistent so I am only going to show the output from one of them (shown below).

Apache and Ghost Siege Results

Load test results of Apache + Ghost.

Load test results for Nginx + Ghost for easy comparison with the results above.

Load test results for Nginx + Ghost for easy comparison with the results above.

The results shown above are interesting. Apache + Ghost was actually slightly FASTER than running Ghost with Nginx. Ghost is still super fast regardless of using Apache or Nginx as the web server.

The Conclusion

Ghost is way faster and can handle way more load than WordPress while also consuming much less CPU resource (Ghost also has considerably less functionality than WordPress but that’s not relevant for the purpose of this test). It would be interesting to run Ghost in a 2 process Node.js cluster and see  what difference it makes in throughput and CPU utilization. Hmmmm, that sounds like a really good subject for another blog post…

Another interesting topic that I didn’t cover here is monitoring for both of these platforms. In my mind it’s not enough to just observe the behavior of these platforms from the outside. I want to see what’s going on from the inside. In a future blog post I am going to monitor Ghost with Nodetime and will monitor WordPress with AppDynamics. I can’t wait to see how they both look from the inside!

Update on 11/14/2013: Due to popular request I have performed more testing, this time with an opcode cache for PHP. You can read all about it in An Example of How Node.js is Faster Than PHP – Part 2

Appendix

Here are the links to the information I used to build out my blogging and testing platforms (I used the relevant portions of each article since my configuration was different than what was in each article alone):

How to install Node.js: https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager

How to install Ghost: http://docs.ghost.org/installation/

How to install MySQL: http://www.cyberciti.biz/faq/how-to-install-mysql-under-rhel/

How to install and configure NginX and use MySQL: http://0v.org/installing-ghost-on-ubuntu-nginx-and-mysql/#.Ul26n2RATL4

Where to find Siege: http://www.joedog.org/siege-home/

Where to find Sproxy: http://www.joedog.org/sproxy-home/

More good information on using Siege and Sproxy: http://www.euperia.com/linux/tools-and-utilities/speed-testing-your-website-with-siege-part-two/771

AppDynamics 3.7.8 – New Features Abound

I love it when my favorite applications release new features. AppDynamics has shifted to a new release model which enables us to release more features faster while maintaining high quality code. The end result is that you, the users, get the cool features you need without having to wait around for 6 months for a big bang release. So what goodness have we cooked up for you in this release?

Core Product

Awesome RabbitMQ Support – RabbitMQ worked just fine in prior version of AppDynamics but we wanted to make it super easy for you to get the exact instrumentation details you want. You asked for it, we listened, and here it is…

RabbitMQ

Percentile Metrics – Averages have long been the standard for measuring response time over time but we understand that percentiles can provide additional insight into the distribution of your transaction response times. To arm you with the best information possible we have introduced percentile metrics for all Business Transactions. By default we show 95th percentile but you can change that to whatever you want.

Percentiles

HTML5 Navigation – We’re in the process of shifting our entire UI to HTML. We started with the navigation menus to provide a faster user experience for everyone. HTML5 Navigation menus are now turned on by default.

Java Specific

Open Mbeans – Open Mbeans enable functionality that normal mbeans can’t handle (like gathering tabular data). Open MBeans are now supported by AppDynamics Pro.

GlassFish Appserver Management Extensions (AMX) – If you use GlassFish and monitor using AMX you’re in luck. We now support AMX out of the box and have preconfigured rules to get your AMX data into AppDynamics.

GlassFish AMX

Database Specific

Sybase IQ – Extending our industry leading database support, AppDynamics now officially supports Sybase IQ. If you’ve got a Sybase IQ data warehouse that needs monitoring you’ll want to check out AppDynamics for Databases.

PHP Specific

For PHP here is our list of new features

  • 32-bit x86 support
  • Apache 2.4 support
  • RPM Installer
  • memcache as an auto-detected cache exit point

memcache

.NET Specific

  • New unified configuration
  • New node naming
  • Improved URL rewrites and BT naming
  • Parameterized installer

This rapid release has more new features than most companies major releases. We’re going to blog in more detail about some of these features in upcoming weeks so keep an eye out. We’ll also keep you updated every time we have a rapid release right here in our blog so be sure to subscribe to our RSS feed (contrary to Googles opinion, RSS is not dead) so you don’t miss out. If you’re not already an AppDynamics customer you can take a free self service trial by clicking here.