Monitoring NGINX with AppDynamics

Did you know the AppDynamics Application Intelligence Platform includes monitoring for hundreds of IT components out of the box, including NGINX and NGINX Plus? They’re all featured in the AppDynamics Exchange.

With the AppDynamics extension for NGINX, you gain powerful capabilities and insights including:

  • Business Context: Application transaction tracing automatically associates IT components with business activities
  • Self-learning: Dynamically baselines behavior, eliminating static threshold configuration and false positives
  • Unified Monitoring: One platform and a single UI for end-user, application, and infrastructure monitoring
  • DevOps Automation: From remediation to cloud capacity scaling, automate for proactivity and efficiency

I’m getting ahead of myself though. Let’s back up a minute and cover the basics.

First and foremost, what exactly is NGINX?

From NGINX, they “accelerates content and application delivery, improves security, facilitates availability and scalability for the busiest websites on the Internet.” Next layer explanation: NGINX (open source) and NGINX Plus (commercial) are an HTTP server and reverse proxy, and an IMAP/POP3 proxy server. And if you want to get technical: NGINX is one of a handful of servers written to address the C10K problem. Unlike traditional servers, NGINX doesn’t rely on threads to handle requests. Instead, it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly, predictable amounts of memory under load.

That last part is a big deal, especially given the large-scale, complex, distributed web application architectures of today. Which is where AppDynamics comes in.

AppDynamics provides 24/7 visibility and management of your entire application environment, whether you’re running 10 servers or 10 thousand. We do this via our Application Intelligence Platform, a core pillar of which provides Unified Monitoring for your end-users, applications, and infrastructure.

Unifying monitoring sounds great and all, but it’s how we do it that makes the difference, which brings me back to the beginning of this blog.

Business Context and why it matters

Screen Shot 2015-01-28 at 4.13.56 PMTraditional monitoring solutions take a bottom-up approach: start with the infrastructure and attempt to logically associate it with the business. This barely worked with the static, monolithic IT systems of yesterday and is hopelessly inadequate for the service-oriented, high-distributed systems and agile processes of today.

A new top-down approach to monitoring is required, and it starts with the Business Transaction (BT): a unit of measure based on a business-level application transaction (e.g. customer checkout) that is automatically associated with the IT components used for its execution, including application code and infrastructure resources. This is accomplished by automatically tagging and tracing the execution of each and every application transaction through underlying IT systems. I wrote more in depth about this BT-focused method in a previous blog post.

Self-learning, so what?

If you’ve ever been a part of an IT operations team, you know what a pain it is to define altering thresholds. The entire process is predicated on knowing what’s normal, and from there when to be warned and when to be alerted there’s a critical problem. Static thresholds, on the other hand, are too rigid and in danger of either alert storms and/or missing performance issues.

AppDynamics dynamically baselines every metric – from CPU, memory, network, and disk utilization to end-user, transaction, and database query response times. This proactively detects anomalies and alerts intelligently.

Okay, so what about NGINX specifically? Take all the above and apply it to what we track directly from NGINX, including:

  • Active connections
  • Total number of requests
  • Accepted and handled requests
  • Reading, writing, and waiting metrics

Once you have the NGINX extension configured and integrated with AppDynamics, you can see all the NGINX metrics in the AppDynamics Metric Browser. For example, in the screen shot below you can see all the Active Connections to the NGINX during a certain time interval.

Screen Shot 2015-01-28 at 4.11.09 PM

As mentioned earlier, you can monitor the entire business transaction all the way from a mobile device or web browser to the NGINX web server through to the database. For example, you can analyze the NGINX metrics against the connected server. In the screenshot below, you can see the correlation between the number of NGINX active connections and the app server threads. As the traffic increased you can see the NGINX connection increase as well with the server responding with more threads serving more web requests. This causes the number of active connections to decrease.

Screen Shot 2015-01-28 at 4.11.16 PM

You can also set-up health rules triggering actions such as sending emails, executing remedial scripts, etc. after crossing a certain threshold.

Screen Shot 2015-01-28 at 4.11.23 PM

With AppDynamics, you can create custom dashboards for specific KPIs and metrics.

Screen Shot 2015-01-28 at 4.11.30 PM

Monitoring NGINX with AppDynamics is more than just stats and graphs. With automatic transaction tracing, you’ll see your web application in business context, while the self-learning baselines alert you with what’s normal – and abnormal – without configuring a thing.

If you’d like to learn more, check out our webinar with NGINX!

An example of how Node.js is faster than PHP

I wanted to see what all the Node.js hype was about so I decided to run some head to head load tests using Ghost (Node.js) and WordPress (php). The results were incredible with Ghost soundly trouncing WordPress. It was like watching a starship racing an airplane (well, what I imagine that would be like anyway).

There is a new blogging platform that was recently made available to the general public called Ghost. What’s interesting about Ghost is that it is built on the Node.js platform. If you’re not familiar with Node.js you should read my blog post about it here. If your not familiar with Ghost you can read about this kickstarter project here.

The Setup

To provide a little background, Ghost is just a blogging platform and nothing more while WordPress is a full up CMS. I wanted to make this comparison as fair as possible so I limited my load testing scripts to executing against only the blog pages. I also wanted to test the “out of the box” experience so I did not make configuration changes to either platform (besides hooking them both up to MySQL). I spun up a single 64-bit RHEL m1.large (reference server sizing image below for specs) instance on Amazon EC2 to host both blogging platforms.

Amazon Server Sizes

I wanted to test the most common configurations so I used NginX to front end Ghost and used Apache to front end WordPress. Both platforms shared the same local MySQL backend database instance (Ghost comes with SQLite by default but I wanted to make sure I provided a level playing field on the back end).

I had both Ghost (listening on port 80) and WordPress (listening on port 8080) running at the same time but only applied load to one blogging platform at any given time.

That brings me to the load generation portion of this little experiment. I spun up another EC2 instance (64-bit ubuntu, size m1.medium – reference server sizing image above for specs) in the same availability zone in an attempt to minimize network impact on test results. I asked my colleague @dustinwhittle to recommend a load test configuration and he referred me to his blog post about load test tools and recommended I used a combination of Siege and Sproxy.

After I had the blogging platforms installed and tested as working I added an 8 part blog series in plain text (no images) to each site and removed any pre-existing blogs. In WordPress I left the standard URL pattern in place and did NOT implement permalinks so that I would not slow things down by using that feature. I also did not turn on any caching technology for WordPress as I was trying to measure the out of the box experience. Basically I didn’t attempt any sort of tuning at all on either platform.

The other major configuration to note was that I used the AppDynamics machine agent to collect and chart OS metrics during these load tests.

The Tests

In order to use Siege to test many concurrent connections against many URLs I had to create a list of the URLs in a text file. For this I used Sproxy. Reference slides 20-23 of the following presentation for the details on using Sproxy https://speakerdeck.com/dustinwhittle/agile-performance-testing-checklist

I ran Sproxy against both Ghost and WordPress and ended up with my list of URLs. I modified each of these files to include the exact same list of blog posts so that the load tests would be as similar as possible. You can see the contents of each file below.

Ghost Load Test URLS

Wordpress Load Test URLS

So now I was ready to fire up Siege and start hitting each blog with load. Siege is a nice tool that allows you to manipulate some key parameters. The ones I played with the most were the number of concurrent connections (-c) and the delay (-d in seconds) between batches of requests. Here is the command for your reference… siege -v -c 100 -i -t 10M -f urls.txt -d 1

The Results

In a word, staggering! I ran siege for 10 minutes with 100 concurrent connections and a 1 second delay between batches of web requests. The results are shown below…

Ghost Performance Under Heavy Load

Siege load test results for Ghost with Nginx under heavy load.

Wordpress Siege Results

Siege load test results for WordPress and Apache under heavy load.

As you can see from the output shown above, Ghost with Nginx outperformed WordPress with Apache by about 678% when looking at total transactional throughput over a 10 minute test. Impressively, the longest transaction response time for Ghost was 2.62 seconds compared the an abysmal 33.41 seconds for WordPress. I repeated these test runs multiple times and got very similar results so I am not going to show the rest of the test results since they are redundant. My goal here was not to run an exhaustive analysis of performance at varying loads but instead to create a substantial load and to see how each platform handled it.

Some other interesting data points to note. During the load test, Ghost ran with only 1 process and Nginx had a total of 2 processes. WordPress and Apache on the other hand spawned a total of ~110 httpd processes which makes sense since Siege was throwing 100 concurrent connections at it. The interesting part is in the CPU data during the load tests. I have plotted Average, Min, and Max CPU utilization on the charts below. You can clearly see that Ghost CPU consumption was about 40% while WordPress consumption was about 70%.

Ghost Heavy Load - AppDynamics

CPU consumption during Ghost load test showing Average, Min, and Max values.

Wordpress Heavy Load - AppDynamics

CPU consumption during WordPress load test showing Average, Min, and Max values.

Now don’t think that I have forgot about normal loading patterns. How do things look with a moderate load as compared to the super high load that I placed on these platforms with the 100 concurrent connections test? To find out I dropped the number of concurrent connections to 10 and set the delay between batches of connections to 5 seconds. The results are shown below and are still incredibly impressive for Ghost. WordPress was outperformed in every way possible. Ghost had higher throughput and most importantly the slowest transaction response time was .18 seconds compared to 2.72 seconds for WordPress. From a CPU perspective Ghost only consumed ~4% on average during this test while WordPress consumed ~30% on average.

Siege load test results for Ghost with Nginx under light load.

Siege load test results for Ghost with Nginx under light load.

Siege load test results for WordPress with Apache under light load.

Siege load test results for WordPress with Apache under light load.

Update on 10/18/2013 – It’s not fair!!! Apples and Oranges!!!

There have been some who say I’m comparing apples to oranges. To this I say, you’re damn right! In this post I set out to compare the common combinations of Nginx + Ghost and Apache + WordPress. I set out to compare these in their most basic forms, no tuning, no caching, just what you get out of the box. But I understand the outcry and I decided to level the playing field. Some people thought that Apache was a bottleneck so I decided to use Apache as the front end web server for Ghost and to re-run my load tests. I ran multiple tests again but they were all very consistent so I am only going to show the output from one of them (shown below).

Apache and Ghost Siege Results

Load test results of Apache + Ghost.

Load test results for Nginx + Ghost for easy comparison with the results above.

Load test results for Nginx + Ghost for easy comparison with the results above.

The results shown above are interesting. Apache + Ghost was actually slightly FASTER than running Ghost with Nginx. Ghost is still super fast regardless of using Apache or Nginx as the web server.

The Conclusion

Ghost is way faster and can handle way more load than WordPress while also consuming much less CPU resource (Ghost also has considerably less functionality than WordPress but that’s not relevant for the purpose of this test). It would be interesting to run Ghost in a 2 process Node.js cluster and see  what difference it makes in throughput and CPU utilization. Hmmmm, that sounds like a really good subject for another blog post…

Another interesting topic that I didn’t cover here is monitoring for both of these platforms. In my mind it’s not enough to just observe the behavior of these platforms from the outside. I want to see what’s going on from the inside. In a future blog post I am going to monitor Ghost with Nodetime and will monitor WordPress with AppDynamics. I can’t wait to see how they both look from the inside!

Update on 11/14/2013: Due to popular request I have performed more testing, this time with an opcode cache for PHP. You can read all about it in An Example of How Node.js is Faster Than PHP – Part 2

Appendix

Here are the links to the information I used to build out my blogging and testing platforms (I used the relevant portions of each article since my configuration was different than what was in each article alone):

How to install Node.js: https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager

How to install Ghost: http://docs.ghost.org/installation/

How to install MySQL: http://www.cyberciti.biz/faq/how-to-install-mysql-under-rhel/

How to install and configure NginX and use MySQL: http://0v.org/installing-ghost-on-ubuntu-nginx-and-mysql/#.Ul26n2RATL4

Where to find Siege: http://www.joedog.org/siege-home/

Where to find Sproxy: http://www.joedog.org/sproxy-home/

More good information on using Siege and Sproxy: http://www.euperia.com/linux/tools-and-utilities/speed-testing-your-website-with-siege-part-two/771