Performance testing tools explained: The client side

In my last post I showed different approaches for load testing the server side. In this post I will highlight some tools I use to monitor the performance of the client side.

In modern Javascript-intensive web applications, users spend more time waiting on client-side rendering than the server-side processing. The reality is that as much effort that goes into optimizing the server-side, more effort should be made to optimize the client-side. End user monitoring has never been so important.

Why does performance matter?

Just to recap a few statistics about the business impact of performance at major internet companies:

Performance Impact

As you iterate through the software development cycle it is important to measure application performance on both the server side and the client side and understand the impact of every release. Here are some tools you can use to test the performance of the entire end user experience:

Google PageSpeed Insights

Google PageSpeed Insights provides actionable advice for improving the performance of client-side web applications. PageSpeed Insights analyzes the content of a web page, then generates suggestions to make that page faster. Reducing page load times can reduce bounce rates and increase conversion rates. The service is available as part of the Google Chrome Developer Tools as an extension, a web service, and extensions for Apache and Nginx.

Use the Google PageSpeed Insight API to integrate client-side optimizations into your continuous integration setup.

curl "https://www.googleapis.com/pagespeedonline/v1/runPagespeed?url=http://dustinwhittle.com/&key=xxx"

WBench

WBench is a tool that uses the HTML5 navigation timing API to benchmark end user load times for websites.

1) Install WBench:

gem install wbench

2) Run WBench:

wbench http://dustinwhittle.com/

WebPageTest.org

WebPageTest.org enables anyone to test the client-side performance on a range of browsers from anywhere in the world for free. This service is great and worth every penny I didn’t pay for it. Not only does it provide a range of mobile/desktop browsers and locations, but it also shows a waterfall timeline and a video of the rendering.

Screen Shot 2013-10-23 at 3.52.50 PM

AppDynamics

With AppDynamics Pro you get in-depth performance metrics to evaluate the scalability and performance of your application. Use the AppDynamics Pro Metrics Browser to track end user experience times and errors over the duration of the load tests:

With AppDynamics Pro End User Experience Dashboard you get visibility into both the server-side and the client-side:

Use AppDynamics Pro to compare multiple application releases to see the change in performance and stability:

release

In my next post in this series I will cover load testing tools for native mobile applications. Get started with AppDynamics Pro today for in-depth application performance management.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Performance testing tools explained: the server side

The performance of your application affects your business more than you might think. Top engineering organizations think of performance not as a nice-to-have, but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately, most engineering teams do not regularly test the performance and scalability of their infrastructure. In my last post on performance testing I highlighted a long list of tools that can be used for load testing. In this post we will walk through three performance testing tools: Siege, MultiMechanize, and Bees with Machine Guns. I will show simple examples to get started performance testing your web applications regardless of the language.

Why performance matters?

A few statistics about the business impact of performance at major internet companies:

Performance Impact

As you iterate through the software development cycle it is important to measure application performance and understand the impact of every release. As your production infrastructure evolves you should also track the impact of package and operating system upgrades. Here are some tools you can use to load test your production applications:

Apache Bench

Apache bench is simple tool for load testing applications provided by default with the Apache httpd server. A simple example to load test example.com with 10 concurrent users for 10 seconds.

Install Apache Bench:

apt-get install apache2-utils

Apache bench a web server with 10 conncurrent connections for 10 seconds:

ab -c 10 -t 10 -k http://example.com/

Benchmarking example.com (be patient)
Finished 286 requests

Server Software:        nginx
Server Hostname:        example.com
Server Port:            80

Document Path:          /
Document Length:        6642 bytes

Concurrency Level:      10
Time taken for tests:   10.042 seconds
Complete requests:      286
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      2080364 bytes
HTML transferred:       1899612 bytes
Requests per second:    28.48 [#/sec] (mean)
Time per request:       351.133 [ms] (mean)
Time per request:       35.113 [ms] (mean, across all concurrent requests)
Transfer rate:          202.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        9   20  14.2     17     148
Processing:   117  325  47.8    323     574
Waiting:      112  317  46.3    314     561
Total:        140  346  46.0    341     589

Percentage of the requests served within a certain time (ms)
  50%    341
  66%    356
  75%    366
  80%    372
  90%    388
  95%    408
  98%    463
  99%    507
 100%    589 (longest request)

 

Siege + Sproxy

Personally, I prefer Siege to Apache Bench for simple load testing as it is a bit more flexible.

Install Siege:

apt-get install siege

Siege a web server with 10 conncurrent connections for 10 seconds:

siege -c 10 -b -t 10S http://example.com/

** SIEGE 2.72
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:              263 hits
Availability:              100.00 %
Elapsed time:              9.36 secs
Data transferred:          0.35 MB
Response time:             0.35 secs
Transaction rate:          28.10 trans/sec
Throughput:                0.04 MB/sec
Concurrency:               9.82
Successful transactions:   263
Failed transactions:       0
Longest transaction:       0.54
Shortest transaction:      0.19

 

More often than not you want to load test an entire site and not just a single endpoint. A common approach is to crawl the entire application to discover all urls and then load test a sample of those urls. The makers of Siege also make sproxy which in conjunction with wget enables you to crawl an entire site through a proxy and record all of the urls accessed. It makes for an easy way to compile a list of every possible url in your appplication.

 

1) Enable sproxy and specify to output all the urls to a file urls.txt:

sproxy -o ./urls.txt

2) Use wget with sproxy to crawl all the urls of example.com:

wget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -e robots=on -e "http_proxy = http://127.0.0.1:9001" "http://example.com/"

3) Sort and de-duplicate the list of urls from our application:

sort -u -o urls.txt urls.txt

4) Siege the list of urls with 100 concurrent users for 3 minutes:

siege -v -c 100 -i -t 3M -f urls.txt

** SIEGE 2.72
** Preparing 100 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:              263- hits
Availability:              100.00 %
Elapsed time:              90.36 secs
Data transferred:          3.51 MB
Response time:             0.35 secs
Transaction rate:          88.10 trans/sec
Throughput:                0.28 MB/sec
Concurrency:               9.82
Successful transactions:   2630
Failed transactions:       0
Longest transaction:       0.54
Shortest transaction:      0.19

 

Multi-Mechanize

When testing web applications sometimes you need to write test scripts that simulate virtual user activity against a site/service/api. Multi-Mechanize is an open source framework for performance and load testing. It runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. Multi-Mechanize is most commonly used for web performance and scalability testing, but can be used to generate workload against any remote API accessible from Python. Test output reports are saved as HTML or JMeter-compatible XML.

 

1) Install Multi-Mechanize:

pip install multi-mechanize

2) Bootstrapping a new multi mechanize project is easy:

multimech-newproject demo

import mechanize
import time

class Transaction(object):
    def run(self):
        br = mechanize.Browser()
        br.set_handle_robots(False)

        start_timer = time.time()
        resp = br.open('http://www.example.com/')
        resp.read()
        latency = time.time() - start_timer

        self.custom_timers['homepage'] = latency

        assert (resp.code == 200)
        assert ('Example' in resp.get_data())

3) Run the multi-mechanize project and review the outputted reports

multimech-run demo

multimech

 

Bees with Machine Guns

In the real world you need to test your production infrastructure with realistic traffic. In order to generate the amount of load that realistically represents production, you need to use more than one machine. The Chicago Tribune has invested in helping the world solve this problem by creating Bees with Machine Guns. Not only does it have an epic name, but it is also incredibly useful for load testing using many cloud instances via Amazon Web Services. Bees with Machine Guns is a utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).

1) Install Bees with Machine Guns:

pip install beeswithmachineguns

 

2) Configure Amazon Web Services credentials in ~/.boto:

[Credentials]

aws_access_key_id=xxx
aws_secret_access_key=xxx

[Boto]

ec2_region_name = us-west-2
ec2_region_endpoint = ec2.us-west-2.amazonaws.com

 

3) Create 2 EC2 instances using the default security group in the us-west-2b availabily zone using the ami-bc05898c image and login using the ec2-user user name.

bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k aws-us-west-2 -l ec2-user

Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine guns...
.
.
.
.
Bee i-3828400c is ready for the attack.
Bee i-3928400d is ready for the attack.
The swarm has assembled 2 bees.

 

4) Check if the ec2 instances are ready for battle

bees report

Read 2 bees from the roster.
Bee i-3828400c: running @ 54.212.22.176
Bee i-3928400d: running @ 50.112.6.191

 

5) Attack a url if the ec2 instances are ready for battle

bees attack -n 100000 -c 1000 -u http://example.com/

Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 50000 rounds, 125 at a time.
Stinging URL so it will be cached for the attack.
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 1 is out of ammo.
Bee 0 is out of ammo.
Offensive complete.
     Complete requests:   100000
     Requests per second: 1067.110000 [#/sec] (mean)
     Time per request:    278.348000 [ms] (mean)
     50% response time:   47.500000 [ms] (mean)
     90% response time:   114.000000 [ms] (mean)
Mission Assessment: Target crushed bee offensive.
The swarm is awaiting new orders.

6) Spin down all the EC2 instances

bees down

 

AppDynamics

With AppDynamics Pro you get in-depth performance metrics to evaluate the scalability and performance of your application. Use the AppDynamics Pro Metrics Browser to track key response times and errors over the duration of the load tests:

metrics browser

Use the AppDynamics Scalability Analysis Report to evaluate the performance of your application against load tests:

Screen Shot 2013-10-15 at 5.44.52 PM

Use AppDynamics Pro to compare multiple application releases to see the change in performance and stability:

release

Get started with AppDynamics Pro today for in-depth application performance management.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

Tools of the trade for performance and load testing

Your application is fast and scalable, right? How do you know? How often do you run performance or load tests? In this post I will give an overview of the tools of the trade for performance and load testing web applications.

Open-Source performance testing tools

These tools allow you to load test your application for free. My preferred tool is Bees with Machine Guns — not just because of the epic name, but primarily because it uses Amazon’s EC2 to generate high levels of concurrency with ease.

  • Bees with Machine Guns – A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).
  • MultiMechanize – Multi-Mechanize is an open source framework for performance and load testing. It runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. Multi-Mechanize is most commonly used for web performance and scalability testing, but can be used to generate workload against any remote API accessible from Python.
  • Siege – Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”
  • HttpPerf – Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus of httperf is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httpperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility to new workload generators and performance measurements.
  • Apache Bench – AB is a tool for benchmarking your Apache HTTP server. It is designed to give you an impression of how Apache performs.
  • JMeter – Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, databases and queries, FTP servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

Performance testing tools as a service

Through these services you can build, execute, and analyze performance tests.

  • Apica Load Test – Cloud-based load testing for web and mobile applications
  • Blitz.io – Blitz allows you to continuously monitor your app 24×7 from around the world. You can emulate a single user or hundreds of users all day, everyday and be notified immediately if anything goes wrong.
  • Soasta – Build, execute, and analyze performance tests on a single, powerful, intuitive platform.
  • Blazemeter – BlazeMeter is a self- service performance & load testing cloud, 100% JMeter-compatible. Easily run tests of 30k, 50k, 80k or more concurrent users, on demand.

Performance testing on the client side

The best place to get started is at Google with the Web Performance best practices.

  • Google PageSpeed Insights – PageSpeed Insights analyzes the content of a web page, then generates suggestions to make that page faster. Reducing page load times can reduce bounce rates and increase conversion rates.
  • Google ngx_pagespeed – ngx_pagespeed speeds up your site and reduces page load time. This open-source nginx server module automatically applies web performance best practices to pages, and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.
  • Google mod_pagespeed – mod_pagespeed speeds up your site and reduces page load time. This open-source Apache HTTP server module automatically applies web performance best practices to pages, and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.

Web acceleration services

Through a simple DNS change, your website’s traffic is routed through these services and your content is optimized and cached globally for better performance. This is an easy way to improve performance with minimum efforts.

  • Yottaa – All-in-one web optimization solution delivers speed, scale, security and actionable insight for any website.
  • Cloudflare – Offers free and commercial, cloud-based services to help secure and accelerate websites.
  • Torbit – Torbit helps you accurately measure your website’s performance and quantify how speed impacts your revenue.
  • Incapsula – Incapsula offers state-of-the-art security and performance to websites of all sizes.

Recommended reading

A great book to learn about capacity planning and preparing for production traffic is The Art of Capacity Planning by John Allspaw. John Allspaw wrote the book from his experience scaling Flickr.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.