Test Your Web App on 500+ Browsers? Tips for Automated Smoke Tests Using Sauce Labs

It goes without saying the goal of adding functionality is to add a new feature without it disrupting everything else. It’s also nice when you fix a bug without it breaking the system. But, unfortunately most of us have encountered that awkward moment when a new check-in broke the functionality of our app. Don’t feel bad. It happens a lot.

Luckily we have regression tests to help us out, but in the business of dealing with browsers and end users, how do you deal with the complexity of over 500 combinations of operating system and browser varieties?

At AppDynamics, we use Sauce Labs and Selenium in our automated smoke tests to help do just that. Sauce Labs provides test automation across virtual machines of browsers and platforms. With Sauce Labs’ Platform Configurator, you first specify the API, device, OS, and browsers, then you’re a button click away from configuring the desired capabilities of your test.

Depending on your infrastructure and test environment, you can require the browser to have access to a server inside the firewall. Sauce Connect opens a channel between your local network and the remote browser in the testing environment in Sauce Labs, so the VM can access the test system’s server.

All the above is standard information, and you can find more details about using Sauce Labs from their Wiki. So let’s dig in and discuss some of the gotchas that we discovered using Sauce Labs, and how we deal with them.

Some challenges found when working with Sauce labs

1. Cache busting

Although Sauce Labs spins off pristine browsers for each test session, the browser can take a significant amount of time to start up. To amortize these costs, we reuse a single browser session for all the tests. Cached website data can skew the test results. If you want to get metrics such as page load time and DOM first byte ready, you need to make sure the resource under testing is not cached. When you instruct WebDriver to visit a URL, you need to consider the usage of some cache busting techniques, such as cache control headers and unique fragments. This way ensures a unique URL for each visit, and no confusing results will be generated.

This code is in Java with Selenium. We added UUID to the end of each URL request.

void loadURLInBrowser(String url, boolean reload) {

       UUID uuid = UUID.randomUUID();

       String hashString = "#" + uuid;

       browser.get(server.absoluteUrl(url  + hashString));

       server.beaconSaver.waitForExpectedEvents(beaconCount);

       if (reload) {

           browser.navigate().refresh();

       }

}

2. Unknown_ca errors when testing HTTPS

Browsers don’t send HTTPS request if they can’t find the right SSL Certificate. When the server uses a self-signed certificate (for the convenience of tests), the Sauce Labs browsers don’t trust this certificate. Instead, they modify the browser to trust their own certificate, which is installed in their VPN proxies. Newer versions of Safari and some other browsers are more secure and harder to modify, so they throw this error. Sauce Labs knows this issue, but hasn’t spent the time to figure out how to modify it. This creates problems, and you’ll see unknown_ca errors.

In our testing, we had this problem with:

  • Android 4.4 and 5.0
  • iOS 7.1 and later
  • Safari 8 and later
  • Edge

You have to avoid tests that depend on the Sauce Connect proxy for HTTPS on these browsers.

3. Clock synchronization issues between your local server and remote web browsers

Performance monitoring collects lots of timing metrics data, and we often need to validate a timestamp in our tests. However, the clock on the remote machine may not be synchronized with the clock on the local machine. The timestamps may differ significantly. When you design your tests, you need to keep the clock synchronization issue in mind. Likewise, network latency can be an issue. Requests and responses can arrive out of order, even when you try to add latency to responses. It’s best to skip tests that depend on the order of asynchronous requests.

4. Be sure to check a port’s availability

If your test requires a distinct web server to go with each test, then you need to make sure the server is running and has the port open before you send the address to the browser. When starting tests in parallel in threads, the server could choose a port and report it to the browser before the port is opened, and it could fail because another server could open that port first.

5. Problematic tunnel ports: 6000 and 6666

If your pages load from localhost, then to be compatible with all browsers you need to use the ports tunneled by the Sauce Connect proxy in the VM, in accordance with the FAQ at Sauce Labs (https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy+FAQS). However, we’ve empirically had issues with ports 6000 and 6666, so we removed those from our list. We also grouped the ports into pairs, so we could do HTTP/HTTPS tests.

public class SauceInfo {

   static final int[][] PORTS_TUNNELED_BY_SAUCE_CONNECT = {

           /* 6000 Safari doesn't want to connect to this, even when it's available (!?),*/

            /* 6666 Safari doesn't like ,*/

           {80, 443}, {2000, 2001}, {2020, 2109}, {2222, 2310}, {3000, 3001}, {3030, 3210}, {3333, 4000},

           {4001, 4040}, {4321, 4502}, {4503, 4567}, {5000, 5001}, {5050, 5432}, {6001, 6060}, {6543, 7000},

           {7070, 7774}, {7777, 8000}, {8001, 8003}, {8031, 8081}, {8765, 8777}, {8888, 9000}, {9001, 9080},

           {9090, 9876}, {9877, 9999}, {49221, 55001}

   };

}

Hope this article helps. Have fun playing with Sauce Labs in the smoke tests of your web apps.

 

The Implications of Google’s Mobilegeddon

Google, with one fell swoop, changed their search algorithm and changing the lives (and livelihoods) of billions. Enter Mobilegeddon.

With smartphones becoming as ubiquitous as ever, people are increasingly using them for tasks that they used to do on their PCs, like search. And just as in earlier generations, whole countries skipped widespread landline networks to go directly to wireless feature phones, now whole generations of consumers around the world are entirely skipping PCs and directly accessing the internet from their mobile phones (whether smart or feature).

According to Akamai’s recent State of the Internet report monthly mobile data usage continued to grow exponentially (outstripping voice data by several orders of magnitude) with a growth rate of 10% just between the first and second quarters of 2014. Of that traffic, Apple Mobile Safari constituted 35.9% of requests and Android Webkit trended toward 32.6% of requests.

Now Google is clearly the leader in desktop search with an estimated 75% of PC search market share in January of 2015. Similarly, Google makes a lot of money on its search engine business, an estimated $44.5 billion.

As more and more of global search is going to be done on smartphones vs. PCs, Google has a vested interest to make sure that it provides an excellent mobile search experience.

The problem is that many websites were primarily designed for PCs. As good as smartphones and tablets have become, they are still very constrained computing devices compared to very affordable PCs which, in many cases now, can be had at price points not too dissimilar from top of the line smartphones, yet still have way more computing power and ergonomics that are much better suited to the design, development, deployment, and maintenance of website.

Unfortunately, a website that was initially designed on and probably intended for viewing on a PC will probably not work or perform as well on smartphone which has a much smaller screen and usually also has a much slower data connection via a wireless network. Consequently, all those big fancy graphics and cool animations, transitions, and effects that you put on your site will only wind up making a mobile user wait forever or may not work at all on mobile devices.

As a result of all this Google has decided to change its search engine algorithm in or to rank mobile-friendly website higher. Websites that aren’t mobile-friendly are going to get demoted in the search engine rankings.

Although Google announced this decision back in February, it’s going to catch a lot of people by surprise when all of a sudden they start seeing that their search rankings are dropping like a stone. A lot of businesses are highly dependent on their visibility in searches, and this has a direct impact on their bottom lines

Google understands at a visceral level that user experience is as important if not more important than relevancy. Several years ago, Google teamed with Microsoft to do a study on how performance affects user behavior and they found that a slow down of as little as 500ms can result in a decrease of 1% in revenue, and worse yet, that reduction will continue even after the delay is removed.

Consequently, Google is making these changes in order to reward business that provide not only relevant data but will also provide a good user experience when the customer clicks on a link from Google’s search results.

So what does all this mean for businesses? Well, unless you have tools to understand exactly how your website is performing for all users, on all devices, on any browser and operating system, it could mean a significant drop in your business whether it’s walk-in traffic to a local coffee shop are a major e-commerce retail site, bank, insurance company, government agency, or whatever else.

But don’t panic, AppDynamics has you covered with our Browser Real User Monitoring technology, which gives you real-time end-to-end visibility into every browser request, from any location, on any device, and shows you exactly how your website is performing and all of the factors that affect its performance from web page design (resource loading), third party content and APIs, to back-end systems from servers to databases to network infrastructure, we will identify the cause of the slowdown and show it to you in an easy to ready waterfall diagram so you can pinpoint the culprit, fix the problem, improve the speed of your page loads, get higher ranking, and drive your business where you want it to go.

Want to check out Browser RUM for yourself? Download a FREE trial today!