Test Your Web App on 500+ Browsers? Tips for Automated Smoke Tests Using Sauce Labs

It goes without saying the goal of adding functionality is to add a new feature without it disrupting everything else. It’s also nice when you fix a bug without it breaking the system. But, unfortunately most of us have encountered that awkward moment when a new check-in broke the functionality of our app. Don’t feel bad. It happens a lot.

Luckily we have regression tests to help us out, but in the business of dealing with browsers and end users, how do you deal with the complexity of over 500 combinations of operating system and browser varieties?

At AppDynamics, we use Sauce Labs and Selenium in our automated smoke tests to help do just that. Sauce Labs provides test automation across virtual machines of browsers and platforms. With Sauce Labs’ Platform Configurator, you first specify the API, device, OS, and browsers, then you’re a button click away from configuring the desired capabilities of your test.

Depending on your infrastructure and test environment, you can require the browser to have access to a server inside the firewall. Sauce Connect opens a channel between your local network and the remote browser in the testing environment in Sauce Labs, so the VM can access the test system’s server.

All the above is standard information, and you can find more details about using Sauce Labs from their Wiki. So let’s dig in and discuss some of the gotchas that we discovered using Sauce Labs, and how we deal with them.

Some challenges found when working with Sauce labs

1. Cache busting

Although Sauce Labs spins off pristine browsers for each test session, the browser can take a significant amount of time to start up. To amortize these costs, we reuse a single browser session for all the tests. Cached website data can skew the test results. If you want to get metrics such as page load time and DOM first byte ready, you need to make sure the resource under testing is not cached. When you instruct WebDriver to visit a URL, you need to consider the usage of some cache busting techniques, such as cache control headers and unique fragments. This way ensures a unique URL for each visit, and no confusing results will be generated.

This code is in Java with Selenium. We added UUID to the end of each URL request.

void loadURLInBrowser(String url, boolean reload) {

       UUID uuid = UUID.randomUUID();

       String hashString = "#" + uuid;

       browser.get(server.absoluteUrl(url  + hashString));

       server.beaconSaver.waitForExpectedEvents(beaconCount);

       if (reload) {

           browser.navigate().refresh();

       }

}

2. Unknown_ca errors when testing HTTPS

Browsers don’t send HTTPS request if they can’t find the right SSL Certificate. When the server uses a self-signed certificate (for the convenience of tests), the Sauce Labs browsers don’t trust this certificate. Instead, they modify the browser to trust their own certificate, which is installed in their VPN proxies. Newer versions of Safari and some other browsers are more secure and harder to modify, so they throw this error. Sauce Labs knows this issue, but hasn’t spent the time to figure out how to modify it. This creates problems, and you’ll see unknown_ca errors.

In our testing, we had this problem with:

  • Android 4.4 and 5.0
  • iOS 7.1 and later
  • Safari 8 and later
  • Edge

You have to avoid tests that depend on the Sauce Connect proxy for HTTPS on these browsers.

3. Clock synchronization issues between your local server and remote web browsers

Performance monitoring collects lots of timing metrics data, and we often need to validate a timestamp in our tests. However, the clock on the remote machine may not be synchronized with the clock on the local machine. The timestamps may differ significantly. When you design your tests, you need to keep the clock synchronization issue in mind. Likewise, network latency can be an issue. Requests and responses can arrive out of order, even when you try to add latency to responses. It’s best to skip tests that depend on the order of asynchronous requests.

4. Be sure to check a port’s availability

If your test requires a distinct web server to go with each test, then you need to make sure the server is running and has the port open before you send the address to the browser. When starting tests in parallel in threads, the server could choose a port and report it to the browser before the port is opened, and it could fail because another server could open that port first.

5. Problematic tunnel ports: 6000 and 6666

If your pages load from localhost, then to be compatible with all browsers you need to use the ports tunneled by the Sauce Connect proxy in the VM, in accordance with the FAQ at Sauce Labs (https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy+FAQS). However, we’ve empirically had issues with ports 6000 and 6666, so we removed those from our list. We also grouped the ports into pairs, so we could do HTTP/HTTPS tests.

public class SauceInfo {

   static final int[][] PORTS_TUNNELED_BY_SAUCE_CONNECT = {

           /* 6000 Safari doesn't want to connect to this, even when it's available (!?),*/

            /* 6666 Safari doesn't like ,*/

           {80, 443}, {2000, 2001}, {2020, 2109}, {2222, 2310}, {3000, 3001}, {3030, 3210}, {3333, 4000},

           {4001, 4040}, {4321, 4502}, {4503, 4567}, {5000, 5001}, {5050, 5432}, {6001, 6060}, {6543, 7000},

           {7070, 7774}, {7777, 8000}, {8001, 8003}, {8031, 8081}, {8765, 8777}, {8888, 9000}, {9001, 9080},

           {9090, 9876}, {9877, 9999}, {49221, 55001}

   };

}

Hope this article helps. Have fun playing with Sauce Labs in the smoke tests of your web apps.

 

The Benefits of Migrating from JavaScript to TypeScript

Recently, we moved our Browser RUM agent from JavaScript to TypeScript. In my last post, I focused on walking through the steps of migrating from JavaScript, the challenges, and best practices we uncovered along the way.

This one will focus on more details of the benefits and one missing feature in TypeScript compiler we suggest to implement.

TypeScript’s main benefits:

  1. Class and Module Support
  2. Static Type-checking
  3. ES6 Feature Support
  4. Clear Library API Definition
  5. Build-in Support for JavaScript Packaging
  6. Syntax Similarity to Our Backend Languages (Java, Scala)
  7. Superset of JavaScript

Class and Module Support

Keywords like class, interface, extends and module are available in TypeScript.
You can define a class as

// class define in TypeScript
class VirtualPageTracker extends Tracker {
    private virtualPageName: string = '';
    constructor(name) {
        super(name);
    }

    getName(): void {
        return this.virtualPageName;
    }

    static getTrackerName(): string {
        return  'VirtualPageTracker';
    }
}

TypeScript compiler will transcompile it to

var __extends = (this && this.__extends) || function (d, b) {
    for (var p in b) if (b.hasOwnProperty(p)) d

= b

; function __() { this.constructor = d; } d.prototype = b === null ? Object.create(b) : (__.prototype = b.prototype, new __()); }; // class define in TypeScript var VirtualPageTracker = (function (_super) { __extends(VirtualPageTracker, _super); function VirtualPageTracker(name) { _super.call(this, name); this.virtualPageName = ''; } VirtualPageTracker.prototype.getName = function () { return this.virtualPageName; }; VirtualPageTracker.getTrackerName = function () { return 'VirtualPageTracker'; }; return VirtualPageTracker; })(Tracker);

Static Type-checking

TypeScript compiler will check the type (to surface more typing errors at compiling time)

var name: string;
name = 2; // type error, assign a number to a string type variable

function foo(value: number) {}
foo(''); // type error, use a number as a string type parameter

interface Bar {
    setName: (name: string) => void;
    getName: () => string;
}


var bar: Bar = {
    getName: function() {
        return 'myName';
    }
} // type error, setName function is missing in the object assigned to bar.

A practical example is if we use the wrong data type in the browser beacon, we now get compiling errors. Before migrating to Typescript, they could only be found by testing against the back-end.

ECMAScript 6 Feature Support

It is the current version of the ECMAScript Language Specification with more language features.
With TypeScript, you can start using many ES6 features although it may not be supported in your target browser. TypeScript compile can compile the ts files into “ES3”, “ES5” or “ES6”.

Some of the features are very handy like:

// for..of loops
var arr = ['a', 'b', 'c'];
for (let item of arr) {
    console.log(item);
}

It’s compiled to

// for..of loops
var arr = ['a', 'b', 'c'];
for (var _i = 0; _i < arr.length; _i++) {
    var item = arr[_i];
    console.log(item);
}

Refer to TypeScript ES6 Compatibility Table for more ES6 features you can use.

Clear API Definition

To let other TypeScript libraries use your library, you need to create a .d.ts file to declare all your public types and APIs of your library.
These definition files turned out to be clear and accurate references of your public APIs since they are always maintained and update-to-date because
you always need them if you write your tests in TypeScript too.

Refer to https://github.com/borisyankov/DefinitelyTyped for TypeScript definition files created for large amounts of JavaScript libraries.

Build-in Support for JavaScript Packaging

You can define one main entry ts file and refer all the ts files you need in the output js file.

Compiling the main entry ts file with the –out option, the compiler will concatenate all the directly or indirectly referred ts files into one js file in the order they are referred.

Thus, we can easily tailor our library into multiple versions.
For example, with the same code base, we are able to generate specific versions of browser agents for desktop and mobile respectively.
We just need to create one entry file for each version with different ts files referred in it.

Syntax Similarity to Our Backend Languages (Java, Scala)

We use Java and Scala at the back-end.
TypeScript’s similarity to these languages allows our developers can switch between front-end and back-end programming more smoothly.
Refer http://www.slideshare.net/razvanc/quick-typescript-vs-scala-sample for a quick syntax comparison between TypeScript and Scala.

Superset of JavaScript

As a superset of JavaScript, TypeScript has a smooth learning curve for JavaScript developers.
This means  you can adopt TypeScript in your existing JavaScript projects quickly in an less disruptive way than CoffeeScript or ClojureScript.

One Missing Feature Suggested

In addition to the benefit, we also found some missing features could be implemented.
One of them is to merge the same module into the same function rather than multiple functions.

module A {
    function foo() { }
}

module A {
    function bar() {
        foo();
    }
}

generates below code with compiling error “cannot find name ‘foo’”.

var A;
(function(A) {
    function foo() {}
})(A || (A = {}));
var A;
(function(A) {
    function bar() {
        foo();
    }
})(A || (A = {}));

foo function defined within the first anonymous function call for module A is not visible in the second anonymous function call, so you have to export it as:

module A {
    export function foo() {}
}

module A {
    function bar() {
        foo();
    }
}

generates below code without error:

var A;
(function(A) {
    function foo() {}
    A.foo = foo;
})(A || (A = {}));
var A;
(function(A) {
    function bar() {
        A.foo();
    }
})(A || (A = {}));

The problem here is now A.foo is not only visible to module A. Anyone can call it and modify it now.

There is no module level visible concept which should be similar to Java’s “package-private” when there is no modifier for Java classes or members.

This could be solved by generating:

module A {
    export function foo() {}
}

module A {
    function bar() {
        foo();
    }
}

to

var A;
(function (A) {
    function foo() { }
    A.foo = foo;
})(A || (A = {}));
var A;
(function (A) {
    function bar() {
        A.foo();
    }
})(A || (A = {}));

The problem of merging into one function is a potential name conflict between the same module in two files. But the compiler can report error in this case, and if two people are working independently on the same module in two files, it would be better to create two different sub modules. Merging into one function could be a feasible way support module level visibility.

As I write this article, I notice the /* @internal */ annotation in the ts compiler source code; it’s an experimental option released with typescript 1.5.0-alpha to strip the declarations marked as @internal.

It helps to only include the declarations without @internal (which serves as your external APIs) when generating the .d.ts file from your code. And if your consumer are using TypeScript too, this prevents it from using your internal members.

Generating the .d.ts file for:

module A {
    /* @internal */ export function internal() {}
    export function external() {}
}

by

tsc -d --stripInternal A.ts

will output

declare module A {
    function external(): void;
}

However, if your consumers uses JavaScript, they can still use the internal function.

Conclusion

By and large, it’s a pleasant and rewarding experience to move to TypeScript. Though it adds limitations on your JavaScript implementation, you can either find a good workaround or implement the benefits that outweigh it. Moreover, it’s an active open source project (about 200 commits to master in last month) with well documentation to help you start easily. And just in March this year, Google also announced they would replace AtScript with TypeScript. Angular 2 is now built with TypeScript too. So far, the move to TypeScript has proved beneficial.