After writing an article on Selenium testing with Ruby, JS, and Python I needed to make it available somewhere, and decided to host it from my own site. But that meant adding new functionality, and surely I’d want to test that functionality given the topic I’m writing about!

However my site’s backend is JavaScript and I wasn’t so fond of the JS framework for testing, so what to do?

Well, with great timing Kevin Lamping happened to tweet about how nice WebdriverIO is, so I went to give it a try since I had passed it over in my previous article.

I ended up using WebdriverIO as part of my fully automated deployment process, by combining the continuous integration service Travis and multi-browser testing service Sauce Labs. With this article we’ll go through the details of setting it all up.

Not so fast

I did actually try WebdriverIO for my previous article, but it threw errors during npm install which left a bad taste so I gave it a pass. That was a shame though, because it turns out WebdriverIO is a great framework.

What actually went wrong is that a dependency of webdriverio needs to be compiled if you’re on Node 7, and it can’t be compiled with Python 3 which happens to be my system default. In hindsight it’s a simple fix (activate Python 2.7 for this project’s folder via pyenv), but it took a bit of work to figure out that this was the issue.

Travis fails to install the library for the same reason, and configuring Travis to build the library proved difficult. I actually gave up on this part and instead specified Node 6 for Travis’ configuration… I’m unlikely to hit critical version differences and hopefully no-one gets hurt from this subpar solution :)

WebdriverIO

Thankfully once it’s installed it works great. WebdriverIO has a nice predictable DSL that results in simple and very readable page objects and tests:

That’s all it took to get going with local E2E tests, really quite painless.

So, that’s the hard part done, right?

Not quite.

Travis & Sauce Labs

UI tests need to run across multiple real browsers, because only by running on browsers really used by customers can we know that things actually work. But I don’t want to install and maintain a mess of browsers myself, so we turn to services like Sauce Labs.

It took me a whole day getting all this working but hopefully this article will get you going quickly. There are basically three critical pieces of configuration to be aware of for success:

  • Correctly specify which browsers Sauce Labs should run
  • Ensure Sauce Labs browsers can see your site
  • Configure WebdriverIO to use Sauce Labs

First the basics though: Sauce Labs’ credentials must be added to Travis, this is done via Travis’ encrypted environment variables system. Then we must host the site from Travis’ VM (in my case that means running my site generator’s server), and set Travis to use “Sauce Connect” (a secure tunnel that lets Sauce Labs visit locally hosted sites).

These are pretty simple steps to set up, you can see travis.yml for these settings.

Browsers

Next we need to specify which browsers Sauce Labs should use to run our tests. WebdriverIO specifies a browser like this:

1
2
3
4
5
6
7
8
9
10
11
{
"safari-os-x-10.11-latest": {
"base": "SauceLabs",
"browserName": "safari",
"name": "ui-safari-os-x-10.11-latest",
"platform": "OS X 10.11",
"version": "latest",
"tunnel-identifier": [process.env.TRAVIS_JOB_NUMBER],
"build": [process.env.TRAVIS_BUILD_NUMBER]
}
}

But note how the key and name is duplicated and is a concatenation of the other data, and base and some other fields are just static data. It quickly becomes an inflexible mess to specify multiple browsers like this, it’s more maintainable to DRY it up by generating the data WebdriverIO expects. It’s not so important how that’s done as long as we boil down our browser data to just the relevant bits. Here’s the minimum configuration data I settled on:

(You can see the full file for how I generate the entire data structure)

Enable Sauce Labs

Then WebdriverIO needs to be configured to use Sauce Labs. This boils down to configuring wdio.conf.js, but doing so without requiring Sauce Labs because then we’d lose the ability to run the tests locally.

It comes down to changing a handful of variables depending on if Sauce Labs is present or not. Here’s how I did it:

With all that in place Travis now runs the tests per pull-request, pretty sweet!

Pull-request 41 with automated tests passing

Stability

At this point I recommend investing some time to be certain the tests are stable. I made sure to work on a number of small incremental features, each going into open pull-requests that I updated frequently. This was to stress-test the UI tests against flakiness, because having flaky tests is almost worse than no tests at all.

I also purposefully started with completely simple tests, that way I could stress-test the process itself instead of having to debug the tests. Thankfully the setup runs very stable, the tests basically always run correctly.

Wait, “basically”?

Yeah…

One test failed randomly

That one red entry is where Microsoft Edge failed to start for no apparent reason, and the only action I took was to manually re-invoke the Travis job. That made everything pass just fine. There are unfortunately so many moving parts glued together here that invariably something fails. Maybe a setting can be tweaked to help against this, but for now it happens so rarely it’s not a real impediment.

It sure is annoying though!

In the end

So what’s the point of all this effort?

Well first of all it’s just fantastic seeing a pull-request marked green knowing the site has just been tested on multiple real browsers. It gives confidence that the code coming in isn’t totally ruining the product.

Another huge benefit is that we can visually inspect how the tests ran, Sauce Labs captures screenshots of all the states and we can play them back step by step.

Sauce Labs testing a page

It’s very powerful to flip through the various screenshots to quickly do visual inspection.

As a side bonus we also get this nice badge to proudly display:

saucelabs-badge.svg
SauceLabsBadge of browsers passing our tests

Conclusion

Ultimately WebdriverIO and Sauce Labs are pleasant to work with, despite taking a whole day to set up. At least once it all works it works very nicely.

It’s a huge joy to have this completely automated continuous integration/deployment process, and it’s already caught a couple dumb mistakes I was about to merge to master.

Special thanks to Travis and Sauce Labs for providing free tiers of their services, it’s frankly unbelievable to have free access to professional-grade CI and cross-platform browser testing.

If you have any comments or questions I’d love to hear them, and all the code is freely available if you’d like to play around yourself.