pytest-mozwebqa 1.1 released

It’s been a long time coming, but pytest-mozwebqa 1.1 has finally been released! The main feature of this new version is the ability to specify a proxy server for the browsers launched. It will also use this in conjunction with upcoming plugins pytest-browsermob-proxy (to record and report network traffic) and pytest-zap (to spider and scan for known security vulnerabilities). Check out the complete changelog for 1.1.

FlynnID 0.3

I’ve just released an update to FlynnID. The primary change is that I’ve reintroduced the command line arguments, meaning a single node can be registered without the need for a configuration file. I’ve also hopefully learned my lesson, and in the words of a friend ‘Deprecate, not annihilate!’

Along with this change, 0.3 also introduces a handy feature if you’re running FlynnID on a schedule. If the node you’re registering is already registered then it won’t attempt to register it again. You can override this behavior using the --force command line option.

Lastly, the output is now much more colourful…

You can install/upgrade using pip install -U flynnid.

Mozilla drops usage of Selenium RC

I thought this was important enough to share in a short blog post… Just 10 months ago, Mozilla started to migrate their Selenium projects from the Selenium RC API to the WebDriver API. I’m thrilled to say that this is now complete, and that no Selenium RC projects are actively being run or maintained!

BIDPOM drops support for Selenium RC

As all of the active Mozilla Web QA automation projects are now using WebDriver, there is no longer a need for BIDPOM (Browser ID Page Object Model) to support Selenium RC. I considered keeping this support purely for the community, however I would rather encourage anyone still using Selenium RC to upgrade to WebDriver.

If you require Selenium RC support then I recommend you fork the repository and continue to develop the RC page objects separately. The only difference you will notice if you’re upgrading to the latest version of BIDPOM is that you may need to import from pages now, rather than pages.webdriver.

FlynnID 0.2

In Tron, Flynn’s identity disc is the master key to getting onto the Grid. In the far less exciting real world, FlynnID is the key to registering a Selenium node to Selenium Grid. Yesterday I released FlynnID 0.2, which changes the usage from a list of optional arguments to a single expected argument: a configuration file. This means you can now register several nodes in one go. Below is an example configuration file.

Of course this does unfortunately mean that anyone upgrading from 0.1 may be a little surprised that the command line options have gone, but I strongly feel this is a better approach. This way, your configuration file can be backed up (or added to version control), and it’s much quicker to run. You can install/upgrade FlynnID using pip: pip install -U flynnid.

Announcing pytest-mozwebqa 1.0

Finally I can announce that I have released version 1.0 of the pytest plugin used by Mozilla’s Web QA team! It’s been in use for several months now, but I’ve paid off some long standing technical debt, and now consider it stable!

I should say that although this plugin was primarily developed for Mozilla’s Web QA team, anyone that wants to write test automation for websites in Python can take advantage of it. There’s very little that is specific to Mozilla, and this can easily be overridden on the command line, or you could simply fork the project and create your own version of the plugin. Anyway, as I haven’t previously announced the plugin, it’s probably a good idea for me to explain what it actually does…

Selenium integration
The primary feature of the plugin is the ability for it to launch and interact with a web browser. It does this by integrating with either the RC or WebDriver APIs provided by the Selenium browser automation framework. The browser is launched ahead of every test, unless the test is specifically decorated to indicate that the test does not require a browser:

The plugin works with a local WebDriver instance, with a remote server (RC or WebDriver), and with Selenium Grid (also RC or WebDriver).

Sauce Labs integration

You can also use this plugin to run your tests in the cloud using your Sauce Labs account. The integration allows you to specify a build identifier and tags, which help when filtering the Sauce Labs jobs. To enable Sauce Labs integration, you simply need to specify a credentials file on the command line, which is a YAML file in the following format:

When tests are run using Sauce Labs, there will also be additional items in the HTML report, such as the video of the test and a link to the job.

If you don’t have a Sauce Labs account already, you can sign up for one here. Sauce Labs is used by Mozilla whenever we need to run on browser/platform combinations that our own Selenium Grid doesn’t support, or whenever we need boost up the number of testruns such as before a big deployment.

The plugin allows you to store your application’s credentials in a YAML file as specified on the command line. This is an important feature for Mozilla, where the credentials files are stored in a private repository. Anyone wanting to contribute or run the tests themselves, simply has to create an account and a YAML file.

Fail fast
Have you ever been frustrated when you’ve kicked off your suite of several hundred tests, just for every single one of them to launch a browser despite the application under test being unavailable? It’s happened to me enough times that I added an initial check to ensure the base URL of the application is responding with a 200 OK; This saves so much time.

Protect sensitive sites
I’ll leave the debate on whether you should be running your tests against sensitive environments such as production for someone else, but if you do decide to do this, the plugin gives you a little bit of extra protection. For a start, all tests are considered destructive and therefore will not run by default. You can explicitly mark tests as non-destructive. Having an opt-in system is more maintenance (I know it’s a pain), but much lower risk. I’d rather accidentally not be running a non-destructive test against production than accidentally run a destructive one, and I have felt this pain before!

Of course, for some environments, you will want to run your destructive tests, and you can do so by specifying the --destructive command line option.

There’s also a final safety net just in case you try running destructive tests against a sensitive environment. This skips any destructive tests that are run against a URL that matches a regular expression. For Mozilla, this defaults to mozilla\.(com|org) and any skipped tests will give a suitable reason in your reports.

HTML report

Digging through console logs or JUnit reports can be a little frustrating when investigating failures, so the plugin provides a nicely formatted HTML report. This shows the options used when running the tests, a short summary of the results, and then lists each test along with timings and additional resources where appropriate.

I call the additional resources the test’s “death rattle” as it’s only captured for failing tests (it’s not particularly useful for passing tests, and consumes unnecessary resources). For tests that use Selenium this should at least include a screenshot, the HTML, and the current URL when the failure occurred. If you’re running in Sauce Labs then you should also see the video of the test and a link to the test job.

For full details and documentation of the plugin, take a look over the project’s README file on github. If you find any bugs or have any feature requests please raise them in github issues.

Automating BrowserID with Selenium

BrowserID is an awesome new approach to handling online identity. If you haven’t heard of it then I highly recommend reading this article, which explains what it is and how it works. Several Mozilla projects have already integrated with BrowserID, including Mozillians, Affiliates, and the Mozilla Developer Network.

With all of these sites now integrating with BrowserID (and more on their way) we needed to add support to our test automation to handle the new sign in process. Initially we started to do this independently in our projects, but the thought of updating all of our projects whenever a tweak was made to BrowserID was daunting to say the least! For this reason I have created a project that contains a page object model for BrowserID. This can be included in other projects as a submodule and then updated and maintained centrally.

The new project is called ‘BIDPOM’ (BrowserID Page Object Model) and can be found here. It currently only contains a page object for the Sign In page, however this currently meets the needs of the automation for projects that have integrated with BrowserID. As we have a mix of projects using Selenium’s two APIs (RC and WebDriver), it was necessary for BIDPOM to support both.

By adding BIDPOM as a submodule, we can easily pull the BrowserID page objects into our automation projects and reference them in a very similar way to the main project’s page objects. We can also update the version of BIDPOM simply by updating the git link and updating the submodule. What’s even better is that our continuous test builds running in Jenkins automatically initialise and update the submodule for us!

I hope that in addition to being a dependency for our own automation projects, this page object model can be utilised by others wanting to create or maintain automated tests using Selenium against sites that adopt BrowserID. If you would like to start using BIDPOM then I have provided below a guide to adding the project as a submodule to an existing git repository.

From within your project, add the BIDPOM project as a git submodule:

This will add an entry to .gitmodules and clone the BIDPOM project to the browserid subdirectory. It will also stage the new gitlink and .gitmodules items for commit.

You can now commit these changes to your project’s repository:

Before you can test the new submodule you will need to run the following command to copy the contents of .gitmodules into your .git/config file.

Now you can test the submodule by deleting the browserid directory and allowing it to be recreated:

The BIDPOM project should be cloned to the browserid directory.

You will now be able to integrate your project with BrowserID! Here follow a few examples of how to integrate your project.

Example: Short sign-in using Selenium’s RC API

Example: Long sign-in using Selenium’s RC API

Example: Short sign-in using Selenium’s WebDriver API

Example: Long sign-in using Selenium’s WebDriver API

For the latest documentation on the BIDPOM project refer to the github wiki.

Hooking Android up to the (Selenium) Grid

When I set myself a Q4 goal of getting a small suite of Mozilla’s WebQA tests running on Android I didn’t think it would be much work. The AndroidDriver has been around for some time, and from what I understood it was pretty mature – I had even run a couple of tests locally against it and they worked well. What I found was there were a couple of important gotchas…

The first was that the port forwarding that’s necessary to run tests on an Android emulator or device binds to localhost. This isn’t a problem so long as the test commands originate from the same machine as the emulator/device is attached to, however that doesn’t make it very useful for Selenium Grid, where we want to run several emulators/devices remotely.

I was suprised that this issue had not come up before… I didn’t think I could possibly be the first person to want to hook up an AndroidDriver to Selenium Grid..! After searching around for a solution I decided to leave things for a while to take care of other priorities. I was then pulled back into the problem when my fellow Mozillian Raymond Etornam came across the same issue and was looking for a solution.

Raymond and I approached Dounia Berrada, who heads up the AndroidDriver efforts at Google, and as a result of this discussion, Raymond raised an issue in the Selenium project. Dounia got to the bottom of the issue and found a very simple solution, using socat to listen on another port on all interfaces and forward traffic to the localhost bound port forward configured by ADB. The Selenium project’s wiki has been updated with the solution.

The second issue I had was registering the AndroidDriver with Selenium Grid. This is simply a case of posting the correct JSON to the server, and is something that’s done for you when launching a Selenium server. The quick solution would have been to just start a Selenium server with the details for the Android node. This feels like too much of a hack, plus I didn’t really want the server running constantly when it’s not really doing anything other than consuming resources.

I decided to write a very simple Python package that would take a few arguments, construct the necessary JSON and post it to the Selenium Grid. I’ve released this as FlynnID (there’s a reference to ‘The Grid’ if you look hard enough) and it’s available on PyPI, with the source code on github.

A more long term solution will be to have the AndroidDriver register with a Selenium Grid when it’s started. If anyone is keen to look into this please get in touch!

So with these issues resolved, I was able to finally put together the first mobile test suite. I migrated the Input tests to WebDriver, did a lot of cleaning up, and split them into desktop/mobile tests. There’s only one mobile test there at the moment, but now we have everything in place we can go full speed into test development! If you’re interested in helping us out to write more tests or improve our frameworks/infrastructure then the WebQA page on QMO is a great place to start.

Oh, I’ve also added a cute Android icon  to Selenium Grid, which will be available from version 2.16.0.

Adding Mozmill tests to the Selenium IDE build system

Back in April I blogged about the Mozmill tests I’d written to test Selenium IDE. I followed up in June with a blog post covering how to run these tests. The natural progression is to add these tests into the existing Selenium IDE build environment, which is run using Atlassian’s continuous integration server, Bamboo.

At Mozilla, we want to run the Mozmill tests for the released versions of add-ons against the latest builds of Firefox. This is to determine any regressions in Firefox that will potentially cause issues for the add-on authors. It also gives the add-on authors an early warning if there is a potential compatibility issue with an upcoming release of Firefox. We currently run these tests on a schedule, but will soon be looking to move to a continuous integration solution ourselves. You can see the results of these daily tests on our dashboard.

The add-on authors (Selenium in this case) are typically more interested to know if the latest build of their add-on is functioning in the current version of Firefox, to give them confidence to release bug fixes and new features without regressions. As the Selenium project already uses continuous integration, adding the Mozmill test step to this is a great step towards achieving the same within Mozilla, and immediately benefits the add-on author.

The Selenium IDE project plan in Bamboo has three stages: Build, Test, and Package. I’m going to focus on the Test stage in this blog post, as the other stages are very specific to Selenium. If you’re reading this and you develop a Firefox add-on then you should be able to apply the following to your project without too much tweaking.


There are just two prerequisites for running these tests:

  • Python: The automation scripts are written in Python, so this is required.
  • Mercurial: You must have the Python modules for Mercurial installed, as this is the source code management tool that the automation scripts and tests are using.


There are several tasks to complete for the Tests build. Below I list these tasks, with an explanation and configuration steps for each.

Clone mozmill-tests

Because we’re wanting to run the tests against a specific build of the add-on rather than the latest release, we need to clone the mozmill-tests repository and override the addons.ini file with the location of our target add-on. This is simply a Mercurial task, as follows:

hg clone

Note that if you have Mercurial set up as an executable in your continuous integration server then you may not need the ‘hg’ part of this command, as it will be substituted based on the agent running the command.

Switch branch

By default the mozmill-tests repository will be set to the default branch. This is paired with the mozilla-central branch of Firefox, and therefore is relevant only when running tests against the very latest nightly builds of Firefox. As we’re interested in testing against the latest release, we switch to the mozmill-release branch.

hg checkout mozilla-release

You will need to make sure this command is run within the directory of the cloned mozmill-tests repository, which by default will be simply mozmill-tests.

Create addon.ini

The addon.ini file tells the script where to download/install the add-on from. If you didn’t create your own then it would likely be downloaded from or the add-on author’s preferred download location for the latest release. The file is essentially an ini file, with locations for linux, mac, and windows. If your continuous integration server provides a link to latest artifacts then you can add this, or you can use some sort of artifact sharing such as there is in Bamboo. The following commands create a suitable addon.ini for Selenium IDE.

echo "[download]" > tests/addons/
echo "linux=file://${}/selenium-ide.xpi" >> tests/addons/
echo "mac=file://${}/selenium-ide.xpi" >> tests/addons/
echo "win=file://${}/selenium-ide.xpi" >> tests/addons/

If you’re using Jenkins, then the Copy Artifact Plugin could be useful for sharing artifacts between builds.

Commit addon.ini

It’s necessary to commit the replacement addon.ini file so that it is included when the local repository is cloned when running the tests.

hg commit -m 'Specify latest build of addon.'

Note: It really doesn’t matter what you put for the commit message here as this commit is not preserved between builds.

Download latest Firefox release

Rather than having to make sure your build agent always has the latest version of Firefox installed, there’s a handy script that can download this for you. This is a Python script, and therefore needs to be set up to use the Python executable.

./ --directory=latest-release --platform=mac --type=release --version=latest

Substitute the value of the platform for whatever platform your build agent is running. The latest release of Firefox will be downloaded to the latest-release directory. Note that the version value of ‘latest’ is relying on a symlink on Mozilla’s FTP server, that points to the directory of the latest released version number.

Run Mozmill tests

You can now run the script that executes the Mozmill tests.

./ --junit=results.xml --logfile=results.log --repository=mozmill-tests --with-untrusted latest-release

Here’s an explanation of the command line options:

  • The junit command line option determines where the results will be stored. The JUnit report format is one supported by many continuous integration servers, and often provides some nice reporting and visualizations of the results. This destination filename is substituted with a counter for each file created, for example results.xml will be results_0.xml.
  • Adding the logfile is optional, however this can be a useful build artifact if you have failures.
  • As we’re using a locally modified repository, we need to specify the location of this using the repository command line option. The default location will be mozmill-tests.
  • We need to set the target add-on using the target-addon option. This must match the directory beneath tests/addons, which in the case of Selenium IDE is
  • The with-untrusted flag is necessary if the add-on is not hosted at Any add-on hosted by Mozilla will be implicitly trusted. As Selenium IDE is not currently hosted by Mozilla, this flag is necessary.
  • Finally, the path to the Firefox binary is needed. It’s possible to simply point to a directory that contains a downloaded copy of Firefox, so we just use latest-release as that’s where our download task was told to put Firefox.

Parse test results

As mentioned above, a lot of continuous integration servers support results in JUnit report format, so your final task may be to specify the location of these files. If you used the example given above, then you will specify these using results*.xml.

The Bamboo instance for Selenium is publically viewable, so you can see the results of recent builds for Selenium IDE, and the report for the latest build. You can also see the results on the Mozmill archive dashboard. The Selenium IDE project is built whenever a change is committed to the core or dependent code sections of the repository. It can also be triggered manually.

Unfortunately the Selenium build hardware is experiencing stability issues at the time of writing this, meaning that there is not always a suitable build agent for the Mozmill tests.

Known issues

Currently there is an issue with the Mozmill automation script, in that it will exit without an error code even when tests have failed. Fortunately, the continuous integration servers that I’ve been working with update the build success based on the JUnit reports. If this wasn’t the case then builds with failing tests would be incorrectly reported as successful. We have a bug 626712 on file for this issue, and it will hopefully be resolved soon.

Running the Selenium IDE Mozmill tests

A short while ago I posted about the Mozmill tests I’ve created for Selenium IDE, however I didn’t cover how you can run these tests yourself. Currently I run these manually as needed to ensure that the nightly Firefox builds have not regressed or introduced changes in any areas that the addon depends on. We ultimately intend this to be a scheduled job.

I also have added a job on the Selenium continuous integration server that runs the tests against a released version of Firefox. In the future this will test the latest build of Selenium IDE, and will run every time the addon is built.

In order to run the Selenium IDE tests you will need to have Mercurial and Mozmill installed, which you can do simply by using pip install mercurial mozmill. Once you have these you can clone the mozmill-automation repository, using the following command:

hg clone

Then from the repository directory run the following:

./ --report= --with-untrusted /Applications/

Reports will be sent to our dashboard as specified by the --report parameter, and available to see here.

The --target-addons parameter specifies that we only want to run the Selenium IDE tests, and not all of the addons tests we have, and the --with-untrusted parameter is required because Selenium IDE is not listed on and is therefore ‘untrusted’.

The final parameter is the version of Firefox you want to run the tests against. These tests can currently be run against Nightly (7.0), Aurora (6.0), Beta (5.0), as well as the current releases (4.0, 3.6, and 3.5).

Below is a short screencast demonstrating how to run the tests:

With the recent release of Selenium IDE 1.0.11, I was able to push some new tests. These check a few more commands, and brings the total number of tests up to 40. If you’re interested in helping out and you have any questions, then you can either get in touch with me directly, ask in the #selenium IRC channel on Freenode, or post a message to the selenium-developers Google group.