Working on the Firefox OS automation, it’s often been necessary to populate a device with some sample content. For example, when measuring the launch time of the contacts app it’s more realistic if we already have a bunch of contacts on our phone. To solve this, I created a small Python package called b2gpopulate, which uses Web APIs and mozdevice to push various types of content to a device with Marionette enabled.
To install b2gpopulate you will need Python and can simply run pip install b2gpopulate from the command line. If you don’t have pip installed then you can also use easy_install b2gpopulate. Running b2gpopulate is pretty straightforward, however you will need to have a Firefox OS device connected that’s running Marionette, and you will need to forward port 2828 by running adb forward tcp:2828 tcp:2828. The following example will populate the connected device with 200 of each content type:
Note that before pushing a database the b2g process is stopped, so don’t panic if you see your device restarting. Run b2gpopulate --help for full usage instructions.
Initially I used just the Contacts API to add/remove contacts from the device, but this is a pretty slow process, especially for a large number of contacts. After finding out about the reference workload that Gaia uses in its build I modified this to push a prebuilt database of contacts. This is then topped up using the Contacts API as needed. There are prebuilt databases for 200, 500, 1000, and 2000 contacts.
The most recent addition to b2gpopulate is messages. Like contacts, this pushes a prebuilt database of 200, 500, 1000, or 2000. Unlike the contacts, there is currently no option to top this up.
Pictures & Videos
This uses mozdevice to push a reference picture or video to the device and then performs a remote copy. In a future version I would like to alternate through a number of reference files so there’s some variance.
This has changed in the version of b2gpopulate I released today. Previously it worked in exactly the same way as the pictures and videos, but because the metadata files doesn’t vary, the music app doesn’t distinguish between them. Now, the metadata is modified for each file using mutagen, and the album/artist is changed every ten tracks.
I suspect there will be a need for more content types in the future. For example, we could potentially add events, alarms, history, favourites, bookmarks, emails, etc. If your interested in contributing, you can find the repository on GitHub.
If you’re not already familiar with the Firefox endurance tests, these are Mozmill tests that repeat a small snippet of user interaction over and over again while gathering metrics. This allows us to detect if there’s a memory leak in an very localised area, or if there’s a memory regression within the areas tested. I’ve blogged about them a few times.
We’ve known for a while that the results we’ve been getting aren’t entirely realistic, and this is due to the fact that we only wait for 0.1 seconds between each iteration. This doesn’t give Firefox any time to perform tasks such as garbage collection. Unfortunately we couldn’t just increase this delay as that would cause other Mozmill tests to be queued behind the much longer running endurance tests.
So now that we have our new VMWare ESX cluster in place (which has given us an awesome three VMs per platform) we’ve configured Jenkins to run endurance tests on just one node per platform. This allows other Mozmill tests to continue on the remaining available nodes. We were then finally able to increase the delay to 5 seconds.
The results are as we had hoped. The memory usage has dropped, and the duration has increased. Also, the individual testrun results became a lot less erratic. This can be seen in the following charts:
Results with 0.1s delay
Results with 5s delay
It should now be much easier for us to spot regressions, and hopefully we’ll have less false positives! If you’re interested in the latest endurance results, you can find them in our Mozmill Dashboard, along with the endurance charts.
Bug 788531 – Revise default delay for endurance test to make scenarios more realistic
Issue 173 – Have dedicated nodes for endurance tests
Issue 201 – Revise default delay for all endurance jobs
Issue 203 – Increase build timeout for endurance tests
It’s a little difficult to get your hands on a device that can run Firefox OS right now, but if you’re interested in running the UI tests a device is not essential. This guide will show you how to run the tests on the nightly desktop client builds we provide.
Step 1: Download the latest desktop client
The Firefox OS desktop client lets you run Gaia (the UI for Firefox OS) and web apps in a Gecko-based environment somewhat similar to an actual device. There are certain limitations of the desktop client, including: it doesn’t emulate device hardware (camera, battery, etc), it doesn’t support carrier based operations such as sending/receiving messages or calls, and it relies on the network connection of the machine it’s running on.
You can download the latest build of the desktop client from this location, but make sure you download the appropriate file for your operating system. Unfortunately, due to bug 832469 the nightly desktop builds do not currently work on Windows, so you will need either Mac or Linux (a virtual machine is fine) to continue:
Linux (32bit): b2g-[VERSION].multi.linux-i686.tar.bz2
Linux (64bit): b2g-[VERSION].multi.linux-x86_64.tar.bz2
Once downloaded, you will need to extract the contents to a local folder. For the purposes of the rest of this guide, I’ll refer to this location as $B2G_HOME.
Step 2: Enable Marionette
Marionette is a test framework built into Gecko that allows remote control of the application. The Gaia UI tests use Marionette to launch applications and simulate a user interacting with them. By default, this is enabled in the desktop client but it is necessary for us to set a preference in the default profile before we can run the tests.
Add the following line to your gaia/profile/user.js file, which on Mac is located in $B2G_HOME/B2G.app/Contents/MacOS and on Linux in $B2G_HOME/b2g.
Step 3: Start Firefox OS
You can start Firefox OS by double clicking $B2G_HOME/B2G.app (Mac) or running $B2G_HOME/b2g/b2g (Linux). If everything went well, you should see the ‘powered by’ screen shortly followed by the first launch app. Complete the configuration steps and optionally follow the tour, and you will be presented with the lock screen. Unlock by dragging the bar up and clicking the padlock. You should be presented with the home screen (shown here).
Take a moment to familiarise yourself with Firefox OS. Launch a couple of applications, change some settings. You’ll soon discover the limitations of the simulator. Probably the most noticeable difference is that there’s no home/power/volume buttons as there would be on a device. The most useful of these is the home button, which allows you to return the to the home screen or to switch between open apps. You should be able to use the home key on your keyboard as a substitute. Here are some more usage tips.
Step 4: Run the tests!
Now you’ve got the simulator running, you can clone and run the automated UI tests against it. You will need to have git and Python installed (I recommend using version 2.7), and I highly recommend using virtual environments.
First, clone the gaia-ui-tests repository using the following command line, where $WORKSPACE is your local workspace folder:
git clone git://github.com/mozilla/gaia-ui-tests.git gaia-ui-tests
If you’re using virtual environments, create a new environment and activate it. You will only need to create it once, but will need to activate it whenever you wish to run the tests:
Now you need to install the test harness (gaiatest) and all of it’s dependencies:
python setup.py develop
Once this is done, you will have everything you need to run the tests. Because we’re running against the desktop client we must filter out all tests that are not appropriate. This list may grow, but it currently includes tests that use: antenna, bluetooth, carrier, camera, sdcard, and wifi. You will probably also want to exclude any tests that are expected to fail (xfail). To run the tests, use the following command:
You should then start to see the tests running, with output similar to the following:
running webserver on http://184.108.40.206:47413/
test_get_all_settings (test_settings.TestSettings) ... ok
test_set_named_setting (test_settings.TestSettings) ... ok
test_set_volume (test_settings.TestSettings) ... ok
Ran 3 tests in 3.234s
The first tests that run are unit tests for the gaiatest harness, so you won’t immediately see much happening in the simulator. You may encounter test failures, and we’re currently focusing on getting these resolved. You may also encounter bug 844498, which has the nasty side-effect of causing all remaining tests to fail. If this happens just try running the suite again for now.
The video shows a full suite run against the simulator. Note that where tests time out I have either cropped the video or increased the speed. This is just to keep the video shorter.
Step 5: Contribute?
Now you can run the tests, you’re in a great position to help us out! Our first focus is to get all the tests passing against the desktop build, but then we need to identify missing areas of coverage that are relevant to the simulator.
To contribute, you will need to set up a github account and then fork the main gaia-ui-tests repository. You will then need to update your local clone so it’s associated with your fork rather than the main one. You can do this with the following commands, replacing $USERNAME with your github username:
You can now create a branch, and make your changes. Once done, you should commit your changes and push them to your fork before submitting a pull request. I’m not going to cover these steps in detail here, as they’re fairly standard git practices and will be covered in far better detail elsewhere. In fact, github:help has some fantastic documentation.
If you’re looking for a task, you should first check the desktop issues list on github. If there’s nothing available there, see if you can find an area that needs more coverage. Feel free to add an issue and a comment to say you’ll work on it.
I’ve just released an update to FlynnID. The primary change is that I’ve reintroduced the command line arguments, meaning a single node can be registered without the need for a configuration file. I’ve also hopefully learned my lesson, and in the words of a friend ‘Deprecate, not annihilate!’
Along with this change, 0.3 also introduces a handy feature if you’re running FlynnID on a schedule. If the node you’re registering is already registered then it won’t attempt to register it again. You can override this behavior using the --force command line option.
Lastly, the output is now much more colourful…
Registering 10.250.3.15:8080 to localhost:4444 [SUCCESS]
Registering 10.250.5.90:8080 to localhost:4444 [SKIPPED]
Registering 10.250.1.26:8080 to localhost:4444 [FAILED]
You can install/upgrade using pip install -U flynnid.
I thought this was important enough to share in a short blog post… Just 10 months ago, Mozilla started to migrate their Selenium projects from the Selenium RC API to the WebDriver API. I’m thrilled to say that this is now complete, and that no Selenium RC projects are actively being run or maintained!
As all of the active Mozilla Web QA automation projects are now using WebDriver, there is no longer a need for BIDPOM (Browser IDPage Object Model) to support Selenium RC. I considered keeping this support purely for the community, however I would rather encourage anyone still using Selenium RC to upgrade to WebDriver.
If you require Selenium RC support then I recommend you fork the repository and continue to develop the RC page objects separately. The only difference you will notice if you’re upgrading to the latest version of BIDPOM is that you may need to import from pages now, rather than pages.webdriver.
In Tron, Flynn’s identity disc is the master key to getting onto the Grid. In the far less exciting real world, FlynnID is the key to registering a Selenium node to Selenium Grid. Yesterday I released FlynnID 0.2, which changes the usage from a list of optional arguments to a single expected argument: a configuration file. This means you can now register several nodes in one go. Below is an example configuration file.
Of course this does unfortunately mean that anyone upgrading from 0.1 may be a little surprised that the command line options have gone, but I strongly feel this is a better approach. This way, your configuration file can be backed up (or added to version control), and it’s much quicker to run. You can install/upgrade FlynnID using pip: pip install -U flynnid.
Finally I can announce that I have released version 1.0 of the pytest plugin used by Mozilla’s Web QA team! It’s been in use for several months now, but I’ve paid off some long standing technical debt, and now consider it stable!
I should say that although this plugin was primarily developed for Mozilla’s Web QA team, anyone that wants to write test automation for websites in Python can take advantage of it. There’s very little that is specific to Mozilla, and this can easily be overridden on the command line, or you could simply fork the project and create your own version of the plugin. Anyway, as I haven’t previously announced the plugin, it’s probably a good idea for me to explain what it actually does…
The primary feature of the plugin is the ability for it to launch and interact with a web browser. It does this by integrating with either the RC or WebDriver APIs provided by the Selenium browser automation framework. The browser is launched ahead of every test, unless the test is specifically decorated to indicate that the test does not require a browser:
The plugin works with a local WebDriver instance, with a remote server (RC or WebDriver), and with Selenium Grid (also RC or WebDriver).
Sauce Labs integration
You can also use this plugin to run your tests in the cloud using your Sauce Labs account. The integration allows you to specify a build identifier and tags, which help when filtering the Sauce Labs jobs. To enable Sauce Labs integration, you simply need to specify a credentials file on the command line, which is a YAML file in the following format:
When tests are run using Sauce Labs, there will also be additional items in the HTML report, such as the video of the test and a link to the job.
If you don’t have a Sauce Labs account already, you can sign up for one here. Sauce Labs is used by Mozilla whenever we need to run on browser/platform combinations that our own Selenium Grid doesn’t support, or whenever we need boost up the number of testruns such as before a big deployment.
The plugin allows you to store your application’s credentials in a YAML file as specified on the command line. This is an important feature for Mozilla, where the credentials files are stored in a private repository. Anyone wanting to contribute or run the tests themselves, simply has to create an account and a YAML file.
Have you ever been frustrated when you’ve kicked off your suite of several hundred tests, just for every single one of them to launch a browser despite the application under test being unavailable? It’s happened to me enough times that I added an initial check to ensure the base URL of the application is responding with a 200 OK; This saves so much time.
Protect sensitive sites
I’ll leave the debate on whether you should be running your tests against sensitive environments such as production for someone else, but if you do decide to do this, the plugin gives you a little bit of extra protection. For a start, all tests are considered destructive and therefore will not run by default. You can explicitly mark tests as non-destructive. Having an opt-in system is more maintenance (I know it’s a pain), but much lower risk. I’d rather accidentally not be running a non-destructive test against production than accidentally run a destructive one, and I have felt this pain before!
Of course, for some environments, you will want to run your destructive tests, and you can do so by specifying the --destructive command line option.
There’s also a final safety net just in case you try running destructive tests against a sensitive environment. This skips any destructive tests that are run against a URL that matches a regular expression. For Mozilla, this defaults to mozilla\.(com|org) and any skipped tests will give a suitable reason in your reports.
Digging through console logs or JUnit reports can be a little frustrating when investigating failures, so the plugin provides a nicely formatted HTML report. This shows the options used when running the tests, a short summary of the results, and then lists each test along with timings and additional resources where appropriate.
I call the additional resources the test’s “death rattle” as it’s only captured for failing tests (it’s not particularly useful for passing tests, and consumes unnecessary resources). For tests that use Selenium this should at least include a screenshot, the HTML, and the current URL when the failure occurred. If you’re running in Sauce Labs then you should also see the video of the test and a link to the test job.
For full details and documentation of the plugin, take a look over the project’s README file on github. If you find any bugs or have any feature requests please raise them in github issues.
This week part of the Automation Tools team have gathered at the London Mozilla Space to work on migrating our Firefox automated UI tests to Mozmill 2.0. A considerable part of this work is converting our automation scripts repository, which contains a number of packages that should really be dependencies. Our intention is to take these packages and either merge them to the appropriate mozbase packages, or configure them as suitable packages in their own right.
The first of these packages to be released independently is our impressive download script, which can be used to download a variety of Firefox or Thunderbird builds. We’ve appropriately name it mozdownload, and have released it on PyPI, and the repository can be found on github.
You can install it using pip install mozdownload or easy_install mozdownload and use mozdownload -h for a full list of command line options. A couple of simple examples are provided below:
To download the latest official Firefox release for your platform:
mozdownload -a firefox -v latest
To download the latest official Thunderbird release for your platform:
mozdownload -a thunderbird -v latest
Of course we don’t just use this to download the official releases. You can also download latest (or specific) builds from any of the channels with daily builds. Here are a few more examples for daily builds:
To download the Firefox Nightly build from 23rd May 2012:
mozdownload -a firefox -t daily --branch=mozilla-central --date=2012-05-23
To download the latest Thunderbird Daily build:
mozdownload -a thunderbird -t daily --branch=comm-central
For Firefox there are daily builds for the mozilla-central and mozilla-aurora branches. For Thunderbird these are comm-central and comm-aurora.
Candidate builds can also be downloaded, so for example if you wanted to test a candidate build for the fourth beta of Firefox 13 you could use the following:
mozdownload -a firefox -t candidate -v 13.0b4
Finally, you can also download Tinderbox builds. For example, to download the latest tinderbox build use:
Yesterday WebQA held a combined test day and meetup event in San Francisco. The idea was to bring everyone together to hack on our QA projects alongside us, and in return we would be able to share our own knowledge and experiences, and demonstrate how we test.
The team had managed to pull together over 50 tasks for the evening, ranging from simple beginner level to a few advanced ones, and using a variety of technologies and projects. As people arrived we encouraged them to slap on a name tag and grab a task.
Once we had everyone in the room (I think there were 18 of us in total) we gave a brief introduction from the team over some beer and pizza, and then the fun really started! Of course it didn’t all go smoothly, and we quickly realised that the entire python.org domain was unavailable, which meant that nobody could download the required Python packages, such as Selenium and pytest! Fortunately we found after a little scrambling that we could use the –use-mirrors command line argument to get us unstuck. In fact, we’ve now even added this to our test jobs running on Jenkins to hopefully make them a little more resilient to PyPI downtime, so in a way I’m glad we had the issues.
Not everyone was working on new tests, in fact Zac set up a mini grid of Android devices (tablets and mobile), and was giving demonstrations on running web tests on mobile. If you’ve not tried this yet then you should as it’s great fun – and the future is mobile, right?
We also had some great feedback on our documentation, which will soon lead to some further improvements and simplifications. I was really impressed when one of the attendees told me there were some issues with the documentation on our wiki, and then told me they just logged in and fixed it themselves. That’s totally the kind of involvement we’re encouraging and it’s so great to see if happening.
By the end of the evening we had three pull requests submitted (all for different projects), and I’m almost certain that number will increase as some of the other tasks were near completion. We’ll now work on reviewing those pulls and merging them in, and look forward to any more that come our way!
Finally, I want to say a huge thanks to everyone that was able to attend for making it such a great event. We all hope you enjoyed it as much as we did, and look forward to seeing you at future events (and on #mozwebqa on irc.mozilla.org). Oh and thanks to Kedo for the Ctrl+W keyboard shortcut for Terminal!