Custom Firefox Cufflinks

If you’re interested in a pair of custom Firefox logo cufflinks for $25 plus postage then please get in touch. I’ve been in contact with CuffLinks.com, and if I can place an initial order of at least 25 pairs then the mold and tooling fee will be waived. Check out their website for examples of their work. The Firefox cufflinks would be contoured to the outline of the logo, and the logo itself would be screen printed in full colour. If these go well, I’d also love a pair of dino head cufflinks to complement them!

Unless I can organise with CuffLinks.com to ship to multiple recipients and take payments separately, I will accept payments via Paypal and take care of sending the cufflinks out. As an example of postage costs, sending to USA would cost me approximately $12 per parcel. Let me know if you’re interested via a comment on this post, e-mail (find me in the phonebook), or IRC (:davehunt).

More realistic endurance test results

If you’re not already familiar with the Firefox endurance tests, these are Mozmill tests that repeat a small snippet of user interaction over and over again while gathering metrics. This allows us to detect if there’s a memory leak in an very localised area, or if there’s a memory regression within the areas tested. I’ve blogged about them a few times.

We’ve known for a while that the results we’ve been getting aren’t entirely realistic, and this is due to the fact that we only wait for 0.1 seconds between each iteration. This doesn’t give Firefox any time to perform tasks such as garbage collection. Unfortunately we couldn’t just increase this delay as that would cause other Mozmill tests to be queued behind the much longer running endurance tests.

So now that we have our new VMWare ESX cluster in place (which has given us an awesome three VMs per platform) we’ve configured Jenkins to run endurance tests on just one node per platform. This allows other Mozmill tests to continue on the remaining available nodes. We were then finally able to increase the delay to 5 seconds.

The results are as we had hoped. The memory usage has dropped, and the duration has increased. Also, the individual testrun results became a lot less erratic. This can be seen in the following charts:

It should now be much easier for us to spot regressions, and hopefully we’ll have less false positives! If you’re interested in the latest endurance results, you can find them in our Mozmill Dashboard, along with the endurance charts.

Related bugs/issues:

  1. Bug 788531 – Revise default delay for endurance test to make scenarios more realistic
  2. Issue 173 – Have dedicated nodes for endurance tests
  3. Issue 201 – Revise default delay for all endurance jobs
  4. Issue 203 – Increase build timeout for endurance tests

Mozilla application downloader released

This week part of the Automation Tools team have gathered at the London Mozilla Space to work on migrating our Firefox automated UI tests to Mozmill 2.0. A considerable part of this work is converting our automation scripts repository, which contains a number of packages that should really be dependencies. Our intention is to take these packages and either merge them to the appropriate mozbase packages, or configure them as suitable packages in their own right.

The first of these packages to be released independently is our impressive download script, which can be used to download a variety of Firefox or Thunderbird builds. We’ve appropriately name it mozdownload, and have released it on PyPI, and the repository can be found on github.

You can install it using pip install mozdownload or easy_install mozdownload and use mozdownload -h for a full list of command line options. A couple of simple examples are provided below:

To download the latest official Firefox release for your platform:

To download the latest official Thunderbird release for your platform:

Of course we don’t just use this to download the official releases. You can also download latest (or specific) builds from any of the channels with daily builds. Here are a few more examples for daily builds:

To download the Firefox Nightly build from 23rd May 2012:

To download the latest Thunderbird Daily build:

For Firefox there are daily builds for the mozilla-central and mozilla-aurora branches. For Thunderbird these are comm-central and comm-aurora.

Candidate builds can also be downloaded, so for example if you wanted to test a candidate build for the fourth beta of Firefox 13 you could use the following:

Finally, you can also download Tinderbox builds. For example, to download the latest tinderbox build use:

If you have any feature requests or find any issues please use github’s issue tracker.

Happy downloading!

Q3/2011 in review

In the hope that I might inspire others to do the same, I’ve created a few screencasts showing some of the cool things I worked on in the last quarter. I’ve tried to keep them all short, and they’re all available in HD so no need to squint to see details.

pytest plugin for WebQA

Endurance tests daily results

System graphics details in endurance reports

Running the Mozmill tests in Jenkins

Running the Selenium IDE Mozmill tests in Bamboo

Adding Mozmill tests to the Selenium IDE build system

Back in April I blogged about the Mozmill tests I’d written to test Selenium IDE. I followed up in June with a blog post covering how to run these tests. The natural progression is to add these tests into the existing Selenium IDE build environment, which is run using Atlassian’s continuous integration server, Bamboo.

At Mozilla, we want to run the Mozmill tests for the released versions of add-ons against the latest builds of Firefox. This is to determine any regressions in Firefox that will potentially cause issues for the add-on authors. It also gives the add-on authors an early warning if there is a potential compatibility issue with an upcoming release of Firefox. We currently run these tests on a schedule, but will soon be looking to move to a continuous integration solution ourselves. You can see the results of these daily tests on our dashboard.

The add-on authors (Selenium in this case) are typically more interested to know if the latest build of their add-on is functioning in the current version of Firefox, to give them confidence to release bug fixes and new features without regressions. As the Selenium project already uses continuous integration, adding the Mozmill test step to this is a great step towards achieving the same within Mozilla, and immediately benefits the add-on author.

The Selenium IDE project plan in Bamboo has three stages: Build, Test, and Package. I’m going to focus on the Test stage in this blog post, as the other stages are very specific to Selenium. If you’re reading this and you develop a Firefox add-on then you should be able to apply the following to your project without too much tweaking.

Prerequisites

There are just two prerequisites for running these tests:

  • Python: The automation scripts are written in Python, so this is required.
  • Mercurial: You must have the Python modules for Mercurial installed, as this is the source code management tool that the automation scripts and tests are using.

Tasks

There are several tasks to complete for the Tests build. Below I list these tasks, with an explanation and configuration steps for each.

Clone mozmill-tests

Because we’re wanting to run the tests against a specific build of the add-on rather than the latest release, we need to clone the mozmill-tests repository and override the addons.ini file with the location of our target add-on. This is simply a Mercurial task, as follows:

hg clone http://hg.mozilla.org/qa/mozmill-tests

Note that if you have Mercurial set up as an executable in your continuous integration server then you may not need the ‘hg’ part of this command, as it will be substituted based on the agent running the command.

Switch branch

By default the mozmill-tests repository will be set to the default branch. This is paired with the mozilla-central branch of Firefox, and therefore is relevant only when running tests against the very latest nightly builds of Firefox. As we’re interested in testing against the latest release, we switch to the mozmill-release branch.

hg checkout mozilla-release

You will need to make sure this command is run within the directory of the cloned mozmill-tests repository, which by default will be simply mozmill-tests.

Create addon.ini

The addon.ini file tells the script where to download/install the add-on from. If you didn’t create your own then it would likely be downloaded from addons.mozilla.org or the add-on author’s preferred download location for the latest release. The file is essentially an ini file, with locations for linux, mac, and windows. If your continuous integration server provides a link to latest artifacts then you can add this, or you can use some sort of artifact sharing such as there is in Bamboo. The following commands create a suitable addon.ini for Selenium IDE.

echo "[download]" > tests/addons/ide@seleniumhq.org/addon.ini
echo "linux=file://${bamboo.build.working.directory}/selenium-ide.xpi" >> tests/addons/ide@seleniumhq.org/addon.ini
echo "mac=file://${bamboo.build.working.directory}/selenium-ide.xpi" >> tests/addons/ide@seleniumhq.org/addon.ini
echo "win=file://${bamboo.build.working.directory}/selenium-ide.xpi" >> tests/addons/ide@seleniumhq.org/addon.ini

If you’re using Jenkins, then the Copy Artifact Plugin could be useful for sharing artifacts between builds.

Commit addon.ini

It’s necessary to commit the replacement addon.ini file so that it is included when the local repository is cloned when running the tests.

hg commit -m 'Specify latest build of addon.'

Note: It really doesn’t matter what you put for the commit message here as this commit is not preserved between builds.

Download latest Firefox release

Rather than having to make sure your build agent always has the latest version of Firefox installed, there’s a handy script that can download this for you. This is a Python script, and therefore needs to be set up to use the Python executable.

./download.py --directory=latest-release --platform=mac --type=release --version=latest

Substitute the value of the platform for whatever platform your build agent is running. The latest release of Firefox will be downloaded to the latest-release directory. Note that the version value of ‘latest’ is relying on a symlink on Mozilla’s FTP server, that points to the directory of the latest released version number.

Run Mozmill tests

You can now run the script that executes the Mozmill tests.

./testrun_addons.py --junit=results.xml --logfile=results.log --repository=mozmill-tests --target-addons=ide@seleniumhq.org --with-untrusted latest-release

Here’s an explanation of the command line options:

  • The junit command line option determines where the results will be stored. The JUnit report format is one supported by many continuous integration servers, and often provides some nice reporting and visualizations of the results. This destination filename is substituted with a counter for each file created, for example results.xml will be results_0.xml.
  • Adding the logfile is optional, however this can be a useful build artifact if you have failures.
  • As we’re using a locally modified repository, we need to specify the location of this using the repository command line option. The default location will be mozmill-tests.
  • We need to set the target add-on using the target-addon option. This must match the directory beneath tests/addons, which in the case of Selenium IDE is ide@seleniumhq.org.
  • The with-untrusted flag is necessary if the add-on is not hosted at addons.mozilla.org. Any add-on hosted by Mozilla will be implicitly trusted. As Selenium IDE is not currently hosted by Mozilla, this flag is necessary.
  • Finally, the path to the Firefox binary is needed. It’s possible to simply point to a directory that contains a downloaded copy of Firefox, so we just use latest-release as that’s where our download task was told to put Firefox.

Parse test results

As mentioned above, a lot of continuous integration servers support results in JUnit report format, so your final task may be to specify the location of these files. If you used the example given above, then you will specify these using results*.xml.

The Bamboo instance for Selenium is publically viewable, so you can see the results of recent builds for Selenium IDE, and the report for the latest build. You can also see the results on the Mozmill archive dashboard. The Selenium IDE project is built whenever a change is committed to the core or dependent code sections of the repository. It can also be triggered manually.

Unfortunately the Selenium build hardware is experiencing stability issues at the time of writing this, meaning that there is not always a suitable build agent for the Mozmill tests.

Known issues

Currently there is an issue with the Mozmill automation script, in that it will exit without an error code even when tests have failed. Fortunately, the continuous integration servers that I’ve been working with update the build success based on the JUnit reports. If this wasn’t the case then builds with failing tests would be incorrectly reported as successful. We have a bug 626712 on file for this issue, and it will hopefully be resolved soon.

Running the ‘Mem Buster’ endurance test

I blogged a few weeks ago about how I was able to demonstrate improvements to the memory usage of Firefox using endurance tests. The test I was using was inspired by Stuart Parmenter’s Mem Buster test, and it has now been checked into the repository and available for anyone to run.

At the moment it’s only possible to run the Mem Buster test from the command line (hopefully Mozmill Crowd support won’t be too far off). The first thing you’ll need is the mozmill-automation repository checked out. If you already have this then you’ll need to do a pull and update to make sure you have the latest changes.

hg clone http://hg.mozilla.org/qa/mozmill-automation

Then, from the mozmill-automation directory, run the following command:

./testrun_endurance.py --reserved=membuster --delay=3 --iterations=2 --entities=100 --report=http://mozmill-crowd.brasstacks.mozilla.com/db/ /Applications/Firefox.app

The reserved argument test the script to only run the Mem Buster test and not the general endurance tests. The test opens a site for each entity, so by specifying 100 entities and 2 iterations it will open a total of 200 sites. The delay of 3 seconds is from the original Mem Buster test. I would recommend including the report argument as this shares your results and allows you to see the visualisation of memory usage during the test. The final argument is the location of the version of Firefox you want to run the tests against.

Below is a screencast demonstrating the Mem Buster endurance test on Windows 7:

If you’re interested in following the progress of the endurance tests project, check out the project page. For further help you can find the documentation here, post a comment to this entry, or ask a question in the QMO forums.

Endurance tests demonstrate Firefox’s memory usage improvements

Thanks to the amazing efforts of the MemShrink project, Firefox’s memory usage is seeing some great improvements. In particular, Firefox 7 will be much more efficient with memory than the current version. As endurance tests monitor resources such as memory, it makes sense for us to work together to ensure that we’re moving in the right direction, and that we don’t regress in any of these areas.

At this point there are only five endurance tests, and although these can be run with many hundreds of iterations in order to seek out memory leaks in the tested areas, they do nothing to simulate a user. It was suggested that we have a special endurance test similar to Stuart Parmenter’s Mem Buster test.

Creating an initial version of this new test did not take long. Instead of opening sites in new windows I open them in tabs, and the number of sites opened is controlled by iterations and micro-iterations. I also increased the number of sites so we’d be hitting the same ones less often, and based this new list on Alexa’s top sites. Once I added in handling of modal dialogs that some sites were causing to be displayed then I was able to consistently get results.

This test would appear to be similar to Talos tp5 in that is loads sites from Alexa’s index, however we’re not measuring how long each site takes to load. Instead, we move onto the next site after a delay as specified on the command line. I have kept the same delay as the original Mem Buster test, which is 3 seconds.

After running the Mem Buster Endurance Test five times across five versions of Firefox, I found the results to clearly reflect the MemShrink efforts. Although the memory consumption varies somewhat for each run, the general downward trend is unmistakable.

In the following charts you can see the improvement in memory usage between Firefox 4 & 5. These can be directly compared as the endurance tests were measuring the same metrics (allocated memory & mapped memory).

Charts showing allocated and mapped memory usage in Firefox 4 & 5

In Firefox 6 there were several improvements to memory reporting, and the endurance tests were updated to record new metrics (explicit memory & resident memory). You can see in the following charts that explicit memory usage in Firefox 7 is rough half that of Firefox 6! It appears that this has increased in Firefox 8, which will require some further investigation. The resident memory has continued to decrease in each version.

Charts showing explicit and resident memory usage in Firefox 6, 7, & 8

You can follow the progress of the Mem Buster Endurance Test in Bugzilla. Full reports from the test runs used in this blog post can be found here.

Update: It appears that the explicit memory calculated for Firefox 7 on Mac was artificially low. This explains the slight increase in Firefox 8. If you’re interested you can read further details on Bugzilla.

Micro-iterations in Endurance Tests

Last week micro-iterations landed in Mozmill Endurance Tests. These allow tests to accumulate resources during an iteration. This was previously achieved by leaving the state of the test snippet in a different state to how it started, allowing the iterations themselves to accumulate. The problem with this is that these accumulating tests have a very different pattern compared to other tests that clean up before ending the iteration.

To solve this we decided to add a micro-iteration parameter and to use it to loop within an iteration. An example use for this is the new tab tests. Now, if you specify 5 iterations and 10 micro-iterations then these tests will open 10 new tabs, close them, and repeat that 5 times.

The endurance tests documentation has been updated with details on writing and running tests with micro-iterations.

Endurance Tests in Firefox 6

One of the features of the upcoming Firefox 6 is an improvement to the handling and reporting of memory resources. As you can probably imagine, this is very applicable to the endurance tests project. As a result of the changes, running the endurance tests with the previews of Firefox 6 was failing to gather any metrics at all.

I’m pleased to announce that as of yesterday, the endurance tests now support Firefox 6! One of the main differences you will see is that we’re no longer gathering mapped/allocated memory, and are instead gathering explicit/resident, which we are expecting to provide much more useful results. You don’t need to do anything to get the latest changes, just run the tests as described here (using the command line) or here (using Mozmill Crowd).

If you’re interested, here are the relevant bugs:

  • Bug 633653 – Revamp about:memory
  • Bug 657327 – Merge the “mapped” and “heap used” trees, and make the tree flatter
  • Bug 656869 – No memory results on endurance testrun with Nightly 6.0a1
  • Bug 657508 – Update dashboard to display endurance tests results from Firefox 6.0