Running endurance tests with Mozmill Crowd

Ahead of our Mozmill Crowd testday last Friday we made some changes to the endurance tests, including enabling endurance test run within Mozmill Crowd! Running the endurance tests is now even easier – simply install the Mozmill Crowd extension, and in just a few clicks the tests will be running. We also updated the endurance dashboard reports and pushed them to our Mozmill Crowd report server.

I’ve created a short screencast that demonstrates installing Mozmill Crowd, running the endurance tests, and reviewing the results:

If you want to run with add-ons installed then you’ll still need to use the command line for now (support in Mozmill Crowd is planned).

It’s also important to note that delay is now specified in seconds, and not in milliseconds.

Endurance Results from Test Day

Today I finally finished reviewing the hundreds (yes, hundreds!) of endurance reports that were submitted on our Firefox 4 add-ons test day last Friday and on the days following. It was amazing to see so many reports coming in, and I would like to thank everyone that ran an endurance test run. By far the most active contributor was pxbuz, to whom I’m extra grateful!

Of all of the test runs, it turns out there are three major issues discovered:

The first is that we really need to improve the reporting system. Going through the results was a long and tedious job, so I will be thinking about how I can improve that experience.

Secondly, we need to come up with a way to dismiss any modal dialogs that add-ons might show on first run. There were a couple of these that resulted in what looked like memory leaks, but turns out would be impossible to replicate manually.

I saved the best for last – we found a memory leak when the Greasemonkey add-on is installed! It seems that when entering/leaving private browsing mode there is memory allocated but not released. A bug has been raised and hopefully it’ll soon be resolved. Greasemonkey is one of our most popular add-ons with a current average active daily usage of over 2.5 million users!

Below you can see how the memory leak was spotted. On the left is an example of an endurance test without any add-ons installed, and on the right is a test run with Greasemonkey installed. Those five spikes that start around the 500 checkpoints mark occur during the private browsing test.

You can see the actual reports here and here.

Automated Firefox tests with add-ons installed

Mozmill has a feature that allows the tester to install an add-on during the test run. Until recently this was only used by one of our automated testruns, which was for specifically testing the installed add-on rather than Firefox.

With the recent development on the endurance tests project, it has been necessary to take more advantage of this feature. A bug was reported where if Adblock Plus – our most popular add-on – is installed memory usage increased rapidly when navigating a web page, and none of the memory was being released. To start investigating this I created a very basic (and specific) test for the site mentioned in the bug report, and then simply hacked something together based on the existing add-ons test run. A short time later, the need to run the endurance tests with multiple add-ons installed came up, so I hacked some more to get that in place too. Rather than keep these hacks around, it made sense to allow testers to specify add-ons to be installed during any of our testruns, so I started to work on the necessary patches.

As a result, testers can now run any of our automation scripts with one or more add-ons installed by simply specifying the addons command line parameter. To install multiple add-ons simply repeat the parameter. The argument can either be a path on your machine, or a web/ftp server. In the latter case the add-on will be downloaded to a temporary location before the testrun and removed at the end. The latest version of Mozmill (1.5.2) also now disables the compatibility check, meaning that we can run tests with add-ons that are not marked as compatible with the version of Firefox in use.

An example of running the endurance tests with two add-ons installed:

./testrun_endurance.py --addons=https://addons.mozilla.org/firefox/downloads/latest/748/addon-748-latest.xpi --addons=/Users/dave/Downloads/noscript.xpi --delay=1000 --iterations=10 /Applications/Firefox.app/

Introducing Firefox Endurance Testing

Since late last year I have been working on a prototype of an Endurance Testing project for Firefox. The idea is to use our existing Mozmill framework for automating UI testing of Firefox to write tests that stress and strain the browser over time. I’ve heard many times from people that Firefox needs to be restarted once in a while because it’s become sluggish, and indeed I’ve experienced this myself. The problem is that there are rarely clear steps to reproduce this issues as they normally are an accumulation of many actions over an extended period of time. What actions cause a degradation in performance, and why? This is what we hope to discover with the endurance tests project.

The initial implementation of endurance tests is rather simple: Create a test snippet that exercises a function of Firefox, and execute it repeatedly whilst gathering details of resources in use. Ultimately we may come up with more elaborate tests, and but it’s important to get a proof of concept.

There are several components to the endurance tests:

  1. Command line automation script
  2. Test snippets
  3. Resource gathering
  4. Reporting

The number of iterations each snippet repeats can be set on the command line, as well as an optional delay between each iteration. The normal command line options allow for logging, and reporting to a Mozmill Dashboard instance.

Triggering the endurance tests currently looks something like this:

./testrun_endurance.py --delay=1000 --iterations=50 --report=http://davehunt.couchone.com/mozmill /Applications/Minefield.app

This will then launch Firefox, run through all of the endurance tests (each one iterating over it’s test snippet 50 times), and then close Firefox. Because I’ve included a report parameter, the report will also be sent to the Mozmill Dashboard instance. These reports are currently available here.

Here’s a short screencast that demonstrates running the endurance tests:

If you’re interested in following the progress of the endurance tests project, check out the project page or the tracking bug for phase one.