More realistic endurance test results

If you’re not already familiar with the Firefox endurance tests, these are Mozmill tests that repeat a small snippet of user interaction over and over again while gathering metrics. This allows us to detect if there’s a memory leak in an very localised area, or if there’s a memory regression within the areas tested. I’ve blogged about them a few times.

We’ve known for a while that the results we’ve been getting aren’t entirely realistic, and this is due to the fact that we only wait for 0.1 seconds between each iteration. This doesn’t give Firefox any time to perform tasks such as garbage collection. Unfortunately we couldn’t just increase this delay as that would cause other Mozmill tests to be queued behind the much longer running endurance tests.

So now that we have our new VMWare ESX cluster in place (which has given us an awesome three VMs per platform) we’ve configured Jenkins to run endurance tests on just one node per platform. This allows other Mozmill tests to continue on the remaining available nodes. We were then finally able to increase the delay to 5 seconds.

The results are as we had hoped. The memory usage has dropped, and the duration has increased. Also, the individual testrun results became a lot less erratic. This can be seen in the following charts:

It should now be much easier for us to spot regressions, and hopefully we’ll have less false positives! If you’re interested in the latest endurance results, you can find them in our Mozmill Dashboard, along with the endurance charts.

Related bugs/issues:

  1. Bug 788531 – Revise default delay for endurance test to make scenarios more realistic
  2. Issue 173 – Have dedicated nodes for endurance tests
  3. Issue 201 – Revise default delay for all endurance jobs
  4. Issue 203 – Increase build timeout for endurance tests

Running the ‘Mem Buster’ endurance test

I blogged a few weeks ago about how I was able to demonstrate improvements to the memory usage of Firefox using endurance tests. The test I was using was inspired by Stuart Parmenter’s Mem Buster test, and it has now been checked into the repository and available for anyone to run.

At the moment it’s only possible to run the Mem Buster test from the command line (hopefully Mozmill Crowd support won’t be too far off). The first thing you’ll need is the mozmill-automation repository checked out. If you already have this then you’ll need to do a pull and update to make sure you have the latest changes.

hg clone http://hg.mozilla.org/qa/mozmill-automation

Then, from the mozmill-automation directory, run the following command:

./testrun_endurance.py --reserved=membuster --delay=3 --iterations=2 --entities=100 --report=http://mozmill-crowd.brasstacks.mozilla.com/db/ /Applications/Firefox.app

The reserved argument test the script to only run the Mem Buster test and not the general endurance tests. The test opens a site for each entity, so by specifying 100 entities and 2 iterations it will open a total of 200 sites. The delay of 3 seconds is from the original Mem Buster test. I would recommend including the report argument as this shares your results and allows you to see the visualisation of memory usage during the test. The final argument is the location of the version of Firefox you want to run the tests against.

Below is a screencast demonstrating the Mem Buster endurance test on Windows 7:

If you’re interested in following the progress of the endurance tests project, check out the project page. For further help you can find the documentation here, post a comment to this entry, or ask a question in the QMO forums.