Introducing Firefox Endurance Testing

Since late last year I have been working on a prototype of an Endurance Testing project for Firefox. The idea is to use our existing Mozmill framework for automating UI testing of Firefox to write tests that stress and strain the browser over time. I’ve heard many times from people that Firefox needs to be restarted once in a while because it’s become sluggish, and indeed I’ve experienced this myself. The problem is that there are rarely clear steps to reproduce this issues as they normally are an accumulation of many actions over an extended period of time. What actions cause a degradation in performance, and why? This is what we hope to discover with the endurance tests project.

The initial implementation of endurance tests is rather simple: Create a test snippet that exercises a function of Firefox, and execute it repeatedly whilst gathering details of resources in use. Ultimately we may come up with more elaborate tests, and but it’s important to get a proof of concept.

There are several components to the endurance tests:

  1. Command line automation script
  2. Test snippets
  3. Resource gathering
  4. Reporting

The number of iterations each snippet repeats can be set on the command line, as well as an optional delay between each iteration. The normal command line options allow for logging, and reporting to a Mozmill Dashboard instance.

Triggering the endurance tests currently looks something like this:

./testrun_endurance.py --delay=1000 --iterations=50 --report=http://davehunt.couchone.com/mozmill /Applications/Minefield.app

This will then launch Firefox, run through all of the endurance tests (each one iterating over it’s test snippet 50 times), and then close Firefox. Because I’ve included a report parameter, the report will also be sent to the Mozmill Dashboard instance. These reports are currently available here.

Here’s a short screencast that demonstrates running the endurance tests:

If you’re interested in following the progress of the endurance tests project, check out the project page or the tracking bug for phase one.

3 thoughts on “Introducing Firefox Endurance Testing”

  1. I’d like to be given the option to include an anonymized-but-unique ID to allow for better data collection, too. There is probably no good reason to exclude volunteers wanting to provide more valuable data to the Mozilla developers.

    1. I’m interested to know what additional information you would suggest we gather? Were you aware of Telemetry? It’s been available since Firefox 7 and allows users to opt-in to sending anonymous performance related data back to Mozilla.

Leave a Reply

Your email address will not be published. Required fields are marked *