Running the Selenium IDE Mozmill tests

A short while ago I posted about the Mozmill tests I’ve created for Selenium IDE, however I didn’t cover how you can run these tests yourself. Currently I run these manually as needed to ensure that the nightly Firefox builds have not regressed or introduced changes in any areas that the addon depends on. We ultimately intend this to be a scheduled job.

I also have added a job on the Selenium continuous integration server that runs the tests against a released version of Firefox. In the future this will test the latest build of Selenium IDE, and will run every time the addon is built.

In order to run the Selenium IDE tests you will need to have Mercurial and Mozmill installed, which you can do simply by using pip install mercurial mozmill. Once you have these you can clone the mozmill-automation repository, using the following command:

hg clone http://hg.mozilla.org/qa/mozmill-automation

Then from the repository directory run the following:

./testrun_addons.py --report=http://mozmill-crowd.brasstacks.mozilla.com/db/ --target-addons=ide@seleniumhq.org --with-untrusted /Applications/Firefox.app

Reports will be sent to our dashboard as specified by the --report parameter, and available to see here.

The --target-addons parameter specifies that we only want to run the Selenium IDE tests, and not all of the addons tests we have, and the --with-untrusted parameter is required because Selenium IDE is not listed on addons.mozilla.org and is therefore ‘untrusted’.

The final parameter is the version of Firefox you want to run the tests against. These tests can currently be run against Nightly (7.0), Aurora (6.0), Beta (5.0), as well as the current releases (4.0, 3.6, and 3.5).

Below is a short screencast demonstrating how to run the tests:

With the recent release of Selenium IDE 1.0.11, I was able to push some new tests. These check a few more commands, and brings the total number of tests up to 40. If you’re interested in helping out and you have any questions, then you can either get in touch with me directly, ask in the #selenium IRC channel on Freenode, or post a message to the selenium-developers Google group.

Running endurance tests with Mozmill Crowd

Ahead of our Mozmill Crowd testday last Friday we made some changes to the endurance tests, including enabling endurance test run within Mozmill Crowd! Running the endurance tests is now even easier – simply install the Mozmill Crowd extension, and in just a few clicks the tests will be running. We also updated the endurance dashboard reports and pushed them to our Mozmill Crowd report server.

I’ve created a short screencast that demonstrates installing Mozmill Crowd, running the endurance tests, and reviewing the results:

If you want to run with add-ons installed then you’ll still need to use the command line for now (support in Mozmill Crowd is planned).

It’s also important to note that delay is now specified in seconds, and not in milliseconds.

Endurance Results from Test Day

Today I finally finished reviewing the hundreds (yes, hundreds!) of endurance reports that were submitted on our Firefox 4 add-ons test day last Friday and on the days following. It was amazing to see so many reports coming in, and I would like to thank everyone that ran an endurance test run. By far the most active contributor was pxbuz, to whom I’m extra grateful!

Of all of the test runs, it turns out there are three major issues discovered:

The first is that we really need to improve the reporting system. Going through the results was a long and tedious job, so I will be thinking about how I can improve that experience.

Secondly, we need to come up with a way to dismiss any modal dialogs that add-ons might show on first run. There were a couple of these that resulted in what looked like memory leaks, but turns out would be impossible to replicate manually.

I saved the best for last – we found a memory leak when the Greasemonkey add-on is installed! It seems that when entering/leaving private browsing mode there is memory allocated but not released. A bug has been raised and hopefully it’ll soon be resolved. Greasemonkey is one of our most popular add-ons with a current average active daily usage of over 2.5 million users!

Below you can see how the memory leak was spotted. On the left is an example of an endurance test without any add-ons installed, and on the right is a test run with Greasemonkey installed. Those five spikes that start around the 500 checkpoints mark occur during the private browsing test.

You can see the actual reports here and here.

Automated Firefox tests with add-ons installed

Mozmill has a feature that allows the tester to install an add-on during the test run. Until recently this was only used by one of our automated testruns, which was for specifically testing the installed add-on rather than Firefox.

With the recent development on the endurance tests project, it has been necessary to take more advantage of this feature. A bug was reported where if Adblock Plus – our most popular add-on – is installed memory usage increased rapidly when navigating a web page, and none of the memory was being released. To start investigating this I created a very basic (and specific) test for the site mentioned in the bug report, and then simply hacked something together based on the existing add-ons test run. A short time later, the need to run the endurance tests with multiple add-ons installed came up, so I hacked some more to get that in place too. Rather than keep these hacks around, it made sense to allow testers to specify add-ons to be installed during any of our testruns, so I started to work on the necessary patches.

As a result, testers can now run any of our automation scripts with one or more add-ons installed by simply specifying the addons command line parameter. To install multiple add-ons simply repeat the parameter. The argument can either be a path on your machine, or a web/ftp server. In the latter case the add-on will be downloaded to a temporary location before the testrun and removed at the end. The latest version of Mozmill (1.5.2) also now disables the compatibility check, meaning that we can run tests with add-ons that are not marked as compatible with the version of Firefox in use.

An example of running the endurance tests with two add-ons installed:

./testrun_endurance.py --addons=https://addons.mozilla.org/firefox/downloads/latest/748/addon-748-latest.xpi --addons=/Users/dave/Downloads/noscript.xpi --delay=1000 --iterations=10 /Applications/Firefox.app/

Introducing Firefox Endurance Testing

Since late last year I have been working on a prototype of an Endurance Testing project for Firefox. The idea is to use our existing Mozmill framework for automating UI testing of Firefox to write tests that stress and strain the browser over time. I’ve heard many times from people that Firefox needs to be restarted once in a while because it’s become sluggish, and indeed I’ve experienced this myself. The problem is that there are rarely clear steps to reproduce this issues as they normally are an accumulation of many actions over an extended period of time. What actions cause a degradation in performance, and why? This is what we hope to discover with the endurance tests project.

The initial implementation of endurance tests is rather simple: Create a test snippet that exercises a function of Firefox, and execute it repeatedly whilst gathering details of resources in use. Ultimately we may come up with more elaborate tests, and but it’s important to get a proof of concept.

There are several components to the endurance tests:

  1. Command line automation script
  2. Test snippets
  3. Resource gathering
  4. Reporting

The number of iterations each snippet repeats can be set on the command line, as well as an optional delay between each iteration. The normal command line options allow for logging, and reporting to a Mozmill Dashboard instance.

Triggering the endurance tests currently looks something like this:

./testrun_endurance.py --delay=1000 --iterations=50 --report=http://davehunt.couchone.com/mozmill /Applications/Minefield.app

This will then launch Firefox, run through all of the endurance tests (each one iterating over it’s test snippet 50 times), and then close Firefox. Because I’ve included a report parameter, the report will also be sent to the Mozmill Dashboard instance. These reports are currently available here.

Here’s a short screencast that demonstrates running the endurance tests:

If you’re interested in following the progress of the endurance tests project, check out the project page or the tracking bug for phase one.

Mozilla in London for Selenium Meetup #3

On Wednesday, the third London Selenium meetup went ahead. Attendance was good considering there was a London Underground strike in action, however it had clearly made an impact. I arrived at midday and enjoyed a chilled out afternoon at Google’s offices – much more relaxed than my previous visit as I wasn’t actually presenting this time!

The event started with a presentation on how Mozilla uses Selenium to test their sites. Stephen Donner described the infrastructure and technologies in place, and explained the benefits of a page object approach to testing.

This was followed by a rather anticipated update from Simon Stewart on the progress of Selenium 2 (beta soon!) We were also fortunate enough to have Jason (Mr Selenium) Huggins at the event, and he even stepped up to answer some questions on the advantages of moving to Selenium 2.

David Burns then demonstrated a much updated version of his GTAC 2009 presentation on automating gathering client side performance metrics – now using Selenium 2! There was also a peek into using the bleeding edge WebTimings API.

In my personal favourite presentation of the evening (probably because it’s my new area of work), Henrik Skupin gave a demonstration of how Mozilla are approaching crowdsourcing their test automation with the upcoming MozMill Crowd Testing addon for Firefox. There were definitely a few ‘wow’s from the audience when Henrik ran the tests.

Something I found very encouraging was the quality of questions coming from the audience. I’d like to thank those that came, it’s great to get everyone together, and really great to recognise people from previous events!

This was also the first event where the #ldnse hashtag was actively used on Twitter, which is also encouraging. After LDNSE2 I was considering trying to find someone else to continue the events, but I’m glad I didn’t as I’m now really looking forward to organising LDNSE4! Our frequency at the moment is about one every six months, so I’ll be looking to at least keep this regular.

Thanks again to everyone for coming, to all of our presenters and contributors, and to Google for hosting again!

Update: slides/videos/blogs available below.

YouTube channel (all the videos):
http://www.youtube.com/user/londonselenium

How Mozilla Uses Selenium – Stephen Donner
Slides: http://www.slideshare.net/stephendonner/selenium-londonmeetup-5671730
Video (Part 1): http://www.youtube.com/watch?v=Kvd_TIxLziI
Video (Part 2): http://www.youtube.com/watch?v=ATtXDuUlt9Q

Update on Selenium 2 – Simon Stewart & Jason Huggins
Video (Part 1): http://www.youtube.com/watch?v=AYJMct82YXg
Video (Part 2): http://www.youtube.com/watch?v=HYSJUSI3_VU

Client-side profiling with Selenium 2 – David Burns
Slides: http://prezi.com/dgqpq7bywuin/client-side-profiling-with-selenium-2/
Video (Part 1): http://www.youtube.com/watch?v=2TSJHJfbOHE
Video (Part 2): http://www.youtube.com/watch?v=NrvN8HwmpQ4

Crowd-sourcing Automated Firefox UI Testing – Henrik Skupin
Slides: http://www.slideshare.net/hskupin/crowdsourced-automated-firefox-ui-testing
Video (Part 1): http://www.youtube.com/watch?v=O8NaG07NoLc
Video (Part 2): http://www.youtube.com/watch?v=TIfH5Bku20U
Blog: http://www.hskupin.info/2010/11/19/mozmill-crowd-talk-at-selenium-meetup-3-in-london/