QA Automation Services Work Week 2011 – Day 1

QAASWW11 kicked off yesterday with a day of planning at IdeaSpace in Cambridge, UK. We had a meeting room for the day – kindly offered up by our new friends at Springboard – and plenty of instant whiteboard! As with the last work week I attended, it was organised UnCon style, which worked really well before. I will say that the first day is usually the most painful, as the entire team filled out their thoughts/needs for the week onto post-it notes, which gradually were organised into groups and ultimately sessions with agendas. Once this was finally done, the schedule for the week was set out. Although the schedule is incredibly flexible, it really helps to have this set of intentions outlined on the first day.

In the evening, the Springboard teams were kind enough to practice their investor pitches on us, and we saw 10 very promising ideas presented by an incredibly smart and enthusiastic bunch of people. I noticed a very strong lean towards mobile devices in the pitches, which really reflects the current state of the market and the direction things are heading.

After the pitches, everybody relaxed with will deserved beer and pizza, and we had an opportunity to talk with the Springboard guys, who are in their last week.

So ends the first day. Tomorrow we will be working from a cottage, which unfortunately we have already discovered has slow connection to the Internet. This won’t affect our work week sessions, but will be an obstacle during the time we have scheduled to get on with our day-to-day work activities, as well as communicating with our colleagues and community around the world! If you want to follow our activities for the week, we have created a Twitter hashtag of #mozautoqa.

Good luck to everyone at Springboard for the investor pitches on Friday, and thank you so much for inviting us to spend the day with you at IdeaSpace!

Endurance tests demonstrate Firefox’s memory usage improvements

Thanks to the amazing efforts of the MemShrink project, Firefox’s memory usage is seeing some great improvements. In particular, Firefox 7 will be much more efficient with memory than the current version. As endurance tests monitor resources such as memory, it makes sense for us to work together to ensure that we’re moving in the right direction, and that we don’t regress in any of these areas.

At this point there are only five endurance tests, and although these can be run with many hundreds of iterations in order to seek out memory leaks in the tested areas, they do nothing to simulate a user. It was suggested that we have a special endurance test similar to Stuart Parmenter’s Mem Buster test.

Creating an initial version of this new test did not take long. Instead of opening sites in new windows I open them in tabs, and the number of sites opened is controlled by iterations and micro-iterations. I also increased the number of sites so we’d be hitting the same ones less often, and based this new list on Alexa’s top sites. Once I added in handling of modal dialogs that some sites were causing to be displayed then I was able to consistently get results.

This test would appear to be similar to Talos tp5 in that is loads sites from Alexa’s index, however we’re not measuring how long each site takes to load. Instead, we move onto the next site after a delay as specified on the command line. I have kept the same delay as the original Mem Buster test, which is 3 seconds.

After running the Mem Buster Endurance Test five times across five versions of Firefox, I found the results to clearly reflect the MemShrink efforts. Although the memory consumption varies somewhat for each run, the general downward trend is unmistakable.

In the following charts you can see the improvement in memory usage between Firefox 4 & 5. These can be directly compared as the endurance tests were measuring the same metrics (allocated memory & mapped memory).

Charts showing allocated and mapped memory usage in Firefox 4 & 5

In Firefox 6 there were several improvements to memory reporting, and the endurance tests were updated to record new metrics (explicit memory & resident memory). You can see in the following charts that explicit memory usage in Firefox 7 is rough half that of Firefox 6! It appears that this has increased in Firefox 8, which will require some further investigation. The resident memory has continued to decrease in each version.

Charts showing explicit and resident memory usage in Firefox 6, 7, & 8

You can follow the progress of the Mem Buster Endurance Test in Bugzilla. Full reports from the test runs used in this blog post can be found here.

Update: It appears that the explicit memory calculated for Firefox 7 on Mac was artificially low. This explains the slight increase in Firefox 8. If you’re interested you can read further details on Bugzilla.

Micro-iterations in Endurance Tests

Last week micro-iterations landed in Mozmill Endurance Tests. These allow tests to accumulate resources during an iteration. This was previously achieved by leaving the state of the test snippet in a different state to how it started, allowing the iterations themselves to accumulate. The problem with this is that these accumulating tests have a very different pattern compared to other tests that clean up before ending the iteration.

To solve this we decided to add a micro-iteration parameter and to use it to loop within an iteration. An example use for this is the new tab tests. Now, if you specify 5 iterations and 10 micro-iterations then these tests will open 10 new tabs, close them, and repeat that 5 times.

The endurance tests documentation has been updated with details on writing and running tests with micro-iterations.