Endurance tests demonstrate Firefox’s memory usage improvements

Thanks to the amazing efforts of the MemShrink project, Firefox’s memory usage is seeing some great improvements. In particular, Firefox 7 will be much more efficient with memory than the current version. As endurance tests monitor resources such as memory, it makes sense for us to work together to ensure that we’re moving in the right direction, and that we don’t regress in any of these areas.

At this point there are only five endurance tests, and although these can be run with many hundreds of iterations in order to seek out memory leaks in the tested areas, they do nothing to simulate a user. It was suggested that we have a special endurance test similar to Stuart Parmenter’s Mem Buster test.

Creating an initial version of this new test did not take long. Instead of opening sites in new windows I open them in tabs, and the number of sites opened is controlled by iterations and micro-iterations. I also increased the number of sites so we’d be hitting the same ones less often, and based this new list on Alexa’s top sites. Once I added in handling of modal dialogs that some sites were causing to be displayed then I was able to consistently get results.

This test would appear to be similar to Talos tp5 in that is loads sites from Alexa’s index, however we’re not measuring how long each site takes to load. Instead, we move onto the next site after a delay as specified on the command line. I have kept the same delay as the original Mem Buster test, which is 3 seconds.

After running the Mem Buster Endurance Test five times across five versions of Firefox, I found the results to clearly reflect the MemShrink efforts. Although the memory consumption varies somewhat for each run, the general downward trend is unmistakable.

In the following charts you can see the improvement in memory usage between Firefox 4 & 5. These can be directly compared as the endurance tests were measuring the same metrics (allocated memory & mapped memory).

Charts showing allocated and mapped memory usage in Firefox 4 & 5

In Firefox 6 there were several improvements to memory reporting, and the endurance tests were updated to record new metrics (explicit memory & resident memory). You can see in the following charts that explicit memory usage in Firefox 7 is rough half that of Firefox 6! It appears that this has increased in Firefox 8, which will require some further investigation. The resident memory has continued to decrease in each version.

Charts showing explicit and resident memory usage in Firefox 6, 7, & 8

You can follow the progress of the Mem Buster Endurance Test in Bugzilla. Full reports from the test runs used in this blog post can be found here.

Update: It appears that the explicit memory calculated for Firefox 7 on Mac was artificially low. This explains the slight increase in Firefox 8. If you’re interested you can read further details on Bugzilla.

This entry was posted in Mozilla, QMO and tagged , , , , . Bookmark the permalink.

24 Responses to Endurance tests demonstrate Firefox’s memory usage improvements

  1. Roberto says:

    Hi, Dave, I’m Roberto Alvarado I’m working for Mozilla Mexico, Can I translate your article in spanish for our web? Thanks.

    • Dave says:

      Hi Roberto, thanks for commenting! Feel free to translate whatever you like from my blog. Let me know if there’s anything you need. Thanks.

  2. Caspy7 says:

    Thank you for this.
    Would enjoy a comparison of 5 & 7 as this would be a nice demo (stark contrast) of the current reality vs what’s coming.

    • Dave says:

      Unfortunately it’s not easy to compare Firefox 5 with Firefox 7 using endurance tests. In Firefox 6 the memory reporters have been vastly improved, and as a result the metrics we gather changed from allocated/mapped memory to explicit/resident memory. It would not be fair to compare these different metrics, so having a 5->7 comparison is not possible – at least with these tests.

      • You could use an external tool, such as the task manager built into the OS, to make measurements across versions.

        Which OS did you use, by the way?

        • Dave says:

          Hi Jesse,

          The endurance tests don’t currently use external tools to gather metrics, but it’s certainly a possibility for the future.

          The results shown in this blog post were all using Mac OS X 10.6.8.

          Thanks,
          Dave

  3. Nick Tulett says:

    Is 3 seconds long enough to use all the necessary memory? Some websites are quite badly written and might still be executing, rendering or fetching at that point.

    • Dave says:

      Hi Nick,

      3 seconds is how long the test waits before opening a new site, but that’s always in another tab. There is a maximum of 30 tabs defined for the test, so each tab is open for around 90 seconds. This is based on the original Mem Buster and can certainly be tweaked for better results.

      Dave.

  4. Jigar Shah says:

    Hi,
    I am not very power user (though i am a software developer) but i can see memory usage in Task / Performance Manager of Ubuntu/Windows that FF keeps increasing its memory consumption even when i am not using it. Please let me know what additional information i can provide to your team such that you can investigate. In most of the cases i have to kill FF and restart again. And this happens consistently if i keep browser open for say a day or 2.

  5. Bernhard says:

    It would have been nice to include Firefox 3.6 memory usage as well. I have just upgraded from FF3.6 to FF7a (Aurora) and FF7a uses way more memory than FF3.6. In fact, just starting FF7a with all my saved windows and tabs brings FF7a close to the crash limit of 2GB Virtual Size. The Working set is about 400MB above FF3.6 (1300MB versus 900MB). This is an almost 50% increase in memory consumption going from FF3.6 to FF7a. I chose FF7a because it should have the least memory usage of the FF5/6/7 but I see myself reverting to FF3.6 in the near future again. :-/

    • Dave says:

      Unfortunately the endurance tests have been written to work with Firefox 4 and above. If you have some examples where you can replicate higher memory usage compared to 3.6 then please raise a bug with as much detail as possible. Thanks.

      • Bernhard says:

        I would, but all I have is my current usage which includes several windows and *many* tabs. And as this is a personal list of sites I visit, I prefer not to share them to the development team. But Firefox 7.0a2 uses about 45-50% more memory for me than Firefox 3.6.18.

        While it seem exessive, I have about 6 windows and about 150 tabs. Firefox 3.6.18 kept that in memory with about 900MB-1.0GB Working Set (about 1.3-1.5GB Virtual Size) whereas Firefox 7.0a2 uses 1.4GB Working Set and about 1.8-1.9GB Virtual Size (i.e. _very_ close to the crash limit of a 32bit application).

        I could use Firefox 3.6.18 for weeks before having to restart my pc but Firefox 7.0a2 I have to restart daily due to memory usage.

        However, not all is bad with FF7.0a: The UI doesn’t freeze as long as it does with 3.6.18 that could freeze for up to 2-3 seconds before showing any interaction (everything freezed, including flash and java). I don’t know why FF3.6.18 causes these freezes (happens every 10-20 seconds) but I suspect it is javascript related (got a few tabs like gmail polling for email). If it weren’t for the freeze issues, I wouldn’t consider FF4+ at all unless memory usage was kept low.

        • Bernhard says:

          Ok, now I have hope for FF7. :) Just upgraded from 7.0a2 to 7.0a2 (daily builds. :-P ) and now FF7 memory usage is approximately the same as with FF3.6 :) I guess I hit a bad FF7 release.

          I hope it stays like this. :)

          • Bernhard says:

            Well, Working Set is no 1.3GB but Virtual Size has reached 1.95GB and FF7.0a2 is now unstable. It has only run 1 day. I guess memory fragmentation in FF7.0a2 still is a big issue compared to FF3.6.

          • Bernhard says:

            I have now switched to FF9.0a1 64-bit build. FF is now stable but with even larger memory usage: Working Set 1.3GB, Virtual Size 2.6GB, Private 1.8GB. I don’t think Mozilla knows how to bring the memory usage down to FF3.6 values.

  6. Pingback: Firefox 7: Drastisch schneller durch weniger Speicherhunger » t3n News

  7. Pingback: Firefox 7 will use up to 50% less memory…what about google chrome? « jimboski36tv

  8. Pingback: MemShrink: Firefox 7 usará un 50% de memoria

  9. Pingback: Mozilla | Proyecto MemShrink para la Reducción de memoria en Firefox

  10. Pingback: Anonymous

  11. Pingback: Firefox 6 released: Bliss for Javascript and web developers « salorenz

  12. Pingback: Running the ‘Mem Buster’ endurance test | Blargon7

  13. Pingback: Firefox 7 is lean and fast ✩ Mozilla Hacks – the Web developer blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">