Python testing sprint 2016

In June, the pytest developer community are gathering in Freiburg, Germany for a development sprint. This is being funded via an Indiegogo campaign, which needs your help to reach the goal! I am excited to say that I will be attending, which means that after over 5 years of using pytest, I’ll finally get to meet some of the core contributors.

I first learned about pytest when I joined Mozilla in late 2010. Much of the browser based automation at that time was either using Selenium IDE or Python’s unittest. There was a need to simplify much of the Python code, and to standardise across the various suites. One important requirement was the generation of JUnit XML reports (considered essential for reporting results in Jenkins) without compromising the ability to run tests in parallel. Initially we looked into nose, but there was an issue with this exact requirement. Fortunately, pytest didn’t have a problem with this – JUnit XML was supported in core and was compatible with the pytest-xdist plugin for running tests in parallel.

Ever since the decision to use pytest was made, I have not seen a compelling reason to switch away. I’ve worked on various projects, some with overly complex suites based on unittest, and I’ve always been grateful when I’ve been able to return to pytest. The active development of pytest has meant we’ve never had to worry about the project becoming unsupported. I’ve also always found the core contributors to be extremely friendly and helpful on IRC (#pylib on whenever I need help. I’ve also more recently been following the pytest-dev mailing list.

I’ve recently written about the various plugins that we’ve released, which have allowed us to considerably reduce the amount of duplication between our various automation suites. This is even more critical as the Web QA team shifts some of the responsibility and ownership of some of their suites to the developers. This means we can continue to enhance the plugins and benefit all of the users at once, and our users are not limited to teams at Mozilla. The pytest user base is large, and that means our plugins are discovered and used by many. I always love hearing from users, especially when they submit their own enhancements to our plugins!

There are a few features I particularly like in pytest. Highest on the list is probably fixtures, which can really simplify setup and teardown, whilst keeping the codebase very clean. I also like being able to mark tests and use this to influence the collection of tests. One I find myself using a lot is a ‘smoke’ or ‘sanity’ marker, which collects a subset of the tests for when you can’t afford to run the entire suite.

During the sprint in June, I’d like to spend some time improving our plugins. In particular I hope to learn better ways to write tests for plugins. I’m not sure how much I’ll be able to help with the core pytest development, but I do have my own wishlist for improvements. This includes the following:

Maybe I’ll even be able to work on one of these, or any of the open issues on pytest with guidance from the experts in the room.

Selenium tests with pytest

When you think of Mozilla you most likely first associate it with Firefox or our mission to build a better internet. You may not think we have many websites of our own, beyond perhaps the one where you can download our products. It’s only when you start listing them that you realise how many we actually have; addons repository, product support, app marketplace, build results, crash statistics, community directory, contributor tasks, technical documentation, and that’s just a few! Each of these have a suite of automated functional tests that simulate a user interacting with their browser. For most of these we’re using Python and the pytest harness. Our framework has evolved over time, and this year there have been a few exciting changes.

Over four years ago we developed and released a plugin for pytest that removed a lot of duplicate code from across our suites. This plugin did several things; it handled starting a Selenium browser, passing credentials for tests to use, and generating a HTML report. As it didn’t just do one job, it was rather difficult to name. In the end we picked pytest-mozwebqa because it was only specific in addressing the needs of the Web QA team at Mozilla. It really took us to a new level of consistency and quality across all our our web automation projects.

Enhanced HTML report generated by pytest-htmlThis year, when I officially joined the Web QA team, I started working on breaking the plugin up into smaller plugins, each with a single purpose. The first to be released was the HTML report generation (pytest-html), which generates a single file report as an alternative to the existing JUnit report or console output. The plugin was written such that the report can be enhanced by other plugins, which ultimately allows us to include screenshots and other useful things in the report.

Next up was the variables injection (pytest-variables). This was needed primarily because we have tests that require an existing user account in the application under test. We couldn’t simply hard-code these credentials into our tests, because our tests are open source, and if we exposed these credentials someone may be able to use them and adversely affect our test results. With this plugin we are able to store our credentials in a private JSON file that can be simply referenced from the command line.

The final plugin was for browser provisioning (pytest-selenium). This started as a fork of the original plugin because much of the code already existed. There were a number of improvements, such as providing direct access to the Selenium object in tests, and avoiding setting a default implicit wait. In addition to supporting Sauce Labs, we also added support for BrowserStack and TestingBot.

Now that pytest-selenium has been released, we have started to migrate our own projects away from pytest-mozwebqa. The migration is relatively painless, but does involve changes to tests. If you’re a user of pytest-mozwebqa you can check out a few examples of the migration. There will no longer be any releases of pytest-mozwebqa and I will soon be marking this project as deprecated.

The most rewarding consequence of breaking up the plugins is that we’ve already seen individual contributors adopting and submitting patches. If you’re using any of these plugins let us know – I always love hearing how and where our tools are used!

Custom Firefox Cufflinks

If you’re interested in a pair of custom Firefox logo cufflinks for $25 plus postage then please get in touch. I’ve been in contact with, and if I can place an initial order of at least 25 pairs then the mold and tooling fee will be waived. Check out their website for examples of their work. The Firefox cufflinks would be contoured to the outline of the logo, and the logo itself would be screen printed in full colour. If these go well, I’d also love a pair of dino head cufflinks to complement them!

Unless I can organise with to ship to multiple recipients and take payments separately, I will accept payments via Paypal and take care of sending the cufflinks out. As an example of postage costs, sending to USA would cost me approximately $12 per parcel. Let me know if you’re interested via a comment on this post, e-mail (find me in the phonebook), or IRC (:davehunt).

Whistler Work Week 2015

Last week was Mozilla’s first work week of 2015 in Whistler, BC. It was my first visit to Whistler having joined Mozilla shortly after their last summit there in 2010, and it was everything I needed it to be. Despite currently feeling jetlagged, I have been recharged and I have renewed enthusiasm for the mission and I’m even more than a little excited about Firefox OS again! I’d like to share a few of my highlights from last week…

  • S’mores – The dinner on Wednesday evening was followed by a street party, where I had my first S’more. It was awesome!
  • Firefox OS – Refreshing honesty over past mistakes and a coherant vision for the future has actually made me enthusiastic about this project again. I’m no longer working directly on Firefox OS, but I’ve signed up for the dogfoxfooding program and I’m excited about making a difference again.
  • LEGO – I got to build a LEGO duck, and we heard from David Robertson about the lessons LEGO learned from near bankruptcy.
  • Milkshake – A Firefox Q&A was made infinitely better by taking a spontaneous walk to Cow’s for milkshakes and ice cream with my new team!
  • Running – I got to run with #running friends old and new on Tuesday morning around Lost Lake. Then on Thursday morning I headed back and took on the trails with Matt. These were my first runs since my marathon, and running through the beautiful scenary was exactly what I needed to get me back into it.
  • Istanbul – After dinner on Tuesday night, Stephen and I sat down with Bob to play the board game Istanbul.
  • Hacking – It’s always hard to get actual code written during these team events, but I’m pleased to say we thought through some challenging problems, and actually even managed to land some code.
  • Hike – On Friday morning I joined Justin and Matt on a short hike up Whistler mountain. We didn’t have long before breakfast, but it was great to spend more time with these guys.
  • Whistler Mountain – The final party was at the top of Whistler Mountain, which was just breathtaking. I can’t possibly do the experience justice – so I’m not even going to try.

Thank you Whistler for putting up with a thousand Mozillians, and thank you Mozilla for organising such a perfect week. We’re going to keep rocking the free web!

Rock ‘n’ Roll Liverpool Marathon 2015

On Sunday I completed my first ever marathon, and I did it in under 4 hours! It’s only just sinking in what an achievement that is! When I signed up for it last year I predicted a finish time of under 5 hours, but after following a sub 4 training program I started feeling that I could actually do it.

I don’t have any other marathon experience to compare it to, but it was an incredible event. I’d like to thank everybody that helped to organise or came out to support the runners – even those saying we were nearly there before we hit the halfway point, or saying we were halfway at the 18 mile marker! There were times the support really helped – In particular those last 4 windy miles along the river. A special thanks go to my family and friends that cheered me at the start, at the 10 mile marker, and at the end. Also, thanks to my family for putting up with all those long runs during my training.

Inevitably I’ve already been asked about my next marathon, but I’m going to allow myself to recover before I give it too much thought. I have already entered the ballot for next year’s London marathon though, so who knows…

Joining Web QA

Dylan with origami foxI’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!

Performance testing Firefox OS on reference devices

A while back I wrote about the LEGO harness I created for Eideticker to hold both the device and camera in place. Since then there has been a couple of iterations of the harness. When we started testing against our low-cost prototype device, the harness needed modifying due to the size difference and position of the USB socket. At this point I tried to create a harness that would fit all of our current devices, with the hope of avoiding another redesign.

Eideticker harness v2.0 If you’re interested in creating one of these yourself, here’s the LEGO Digital Designer file and building guide.

Unfortunately, when I first got my hands on our reference device (codenamed ‘Flame’) it didn’t fit into the harness. I had to go back to the drawing board, and needed to be a little more creative due to the width not matching up too well with the dimensions of LEGO bricks. In the end I used some slope bricks (often used for roof tiles) to hold the device securely. A timelapse video of constructing the latest harness follows.



We now are 100% focused on testing against our reference device, so in London we have two dedicated to running our Eideticker tests, as shown in the photo below.

Eideticker harness for FlameAgain, if you want to build one of these for yourself, download the LEGO Digital Designer file and building guide. If you want to learn more about the Eideticker project check out the project page, or if you want to see the dashboard with the latest results, you can find it here.

A new home for the gaiatest documentation

The gaiatest python package provides a test framework and runner for testing Gaia (the user interface for Firefox OS). It also provides a handy command line tool and can be used as a dependency from other packages that need to interact with Firefox OS.

Documentation for this package has now been moved to, which is generated directly from the source code whenever there’s an update. In order to make this more useful we will continue to add documentation to the Python source code. If you’re interested in helping us out please get in touch by leaving a comment, or joining #ateam on and letting us know.

Giving up and getting fit

Last July, on my 31st birthday I removed coffee from my diet. On the same day of the month since then I have removed something new from my diet, and pretty soon I will have completed a year of this. I didn’t keep a diary, but I’ve been considering for a while that I should write about my experience, and how it ultimately led to me getting back into fitness and losing over 40lbs.

It started on the eve of my birthday when I decided to go out and pick up some fast food for dinner. I guess it was meant to be a ‘treat’ although it’s rarely worth the effort and cost. In this case I put it down as the reason I felt terribly sick on my birthday, and as a result I completely lost my appetite for a few days. In these days one of the things I didn’t consume was coffee – I had essentially detoxed and removed caffeine from my system. Before this I was probably only really having 2-3 cups a day, so it’s not like I had a really bad addiction. I then decided it would be worth seeing if I can last out a full month without coffee.

It was a little surprising to me how easy it was to just cut one thing out of my diet, and I have to confess that it wasn’t too long before I’d substituted the caffeine deficit from tea or soft drinks. Towards the end of this first month I had already started to think about how else I could experiment with my diet. I’d already gone a month without coffee without much effort, so why should I start drinking it again now? I decided that I’d find something else in my diet that has a perceived negative impact on health and eliminate it in addition to coffee.

So my second month I gave up chocolate, and in my third I gave up alcohol. I had already recognised a pattern of substitution, so rather than give up beer (and probably end up substituting wine or other alcoholic drinks) I decided to just remove all alcohol from my diet. In my fourth month I gave up pizza, and in my fifth and sixth months I gave up crisps (potato chips) and biscuits (cookies). All the while I had been regularly weighing myself and measuring my body fat percentage, and although I wasn’t expecting to see a weight change I was curious to see if there would be an impact. I saw that my body fat had decreased by about 10%, but my weight had increased slightly.

Having been conscious of my weight for a while, I decided at New Year to do something about it. I was at the high end of the ‘overweight’ range according to the body mass index, so I set myself the goal of losing enough to get myself into the ‘normal’ range, which was around 40lbs. It was obvious that my dietary experimentation was not causing me to lose weight (and I hadn’t expected it to) so it was going to take something else to help me reach my goal. What better than a fad diet?

I had some success in the past with the 5:2 diet, where you essentially eat what you want on five days of the week, and on two non-consecutive days you fast. I picked Monday and Wednesday as my ‘fast days’ and decided that rather than continuously calorie count on these days I’d just work out a couple of low calorie meals and then eat the same thing every week. So for the last six months I’ve eaten omelette and stir-fry on Mondays, and bircher muesli and fish with roasted vegetables on Wednesdays. These are all things I like, so it wasn’t too hard, and the great thing about this diet is that when you’re fasting, you can always eat whatever you want the very next day.

I continued to cut things out of my diet too, so on my seventh month I gave up ice cream. Around the time I gave up sweets (candy) for my eighth month I had lost about 10lbs, but I could already see that my weight loss was slowing. This is when I decided to go running for at least 30 minutes, three times a week. Before long I had explored some great areas to run nearby, the weight kept coming off, and I was steadily improving my pace.

For my ninth month I gave up fizzy drinks (soda), and for my tenth month I gave up chips (fries). I had decided early on to not give up things that are common ingredients such as bread or cheese, as that would be too difficult to constantly remember and check for. Everything I gave up was really easy to avoid, although I did get some strange looks when ordering a burger with no fries.

With just two months left I wanted to really challenge myself. I had given up coffee, but whenever that came up in conversation I was inevitably asked if I drink tea. So this became my eleventh item, which again surprised me how easy it was to give up. I now drink a lot more water than I ever used to, and taking away the choice of what to drink has been somewhat liberating. For my last month it was obvious to me what I needed to give up: cakes!

Probably because I hadn’t already excluded it, I was eating a lot of cake. My wife likes to bake, and so there’s often something in, and I had got into the habit of eating them on most of my non-fasting days. It didn’t feel right that I should have a year of purging bad foods from my diet knowing that I had continued all the while to eat cake. So this last month, I have not been eating cake, and it wasn’t that hard!

This week I met my weight target of 168lbs. Next week is my 32nd birthday, and I’m taking the family out for pizza. Of all the things I’ve given up, I’ve missed pizza the most.

To track my weight loss I used Fitbit Aria scales and TrendWeight. For tracking activity I used Fitbit Flex, my iPhone 5, and Zombies, Run! which I’ve synced to RunKeeper and Strava.

The full list of foods I excluded from my diet each month is: coffee, chocolate, alcohol, pizza, crisps (potato chips), biscuits (cookies), ice cream, sweets (candy), fizzy drinks (soda), chips (fries), tea, cake. From next week I’ll be reintroducing most of these into my diet gradually, and in moderation.