Selenium tests with pytest

When you think of Mozilla you most likely first associate it with Firefox or our mission to build a better internet. You may not think we have many websites of our own, beyond perhaps the one where you can download our products. It’s only when you start listing them that you realise how many we actually have; addons repository, product support, app marketplace, build results, crash statistics, community directory, contributor tasks, technical documentation, and that’s just a few! Each of these have a suite of automated functional tests that simulate a user interacting with their browser. For most of these we’re using Python and the pytest harness. Our framework has evolved over time, and this year there have been a few exciting changes.

Over four years ago we developed and released a plugin for pytest that removed a lot of duplicate code from across our suites. This plugin did several things; it handled starting a Selenium browser, passing credentials for tests to use, and generating a HTML report. As it didn’t just do one job, it was rather difficult to name. In the end we picked pytest-mozwebqa because it was only specific in addressing the needs of the Web QA team at Mozilla. It really took us to a new level of consistency and quality across all our our web automation projects.

Enhanced HTML report generated by pytest-htmlThis year, when I officially joined the Web QA team, I started working on breaking the plugin up into smaller plugins, each with a single purpose. The first to be released was the HTML report generation (pytest-html), which generates a single file report as an alternative to the existing JUnit report or console output. The plugin was written such that the report can be enhanced by other plugins, which ultimately allows us to include screenshots and other useful things in the report.

Next up was the variables injection (pytest-variables). This was needed primarily because we have tests that require an existing user account in the application under test. We couldn’t simply hard-code these credentials into our tests, because our tests are open source, and if we exposed these credentials someone may be able to use them and adversely affect our test results. With this plugin we are able to store our credentials in a private JSON file that can be simply referenced from the command line.

The final plugin was for browser provisioning (pytest-selenium). This started as a fork of the original plugin because much of the code already existed. There were a number of improvements, such as providing direct access to the Selenium object in tests, and avoiding setting a default implicit wait. In addition to supporting Sauce Labs, we also added support for BrowserStack and TestingBot.

Now that pytest-selenium has been released, we have started to migrate our own projects away from pytest-mozwebqa. The migration is relatively painless, but does involve changes to tests. If you’re a user of pytest-mozwebqa you can check out a few examples of the migration. There will no longer be any releases of pytest-mozwebqa and I will soon be marking this project as deprecated.

The most rewarding consequence of breaking up the plugins is that we’ve already seen individual contributors adopting and submitting patches. If you’re using any of these plugins let us know – I always love hearing how and where our tools are used!

Joining Web QA

Dylan with origami foxI’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!