Effective Functional Web Tests
• Julian Simpson
Douglas Squirrel just posted about the issues he's having with slow functional tests. My last big project was a large website that used Selenium for functional testing. I thought I'd share what worked:
- Make damn sure that Selenium doesn't spawn a new browser. If you use (for example) JUnit tests that call Selenium objects for all your functional tests, make sure you don't spawn a new VM with your tests. That will certainly cause Selenium RC to spawn a new browser. Fixing this dropped the test execution from 30 minutes to 15.
- Don't let the pages render any external objects. Generally you're just trying to prove that your pages render with enough form that you can trace some functionality through them. Displaying ads or any other external component will just get in the way. If you can't cause your pages to render in such a way, you can use a firewall to block those elements. You can't just drop the packets, however. You need to kill the entire TCP session. The fantastic infrastructure architect at this project used the Iptables REJECT commmand to make the browser give up immediately on rendering the external components. This sped up the tests a little and more importantly made them rock solid.
- Make the first failure fail the build. When the functional test run was 30 minutes, you could be left waiting up to an hour to find out if your checkin had caused a test failure. The ah-ha moment came when we realised that we could fail the build immediately the moment a functional test failed. That gave feedback to the rest of the team that the build was broken. We learned to be very disciplined about checking in our code.