Speed up your tests with Hydra

Published on and tagged with cucumber  performance  testing

Nowadays most computers come with more than one processor. And so it makes sense to use the additional processing power to speed up your tests by distributing them across the available processors.

One tool that helps with this is Hydra. It allows you to distribute tests across multiple processors and machines, and currently supports the Test::Unit, Cucumber, and RSpec frameworks. In this article I will focus on running Cucumber tests on a single machine.

As Hydra is distributed as a Ruby Gem its installation is pretty simple:

gem install hydra

After that, you have to add Hydra to the Rakefile of the respective project. And you have to specify the Cucumber task and the files it should use.

# Rakefile
require 'hydra'
require 'hydra/tasks'

Hydra::TestTask.new('hydra:cucumber') do |t|
  t.add_files 'features/**/*.feature'
end

As last step you have to create a “hydra.yml” file in the “config” folder of your project. Here you define whether the tests should run locally, and how many runners Hydra should start. Runners correspond to processors, and so the setting below is for a dual core machine.

# config/hydra.yml
workers:
  - type: local
    runners: 2

With everything set up, you can now run the tests with the following command:

$ RAILS_ENV=test rake hydra:cucumber

To verify whether there is really a speed gain when running the tests in this way you have to do some kind of a benchmark (see the example below), because running something on multiple processors doesn’t necessarily mean it is faster…

Happy testing!
 


For my (unscientific) benchmark I ran my test suite (consisting of 12 files with 43 scenarios and 193 steps) with the following commands five times and averaged the results:

$ time cucumber features
$ time rake cucumber
$ time RAILS_ENV=test rake hydra:cucumber

In the first scenario I ran the benchmark with no other user applications running, i.e. both processors were available for the task at hand. From the results you can see that using Hydra in such a scenario is the fastest solution.

cucumber: 24.1s
cucumber with rake: 31.4s
hydra:cucumber: 22.4s

In the second scenario I ran the benchmark in a typical development environment with many open applications, i.e. around 1.5 processors were available. This time, using Hydra is not the fastest solution, the overhead is bigger than the speed gain from having more than one processor .

cucumber: 25.9s
cucumber with rake: 34.3s
hydra:cucumber: 30.6s

4 comments baked

  • Nick Gauthier

    Hey thank for the shout out on Hydra!

    One thing to note on benchmarks is that Hydra learns about your test suite each time you run it, taking heuristics on your test file run time. So please run Hydra at least a second time to allow it to use its heuristics.

    My guess at this point is that your hard disk is limiting Hydra’s progress, but it’s worth investigating. For more tips on faster testing, check out this presentation:

    http://bit.ly/grease-your-suite-html

    (use arrow keys to navigate)

    -Nick

  • cakebaker

    @Nick: I guess Hydra used its heuristics in my benchmarks, at least there was no big difference in my results.

    Thanks for the link to your presentation!

  • Dima Samodurov

    Thanks! this tends to be very helpful running test suites.

    What about database access, transactions and concurrency?
    What about functional testing using browser (will it open
    multiple browser instances correctly?)

  • cakebaker

    @Dima: In my test runs with a single test database I didn’t encounter any issues with database access, transactions and concurrency. However, others reported some problems with deadlocks, and there is a fork that supports multiple test databases.

    I can’t say anything about functional testing using browsers, because so far I don’t have used this testing approach…

© daniel hofstetter. Licensed under a Creative Commons License