Before making the switch, I recommend to run your Cucumber tests to ensure all tests pass:

$ cucumber features

As usual when adding new dependencies, you have to modify the Gemfile of your app. Replace the Webrat entry with:

gem "capybara", "0.3.9"

and run:

$ bundle update

With Capybara installed, you have to setup Cucumber to use Capybara by running one of the following commands:

// Rails 2.3.9
$ script/generate cucumber --rspec --capybara

// Rails 3.0.0
$ rails generate cucumber:install --rspec --capybara

Now, you are ready to run the tests again. If you are lucky, all tests pass, which means there is nothing more to do. However, I think it is more likely you will get a bunch of errors and maybe even some failing tests.

The errors I got were “undefined method” errors. They are relatively easy to fix by replacing the no longer existing Webrat methods with their Capybara counterparts. In the following code block I list some examples:

# Webrat
field_with_id('openid_identifier').value.should =~ /invalid OpenID/
# Capybara
find_field('openid_identifier').value.should =~ /invalid OpenID/

# Webrat
response.should contain('Previous')
# Capybara
page.should have_content('Previous')

# Webrat
assert_have_selector('.author', :count => 1)
# Capybara
page.should have_css('.author', :count => 1)

# Webrat
assert_have_xpath("//span[@id='#{id}']", :content => expected_count)
# Capybara
page.should have_xpath("//span[@id='#{id}']", :text => expected_count)

The trickiest part to fix was the failing test. To test some “remember me” functionality I manually set a cookie, and this no longer worked with Capybara:

cookies[:remember_me_id] = remember_me_id

After some searching I found a Gist from Nicholas Rutherford in which he dealt with cookies. Thanks to his code I could set my cookie with the following snippet:

cookies = Capybara.current_session.driver.current_session.instance_variable_get(:@rack_mock_session).cookie_jar
cookies[:remember_me_id] = remember_me_id

And with that, my switch to Capybara was complete. Any questions?

One tool that helps with this is Hydra. It allows you to distribute tests across multiple processors and machines, and currently supports the Test::Unit, Cucumber, and RSpec frameworks. In this article I will focus on running Cucumber tests on a single machine.

As Hydra is distributed as a Ruby Gem its installation is pretty simple:

gem install hydra

After that, you have to add Hydra to the Rakefile of the respective project. And you have to specify the Cucumber task and the files it should use.

# Rakefile
require 'hydra'
require 'hydra/tasks'

Hydra::TestTask.new('hydra:cucumber') do |t|
  t.add_files 'features/**/*.feature'
end

As last step you have to create a “hydra.yml” file in the “config” folder of your project. Here you define whether the tests should run locally, and how many runners Hydra should start. Runners correspond to processors, and so the setting below is for a dual core machine.

# config/hydra.yml
workers:
  - type: local
    runners: 2

With everything set up, you can now run the tests with the following command:

$ RAILS_ENV=test rake hydra:cucumber

To verify whether there is really a speed gain when running the tests in this way you have to do some kind of a benchmark (see the example below), because running something on multiple processors doesn’t necessarily mean it is faster…

Happy testing!
 

—
For my (unscientific) benchmark I ran my test suite (consisting of 12 files with 43 scenarios and 193 steps) with the following commands five times and averaged the results:

$ time cucumber features
$ time rake cucumber
$ time RAILS_ENV=test rake hydra:cucumber

In the first scenario I ran the benchmark with no other user applications running, i.e. both processors were available for the task at hand. From the results you can see that using Hydra in such a scenario is the fastest solution.

cucumber: 24.1s
cucumber with rake: 31.4s
hydra:cucumber: 22.4s

In the second scenario I ran the benchmark in a typical development environment with many open applications, i.e. around 1.5 processors were available. This time, using Hydra is not the fastest solution, the overhead is bigger than the speed gain from having more than one processor .

cucumber: 25.9s
cucumber with rake: 34.3s
hydra:cucumber: 30.6s

class BooleanValidatorTest extends CakeTestCase {
    private $validator = null;
	
    public function setUp() {
        $this->validator = new BooleanValidator();
    }
	
    public function testValidate() {
        $this->assertIdentical(true, $this->validator->validate(true));
        $this->assertIdentical(true, $this->validator->validate(false));
        $this->assertTrue(is_string($this->validator->validate(0)));
        $this->assertTrue(is_string($this->validator->validate(1)));
    }
}

The code is quite trivial, though it is not that easy to see what I try to express with this code. And so we have to improve its readability.

For this purpose we will write two custom asserts. Custom asserts are nothing special, they are normal methods. But for some reason it seems like people think they always have to use the assert methods provided by the testing framework.

Let’s have a look at the test method above. What is verified in the first two lines with assertIdentical()? It is simple: it is verified the validation process recognizes the provided data as valid. So, we have the first custom assert method: assertValid().

Now you might think the second custom assert method will be assertInvalid(). This would be a good name if the validation process would return false if a value doesn’t validate. Though, in this specific case a validation error is returned, and so a better name for the second assert method is: assertValidationError().

With this, we can refactor the example from above to:

class BooleanValidatorTest extends CakeTestCase {
    private $validator = null;
	
    public function setUp() {
        $this->validator = new BooleanValidator();
    }
	
    public function testValidate() {
        $this->assertValid($this->validator->validate(true));
        $this->assertValid($this->validator->validate(false));
        $this->assertValidationError($this->validator->validate(0));
        $this->assertValidationError($this->validator->validate(1));
    }

    private function assertValid($validationResult) {
        $this->assertIdentical(true, $validationResult);
    }
	
    private function assertValidationError($validationResult) {
        $this->assertTrue(is_string($validationResult));
    }
}

The result is now much easier to read (at least in my opinion), and as a nice side effect of this refactoring we also have eliminated some duplicate code.

Let’s have a look at a very trivial code example. First the class we want to test:

class Example {
    protected function getHelloWorld() {
        return 'hello world';
    }
}

And now the test for the getHelloWorld() method:

class ExampleTest extends CakeTestCase {
    private $example = null;

    public function setUp() {
        $this->example = new MyExample();
    }

    public function testGetHelloWorld() {
        $this->assertEqual('hello world', $this->example->publicGetHelloWorld());
    }
}

class MyExample extends Example {
    public function publicGetHelloWorld() {
        return $this->getHelloWorld();
    }
}

You can use the same strategy if you have to modify protected properties for your tests.

Happy testing!

PS: If you are a Munich baker you might be interested in the first cakeBar, which takes place on November 20.

You can find additional information about testing controllers in the cookbook.

Happy testing :)

Sometimes you not only want to skip some test methods but an entire test case. For example, the tests of the MySQL datasource test case can only be executed successfully if there is a MySQL database (or else you will get many errors). What do you do in such a case?

You could override the getTests() method and simply return an empty array if there is no MySQL database available:

public function getTests() {
    if (!$this->isMySQLAvailable()) {
        return array();
    }

    return parent::getTests();
}

This works, but it is rather a theoretical approach. Usually you override the method which was designed for this purpose: skip. In this method you use then either the skipIf() or skipUnless() method to determine whether the test case should be skipped. So we can rewrite the example from above in the following way:

public function skip() {
    $this->skipUnless($this->isMySQLAvailable());
    // $this->skipIf(!$this->isMySQLAvailable()); has the same effect
}

If you now run the test case, and you don’t have a MySQL database, you will get informed that the respective test case has been skipped.

Happy testing!

Anyway, I think this is a good opportunity to think about failing tests (with tests I primarily mean unit tests).

Even though “failing tests” may sound quite negative, this is not the case, at least if you practice test driven development (or a similar approach). There, failing tests are an integral part of the process. You start with a failing test, and then you implement the functionality so that the test no longer fails. Or if you make a change, and you break something by accident, then a failing test informs you about it. So, a failing test is quite positive.

On the other hand, failing tests also have a dark side.

The longer you wait to fix them, the higher the “costs”. If you fix a failing test immediately, then the costs are almost zero: you know what you changed, and so you can easily fix it. But if you wait a while, then the code is no longer present in your mind and it takes much more effort to fix the failing test. A side effect of this waiting is that it encourages others to do the same. If X doesn’t fix the tests he has broken, why should I care whether my tests run? As another side effect it may undermine the motivation of bug reporters to contribute tests if they see the project members don’t care about tests…

A special “behavior” of failing tests is that they don’t tell you whether they fail because of an incorrect test case or a bug in the code you test (in practice, this is sometimes obvious, sometimes not, depending on various factors). And they don’t tell you whether they only fail in your environment or everywhere. So, this means, if I run your tests and some of them fail, then all I can say may be: test X fails in my environment (the reason for saying “in my environment” is that I assume you don’t give me broken code, i.e. code with tests you know are failing). As it is an environment-specific issue, I will inform you about it, so it can get fixed. But if you provide broken code, then everyone using it will bother you with the same issue, at least theoretically ;-)

I think it is obvious what helps to deal with those dark sides: don’t provide code with tests you know are failing, it’s that simple!

I hope it wasn’t too confusing, and sorry about the rant at the beginning, but sometimes it’s necessary to vent one’s anger ;-)

Well, the SimpleTest framework — the testing framework used by CakePHP — provides a feature called “partial mocks”. With this feature it is possible to fake certain methods of a class, i.e. we can mock those methods which access the database.

Let’s have a look at an example.

We assume we want to test a method which checks whether the provided tags are in the database, and returns then those tags which are not yet in the database.

As we use findAll() to retrieve the tags, we have to mock this method. We can do it with the following snippet:

Mock::generatePartial('Tag', 'MockTag', array('findAll'));

The first parameter of Mock::generatePartial() specifies the class we want to mock, the second parameter defines the name of the mocked class, and the last parameter defines the methods we want to mock.

In the test we will now use the MockTag class instead of the Tag class. It is identical to the Tag class (technically it is a subclass of it), with the difference that the implementation of findAll() doesn’t “exist” anymore. With setReturnValue() we can now specify what findAll() should return when called. Below you find the test code, it should be self-explanatory (if not, please leave a comment):

Mock::generatePartial('Tag', 'MockTag', array('findAll'));

class TagTest extends CakeTestCase {
    function testExtractNewTags() {
        $existingTags = array(0 => array('Tag' => array('name' => 'tagA')),
			                   1 => array('Tag' => array('name' => 'tagB')));

        $mock = new MockTag($this);
        $mock->setReturnValue('findAll', $existingTags);

        $newTags = $mock->extractNewTags(array('tagA', 'tagB'));
        $this->assertIdentical(array(), $newTags);

        $newTags = $mock->extractNewTags(array('tagA', 'tagC', 'tagD'));
        $this->assertEqual(2, count($newTags));
        $this->assertEqual('tagC', $newTags[0]);
        $this->assertEqual('tagD', $newTags[1]);
    }
}

That’s only one simple example of what you can do with partial mock objects. For more see the SimpleTest documentation.

Happy testing :)

Update 2007-12-08: Something I forgot to mention is that you have to include the file mock_objects.php to make it work. You can do it with

vendor('simpletest'.DS.'mock_objects');

at the top of your test case file.

Thanks to Zach Cox for asking me this by email!

Let’s write a simple helper which creates an author meta tag.

Usually you write first the test, and then the application code. But in this case I will make an exception, as I think it is easier to understand if I write first the application code and then the test.

Writing the helper is straight-forward: we have to get the view and add the meta tag to this view:

// app/views/helpers/head.php
class HeadHelper extends AppHelper {
    function authorMetaTag($authorName) {
        $metaTag = '<meta name="author" content="'.$authorName.'">';
        $view = ClassRegistry::getObject('view');
        $view->addScript($metaTag);
    }
}

Before this helper can be used we have to add the following line to the head section of the (default) layout:

<?php echo $scripts_for_layout; ?>

With that we are ready to write our test (resp. we are finished with our work if we don’t write tests).

The test skeleton is easy:

class HeadHelperTest extends CakeTestCase {
    var $helper = null;

    function setUp() {
        $this->helper = new HeadHelper();
    }

    function testAuthorMetaTag() {
        $this->helper->authorMetaTag('cakebaker');
    }
}

As you have seen in the implementation of the helper, we retrieve a view object from the ClassRegistry. So to test our helper we also have to put a view object into the ClassRegistry. If you look at the source code of the view class constructor, you will see the view is automatically added to the ClassRegistry (to make sure the correct view is in the ClassRegistry we have to remove a possible view due to the behavior of ClassRegistry I explained in an earlier article). And you will also see that the constructor expects a controller object. As we don’t have any controller, we will use the AppController for this purpose.

loadController(null);

class HeadHelperTest extends CakeTestCase {
    var $helper = null;

    function setUp() {
        $this->helper = new HeadHelper();
    }

    function testAuthorMetaTag() {
        ClassRegistry::removeObject('view');
        $view = new View(new AppController());
        $this->helper->authorMetaTag('cakebaker');
    }
}

The last step is to render the view resp. the layout of the view and to check whether our meta tag has been added. For this purpose we can use the regular expression I described in “Regular expression to check for content between tags” and which was improved by BurntSushi.

loadController(null);

class HeadHelperTest extends CakeTestCase {
    var $helper = null;

    function setUp() {
        $this->helper = new HeadHelper();
    }

    function testAuthorMetaTag() {
        ClassRegistry::removeObject('view');
        $view = new View(new AppController());
        $this->helper->authorMetaTag('cakebaker');
        $renderedLayout = $view->renderLayout('');
        $this->assertPattern('#<head>.*?<meta name="author" content="cakebaker">.*?</head>#si', $renderedLayout);
    }
}

Now, we can run the test and should get a green bar.

Happy testing :)

One point I mentioned in the aforementioned comment are the missing tests. Not every class is tested. That’s because CakePHP was not developed with testing in mind originally. The first tests were added at the beginning of this year I think. And since then more and more tests have been written. But still, it will take quite a while to get a high test coverage. So the consequence of this is that when a developer makes changes in an untested class it is possible he will break something by accident that worked before, and he won’t notice it (you can find many such examples on trac). That means a working feature in version x may no longer work in version x+y.

The other point are the failing core tests in the development branch. They signal: “Hey, something doesn’t work”. Maybe it is “only” a broken test, maybe it is a broken feature, whatever. It doesn’t matter, the message is the same: something is broken. Sure, it can always happen that you commit something that breaks the tests, nobody is perfect. But if the tests fail over a longer time period, well, then the impression I get is that the developers don’t care about the tests.

Ok, I am now aware of those “trust problems”. But what can I do? It is no fun to write tests for existing, untested stuff, so writing core tests is not an option. And I cannot change the attitude of other developers towards tests from one day to the other. But I can write tests for my own application, and so test the underlying framework indirectly. If I update the framework, and my tests fail afterwards, then it is very probable they failed due to a newly introduced bug in the framework, and I can report it.

Of course, the aforementioned points are not the only motivation for me to write tests, the article “Top 12 Reasons to Write Unit Tests” lists other points that are important to me.

So my recommendation to everyone using CakePHP (or writing software in general): write tests, tests, tests!