At 7digital we use Cucumber and Watir for running acceptance tests on some of our websites.

These tests can help greatly in spotting problems with configuration, databases, load balancing, etc that unit testing misses.

But because the tests exercise the whole system, from the browser all the way through the the database, they can tend be flakier than unit tests. Then can fail one minute and work the next, which can make debugging them a nightmare.

So, to make the task of spotting the cause of failing acceptance tests easier, how about we set up Cucumber to take a screenshot of the desktop (and therefore browser) any time a scenario fails.

Install Screenshot Software

The first thing we need to do is install something that can take screenshots. The simplest solution I found is a tiny little windows app called SnapIt. It takes a single screenshot of the primary screen and saves it to a location of your choice. No more, no less.

• Download SnapIt and save it a known location (e.g. C:\Tools\SnapIt.exe)

Tell Cucumber to Take a Screenshot When a Scenario Fails

Now we need to tell Cucumber to take a screenshot. To do so we’ll add a function to the Cucumber World that will take a screenshot if needed, and run this in the After scenario hook. To do this modify your features/support/env.rb file.

env.rb

class DefaultWorld
 
  # Screenshot directory, relative to this env.rb file
  DEFAULT_SCREENSHOT_PATH = File.expand_path(File.dirname(__FILE__) + '/../../../output/cucumber/screenshots/')
 
  # Absolute location of SnapIt
  SNAPIT_PATH = 'C:\\Tools\\SnapIt.exe'
 
  def take_screenshot_if_failed(scenario)
    if (scenario.status != :passed)
      scenario_name = scenario.to_sexp[3].gsub /[^\w\-]/, ' '
      time = Time.now.strftime("%Y-%m-%d %H%M")
      screenshot_path = DefaultWorld::DEFAULT_SCREENSHOT_PATH + '/' +  time + ' - ' + scenario_name + '.png'
      cmd = DefaultWorld::SNAPIT_PATH + ' "' + screenshot_path + '"'
      %x{#{cmd}}
    end    
  end
 
  # [...] Other DefaultWorld code here if needed
 
end
 
World do
  DefaultWorld.new
end
 
After do |scenario|
  take_screenshot_if_failed(scenario)
 
  # [...] Other After hook code here if needed  
 
end

Just modify the constants in the above code to point to the locations of SnapIt and a directory to save the screenshots too.

What the Code Does

The code will only take a screenshot if the scenario fails to pass.

It then extracts the name of the scenario, and converts it to a filename friendly string (e.g. Monkey's should eat "things" => Monkey s should eat things). It then prepends the current date and time, and uses this string as the filename for the screenshot.

This allows you to easily find screenshots for a specific scenario or time.

Run a Failing Test and Check Out the Screenshot

Now you can run Cucumber as normal, watch a test fail, and you should see a screenshot appear in the directory you specified. And hopefully it will help you work out what went wrong, enjoy!

If the screenshot fails to appear, it could be because of an error in the ruby code. But Cucumber seems to hide any execptions within the After hook, so you may need to add puts statements to work out what is going wrong.

TeamCity is a great continuous integration server, and has brilliant built in support for running NUnit tests. The web interface updates automatically as each test is run, and gives immediate feedback on which tests have failed without waiting for the entire suite to finish. It also keeps track of tests over multiple builds, showing you exactly when each test first failed, how often they fail etc.

If like me you are using Cucumber to run your acceptance tests, wouldn’t it be great to get the same level of TeamCity integration for every Cucumber test. Well now you can, using the TeamCity::Cucumber::Formatter from the TeamCity 5.0 EAP release.

JetBrains, the makers of TeamCity, released a blog post demostrating the Cucumber test integration, but without any details in how to set it up yourself. So I’ll take you through it here.

Getting a Copy of the TeamCity Cucumber Formatter

The latest TeamCity EAP contains the new Cucumber Formatter hidden deep in it’s bowels. Rather than make you wade through it all, I’ve extracted the relevant files and they are available to download here:

Download the TeamCity Cucumber Formatter

The archive contains the formatter and the TeamCity library files it requires to run. Extract the archive in your project root and it will add the following files:

features/
    support/
        jetbrains-teamcity-formatter.rb
lib/
    teamcity/
        [some support and utility files]

If you want to locate these files within the TeamCity EAP yourself, download the TeamCity 5.0 EAP War file and extract it. Then from within the war unzip WEB-INF/plugins/rake-runner-plugin.zip. And from within the rake-runner-plugin look at rake-runner/lib/rb/patch/bdd/teamcity/cucumber/formatter.rb and all the files in rake-runner/lib/rb/patch/common/teamcity/.

The formatter in my download has been tweaked to look in a new location for the teamcity support files, and has been changed to be a single class in a module named JBTeamCityFormatter (to ease calling it from the command line).

The relevant changes in the file are:

14
15
16
17
18
19
20
21

 
$: << File.expand_path(File.dirname(__FILE__) + '/../../lib/')
require 'teamcity/runner_common'
require 'teamcity/utils/service_message_factory'
require 'teamcity/utils/runner_utils'
require 'teamcity/utils/url_formatter'
 
class JBTeamCityFormatter < ::Cucumber::Ast::Visitor

Setting up Cucumber to use the TeamCity Formatter

Once you have the formatter installed you can use it as with any Cucumber formatter by adding it as a command line parameter:

cucumber features -f JBTeamCityFormatter

To use it with TeamCity, add a profile your cucumber.yml file that runs all your features using the new formatter:

cucumber.yml

default: features -q 
teamcity: features -q --no-c  -f JBTeamCityFormatter

Running Cucumber with TeamCity

Now when you run Cucumber within TeamCity (using the teamcity profile) it will report tests in real time, with all the feedback you are used to. Just add a call to the Cucumber executable to your build script (NAnt, MSBuild, Ant, Rake, etc).

Enjoy the new found treatment of Cucumber tests as first class citizens in TeamCity!

JSpec is a Javascript BDD framework with a lot of great things going for it: It can run without a browser (great for continuous integration servers), it has a Ruby style custom syntax which makes tests easier to write and read, and it uses a BDD style describe/should syntax.

It’s a very tempting framework to use, but I already have a large collection of tests using qunit. I don’t want to use two frameworks for one project, and I don’t want to rewrite 300+ tests, so what to do?

How about a QUnit to JSpec Adapter, in the vein of my QUnit to JS Test Driver Adapter. Just load the adapter into JSpec as a normal javascript file, and you can nowexec() qunit test files in JSpec.

Downloading the QUnit to JSpec Adapter

First up download the QUnit to JSpec Adapter, or copy the code below, and save it somewhere in your project (e.g. a lib folder).

QUnitToJSpecAdapter.js

/*
QUnitToJSpecAdapter
Version: 1.0.0
 
Run qunit tests using JSspec
 
This provides almost the same api as qunit.
 
Tests must run sychronously, which means no use of stop and start methods.
You can use the JSpec mock timers to deal with timeouts, intervals, etc
 
The qunit #main DOM element is not included. If you need to do any DOM manipulation
you need to set it up and tear it down in each test.
 
*/
(function() {
 
    JSpec.addMatchers({
        be_ok : '!!actual'
    });
 
 
    JSpec.context = JSpec.defaultContext;
    JSpec.context.QUnitAdapter = {
        modules: []
    };
 
    module = function(name, lifecycle) {
 
        JSpec.context.QUnitAdapter.modules.push({
            name: name,
            tests: [],
            setup:    (lifecycle && lifecycle.setup)    ? lifecycle.setup    : function() {}, 
            teardown: (lifecycle && lifecycle.teardown) ? lifecycle.teardown : function() {}
        });
 
        JSpec.describe(name, function() {
 
            var length = QUnitAdapter.modules[0].tests.length;
            for (var i = 0; i < length; i++) {
                it(QUnitAdapter.modules[0].tests[i].name, function() {
                    var adapter = {
                        expectedAsserts: 0,
                        calledAsserts: 0,
 
                        expect: function(count) {
                            adapter.expectedAsserts = count;
                        },
 
                        ok: function(actual, msg) {
                            adapter.calledAsserts++;
                            JSpec.expect(actual).to(JSpec.matchers.be_ok);
                        },
 
                        equals: function(a, b, msg) {
                            adapter.calledAsserts++;
                            JSpec.expect(a).to(JSpec.matchers.be, b);
                        },
 
                        start: function() {
                            throw 'start and stop methods are not available when using JSpec.\n' +
                                'Use the JSpec timer to deal with timeouts and intervals:\n' + 
                                'http://github.com/visionmedia/jspec/tree/master';
                        },
 
                        stop: function() {
                            throw 'start and stop methods are not available when using JSpec.\n' +
                                'Use the JSpec timer to deal with timeouts and intervals:\n' + 
                                'http://github.com/visionmedia/jspec/tree/master';
                        },
 
                        same: function(a, b, msg) {
                            adapter.calledAsserts++;
                            JSpec.expect(a).to(JSpec.matchers.eql, b);
                        },
 
                        reset: function() {
                            throw 'reset method is not available when using JSpec';
                        },
 
                        isLocal: function() {
                            return false;
                        }
                    };
 
                    eval('with(adapter) {' +
                        JSpec.contentsOf(QUnitAdapter.modules[0].setup) +
                        'try {' +
                        JSpec.contentsOf(QUnitAdapter.modules[0].tests[0].testCallback) +
                        '} catch(ex) { throw(ex); } finally {' +
                        JSpec.contentsOf(QUnitAdapter.modules[0].teardown) +
                        '} }');
 
                    if (adapter.expectedAsserts > 0) {
                        JSpec.expect(adapter.calledAsserts).to(JSpec.matchers.equal, adapter.expectedAsserts);
                    }
                });
            }
 
            after_each(function() {
                QUnitAdapter.modules[0].tests.shift();
            });
 
            after(function() {
                QUnitAdapter.modules.shift();
            });
 
        });
 
    };
 
    test = function(name, testCallback) {
        JSpec.context.QUnitAdapter.modules[JSpec.context.QUnitAdapter.modules.length - 1].tests.push({
            name: name,
            testCallback: testCallback
        });
    };
 
})();

Having some QUnit tests

First up you need some qunit tests. Having the qunit test files in the spec directory helps simplify loading them.

As an example you can use the following:

qunit-tests.js

module('Examples');
 
test('True is ok', function() {
  expect(1);
  ok(true);
});
 
module('Examples with lifecycle', {
  setup: function() {
    started = 'yes';
  },
  teardown: function() {
    ok(started);
    started = null;
  }
});
 
test('Test has started', function() {
  expect(2);
  equals(started, 'yes');
});

Configuring QUnit Support

Then to enable QUnit tests to be run in JSpec, you must have JSpec load the adapater as a normal javascript file, andexec() the QUnit test file as you would a JSpec spec file.

For JSpec rhino support your spec.rhino.js file would look like:

spec.rhino.js

load('/Library/Ruby/Gems/1.8/gems/visionmedia-jspec-2.10.0/lib/jspec.js')
load('lib/QUnitAdapter.js')
 
JSpec
.exec('spec/qunit-tests.js')
.run({ formatter : JSpec.formatters.Terminal })
.report()

Notice how the QUnit test file is exec’d exactly as you would a normal spec file. You can run specs and Qunit tests along side each other without any interference.

And for running the tests within a browser your spec.dom.html file would look like:

spec.dom.html

<html>
    <head>
        <link type="text/css" rel="stylesheet" href="/Library/Ruby/Gems/1.8/gems/visionmedia-jspec-2.2.1/lib/jspec.css" />
        <script src="/Library/Ruby/Gems/1.8/gems/visionmedia-jspec-2.2.1/lib/jspec.js"></script>
        <script src="../lib/QUnitAdapter.js"></script>
        <script>
            function runSuites() {
                JSpec
                .exec('qunit-tests.js')
                .run()
                .report()
            }
        </script>
    </head>
    <body class="jspec" onLoad="runSuites();">
        <div id="jspec-top"><h2 id="jspec-title">JSpec <em><script>document.write(JSpec.version)</script></em></h2></div>
        <div id="jspec"></div>
        <div id="jspec-bottom"></div>
    </body>
</html>

Running the tests

To run the tests just launch JSpec as normal. My prefered method is to run the tests using rhino, to do this navigate to your project root directory in a terminal window and run:

jspec run --rhino

And you should now see your QUnit tests running in JSpec:

 Passes: 5 Failures: 0
 
 Examples
  True is ok..
 
 Examples with lifecycle
  Test has started...

Limitations

This adapter has many of the same limitations and my QUnit to JS Test Driver adapter.

The tests must run synchronously (which means no use of the qunit stop and start methods).

If you need to test timeouts, intervals, or other asynchronous sections of code , you can use themock timers that come with JSpec.

QUnit DOM support is not included. Consider avoiding interacting directly with the browser within your unit tests. But if you do need to, you’ll need to create and remove the DOM objects yourself with each test, or the setup and teardown methods.

And lastly, tests are broken out of any closures before they run. This means they lose access to any closure variables. For example, the follow test would work in QUnit, but not in JSpec. When running in JSpec access to thesetup variable is lost.

(function(){
  var setup = function() {
    // do setup...
  };
 
  test('name', function() {
    setup();
  });
})();