These tests can help greatly in spotting problems with configuration, databases, load balancing, etc that unit testing misses.

But because the tests exercise the whole system, from the browser all the way through the the database, they can tend be flakier than unit tests. Then can fail one minute and work the next, which can make debugging them a nightmare.

So, to make the task of spotting the cause of failing acceptance tests easier, how about we set up Cucumber to take a screenshot of the desktop (and therefore browser) any time a scenario fails.

Install Screenshot Software

The first thing we need to do is install something that can take screenshots. The simplest solution I found is a tiny little windows app called SnapIt. It takes a single screenshot of the primary screen and saves it to a location of your choice. No more, no less.

• Download SnapIt and save it a known location (e.g. C:\Tools\SnapIt.exe)

Tell Cucumber to Take a Screenshot When a Scenario Fails

Now we need to tell Cucumber to take a screenshot. To do so we’ll add a function to the Cucumber World that will take a screenshot if needed, and run this in the After scenario hook. To do this modify your features/support/env.rb file.

env.rb

class DefaultWorld
 
  # Screenshot directory, relative to this env.rb file
  DEFAULT_SCREENSHOT_PATH = File.expand_path(File.dirname(__FILE__) + '/../../../output/cucumber/screenshots/')
 
  # Absolute location of SnapIt
  SNAPIT_PATH = 'C:\\Tools\\SnapIt.exe'
 
  def take_screenshot_if_failed(scenario)
    if (scenario.status != :passed)
      scenario_name = scenario.to_sexp[3].gsub /[^\w\-]/, ' '
      time = Time.now.strftime("%Y-%m-%d %H%M")
      screenshot_path = DefaultWorld::DEFAULT_SCREENSHOT_PATH + '/' +  time + ' - ' + scenario_name + '.png'
      cmd = DefaultWorld::SNAPIT_PATH + ' "' + screenshot_path + '"'
      %x{#{cmd}}
    end    
  end
 
  # [...] Other DefaultWorld code here if needed
 
end
 
World do
  DefaultWorld.new
end
 
After do |scenario|
  take_screenshot_if_failed(scenario)
 
  # [...] Other After hook code here if needed  
 
end

Just modify the constants in the above code to point to the locations of SnapIt and a directory to save the screenshots too.

What the Code Does

The code will only take a screenshot if the scenario fails to pass.

It then extracts the name of the scenario, and converts it to a filename friendly string (e.g. Monkey's should eat "things" => Monkey s should eat things). It then prepends the current date and time, and uses this string as the filename for the screenshot.

This allows you to easily find screenshots for a specific scenario or time.

Run a Failing Test and Check Out the Screenshot

Now you can run Cucumber as normal, watch a test fail, and you should see a screenshot appear in the directory you specified. And hopefully it will help you work out what went wrong, enjoy!

If the screenshot fails to appear, it could be because of an error in the ruby code. But Cucumber seems to hide any execptions within the After hook, so you may need to add puts statements to work out what is going wrong.

If like me you are using Cucumber to run your acceptance tests, wouldn’t it be great to get the same level of TeamCity integration for every Cucumber test. Well now you can, using the TeamCity::Cucumber::Formatter from the TeamCity 5.0 EAP release.

JetBrains, the makers of TeamCity, released a blog post demostrating the Cucumber test integration, but without any details in how to set it up yourself. So I’ll take you through it here.

Getting a Copy of the TeamCity Cucumber Formatter

The latest TeamCity EAP contains the new Cucumber Formatter hidden deep in it’s bowels. Rather than make you wade through it all, I’ve extracted the relevant files and they are available to download here:

Download the TeamCity Cucumber Formatter

The archive contains the formatter and the TeamCity library files it requires to run. Extract the archive in your project root and it will add the following files:

features/
    support/
        jetbrains-teamcity-formatter.rb
lib/
    teamcity/
        [some support and utility files]

If you want to locate these files within the TeamCity EAP yourself, download the TeamCity 5.0 EAP War file and extract it. Then from within the war unzip WEB-INF/plugins/rake-runner-plugin.zip. And from within the rake-runner-plugin look at rake-runner/lib/rb/patch/bdd/teamcity/cucumber/formatter.rb and all the files in rake-runner/lib/rb/patch/common/teamcity/.

The formatter in my download has been tweaked to look in a new location for the teamcity support files, and has been changed to be a single class in a module named JBTeamCityFormatter (to ease calling it from the command line).

The relevant changes in the file are:

14
15
16
17
18
19
20
21

 
$: << File.expand_path(File.dirname(__FILE__) + '/../../lib/')
require 'teamcity/runner_common'
require 'teamcity/utils/service_message_factory'
require 'teamcity/utils/runner_utils'
require 'teamcity/utils/url_formatter'
 
class JBTeamCityFormatter < ::Cucumber::Ast::Visitor

Setting up Cucumber to use the TeamCity Formatter

Once you have the formatter installed you can use it as with any Cucumber formatter by adding it as a command line parameter:

cucumber features -f JBTeamCityFormatter

To use it with TeamCity, add a profile your cucumber.yml file that runs all your features using the new formatter:

cucumber.yml

default: features -q 
teamcity: features -q --no-c  -f JBTeamCityFormatter

Running Cucumber with TeamCity

Now when you run Cucumber within TeamCity (using the teamcity profile) it will report tests in real time, with all the feedback you are used to. Just add a call to the Cucumber executable to your build script (NAnt, MSBuild, Ant, Rake, etc).

Enjoy the new found treatment of Cucumber tests as first class citizens in TeamCity!

Install Ruby, Growl, and Gems

First up we need to ensure that we have all our dependancies installed.

If you haven’t already, download and install Ruby.

Then we need to install the Autotest, Cucumber, and Growl Ruby gems. We can do this using the gem command that comes packaged with Ruby.

# Autotest test is part of the ZenTest gem
sudo gem install ZenTest
sudo gem install cucumber
# Use my modified autotest-growl gem (until the changes are merged into the official gem)
sudo gem install karl-autotest-growl --source http://gems.github.com

Next we need to ensure that the Growl application itself is installed. If not download and install Growl.

Create Project

Now we are ready to create our project. The either download the example project or create the file heirarchy below:

• [dir] Autotest-Cucumber
  • [file] .autotest
  • [dir] features
    • [file] test.feature

Copy the code below into the .autotest file:

.autotest

require 'autotest/growl'

And the copy the following into the test.feature file:

test.feature

Feature:
 
    Scenario:
        Given I save 1
        Then I have 1

Set Environment

Lastly we need to set the AUTOFEATURE environment variable to true, so that autotest will run the Cucumber tests automatically:

AUTOFEATURE=true

Run Autotest!

Finally we can run autotest, and watch as it picks up the Cucumber tests, runs them, and notifies us via Growl. Open terminal and navigate to the project directory. Then run autotest:

autotest

Autotest will initiate a Cucumber run. Cucumber will pick up the test.feature file (because it looks for a features folder by default). The Cucumber run will show that you have 1 undefined scenario (and be kind enough to give you the code for your undefined steps). And finally Growl will display a notification that you have 1 undefined scenario, yay!

c:/ruby/bin/ruby c:/ruby/lib/ruby/gems/1.8/gems/aslakhellesoy-cucumber-0.3.11.3/bin/cucumber --format progress --format rerun --out C:/Temp/autotest-cucumber.17824.1 features
UU
 
1 scenario (1 undefined)
2 steps (2 undefined)
0m0.000s
 
You can implement step definitions for undefined steps with these snippets:
 
Given /^I save 1$/ do
  pending
end
 
Then /^I have 1$/ do
  pending
end

You find my patched version of autotest-growl on GitHub.

You can switch to this version of autotest-growl by uninstalling any existing version, and then installing from my fork on GitHub:

sudo gem uninstall autotest-growl
sudo gem install karl-autotest-growl --source http://gems.github.com

Cucumber is an awesome tool for doing behavioural driven development. It allows you to write plain text automated acceptance tests.

Autotest and Cucumber work together seamlessly, you just need to set the AUTOFEATURE environment variable to true:

$ AUTOFEATURE=true

Growl is an excellent notification system for OSX that allows applications to popup unobtrusive messages on the users desktop. (Snarl is the equivalent for windows)

You can add Growl support to Autotest (using the autotest-growl gem) so that you get popup notifications of test results.

But the current version of autotest-growl doesn’t provide notifications for the result of Autotest Cucumber runs.

Cucumber Growl support in Autotest

So I’ve added support for notification of Autotest Cucumber results using growl. It’s super simple at the moment as I’m still pretty new at Ruby.

You need to have Autotest (part of the ZenTest gem), Autotest-Growl, and Growl already installed.

Save the code below to the file growl-cucumber.rb.

growl-cucumber.rb

require 'autotest'
 
module Autotest::Growl
 
  # Growl results of Cucumber
  Autotest.add_hook :ran_features do |autotest|
 
	gist = autotest.results.grep(/\d+\s+scenario.*\)/).join(" / ").strip()
	if gist == ''
	  growl @label + 'Cannot run features.', '', 'error'
	else
	  if gist =~ /[1-9]\d*\s+(failed)/
	    growl @label + 'Some features have failed.', gist, 'failed'
	  elsif gist =~ /pending/
	    growl @label + 'Some features are skipped.', gist, 'skipped'
	  else
	    growl @label + 'All features have passed.', gist, 'passed'
	  end
	end
    false
  end
 
end

Then update your .autotest configuration file to include the new growl-cucumber file:

.autotest

# Add the growl and growl-cucumber requires to your .autotest config file 
require 'autotest/growl'
require 'autotest/growl-cucumber'

To Do

I need to see if this can be integrated with the existing autotest-growl gem.