A few months ago I wrote about Loris, a small tool that will automatically run your javascript tests whenever a file changes.

I’ve just released a new version of the Loris gem with a couple of minor fixes:

• The dependency on win32-process has now been removed from the gem. This means the gem now installs correctly on OSX. Windows users will manually need to install the win32-process gem.

• The dependancy on visionmedia-bind has been updated to just bind, reflecting the gems new name on RubyGems.

• The JS Test Driver server is now reset between all tests. This stops it getting into a situation where it failed to pick up changes to files.

Because RubyGems is now the default gem host, this install process is a little simpler.

sudo gem install loris

or on Windows

gem install loris
gem install win32-process

I’ve updated the JS Test Driver QUnitAdapter to improve compatibility with QUnit.

Variables set on the this object within are now available within setup, teardown, and the tests themselves.

module('Lifecycle', {
  setup: function() {
    this.mockString = "some string";
  },
  teardown: function() {
    equals(this.mockString, "some string");
  }
});
 
test("Things assigned to this in setup are available in test", function() {
  equals(this.mockString, "some string");
});

The test function now supports the optional second parameter of the expected number of assertions.

// declare that this test has expects 1 assertion
test('Test with expected defined as 2nd param', 1, function(){
  ok(true);
});

My thanks go to anotherhero for providing the patch to fix both these issues.

You can always download the latest version of QUnitAdapter from Google Code.

Just a small update to the JS Test Driver QUnitAdapter. Version 1.0.3 has been released, and fixes a single bug:

Issue 64: QUnit Adapter fails to run tests if you don’t include a module

You can now declare tests without a module, and they will run under the Default Module. In previous versions these tests would be silently ignored (whoops!).

You can always download the latest version of QUnitAdapter from Google Code.

I’ve previously written a number of posts on javascript and autotest. Explaining how to integrate javascript lint, unit tests, and growl with the ruby Autotest project.

While this all worked, it felt a little clunky as Autotest doesn’t natively support the idea of running multiple tasks one after the other. Rather than hack at the Autotest codebase, I thought I’d get some ruby experience by rolling my own autotest-style framework. Not great for reuse of code, but a great way for me to learn :)

Loris

Loris will monitor your project and run Javascript Lint and JS Test Driver whenever a file changes, it will report the results to the command line and using Growl. If required, Loris will automatically start the JS Test Driver server and register your default browser with it.

Installing

Loris is hosted on Gemcutter, so you need to install their gem if you haven’t already.

sudo gem update --system
sudo gem install gemcutter
gem tumble

Then to install Loris, just run the following:

sudo gem install loris

Loris has no command line options, and no configuration file (at the moment). It looks for configurations files to decide which tasks to run.

Configuring Javascript Lint

To enable Javascript Lint, create a jsl.conf file in the folder where you run Loris. This should be a standard Javascript Lint config file. If you need here is an example Javascript Lint config file

You just need to specify which files Javascipt Lint should process. For example:

### Files
# Specify which files to lint
# Use "+recurse" to enable recursion (disabled by default).
# To add a set of files, use "+process FileName", "+process Folder\Path\*.js",
# or "+process Folder\Path\*.htm".
#
+process src/js/*.js
+process tests/js/*.js

If no jsl.conf file is found, the Javascript Lint task is silently skipped.

Configuring JS Test Driver

To enable JS Test Driver, create a jsTestDriver.conf file in the folder where you run Loris. This should be a standard JS Test Driver config file.

This should specify which files JS Test Driver should process, and how it connects to the JS Test Driver server. For example:

server: http://localhost:9876
 
load:
  - tests/qunit/equiv.js
  - tests/qunit/QUnitAdapter.js
 
  - src/js/*.js
  - tests/js/*.js

If no jsTestDriver.conf file is found, the JS Test Driver task is silently skipped.

To make it really simple to run JS Test Driver tests, if the server is set to run on localhost, and Loris doesn’t detect one running, it will automatically start one, and register your default browser with it.

This makes it a one step process to get automated tests up and running.

Running

To run, open a command line window, navigate to the root folder of your project, and run:

loris

Loris will run Javascript Lint, and JS Test Driver tasks (if it finds their configuration files), and will output the results on the command line.

Example output

Javascript Lint
success
All files are clean
 
0 error(s), 0 warning(s)
JS Test Driver
success
All tests pass
[PASSED] GreeterTest.testGreet
  [LOG] JsTestDriver Hello World!
[PASSED] GreeterTest.testGoodbye
[PASSED] GreeterTest.testSetName
[PASSED] GreeterTest.testSetNameAndNameParamter
[PASSED] Asserts.test OK true succeeds
[PASSED] Asserts.test Equals succeeds
  [LOG] about to call assertEquals
[PASSED] Asserts.test Same assert succeeds
[PASSED] Lifecycle.test Setup and Teardown are run, and can contain assertions
Total 8 tests (Passed: 8; Fails: 0; Errors: 0) (3.00 ms)
  Firefox 1.9.1.4 MacIntel: Run 8 tests (Passed: 8; Fails: 0; Errors 0) (3.00 ms)

Every time you make a change to a Javascript file, or a configuration file, Loris will automatically re-run Javascript Lint and JS Test Driver. So you can instant feedback on your changes.

Loris will clear the command line when re-running tasks. So the latest run is always at the top of you command line.

Loris will also report a summary of each task using Growl (if it is installed). This allows you to get quick feedback without needing to refer back to the command line on every change.

Requirements

JS Test Driver is written in Java, so you will need to have Java installed to run it.

To get Growl notifications, you will need to install either Growl for OSX or Growl for Windows. Growl for Windows requires the .NET Framework 2.0+.

Caveat

Loris is pretty limited at the moment, I just wired up the basics to get it running for a work project.

It doesn’t have any configuration options at the moment, so you have to follow it’s assumptions for now. I’m happy to add configuration options for any element as required.

Loris only comes with a few tasks (Javascript Lint, JS Test Driver, JSpec, and RSpec), but I hope to allow it have new tasks added via new gems (kind of similar to Autotest).

It comes packaged with a version of Javascript Lint, and JS Test Driver, and will use it’s own versions. It only includes the OSX and Windows versions of Javascript Lint.

If you want to modify the code, just fork the Loris github project

A new version of the JS Test Driver QUnit Adapter is available.

Version 1.0.2 fixes a small bug where a module lifecycle object without Setup or Teardown methods would cause a test to error. For example:

module('Lifecycle', {});
 
test('', function() {
	expect(1);
	ok(true, 'tests still run successfully even if Setup and Teardown are undefined');
});

Would give the error Lifecycle.test error (1.00 ms): Result of expression 'l.setUp' [undefined] is not a function..

This is now fixed so the test behaves as if no lifecycle was defined.

You can get the new 1.0.2 verison of the QUnit Adapter from Google Code.

JS Test Driver is a great new Javascript Testing Framework from the guys at Google. It provides a blisteringly fast, and easily automated way of running your Javascript unit tests. See this introduction to JS Test Driver by Miško Hevery for a great overview.

Getting JS Test Driver up and running on your development workstation is easy enough. But how about on a continuous integration server such as TeamCity?

It’s easy, just follow the instructions below!

Install Java and the JS Test Driver jar on all the build agents

JS Test Driver is packaged as a Java jar, and therefore all the TeamCity build agents will need a copy of Java and the JS Test Driver jar in order to run the tests.

• For each Build Agent
• Download Java and install it.
• Download the JS Test Driver jar and save it to c:\TeamCityBuildTools\JSTestDriver\ (or whichever location you wish)

Configure JS Test Driver

This guide assumes you already have JS Test Driver and some tests set up on your local machine. If not see this introduction to JS Test Driver by Miško Hevery.

Just make sure that the JS Test Driver config file is set to connect to the local machine, on port 9876 (on another port of your choice). E.g.

server: http://localhost:9876
 
load:
  # list of files to load here...

Configure your build process to call JS Test Driver

Now that we have JS Test Driver installed on our build machines, and configured correctly, let’s call it from our build process. At 7digital we are using MSBuild, so we can create a JSTestDriver Target as so:

<Target Name="JSTestDriver">
  <Message Text="##teamcity[progressMessage 'Running JS Test Driver']" Importance="high" />
  <Message Text="-------- Running JS Test Driver --------" Importance="high" />
 
  <PropertyGroup>
    <JSTestDriverJar>&quot;$(BuildToolsPath)\JSTestDriver\jsTestDriver.jar&quot;</JSTestDriverJar>
    <ConfigFile>&quot;$(SolutionFolder)\jsTestDriver.conf&quot;</ConfigFile>
    <IEPath>&quot;C:\Program Files\Internet Explorer\iexplore.exe&quot;</IEPath>
    <OutputDir>&quot;$(SolutionFolder)\output\JSTestDriver&quot;</OutputDir>
  </PropertyGroup>
 
  <Exec Command="java -jar $(JSTestDriverJar) --port 9876 --browser $(IEPath) --config $(ConfigFile) --tests all --testOutput $(OutputDir)" />
 
</Target>

The important points here are that you have the properties configured correctly (JSTestDriverJar, ConfigFile, etc) for your individual setup.

The Exec command will automatically create a JS Test Driver server, launch IE, and run all the tests. The test results will be output to the OutputDir in a JUnit compatible xml format. See the JS Test Driver continuous integration page for more information.

Once you have created this Target, ensure it is called by your standard bulid Target by adding it as a dependancy:

<Target Name="BuildAndUnitTest" DependsOnTargets="Compile;JSLint;JSTestDriver;NUnit" />

Have TeamCity pick up the test results

If you run your build through TeamCity now, you will see that it runs all your Javascript tests using JS Test Driver. But it doesn’t report any results. And test failures don’t stop the build.

Luckily TeamCity can import the JS Test Driver output xml, because it is JUnit compatible.

• Select the build which is running the JS Test Driver tests
• Click Edit Configuration Settings
• Click on build step 3 (e.g. Runner: MSBuild)
• Under the XML Report Processing section, choose Ant JUnit from the Import data from dropdown
• In the Report paths text area add the relative path the the JS Test Driver output file (e.g. output\JSTestDriver\TEST-com.google.jstestdriver.1.xml)
• Click save!

Now when you run your build again you will see your JS Test Driver tests showing up in the Tests tab, and test failures now stop the build.

Your Javascript unit tests now sit as first class citizens within TeamCity!

I’ve made a small update to the QUnit Adapter for JS Test Driver.

This fixes a bug where the ok() assertion was much stricter than the qunit equivalent. It was only succeeding when passed a boolean true value.

Now it behaves the same as the qunit version, and succeeds with all values other than 0, false, or null.

You can get the new 1.0.1 verison of the QUnit Adapter from Google Code.

As part of my experimenting with JS Test Driver, I have been porting over a suite of about 300 existing tests.

Some of these tests deal with time based logic, such as delayed callbacks or time limited user interaction. Currently these tests are run in a HTML page using qunit.

To deal with the time based logic I use the jsUnit’s Mock Timeout. See Simulating Time in jsUnit Tests on the Google Testing Blog.

jsUnit’s Mock Timeout overrides the built in setTimeout, clearTimeout, setInterval, and clearInterval methods, and provides a Clock object that allows you to control the passage of time.

Problem

This works fine in qunit, but it fails catastrophically in JS Test Driver. This is because JS Test Driver uses the setTimeout and setInterval methods to control it’s own behaviour. Once these have been overridden with the mock versions JS Test Driver no longer works.

As a temporary fix for this I have created a patched version of JS Test Driver, which creates it’s own copies of the original setTimeout, clearTimeout, setInterval, and clearInterval methods, before loading any code. This ensures it works well with the jsUnit Mock Timeout.

Patched JS Test Driver

Download a patched JsTestDriver with no-confilict timeouts.

Just use this jar in place of the original JsTestDriver-1.0b.jar, and you can include jsUnitMockTimeout.js in your tests without breaking JS Test Driver!

You may also want to download jsUnitMockTimeout.js (to save you finding it within the jsUnit source).

The Future

I see this patched JS Test Driver as very much a temporary stepping stone, while the correct solution is worked out. Beware of relying on it too heavily!

In my last post I introduced a QUnit to JS Test Driver adapter, a small adapter which converts QUnit tests into native JS Test Driver tests.

This adapter is now part of the JS Test Driver project.

There is now a QUnit Adapter wiki page on the JS Test Driver project site. This will be updated to keep up with any changes to the adapter.

The code is now stored within the JS Test Driver subversion repository. You can always download the latest version of the QUnit Adapter using the Google Code web interface.

Thanks to Jeremie Lenfant-engelmann for deeming QUnit Adapter worthy of official regonition :)

In my previous post on QUnit and JS Test Driver I showed how to run your qunit tests with JS Test Driver.

The technique used was to run the tests with qunit, and report either their success or failure to JS Test Driver. This works, but you miss out on the important feedback of exactly which assertions have failed, and why.

This problem has led me to taking a new approach to running qunit tests with JS Test Driver, where I don’t use any of the existing qunit code, and instead just create an interface wrapper that converts qunit style tests and assertions directly into JS Test Driver tests and assertions.

This gives assertion level error reporting, making it much easier to write and debug tests. Essentially this adapter allows you to write native JS Test Driver tests, but using the less verbose qunit syntax.

The new approach also means that qunit lifecycles (setup and teardown) work.

Installing the QUnit Adapter

First up download the equiv.js file, which is required for the qunit same assertion.

Then download the QUnitAdapter.js file (or copy the code below).

QUnitAdapter.js

(function() {
 
    window.module = function(name, lifecycle) {
        QUnitTestCase = TestCase(name);
 
        if (lifecycle) {
            QUnitTestCase.prototype.setUp = lifecycle.setup;
            QUnitTestCase.prototype.tearDown = lifecycle.teardown;
        }
    };
 
    window.test = function(name, test) {
        QUnitTestCase.prototype['test ' + name] = test;
    };
 
    window.expect = function(count) {
        expectAsserts(count);
    };
 
    window.ok = function(actual, msg) {
        assertTrue(msg ? msg : '', actual);
    };
 
    window.equals = function(a, b, msg) {
        assertEquals(msg ? msg : '', b, a);
    };
 
    window.start = window.stop = function() {
        fail('start and stop methods are not available when using JS Test Driver.\n' +
            'Use jsUnit Clock object to deal with timeouts and intervals:\n' + 
            'http://googletesting.blogspot.com/2007/03/javascript-simulating-time-in-jsunit.html.');
    };
 
    window.same = function(a, b, msg) {
        assertTrue(msg ? msg : '', window.equiv(b, a));
    };
 
    window.reset = function() {
    	fail('reset method is not available when using JS Test Driver');
    };
 
    window.isLocal = function() {
    	return false;
    };
 
    window.QUnit = {
    	equiv: window.equiv,
    	ok: window.ok
    };
 
})();

Save both these files to your project (for example tests/qunit/).

Configuring JS Test Driver

To run your qunit tests in JS Test Driver you need to configure it to load the adapter before your qunit tests.

Update your jsTestDriver.conf to load the files:

server: http://localhost:9876
 
load:
  # Add these lines to load the equiv function and adapter in order, before the tests
  # (assuming they are saved to tests/qunit/)
  - tests/qunit/equiv.js
  - tests/qunit/QUnitAdapter.js
 
  # This is where we load the qunit tests
  - tests/js/*.js
 
  # And this loads the source files we are testing
  - src/js/*.js

Running JS Test Driver with qunit tests

Now we can run JS Test Driver and watch as it runs all our qunit tests!

The tests will run as individual JS Test Driver tests, with the format Module Name.Test Name.

Example output:

[PASSED] Module 1.test Test 1
[PASSED] Module 1.test Test 2
[PASSED] Module 2.test Test 1
Total 3 tests (Passed: 3; Fails: 0; Errors: 0) (1.00 ms)
  Safari 530.18: Run 3 tests (Passed: 3; Fails: 0; Errors 0) (1.00 ms)

Limitations

There are a few limitations on which qunit tests will successfully be converted.

The tests must run synchronously (which means no use of the qunit stop and start methods).

If you need to test timeouts, intervals, or other asynchronous sections of code, consider using the jsUnit Clock object to deal with timeouts and intervals.

QUnit DOM support is not included. Consider avoiding interacting directly with the browser within your unit tests. But if you do need to, you’ll need to create and remove the DOM objects yourself with each test, or the setup and teardown methods.