Friday, June 08, 2007

The Test Monad

Lots of people have written clever things about writing monads - their mathematical structure, properties etc. I wrote a Test Monad for my FilePath library, which I've since refined for Hoogle. The inspiration for this comes from Lennart Augustsson, who gave a talk at fun in the afternoon on ways to abuse Haskell syntax for other purposes.

For the testing of the parser in Hoogle, I want to write a list of strings, versus the data structures they should produce:

"/info" === defaultQuery{flags = [Flag "info" ""]}
"/count=10" === defaultQuery{flags = [Flag "count" "10"]}

In Haskell, the most syntactically pleasant way of writing a list of things is using do notation in a monad. This means that each (===) operation should be in a monad, and the (>>) bind operation should execute one test, then the next. We also need some way of "executing" all these tests. Fortunately this is all quite easy:

data Test a = Pass

instance Monad Test where
a >> b = a `seq` b

instance Show (Test a) where
show x = x `seq` "All tests passed"

pass :: Test ()
pass = Pass

The basic type is Test, which has only one value, being Pass. To represent failure, simply call error, which is failure in all Haskell programs and allows you to give a useful error message. The helper function pass is provided to pin down the type, so you don't get ambiguity errors. The Monad instance simply ensures that all the tests are evaluated, so that if any crash then the whole program will crash. The Show instance demands that all the tests passed, then gives a message stating that.

We can then layer our own test combinators on top, for example for parsec:

parseTest f input output =
case f input of
Left x -> err "Parse failed" (show x)
Right x -> if x == output then pass else
err "Parse not equal" (show x)
err pre post = error $ pre ++ ":\n " ++
input ++ "\n " ++
show output ++ "\n " ++

This parseTest function takes a parsec parser, an input and an output. If the parse fails, or doesn't produce the correct answer, an error is raised. If the test passes, then the function calls pass. It's then simple enough to define each test set as:

parse_Query = do
let (===) = parseTest parseQuery

Here (===) is defined differently for each do block. By evaluating one of these test blocks in an interpreter, the show method will automatically be called, executing all the tests.

I've found this "monad" lets you express tests very succinctly, and execute them easily. I moved to this test format from stand-alone files listing sample inputs and expected results, and its proved much easier to maintain and more useful.


Anonymous said...

In most of the testing philosophy I've read, and practice has reinforced the idea, that independent tests are the ideal. That said, stopping when the first test fails might not be ideal behavior.

Neil Mitchell said...

Anon: It depends on your testing method - I prefer to keep a list of things that don't work separately, and have a failing test be an indicator that "the world is falling down". I know some people (Pugs) do test-driven development, where they write failing tests, then gradually try to make them pass - but that doesn't suit me.

Anonymous said...

This is a nice bit of code.
I'm just getting into Haskell but I'm very impressed by its elegance, and this is a good example of it.

Oh, just one other thing I was wondering - what license is the "test monad" code under? Is it public-domain?
- Andy

Neil Mitchell said...

Andy: I'm posting it to my blog in the hope it may be useful to someone. Consider it public domain, or BSD licensed as suits you.

Anonymous said...

Hi Neil -
Ok, thanks very much!

- andy