Tests by Example in Clojure and Common Lisp

clojure lisp code .....

Later: Adding Tail Call Optimization to A Lisp Written in Go
Earlier: (Yet Another) Lisp In Go

Histogram made by a 1980's version of the author.

While playing around with some dice-rolling simulations recently in Common Lisp (a topic for some future post, perhaps), I looked around for a way of making histograms. Finding none close to hand (it being Common Lisp), it wasn't too hard to "roll my own" library (it being… Common Lisp) to create text-based histograms in the tradition of libraries from CERN that I used as a fledgling physicist in the late 1980s.

Since the project was just for fun, I drove most of the code forward just by REPLing. But I am trying to follow the same habits in my "for fun" GitHub repositories as I would for paid work. This means, among other things, unit tests (in fact, for critical work I still prefer to use TDD when I can).

So, in fleshing the tests out after the fact, I found myself writing somewhat repetitive code that looked like this:

(test hist-values-test
  (let ((h (histogram '(1 2) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(1 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 1) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(2 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 2 3) 3)))
    (is (= 3 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 3 (hist-max h)))
    (is (equalp #(1 2 1) (hist-bin-heights h)))
    (is (equalp #(1 2 3) (hist-bin-xs h)))))

This is using the minimalist 1AM testing library, which I like for its simplicity. It is also similar to the standard clojure.test library I'm used to in Clojure. It lacks, however, one important feature of clojure.test, namely the are macro. The macro allows one to represent tests as a table of examples:

;; Clojure code:
(deftest example
    [nums       prod]         ;; [1]

    (= prod (reduce * nums))  ;; [2]

    []          1             ;; [3]
    [1]         1
    [0]         0
    [0 1]       0
    [1 1]       1
    [2 2]       4
    [2 3]       6
    [-1 -1]     1
    [2 2 2]     8
    [1000 1000] 1000000))

An invocation of are has three parts: a vector of symbols to be bound to values from the examples to follow [1]; the expression to be evaluated for each example [2]; and the actual examples [3], with one example per line being the usual, but optional, convention. (In the code above I have added extra spaces to make clear that nums goes with the first column of data, and prod goes with the second column.)

If at first you didn't understand that code being tested ([2]) was related to multiplication, you'd probably be able to figure it out just by looking at the examples. This is the primary advantage of are, namely that examples often provide as good or better documentation than a written description of functionality does.

Missing such a compact testing idiom for my unit tests, I found myself wondering how hard it would be to port it to Common Lisp. The answer, as you might expect, was: Not that hard. As is often the case, there is not much actual Clojure code behind the function in question, and the corresponding Common Lisp code is similarly compact, as we shall see.

If you look at the actual function definition in clojure.test, most of the macro proper is error checking; the actual work is delegated to a completely different namespace, clojure.template (in what follows, I will ignore the error checking):

;; Clojure code:
(defmacro are [argv expr & args]
  ;; ... error checking
  `(clojure.template/do-template ~argv (is ~expr) ~@args)
  ;; ... error checking

A few macro expansions shows what's happening:

;; Clojure code:
 '(are [nums prod]
    (= prod (reduce * nums))
    []      1
    [1]     1
    [2 2 2] 8))


 [nums prod]
 (clojure.test/is (= prod (reduce * nums)))
 []      1
 [1]     1
 [2 2 2] 8)

do-template is also a macro. Simply running macroexpand-1 again shows what it does:

;; Clojure code:
   [nums prod]
   (clojure.test/is (= prod (reduce * nums)))
   []      1
   [1]     1
   [2 2 2] 8))


(do (clojure.test/is (= 1 (reduce * [])))
    (clojure.test/is (= 1 (reduce * [1])))
    (clojure.test/is (= 8 (reduce * [2 2 2]))))

The expanded code is exactly equivalent to what you would write if you didn't have the are macro. This is accomplished by simply repeating the code being invoked, but substituting the actual example values for nums and prod.

In order to do accomplish the same in Common Lisp, we take a bottom-up approach. First, we need to split our test cases into groups corresponding to the number of columns in our argv vector (in this case, 2):

(let ((cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (partition-n 2 2 cases))
'((#()      1)
  (#(1)     1)
  (#(2 2 2) 8))

The output looks very similar to cases, but each pair has been grouped into its own list. Here I use partition-n from cl-oju, a Common Lisp library I wrote specifically to take advantage of sequence idioms from the Clojure core library; partition-n is a translation of Clojure's partition function. (The equivalent code in clojure.template also uses partition for the same purpose.)

We now need to flesh out those cases by making use of each example in the expression being tested, whose Common Lisp translation is

(is (= prod (reduce #'* nums)))

For example, if nums is #(2 2 2), then prod should be 8.

In other words, we want our test cases to be:

(is (= 1 (reduce #'* #())))
(is (= 1 (reduce #'* #(1))))
(is (= 8 (reduce #'* #(2 2 2)))))

The replacement of symbols with their actual values in the Clojure code is accomplished with a tree-walking function called postwalk-replace. We can do it in Common Lisp with the subst function, which works on linear or nested lists:

(subst 3 'b '(* (+ b b) (/ b (- b b))))
'(* (+ 3 3) (/ 3 (- 3 3)))

But because we have multiple bindings or example arguments we have to convert, we need to do it repeatedly:

(defun apply-bindings (bindings case expr)
  (loop with ret = expr
        for b in bindings
        for c in case
        do (setf ret (subst c b ret))
        finally (return ret)))

(apply-bindings '(x y) '(2 3) '(+ x y))
'(+ 2 3)

Now we have everything we need to expand our test cases:

(let ((argv '(nums prod))
      (expr '(is (= prod (reduce #'* nums))))
      (c (length argv))
      (cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (loop for case in (partition-n c c cases)
        collect (apply-bindings argv case expr)))
'((IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

All the macro has to do is wrap this in a progn:

(defmacro are (argv expr &rest cases)
  Analog of clojure.test/are.  Apply multiple assertions
  based on some set up code and variations of one or more
  variable bindings.
  (let ((c (length argv)))
    `(progn ,@(loop for case in (partition-n c c cases)
                    collect (apply-bindings argv case expr)))))

giving, for our example:

 '(are (nums prod)
   (is (= prod (reduce #'* nums)))
   #()      1
   #(1)     1
   #(2 2 2) 8))
  (IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

This is equivalent to the macroexpansion of Clojure's are that we saw above. I leave as an exercise for the reader a further exploration of clojure.template/apply-template and its use in the do-template macro; and the addition of error checking such as making sure the number example values is a multiple of the number of binding arguments.

One rough edge that I glossed over: in my first attempt, my macro couldn't find apply-bindings at compile time until I wrapped that function's definition as follows:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defun apply-bindings (...) ...))

This is apparently needed for functions used by macros defined in the same file; it's not an issue for functions defined in other files. Since Clojure has a single-pass compiler, this sort of ceremony is not needed for Clojure macros.

I expect to use this macro again, so will probably spin it out into a small stand-alone library or incorporate it into a thin wrapper around 1AM. Here it is in action in the hbook library where our discussion started:

(test hist-values-test
  (are (h c mn mx heights xs)
         (is (= c (hist-count h)))
         (is (= mn (hist-min h)))
         (is (= mx (hist-max h)))
         (is (equalp heights (hist-bin-heights h)))
         (is (equalp xs (hist-bin-xs h))))

       (histogram '(1 2)     2)   2 1 2 #(1 1)   #(1 2)
       (histogram '(1 2 1)   2)   2 1 2 #(2 1)   #(1 2)
       (histogram '(1 2 2 3) 3)   3 1 3 #(1 2 1) #(1 2 3)))

The code which are makes possible is usually denser and can be harder to parse initially than more repetitive test code would, as is clear from this example. However, I find the brevity of are tests extremely helpful, especially when comparing the test examples to each another. Such examples are especially helpful when calling out edge cases where behavior changes. I have given such tables to stakeholders many times, in order to make sure we are aligned on what the behavior should be. (Even if the stakeholder doesn't care about the details of your implementation or your test code per se, they probably will care about the specific examples you provide).

Parenthetically, here is the histogramming library in action, showing the distribution of the sum of 1000 six-sided dice:

(defun d () (1+ (random 6)))
(defun dn (n) (loop repeat n sum (d)))

(princ (hbook (loop repeat 100000 collect (dn 1000))

      4962                                       X  X
      4791                                       X  X
      4620                                     X X  X X
      4449                                     X X  X X
      4278                                     X XXXXXX X
      4106                                     X XXXXXX X
      3935                                   X XXXXXXXXXX
      3764                                   XXXXXXXXXXXX
      3593                                   XXXXXXXXXXXX
      3422                                   XXXXXXXXXXXX
      3251                                   XXXXXXXXXXXXX X
      3080                                  XXXXXXXXXXXXXXXX
      2909                                X XXXXXXXXXXXXXXXX
      2738                                XXXXXXXXXXXXXXXXXX
      2566                                XXXXXXXXXXXXXXXXXX
      2395                                XXXXXXXXXXXXXXXXXXXX
      2224                               XXXXXXXXXXXXXXXXXXXXX
      2053                              XXXXXXXXXXXXXXXXXXXXXX
      1882                              XXXXXXXXXXXXXXXXXXXXXX
      1711                              XXXXXXXXXXXXXXXXXXXXXXX
      1540                             XXXXXXXXXXXXXXXXXXXXXXXXX
      1369                             XXXXXXXXXXXXXXXXXXXXXXXXXX
      1198                           XXXXXXXXXXXXXXXXXXXXXXXXXXXX
      1027                           XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       856                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       684                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

                      11244625381526456400925471219232518023527888048930961854221 1
           1   11235564827658185382948873231241519783970927498485883199710549330480234    1


Later: Adding Tail Call Optimization to A Lisp Written in Go
Earlier: (Yet Another) Lisp In Go