Macro-writing Macros

code clojure ..... November 25, 2015

Later: Questions to Ask
Earlier: Time Limits

… in which we explore the power of macros, and macro-writing macros, to DRY out repetitive code.

I’ve been writing Clojure code full time for nearly two years now. I have a pretty good feel for the language, its virtues and its faults. Mostly, I appreciate its virtues (though I still wish the REPL started faster).

For me one of the language’s attractions has always been that it’s a Lisp — a “homoiconic” language, i.e., one defined in terms of its own data structures. Homoiconicity has one primary virtue, which is that it makes metaprogramming more powerful and straightforward than it is in non-homoiconic languages (arguably at some cost to readability).

In Lisp, this metaprogramming is accomplished with macros, which are functions that transform your code during a separate stage of compilation. In other words, you write little programs to change your programs before they execute. In effect, you extend the compiler itself.

I run a Clojure study group at work and find that it can be hard to explain the utility (or appeal) of this to newcomers to Lisp. This is partly because macros do things you can’t easily do in other languages, and because the things you want to do tend to relate to abstractions latent in a particular codebase.

While playing around with 3d rendering in Quil, I recently came across a use case that reminded me of the following quote by Paul Graham:

The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I’m using abstractions that aren’t powerful enough— often that I’m generating by hand the expansions of some macro that I need to write¹.

In Quil, there are multiple situations in which one needs to create a temporary context to carry out a series of operations, restoring the original state afterwards:

1. Save current style with push-style; change style and draw stuff; restore previous style with pop-style.
2. Start shape with begin-shape; draw vertices; end-shape to end.
3. Save current position/rotation with push-matrix; translate / rotate and draw stuff; restore old position/rotation with pop-matrix.

Here’s an example:

(push-matrix)
(try
  (push-style)
  (try
    (fill 255)
    (no-stroke)
    (translate [10 10 10])
    (begin-shape)
    (try
      (vertex x1 y1 0)
      (vertex x2 y2 0)
      (vertex x2 y2 h)
      (vertex x1 y1 h)
      (vertex x1 y1 0)
      (finally
        (end-shape)))
    (finally
      (pop-style)))
  (finally
    (pop-matrix)))

The (try ... (finally ...)) constructions may not be strictly needed for a Quil drawing, but it’s a good habit to guarantee that stateful context changes are undone, even if problems occur.

In a complex Quil drawing the idioms for saving style, translation state, and denoting shapes appear often enough that one hungers for a more compact way of representing each. Here’s one way to do it:

(defmacro with-style [& body]
  `(do
     (push-style)
     (try
        ~@body
       (finally
         (pop-style)))))

(defmacro with-matrix [& body]
  `(do
     (push-matrix)
     (try
        ~@body
       (finally
         (pop-matrix)))))

(defmacro with-shape [& body]
  `(do
     (begin-shape)
     (try
        ~@body
       (finally
         (end-shape)))))

The original code then becomes more compact and easier to read:

(with-matrix
  (with-style
    (fill 255)
    (no-stroke)
    (translate [10 10 10])
    (with-shape
      (vertex x1 y1 0)
      (vertex x2 y2 0)
      (vertex x2 y2 h)
      (vertex x1 y1 h)
      (vertex x1 y1 0))))

In this example code, the contexts with-matrix, etc. appear so often that the resulting savings in lines of code and mental overhead for the reader is substantial.

However, the astute reader will realize that the macro definitions themselves are pretty repetitive—in fact, they look almost identical except for the setup and teardown details (this kind of “context manager” pattern is common enough that Python has its own language construct for it).

I generally reach for macros when I have a pattern that occurs with obvious repetition that’s not easy to abstract out using just pure functions. Control abstractions such as loops or exception handling are common examples. (I find this situation occurs especially frequently when writing test code).

In any case, the solution for our repetitive macros could be something like:

(defmacro defcontext
  [nom setup teardown]
  `(defmacro ~(symbol (str “with-” nom))
     [~'& body#]
     `(do
        ~'~setup
        (try
          ~@body#
          (finally
            ~'~teardown)))))

Yikes! I have to admit I had to write a lot of macros, and also refer to this helpful page for reference, before I could write (and grok) this macro.

With defcontext in hand, our repetitive macro code just becomes:

(defcontext style (push-style) (pop-style))
(defcontext shape (begin-shape) (end-shape))
(defcontext matrix (push-matrix) (pop-matrix))

These are exactly equivalent to the three context macros (with-*) defined above.

With a little effort, it’s actually not too hard to construct such a nested macro. It’s largely a matter of writing out the code you want to generate, and then writing the code that generates it, testing with macroexpand-1 at the REPL as you go. This page by A. Malloy has a lot of helpful remarks, including this cautionary note: “Think twice before trying to nest macros: it’s usually the wrong answer.” In this case, I actually think it’s the right answer, because the pattern of a context with setup and teardown is so common that I know I’ll reuse this macro for many other things—we have effectively added one of my favorite Python features to Clojure in just a few lines of code[fn::To be even more like Python’s context managers, defcontext would want to enable the user to bind some local state resulting from the setup phase of execution (“ with x() as y: ” idiom); examples include file descriptors or database connections. This is left as an exercise for the reader.]

There’s a saying in the Clojure community: data > functions > macros. I’m a big believer in this. Clojure’s powerful built-in abstractions for wrangling data in all its forms make it the language I prefer above all others these days. But occasionally that means wrangling the data that is the code itself, thereby reaping the benefits in power, brevity and expressiveness.

Later: Questions to Ask
Earlier: Time Limits