Lisp on John Jacobsen

Lisp Testing Fun

Fri, 11 Apr 2025 00:00:00 +0000

I've been making some progress with steelcut, my Common Lisp template generator (does every Lisper write one of these as a rite of passage?), and I keep running into interesting problems which have been fun to solve.

Whenever my code starts to accumulate any sort of complexity, I try to err on the side of writing more tests, even for "recreational" projects such as this one. The tests help make it safe to make changes, while also documenting the expected functionality. But test code can be fairly repetitive, often with many small variations and lots of boilerplate, in the midst of which it can be hard to see the forest for the trees.

Such was the case for steelcut tests. I found myself remembering Paul Graham's line, "Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough – often that I'm generating by hand the expansions of some macro that I need to write."

In my current non-Lisp programming work, I most often miss macros when writing test code. But in this case, the macro I (apparently) needed to write is here. Very roughly speaking, with-setup creates a temporary directory, runs the template generator with a given list of requested features, and allows the user to check properties of the generated files.

The macro grew and shrank as I DRYed out and simplified the test code. I realized many of my tests were checking the generated dependencies for the projects (from the .ASD file), so it would be helpful if the macro would make those available to the test code in the form of a function the user could name.

This wound up being a sweet bit of sugar for my tests, but not every test needed such a helper function. Stubborn "unused variable" warnings ensued, for those tests which don't use the deps symbol bound by the macro. I went back and forth with ChatGPT on this one (it made several wrong suggestions) until I realized I could give the variable an explicit _ name and change the behavior based on the name. This is something I've seen in other languages and was tickled I could get it so easily here.

I find the resulting tests pretty easy to read:

(test needed-files-created
  (with-setup appname "testingapp" appdir _ +default-features+
    (loop for file in '("Makefile"
                        "Dockerfile"
                        ".github/workflows/build.yml"
                        "build.sh"
                        "test.sh"
                        "src/main.lisp"
                        "src/package.lisp"
                        "test/test.lisp"
                        "test/package.lisp")
          do
             (is (uiop:file-exists-p (join/ appdir file))))))

(test cmd-feature-adds-dependency-and-example
  (with-setup appname "testingapp" appdir deps +default-features+
    (let ((main-text (slurp (join/ appdir "src/main.lisp"))))
      ;; :cmd is not there:
      (is (not (find :cmd (deps))))
      ;; It doesn't contain the example function:
      (is (not (has-cmd-example-p main-text)))))

  ;; Add :cmd to features, should get the new dependency:
  (with-setup appname "test2" appdir deps (cons :cmd +default-features+)
    (let ((main-text (slurp (join/ appdir "src/main.lisp"))))
      ;; :cmd is now there:
      (is (find :cmd (deps)))
      ;; It contains the example function:
      (is (has-cmd-example-p main-text)))))

N.B.: A keyword argument would also work, obviously. For a macro in production code, I would probably go that route instead.

Fun

I've also been thinking about a recent post on Hacker News, and the ensuing discussion, in which both the author and commenters point out that programming Lisp is fun. While I must acknowledge that Lisp has been a life-long attraction/distraction/diversion, I'm noticing that the way it is fun for me is evolving somewhat.

First, I used to find Common Lisp pretty painful because of all its sharp edges – the language is so powerful and flexible, you can shoot yourself in the foot pretty much any way you like, and it's not as easy as some newer languages to get started with. But now, with LLMs, it's much easier to get help when you get into the weeds. Sure, "the corpus of training data was smaller for Lisp than for mainstream languages," blah, blah. But often even the wrong answers from ChatGPT point me in the right direction (some of this is probably just rubberducking). When one's own ignorance is less of an obstacle, the pointy bits become less intimidating.

Second, there are some really good, and fun, books on Lisp. PAIP, Let Over Lambda, Practical Common Lisp, Land of Lisp (plus video!). Lisp has some pretty mind-bending things in it, and I'm enjoying taking the time to dig into some of these books again, understanding things better than the first time I swung by.

Third, I'm regularly stunned by how fast Lisp is. All the tests I've been writing run effectively instantly at the REPL on my M1 Macbook Air. And now that I'm starting to get good enough at the language that it is getting out of my way, that speed is more noticeable, and, well, fun.

Finally, I am intrigued by the history of computing, of which Lisp has been an important part. This is worth saying more about in a future post, but for the moment I find a stable, highly-performant, fun language that has intrigued people for decades to be more of interest than the bleeding edge. (I'm not hating on Rust. Rust is nice. But I don't find it fun, at least not yet.)

Beyond fun, I'm also enjoying "escaping the hamster wheel of backwards incompatibility" for awhile. While so many things around us crumble and whirl, and new forms of AI scare the pants out of us as much as they intrigue, older tools and traditions start to feel more like comfortable tools that weigh down one's hands satisfyingly and invite calm creation.

To The Metal... Compiling Your Own Language(s)

Tue, 06 Aug 2024 00:00:00 +0000

Like many programmers, I have programming hobbies. One of these is implementing new languages. My most recent language project, l1, was a Lisp dialect whose primary data types are symbols and arbitrarily-large integers.

I've been happy with l1, but it is interpreted; since I was actively working on it last (2022), I've been wondering about the best way to generate compiled standalone executable programs, written in l1 or any other language.

The Problem In General

Execution models for programming languages take three basic approaches, listed in increasing order of speed:

Tree-walking interpreter: Programs are read and parsed into ASTs in memory, then executed step-by-step by an interpreter. This is the approach l1 uses.
Bytecode VM: Programs are compiled into a sort of abstract machine language, simpler than the physical processor's, and executed by a virtual machine (VM). Java and Python work this way.
Machine code generation: The code is directly compiled into machine language and executed on the user's hardware. C and C++ programs work this way.

Languages using Option 2 often add just-in-time compilation to machine code, for extra performance. Option 3 is typically fastest, but is sometimes skipped in introductory compiler classes and tutorials. For example, Robert Nystrom's excellent Crafting Interpreters book devotes the first section to implementing a tree-walking interpreter implementation in Java and the second half to a compiler and bytecode VM written in C, with minimal coverage of how to target physical hardware. And the (also excellent) class on compiler writing that I took from David Beazley, in its first incarnation, stopped at the point of generating of so-called intermediate representation (IR) output (though students in the current iteration of the class do compile to native code, using LLVM).

Compiling to machine code is tricky because CPUs are inherently complex. Real hardware is intricate, cumbersome, and unintuitive if you're primarily accustomed to high-level languages. Additionally, there are numerous significant variants to consider (e.g., CPU/GPU, ARM/Intel, 32-bit/64-bit architectures).

But targeting machine code rather than interpreters or bytecode VMs is appealing, not just because it is an interesting challenge, but also because the resulting artifacts are small, stand-alone, and typically very fast. While running Python, Ruby, and Java programs require the appropriate infrastructure to be in place on the target machine at all times, Go, Rust, and C programs (among others) benefit from targeting the physical hardware: their programs tend to be smaller, and can be deployed to identical computers simply by copying the executable file, needing to deploy the interpreter, extra libraries, etc. to the target machine(s).

Small Is Beautiful

As a programmer who came up during the dawn of personal computers, I have some nostalgia for an era when programs or even entire operating systems fit on a few-hundred-kB floppy disk. Much existing software feels bloated to me, though some widespread tools are still lean and fast. For illustration purposes, here are the physical sizes of some of the venerable command-line Unix programs I use on a daily basis (this is on MacOS):

Program	Size (kB)
`wc`	100
`cat`	116
`df`	116
`more`	360

These were chosen more or less at random from my bash history and are representative of old-school Unix utilities. For comparison, iMovie on my Mac is 2.8 GB, several thousand times larger than the largest of these. Of course, the comparison is somewhat ludicrous - iMovie does many amazing things… but I use all the above programs hundreds or thousands of times more often than I do iMovie, so it's good that that they are compact and run quickly. In a time of increasingly bloated software stacks, I find myself especially drawn to simple tools with small footprints.

An Approach

If targeting physical hardware is hard, what tools can we use to make the job easier?

I recently started learning about LLVM, a modular set of compiler tools which "can be used to develop a frontend for any programming language and a backend for any instruction set architecture" (Wikipedia). LLVM has been used heavily in the Rust toolchain and in Apple's developer tools.

The "modular" adjective is critical here: LLVM is separated into front-end, back-end and optimizing parts thanks to a shared "intermediate representation" (IR) – a sort of portable assembly language which represents simple computation steps in a machine-independent but low-level manner.

The LLVM IR takes a little getting used to but, with a little practice, is reasonably easy to read, and, more importantly, to generate.

As an example, consider the following simple C program, three.c, which stores the number 3 in a variable and uses it as its exit code. We will use clang, the LLVM C/C++/Obj-C/… compiler for the LLVM ecosystem:

$ cat three.c
int x = 3;

int main() {
  return x;
}
$ clang three.c -o three
$ ./three; echo $?
3

One can easily view, and possibly even understand, the assembler output for such a simple program:

$ clang -O3 -S three.c -o three.s
$ cat -n three.s
     1		.section	__TEXT,__text,regular,pure_instructions
     2		.build_version macos, 14, 0	sdk_version 14, 4
     3		.globl	_main                           ; -- Begin function main
     4		.p2align	2
     5	_main:                                  ; @main
     6		.cfi_startproc
     7	; %bb.0:
     8	Lloh0:
     9		adrp	x8, _x@PAGE
    10	Lloh1:
    11		ldr	w0, [x8, _x@PAGEOFF]
    12		ret
    13		.loh AdrpLdr	Lloh0, Lloh1
    14		.cfi_endproc
    15	                                        ; -- End function
    16		.section	__DATA,__data
    17		.globl	_x                              ; @x
    18		.p2align	2, 0x0
    19	_x:
    20		.long	3                               ; 0x3
    21
    22	.subsections_via_symbols

In comparison, here is the LLVM IR for the same program:

$ clang -S -emit-llvm three.c -o three.ll
$ cat -n three.ll
     1	; ModuleID = 'three.c'
     2	source_filename = "three.c"
     3	target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
     4	target triple = "arm64-apple-macosx14.0.0"
     5
     6	@x = global i32 3, align 4
     7
     8	; Function Attrs: noinline nounwind optnone ssp uwtable(sync)
     9	define i32 @main() #0 {
    10	  %1 = alloca i32, align 4
    11	  store i32 0, ptr %1, align 4
    12	  %2 = load i32, ptr @x, align 4
    13	  ret i32 %2
    14	}
    15
    16	attributes #0 = { noinline nounwind optnone ssp ;; .... very long list...
    17
    18	!llvm.module.flags = !{!0, !1, !2, !3, !4}
    19	!llvm.ident = !{!5}
    20
    21	!0 = !{i32 2, !"SDK Version", [2 x i32] [i32 14, i32 4]}
    22	!1 = !{i32 1, !"wchar_size", i32 4}
    23	!2 = !{i32 8, !"PIC Level", i32 2}
    24	!3 = !{i32 7, !"uwtable", i32 1}
    25	!4 = !{i32 7, !"frame-pointer", i32 1}
    26	!5 = !{!"Apple clang version 15.0.0 (clang-1500.3.9.4)"}

There is a fair amount of stuff here, but a lot of it looks suspiciously like metadata we don't really care about for our experiments going forward. The, uh, main region of interest is from lines 9-14 – notice that the function definition itself looks a little more readable than the assembly language version, but slightly lower-level than the original C program.

You can turn the IR into a runnable program:

$ clang -O3 three.ll -o three
$ ./three; echo $?
3

The approach I explore here is to generate LLVM IR "by fair means or foul." Here, let's just edit our IR down to something more minimal and see how it goes. I suspect the store of 0 in "register" %1 is gratuitous, so let's try to remove it, along with all the metadata:

$ cat 3.ll
target triple = "x86_64-apple-macosx14.0.0"

@x = global i32 3, align 4

define i32 @main() {
  %1 = load i32, ptr @x, align 4
  ret i32 %1
}
$ clang -O3 3.ll -o 3
$ ./3; echo $?
3

This is frankly not much more complicated than the C code, and it shows a helpful strategy at work:

Step 1: To understand how to accomplish something in LLVM IR, write the corresponding C program and use clang to generate the IR, being alert for possible "extra stuff" like we saw in the example.

Step 2. Try to generate, and test, working programs from the IR you write or adapt, making adjustments as desired.

There is another, optional step as well:

Step 3. Use, or write, language "bindings" to drive LLVM generation from the language of your choice. This is the step we will consider next.

Enter Babashka

While one can write LLVM IR directly, as we have seen, we are interested in compiling other languages (possibly higher-level ones of our own invention), so we will want to generate IR somehow. For this project I chose Babashka, an implementation of the Clojure programming language I have found ideal for small projects where both start-up speed and expressiveness are important.

(I assume some familiarity with Lisp and Clojure in this post; for those just getting started, Clojure for the Brave and True is a good introduction.)

The repo https://github.com/eigenhombre/llbb contains the source files discussed here. The bulk of the code in this repo is in llir.bb, a source file which provides alternative definitions in Clojure for common LLVM idioms. Some of these are trivial translations:

(def m1-target "arm64-apple-macosx14.0.0")

(defn target [t] (format "target triple = \"%s\"" t))

… whereas other expressions leverage the power of Clojure to a greater degree. For example, this section defines translations used to represent arithmetic operations:

(defn arithm [op typ a b]
  (format "%s %s %s, %s"
          (name? op)
          (name? typ)
          (sigil a)
          (sigil b)))

(defn add [typ a b] (arithm :add typ a b))
(defn sub [typ a b] (arithm :sub typ a b))
(defn mul [typ a b] (arithm :mul typ a b))
(defn div [typ a b] (arithm :sdiv typ a b))

(comment
  (div :i32 :a :b)
  ;;=>
  "sdiv i32 %a, %b"

  (add :i8 :x 1)
  ;;=>
  "add i8 %x, 1")

You can see this approach at work by representing the C program discussed earlier:

(module
 (assign-global :i :i32 3)
 (def-fn :i32 :main []
   (assign :retval (load :i32 :i))
   (ret :i32 :retval)))

which evaluates to

target triple = "arm64-apple-macosx14.0.0"

@i = global i32 3

define i32 @main() nounwind {
  %retval = load i32, i32* @i, align 4
  ret i32 %retval
}

I use here a slightly different but equivalent pointer syntax for the load expression than output by clang in the example above.

Two very small helper functions allow me to test out small programs quickly:

(require '[babashka.process :as sh])

(defn sh
  "
  Use `bash` to run command(s) `s`, capturing both stdout/stderr
  as a concatenated string.  Throw an exception if the exit code
  is nonzero.
  "
  [s]
  (let [{:keys [out err]}
        (sh/shell {:out :string, :err :string}
                  (format "bash -c '%s'" s))]
    (str/join "\n" (remove empty? [out err]))))

and

(require '[babashka.fs :as fs])

(defn compile-to
  "
  Save IR `body` to a temporary file and compile it, writing the
  resulting binary to `progname` in the current working directory.
  "
  [progname body]
  (let [ll-file
        (str (fs/create-temp-file {:prefix "llbb-", :suffix ".ll"}))]
    (spit ll-file body)
    (sh (format "clang -O3 %s -o %s"
                ll-file
                progname))))

These two together allow me to test small programs out quickly at the REPL. Some examples follow, obtained by running the C compiler for equivalent programs, generating and inspecting the LLVM IR, and translating them into new Clojure bindings as cleanly as possible.

Minimum viable program: just return zero:

(compile-to "smallest-prog"
            (module
             (def-fn :i32 :main []
               (ret :i32 0))))

(sh "./smallest-prog; echo -n $?")
;;=>
"0"

Argument count: return, as the exit code, the number of arguments, including the program name:

;; Argument counting: return number of arguments as an exit code:
(compile-to "argcount"
            (module
             (def-fn :i32 :main [[:i32 :arg0]
                                 [:ptr :arg1_unused]]
               (assign :retptr (alloca :i32))
               (store :i32 :arg0 :ptr :retptr)
               (assign :retval (load :i32 :retptr))
               (ret :i32 :retval))))

(sh "./argcount; echo -n $?") ;;=> "1"
(sh "./argcount 1 2 3; echo -n $?") ;;=> "4"

Hello, world:

(let [msg "Hello, World."
      n (inc (count msg))] ;; Includes string terminator
  (compile-to "hello"
              (module
               (external-fn :i32 :puts :i8*)
               (def-global-const-str :message msg)
               (def-fn :i32 :main []
                 (assign :as_ptr
                         (gep (fixedarray n :i8)
                              (star (fixedarray n :i8))
                              (sigil :message)
                              [:i64 0]
                              [:i64 0]))
                 (call :i32 :puts [:i8* :as_ptr])
                 (ret :i32 0)))))

(sh "./hello") ;;=> "Hello, World.\n"

This is the first program one typically writes in a new programming language. Note that we use here idioms (external-fn, call) to define and invoke an external function from the C standard library.

Let's see how big the resulting program is:

(sh "ls -l hello")
;;=>
"-rwxr-xr-x  1 jacobsen  staff  33432 Aug 14 21:09 hello\n"

At this point I want to pause to reconsider one of the points of this exercise, which is to produce small programs. Here are the rough executable sizes for a "Hello, World" example program in various languages that I use frequently:

Language	Size	Relative Size
Common Lisp	38 MB	1151
Clojure	3.4 MB	103
Go	1.9 MB	58
C	33 kB	1
LLVM IR	33 kB	1

I threw Clojure in there for comparison even though, unlike the other examples, the resulting überjar also requires a Java bytecode VM in order to run. The programs generated from C and from LLVM IR are equivalent; this is not surprising, given that I used the C program to guide my writing and translation of the LLVM IR.

Building A Compiling Calculator

We are now ready to implement a "compiler" for something approaching a useful language, namely, greatly reduced subset of Forth. Forth is a stack-based language created in the 1970s and still in use today, especially for small embedded systems.

LLVM will handle the parts commonly known as "compiler backend" tasks, and Babashka will provide our "frontend," namely breaking the text into tokens and parsing them. This task is made easy for us, because Forth is syntactically quite simple, and Babashka relatively powerful. Here are the language rules we will adopt:

Program tokens are separated by whitespace.
Non-numeric tokens are math operators.
Only integer operands are allowed.
Comments begin with \\.

Forth expressions typically place the arguments first, and the operator last (so-called "reverse-Polish notation"). Here is an example program which does some math and prints the result:

(def example "

2 2 +  \\ 4
5 *    \\ multiply by five to get 20
2 /    \\ divide by 2 -> 10
-1 +   \\ add -1 -> 9
8 -    \\ subtract 8 -> 1
.      \\ prints '1'

")

The code in forth.bb handles the parser, whose goal is to consume the raw program text and generate an abstract syntax tree (in our case, just a list) of operations to translate into IR:

(defn strip-comments
  "
  Remove parts of lines beginning with backslash
  "
  [s]
  (str/replace s #"(?sm)^(.*?)\\.*?$" "$1"))

(defn tokenize
  "
  Split `s` on any kind of whitespace
  "
  [s]
  (remove empty? (str/split s #"\s+")))

(defrecord node
    [typ val] ;; A node has a type and a value
  Object
  (toString [this]
    (format "[%s %s]" (:typ this) (:val this))))

;; Allowed operations
(def opmap {"+" :add
            "-" :sub
            "/" :div
            "*" :mul
            "." :dot
            "drop" :drop})

(defn ast
  "
  Convert a list of tokens into an \"abstract syntax tree\",
  which in our Forth is just a list of type/value pairs.
  "
  [tokens]
  (for [t tokens
        :let [op (get opmap t)]]
    (cond
      ;; Integers (possibly negative)
      (re-matches #"^\-?\d+$" t)
      (node. :num (Integer. t))

      ;; Operations
      op (node. :op op)

      :else (node. :invalid :invalid))))

Running this on our example,

(->> example
     strip-comments
     tokenize
     ast
     (map str))

;;=>
("[:num 2]"
 "[:num 2]"
 "[:op :add]"
 "[:num 5]"
 "[:op :mul]"
 "[:num 2]"
 "[:op :div]"
 "[:num -1]"
 "[:op :add]"
 "[:num 8]"
 "[:op :sub]"
 "[:op :dot]")

The remainder of forth.bb essentially just implements the needed operators, as well as the required stack and the reference to the printf C library function. It is perhaps a bit lengthy to go through here in its entirety, so I will share one example where Babashka helps:

(defn def-arithmetic-op [nam op-fn]
  (def-fn :void nam []
    (assign :sp (call :i32 :get_stack_cnt))
    (if-lt :i32 :sp 2
           (els)  ;; NOP - not enough on stack
           (els
            (assign :value2
                    (call :i32 :pop))
            (assign :value1
                    (call :i32 :pop))
            (assign :result (op-fn :i32 :value1 :value2))
            (call :void :push [:i32 :result])))
    (ret :void)))

This makes LLVM IR which does the following, in pseudo-code:

get current stack position; ensure at least two entries (else return)
pop the operands off the stack
apply the arithmetic operator to the operands
put the result on the stack

Implementing the four arithmetic operators is then as simple as invoking

;; ...
(def-arithmetic-op :mul mul)
(def-arithmetic-op :add add)
(def-arithmetic-op :sub sub)
(def-arithmetic-op :div div)
;; ...

when generating the IR.

Aside from the general-purpose LLVM IR code (llir.bb), the Forth implementation is under two hundred lines. It includes the invocation to clang to compile the temporary IR file to make the runnable program. Here's an example compilation session:

$ cat example.fs

       \\ initial state          stack: []
3      \\ put 3 on stack.        stack: [3]
99     \\ put 99 on stack.       stack: [3, 99]
drop   \\ discard top item.      stack: [3]
drop   \\ discard top item.      stack: []
2 2    \\ put 2 on stack, twice: stack: [2, 2]
+      \\ 2 + 2 = 4.             stack: [4]
5 *    \\ multiply 4 * 5.        stack: [20]
2 /    \\ divide by 2.           stack: [10]
-1 +   \\ add -1                 stack: [9]
8 -    \\ subtract 8 -> 1        stack: [1]
.      \\ prints '1'             stack: [1]
drop   \\ removes 1.             stack: []

$ ./forth.bb example.fs
$ ./example
1

The resulting program is fast, as expected…

$ time ./example
1

real	0m0.007s
user	0m0.002s
sys	0m0.003s

… and small:

$ ls -l ./example
-rwxr-xr-x  1 jacobsen  staff  8952 Aug 16 09:52 ./example

Let's review what we've done: we have implemented a small subset of Forth, writing a compiler front-end in Babashka/Clojure to translate source programs into LLVM IR, and using clang to turn the IR into compact binaries. The resulting programs are small and fast.

Sensible next steps would be to implement more of Forth's stack operators, and maybe start to implement the : (colon) operator, Forth's mechanism for defining new symbols and functions.

Lisp

Instead, let's implement a different variant of our arithmetic calculating language, using Lisp syntax (S-expressions). Consider our first Forth example:

2 2 +
5 *
2 /
-1 +
8 -
.

In Lisp, this looks like:

$ cat example.lisp
(print (- (+ -1
             (/ (* 5
                   (+ 2 2))
                2))
          8))

Rather than coming last, as they did before ("postfix" notation), our operators come first ("prefix" notation). The order of operations is determined by parentheses, as opposed to using stack as we did for our Forth implementation.

Here Babashka helps us tremendously because such parenthesized prefix expressions are valid EDN data, which the Clojure function clojure.edn/read-string can parse for us. But we need to convert the resulting nested list into the "SSA" (single static assignment) expressions LLVM understands. This is relatively straightforward with a recursive function which expands leaves of the tree and stores the results as intermediate values:

(defn to-ssa [expr bindings]
  (if (not (coll? expr))
    expr
    (let [[op & args] expr
          result (gensym "r")
          args (doall
                (for [arg args]
                  (if-not (coll? arg)
                    arg
                    (to-ssa arg bindings))))]
      (swap! bindings conj (concat [result op] args))
      result)))

(defn convert-to-ssa [expr]
  (let [bindings (atom [])]
    (to-ssa expr bindings)
    @bindings))

We use gensym here to get a unique variable name for each assignment, and doall to force the evaluation of the lazy for expansion of the argument terms. The result:

(->> "example.lisp"
     slurp
     (edn/read-string)
     convert-to-ssa)
;;=>
[(r623 + 2 2)
 (r622 * 5 r623)
 (r621 / r622 2)
 (r620 + -1 r621)
 (r619 - r620 8)
 (r618 print r619)]

The next step will be to actually write out the corresponding LLVM IR. The rest of lisp.bb is satisfyingly compact. Operators (we have five, but more can easily be added), are just a map of symbols to tiny bits of LLVM code:

(def ops
  {'* #(mul :i32 %1 %2)
   '+ #(add :i32 %1 %2)
   '/ #(div :i32 %1 %2)
   '- #(sub :i32 %1 %2)
   'print
   #(call "i32 (i8*, ...)"
          :printf
          [:i8* :as_ptr]
          [:i32 (sigil %1)])})

Similar to our Forth implementation, but even more compact, the main Babashka function, after a brief setup for printf, generates a series of SSA instructions.

(defn main [[path]]
  (when path
    (let [assignments (->> path
                           slurp
                           edn/read-string
                           convert-to-ssa)
          outfile (->> path
                       fs/file-name
                       fs/split-ext
                       first)
          ir (module
              (external-fn :i32 :printf :i8*, :...)
              (def-global-const-str :fmt_str "%d\n")
              (def-fn :i32 :main []
                (assign :as_ptr
                        (gep (fixedarray 4 :i8)
                             (star (fixedarray 4 :i8))
                             (sigil :fmt_str)
                             [:i64 0]
                             [:i64 0]))

                ;; Interpolate SSA instructions / operator invocations:
                (apply els
                       (for [[reg op & args] assignments
                             :let [op-fn (ops op)]]
                         (if-not op-fn
                           (throw (ex-info "bad operator" {:op op}))
                           (assign reg (apply op-fn args)))))
                (ret :i32 0)))]
      (compile-to outfile ir))))

(main *command-line-args*)

Putting these parts together (see lisp.bb on GitHub), we have:

$ ./lisp.bb example.lisp
$ ./example
1

It, too, is small and fast:

$ time ./example
1

real	0m0.006s
user	0m0.001s
sys	0m0.003s
$ ls -al ./example
-rwxr-xr-x  1 jacobsen  staff  33432 Aug 16 20:52 ./example

To say this is a "working Lisp compiler" at this point would be grandiose (we still need functions, lists and other collection types, eval, macros, …) but we have developed an excellent foundation to build upon.

To summarize, the strategy we have taken is as follows:

Use a high level language (in our case, Babashka/Clojure) to parse input and translate into LLVM IR;
When needed, write and generate small C programs to understand the equivalent IR to generate.
Compile the IR to small, fast binaries using clang.

Alternatives and Future Directions

I should note that C itself has long been used as an intermediate language, and we could have used it here instead of LLVM IR; I don't have a strong sense of the tradeoffs involved yet, but wanted to take the opportunity to learn more about LLVM for this project.

LLVM is interesting to me because of the modularity of its toolchain; it also provides a JIT compiler which allows one to build and execute code at run-time. We didn't investigate tooling for that here (it needs deeper LLVM language bindings than the homegrown Babashka code I used), but it could provide a way to do run-time compilation similar to what SBCL (a Common Lisp implementation which can compile functions at run-time) does.

Here are some directions I'm considering going forward:

Try interfacing with external libraries, e.g. a bignum library;
Implement more Forth functionality;
Implement more Lisp, possibly including a significant subset of l1;
Try a JIT-based approach, possibly using Rust as the host language.

Conclusion

Whenever possible, I want to make small, fast programs, and I like playing with and creating small programming languages. LLVM provides a fascinating set of tools and techniques for doing so, and using Babashka to make small front-ends for IR generation turns out to be surprisingly effective, at least for simple languages.

Adding Tail Call Optimization to A Lisp Written in Go

Mon, 08 Aug 2022 00:00:00 +0000

The last few days have been devoted to improving l1, the homegrown lisp I wrote about earlier this year. A number of changes have landed in the last week:

Implemented let, defn and sugar for quote;
Figured out basic REPL integration with Emacs;
Added numeric comparison operators;
Reviewed my Minimum Viable Repo checklist for this project;
Fixed four bugs.

I also implemented the bulk of the automated tests in the language itself. This was a decisive step forward in both ease of creating new tests and confidence that the language was approaching something usable.

The work I'm happiest with, though, because it taught me the most, was implementing tail call optimization (TCO) in the language, which the rest of this post will be about.

Motivation

The need for some form of TCO became clear as I started to write more small programs in l1. Perhaps the simplest example is one that sums all the natural numbers up to $n$:

(defn sum-to-acc (n acc)
  (cond ((zero? n) acc)
        (t (sum-to-acc (- n 1) (+ n acc)))))

(defn sum-to (n)
  (sum-to-acc n 0))

Calling sum-to for small $n$ worked fine:

(sum-to 100)
;;=>
5050

However, larger $n$ blew up spectacularly:

(sum-to (* 1000 1000))
;;=>
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x14020500360 stack=[0x14020500000, 0x14040500000]
fatal error: stack overflow

runtime stack:
runtime.throw({0x10289aa2b?, 0x10294ddc0?})
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/panic.go:992 +0x50
runtime.newstack()
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/stack.go:1101 +0x46c
runtime.morestack()
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/asm_arm64.s:314 +0x70

goroutine 1 [running]:
strings.(*Reader).ReadByte(0x1401c2820e0?)
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/strings/reader.go:66 +0x98 fp=0x14020500360 sp=0x14020500360 pc=0x102883348
math/big.nat.scan({0x0, 0x1401c2820e0?, 0x0}, {0x1028dc7c8, 0x1401c2820e0}, 0xa, 0x0)
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/math/big/natconv.go:126 +0x80 fp=0x14020500430 sp=0x14020500360 pc=0x10288b1e0
;; ...

This happens, of course, because sum-to-acc calls itself a million times, each time storing a copy of its local bindings on the stack, which eventually consumes all the space on the stack.

Getting simple recursive functions like this to work for large $n$ is especially important because l1 doesn't have loops (yet)!

The Optimization

The solution is hinted at already in my test case. Note that I did not write sum-to as a single recursive function, as follows:

(defn sum-to-notail (n)
  (cond ((zero? n) 0)
        (t (+ n (sum-to-notail (- n 1))))))

While this function looks slightly simpler, it is harder for a compiler or interpreter to optimize. The difference is that, whereas sum-to-notail does some work after calling itself (by adding n to the result), sum-to-acc calls itself from the tail position; that is, the function returns immediately after calling itself.

People have long realized that function calls from the tail position can be replaced by updating the return address and then jumping directly to the the new function without adding new information to the stack. This is something I had heard about for years and used in various "functional" languages, without ever really implementing myself (and therefore fully understanding). It's an easy thing to take for granted without knowing anything about how it's actually implemented under the hood. The failure of sum-to-acc and similar recursive functions, described above, meant I would have to learn.

Two very different blog posts were helpful to me in pointing the way forward: Adding tail call optimization to a Lisp interpreter in Ruby, and How Tail Call Optimization Works. The posts focus on very different languages (Ruby vs. C / assembler), but they each revolve around what are effectively GOTO statements. I'm old enough to remember BASIC and the pernicious GOTO statement leading to "spaghetti code." I doubt I've ever used a GOTO statement in production code, whose use in modern programming languages fell out of favor in the aftermath of Dijkstra's famous Go To Statement Considered Harmful paper. But the ability to transfer control to another part of your program without invoking a function call is key to the optimization.

The Approach

Since the strategy is general, let's lose all the parentheses for a moment and rewrite sum-to-acc in language-agnostic pseudo-code:

function sum-to-acc(n sum)
   if n == 0, then return sum
   return sum-to-acc(n - 1, n + sum)

In most languages (without TCO), when this function is called, the values of n and sum, as well as the return address, will be put on the stack, whose evolution looks something like the following.¹:

 first invocation: [n=5, sum=0,  ret=sum-to:...]

second invocation: [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 third invocation: [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

fourth invocation: [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 fifth invocation: [n=1, sum=14, ret=sum-to-acc:...]
                   [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 sixth invocation: [n=0, sum=15, ret=sum-to-acc:...]
                   [n=1, sum=14, ret=sum-to-acc:...]
                   [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

At the sixth invocation, our terminating condition is reached, and 15 is returned, with all the pending stack frames popped off the stack.

With TCO, the implementation looks more like the following:

function sum-to-acc(n sum)
TOP:
   if n == 0, then return sum
   n = n - 1
   sum = sum + n
   GOTO TOP

as a result, the evolution of the stack looks as follows:

 first invocation: [n=5, sum=0, ret=sum-to:...]

second invocation: [n=4, sum=5, ret=sum-to:...]

 third invocation: [n=3, sum=9, ret=sum-to:...]

fourth invocation: [n=2, sum=12, ret=sum-to:...]

 fifth invocation: [n=1, sum=14, ret=sum-to:...]

 sixth invocation: [n=0, sum=15, ret=sum-to:...]

All those extra stack frames are gone: recursion has turned into a form of iteration.

Implementing TCO, then, has two ingredients:

Replace the values of the current arguments with their new values directly.
Jump straight to the next call of the function without adding to the stack;

This low-level, imperative optimization makes high-level, functional, recursive implementations efficient.

Implementation

In thinking about the implementation for l1, I was pleased to learn that Go actually has a goto statement. However, my implementation was poorly set up to use it.

Early in the implementation of l1, I noticed that each data type (numbers, atoms, and lists) had its own evaluation rules, so it made sense to make use of Go's features supporting polymorphism, namely interfaces and receivers. I had a Sexpr interface which looked like the following:

type Sexpr interface {
	String() string
	Eval(*env) (Sexpr, error)  // <--------
	Equal(Sexpr) bool
}

Numbers and atoms, for example, had fairly simple Eval implementations. For example,

func (a Atom) Eval(e *env) (Sexpr, error) {
	if a.s == "t" {
		return a, nil
	}
	ret, ok := e.Lookup(a.s)
	if ok {
		return ret, nil
	}
	ret, ok = builtins[a.s]
	if ok {
		return ret, nil
	}
	return nil, fmt.Errorf("unknown symbol: %s", a.s)
}

And, of course, numbers eval to themselves:

func (n Number) Eval(e *env) (Sexpr, error) {
	return n, nil
}

Lists, as you would expect, were more complicated – evaluating a list expression needs to handle special forms², user-defined functions, and built-in functions. Following the classic Structure and Interpretation of Computer Programs, I separated the core logic for function application into separate Eval and Apply phases. And to prevent the Eval for lists from getting too large, I broke out the evaluation rules for different cases (e.g. for let and cond special forms and for function application) into their own functions.

In other words, I had evaluation logic spread over ten functions in five files. Sadly, the need to jump back to the beginning of an evaluation rather than recursively calling Eval again meant that several of those nicely broken out functions had to be brought together into a single function, because goto does not support jumping from one function to another. (C has setjmp and longjmp, which effectively do this, but I would want to upgrade my IQ by a few points before applying them in this situation.)

There were actually three cases where I was performing an evaluation step right before returning, and the goto pattern could be used:

When evaluating code in the tail position of a user-defined function;
When evaluating code in the last expression in a let block;
When evaluating code in the chosen branch of a cond clause.

I wound up with code which with looks like the following. Several steps are indicated only with comments. Note the tiny, easy-to-miss top: label at the very beginning:

// lisp.go
//
func eval(expr Sexpr, e *env) (Sexpr, error) {
top:
	switch t := expr.(type) {
	case Atom:
		return evAtom(t, e)
	case Number:
		return expr, nil
	// ...
	case *ConsCell:
		if t == Nil {
			return Nil, nil
		}
		// special forms:
		if carAtom, ok := t.car.(Atom); ok {
			switch {
			case carAtom.s == "quote":
				return t.cdr.(*ConsCell).car, nil
			case carAtom.s == "cond":
				pairList := t.cdr.(*ConsCell)
				if pairList == Nil {
					return Nil, nil
				}
				for {
					if pairList == Nil {
						return Nil, nil
					}
					pair := pairList.car.(*ConsCell)
					ev, err := eval(pair.car, e)
					if err != nil {
						return nil, err
					}
					if ev == Nil {
						pairList = pairList.cdr.(*ConsCell)
						continue
					}
					expr = pair.cdr.(*ConsCell).car
					goto top
				}
			// ...

The code so far shows the evaluation for atoms, numbers, and cond statements. cond does not introduce any new bindings, but when the first truthy condition is encountered, it evaluates the next argument as its final act. So the code above simply replaces the expression to be evaluated, expr, with the expression from the matching clause, and then restarts the evaluation via goto, without the overhead of a separate function call.

The code for let is somewhat similar:

			case carAtom.s == "let":
				args := t.cdr.(*ConsCell)
				if args == Nil {
					return nil, fmt.Errorf("let requires a binding list")
				}
				// ... code to set up let bindings ...
				body := args.cdr.(*ConsCell)
				var ret Sexpr = Nil
				for {
					var err error
					if body == Nil {
						return ret, nil
					}
					// Implement TCO for `let`:
					if body.cdr == Nil {
						expr = body.car
						e = &newEnv
						goto top
					}
					ret, err = eval(body.car, &newEnv)
					if err != nil {
						return nil, err
					}
					body = body.cdr.(*ConsCell)
				}

The for loop invokes a new eval for each expression in the body of the let, except for the last one: when the last expression is reached, (the cdr is Nil), the last eval is done by jumping to the beginning of the function, once it has updated its environment to point to include the new bindings.

The last use of this pattern is in function invocation proper, which looks similar:

			// (... code to set up new environment based on passed arguments ...)
			var ret Sexpr = Nil
			for {
				if lambda.body == Nil {
					return ret, nil
				}
				// TCO:
				if lambda.body.cdr == Nil {
					expr = lambda.body.car
					e = &newEnv
					goto top
				}
				ret, err = eval(lambda.body.car, &newEnv)
				if err != nil {
					return nil, err
				}
				lambda.body = lambda.body.cdr.(*ConsCell)
			}

I've skipped various parts of eval that aren't relevant for TCO optimization – if you're interested, you can check out the code yourself.

To be clear, what we are optimizing is all tail calls, not just recursive ones – though the recursive ones were the primary objective due to the stack overflows reported above.

The end result is that sum-to now can complete for large values of $n$:

(sum-to (* 1000 1000))
;;=>
500000500000

Incidentally, a variant of our test case failed before I added the TCO optimization to let shown above; this now works, as well:

(defn sum-to-acc-with-let (n acc)
  (let ((_ 1))
    (cond ((zero? n) acc)
          (t (sum-to-acc-with-let (- n 1) (+ n acc))))))

(defn sum-to-with-let (n) (sum-to-acc-with-let n 0))

(sum-to-with-let (* 1000 1000))
;;=>
500000500000

Conclusion

Getting tail-call optimization to work was very satisfying… though the eval implementation is certainly more complex than before. (Ah, optimization!)

To ensure TCO continues to work, variants of sum-to with and without let are run on every build, along with a few other short example programs.

After implementing TCO in my own code, I can appreciate and understand the optimization better when I see it in the wild. I fully expect to use the pattern again when implementing future lisps (yes, I hope there will be more).

Note that this is a somewhat abstract representation: the details are language-specific. The ret=sum-to:... notation means that when the function returns, control will pass back to where it left off inside the sum-to function.

A special form is one that does not follow the normal evaluation rule for functions – it may evaluate its arguments once, many times, or not at all. (I am glossing over macros for the time being; l1 does not have them yet.)

Tests by Example in Clojure and Common Lisp

Wed, 13 Jul 2022 00:00:00 +0000

Histogram made by a 1980's version of the author.

While playing around with some dice-rolling simulations recently in Common Lisp (a topic for some future post, perhaps), I looked around for a way of making histograms. Finding none close to hand (it being Common Lisp), it wasn't too hard to "roll my own" library (it being… Common Lisp) to create text-based histograms in the tradition of libraries from CERN that I used as a fledgling physicist in the late 1980s.

Since the project was just for fun, I drove most of the code forward just by REPLing. But I am trying to follow the same habits in my "for fun" GitHub repositories as I would for paid work. This means, among other things, unit tests (in fact, for critical work I still prefer to use TDD when I can).

So, in fleshing the tests out after the fact, I found myself writing somewhat repetitive code that looked like this:

(test hist-values-test
  (let ((h (histogram '(1 2) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(1 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 1) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(2 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 2 3) 3)))
    (is (= 3 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 3 (hist-max h)))
    (is (equalp #(1 2 1) (hist-bin-heights h)))
    (is (equalp #(1 2 3) (hist-bin-xs h)))))

This is using the minimalist 1AM testing library, which I like for its simplicity. It is also similar to the standard clojure.test library I'm used to in Clojure. It lacks, however, one important feature of clojure.test, namely the are macro. The macro allows one to represent tests as a table of examples:

;; Clojure code:
(deftest example
  (are
    [nums       prod]         ;; [1]

    (= prod (reduce * nums))  ;; [2]

    []          1             ;; [3]
    [1]         1
    [0]         0
    [0 1]       0
    [1 1]       1
    [2 2]       4
    [2 3]       6
    [-1 -1]     1
    [2 2 2]     8
    [1000 1000] 1000000))

An invocation of are has three parts: a vector of symbols to be bound to values from the examples to follow [1]; the expression to be evaluated for each example [2]; and the actual examples [3], with one example per line being the usual, but optional, convention. (In the code above I have added extra spaces to make clear that nums goes with the first column of data, and prod goes with the second column.)

If at first you didn't understand that code being tested ([2]) was related to multiplication, you'd probably be able to figure it out just by looking at the examples. This is the primary advantage of are, namely that examples often provide as good or better documentation than a written description of functionality does.

Missing such a compact testing idiom for my unit tests, I found myself wondering how hard it would be to port it to Common Lisp. The answer, as you might expect, was: Not that hard. As is often the case, there is not much actual Clojure code behind the function in question, and the corresponding Common Lisp code is similarly compact, as we shall see.

If you look at the actual function definition in clojure.test, most of the macro proper is error checking; the actual work is delegated to a completely different namespace, clojure.template (in what follows, I will ignore the error checking):

;; Clojure code:
(defmacro are [argv expr & args]
  ;; ... error checking
  `(clojure.template/do-template ~argv (is ~expr) ~@args)
  ;; ... error checking
)

A few macro expansions shows what's happening:

;; Clojure code:
(macroexpand-1
 '(are [nums prod]
    (= prod (reduce * nums))
    []      1
    [1]     1
    [2 2 2] 8))

gives

(clojure.template/do-template
 [nums prod]
 (clojure.test/is (= prod (reduce * nums)))
 []      1
 [1]     1
 [2 2 2] 8)

do-template is also a macro. Simply running macroexpand-1 again shows what it does:

;; Clojure code:
(macroexpand-1
 '(clojure.template/do-template
   [nums prod]
   (clojure.test/is (= prod (reduce * nums)))
   []      1
   [1]     1
   [2 2 2] 8))

gives

(do (clojure.test/is (= 1 (reduce * [])))
    (clojure.test/is (= 1 (reduce * [1])))
    (clojure.test/is (= 8 (reduce * [2 2 2]))))

The expanded code is exactly equivalent to what you would write if you didn't have the are macro. This is accomplished by simply repeating the code being invoked, but substituting the actual example values for nums and prod.

In order to do accomplish the same in Common Lisp, we take a bottom-up approach. First, we need to split our test cases into groups corresponding to the number of columns in our argv vector (in this case, 2):

(let ((cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (partition-n 2 2 cases))
;;=>
'((#()      1)
  (#(1)     1)
  (#(2 2 2) 8))

The output looks very similar to cases, but each pair has been grouped into its own list. Here I use partition-n from cl-oju, a Common Lisp library I wrote specifically to take advantage of sequence idioms from the Clojure core library; partition-n is a translation of Clojure's partition function. (The equivalent code in clojure.template also uses partition for the same purpose.)

We now need to flesh out those cases by making use of each example in the expression being tested, whose Common Lisp translation is

(is (= prod (reduce #'* nums)))

For example, if nums is #(2 2 2), then prod should be 8.

In other words, we want our test cases to be:

(is (= 1 (reduce #'* #())))
(is (= 1 (reduce #'* #(1))))
(is (= 8 (reduce #'* #(2 2 2)))))

The replacement of symbols with their actual values in the Clojure code is accomplished with a tree-walking function called postwalk-replace. We can do it in Common Lisp with the subst function, which works on linear or nested lists:

(subst 3 'b '(* (+ b b) (/ b (- b b))))
;;=>
'(* (+ 3 3) (/ 3 (- 3 3)))

But because we have multiple bindings or example arguments we have to convert, we need to do it repeatedly:

(defun apply-bindings (bindings case expr)
  (loop with ret = expr
        for b in bindings
        for c in case
        do (setf ret (subst c b ret))
        finally (return ret)))

(apply-bindings '(x y) '(2 3) '(+ x y))
;;=>
'(+ 2 3)

Now we have everything we need to expand our test cases:

(let ((argv '(nums prod))
      (expr '(is (= prod (reduce #'* nums))))
      (c (length argv))
      (cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (loop for case in (partition-n c c cases)
        collect (apply-bindings argv case expr)))
;;=>
'((IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

All the macro has to do is wrap this in a progn:

(defmacro are (argv expr &rest cases)
  "
  Analog of clojure.test/are.  Apply multiple assertions
  based on some set up code and variations of one or more
  variable bindings.
  "
  (let ((c (length argv)))
    `(progn ,@(loop for case in (partition-n c c cases)
                    collect (apply-bindings argv case expr)))))

giving, for our example:

(macroexpand-1
 '(are (nums prod)
   (is (= prod (reduce #'* nums)))
   #()      1
   #(1)     1
   #(2 2 2) 8))
;;=>
'(PROGN
  (IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

This is equivalent to the macroexpansion of Clojure's are that we saw above. I leave as an exercise for the reader a further exploration of clojure.template/apply-template and its use in the do-template macro; and the addition of error checking such as making sure the number example values is a multiple of the number of binding arguments.

One rough edge that I glossed over: in my first attempt, my macro couldn't find apply-bindings at compile time until I wrapped that function's definition as follows:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defun apply-bindings (...) ...))

This is apparently needed for functions used by macros defined in the same file; it's not an issue for functions defined in other files. Since Clojure has a single-pass compiler, this sort of ceremony is not needed for Clojure macros.

I expect to use this macro again, so will probably spin it out into a small stand-alone library or incorporate it into a thin wrapper around 1AM. Here it is in action in the hbook library where our discussion started:

(test hist-values-test
  (are (h c mn mx heights xs)
       (progn
         (is (= c (hist-count h)))
         (is (= mn (hist-min h)))
         (is (= mx (hist-max h)))
         (is (equalp heights (hist-bin-heights h)))
         (is (equalp xs (hist-bin-xs h))))

       (histogram '(1 2)     2)   2 1 2 #(1 1)   #(1 2)
       (histogram '(1 2 1)   2)   2 1 2 #(2 1)   #(1 2)
       (histogram '(1 2 2 3) 3)   3 1 3 #(1 2 1) #(1 2 3)))

The code which are makes possible is usually denser and can be harder to parse initially than more repetitive test code would, as is clear from this example. However, I find the brevity of are tests extremely helpful, especially when comparing the test examples to each another. Such examples are especially helpful when calling out edge cases where behavior changes. I have given such tables to stakeholders many times, in order to make sure we are aligned on what the behavior should be. (Even if the stakeholder doesn't care about the details of your implementation or your test code per se, they probably will care about the specific examples you provide).

Parenthetically, here is the histogramming library in action, showing the distribution of the sum of 1000 six-sided dice:

(defun d () (1+ (random 6)))
(defun dn (n) (loop repeat n sum (d)))

(princ (hbook (loop repeat 100000 collect (dn 1000))
              80
              30))
;;=>

      4962                                       X  X
      4791                                       X  X
      4620                                     X X  X X
      4449                                     X X  X X
      4278                                     X XXXXXX X
      4106                                     X XXXXXX X
      3935                                   X XXXXXXXXXX
      3764                                   XXXXXXXXXXXX
      3593                                   XXXXXXXXXXXX
      3422                                   XXXXXXXXXXXX
      3251                                   XXXXXXXXXXXXX X
      3080                                  XXXXXXXXXXXXXXXX
      2909                                X XXXXXXXXXXXXXXXX
      2738                                XXXXXXXXXXXXXXXXXX
      2566                                XXXXXXXXXXXXXXXXXX
      2395                                XXXXXXXXXXXXXXXXXXXX
      2224                               XXXXXXXXXXXXXXXXXXXXX
      2053                              XXXXXXXXXXXXXXXXXXXXXX
      1882                              XXXXXXXXXXXXXXXXXXXXXX
      1711                              XXXXXXXXXXXXXXXXXXXXXXX
      1540                             XXXXXXXXXXXXXXXXXXXXXXXXX
      1369                             XXXXXXXXXXXXXXXXXXXXXXXXXX
      1198                           XXXXXXXXXXXXXXXXXXXXXXXXXXXX
      1027                           XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       856                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       684                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       513                       XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       342                       XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       171                    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
         0 X   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX    X

                                     111222233344544544443332211111
                            11223568933623981977003313702313558540076333111
                      11244625381526456400925471219232518023527888048930961854221 1
           1   11235564827658185382948873231241519783970927498485883199710549330480234    1

           33333333333333333333333333333333333333333333333333333333333333333333333333333333
           22222222233333333333333334444444444444445555555555555555666666666666666677777777
           45566788900122344556778990112334456678890011233455677899001223445667889901123345
           40639628518417306395285184073062952851740739629528417406396295184173063962851841
           ................................................................................
           04826059371504826159371604827159372604827159382604837159382604937159482604937150
           02457912468913578024579134680135780246791346802357902467913568023579124689135680

(Yet Another) Lisp In Go

Sun, 27 Mar 2022 00:00:00 +0000

For the past month I've been spending some free time writing another Lisp implementation, this time in Go. My previous attempt was a subset of Scheme made in preparation for a class on the classic Structure and Interpretation of Computer Programs. That project was great fun and I learned a lot.

I started learning Go late last year and began thinking about writing another Lisp. Go has garbage collection, a feature that has long been an essential part of most Lisps¹. Getting GC essentially for free would save a lot of time and effort.

Go is also extremely fast. For applications where startup performance is not an issue, I'm pretty happy with Clojure most of the time. But working with Common Lisp in recent years, and Go more recently, has spoiled me and made me hungry for more performance, especially for short-running command-line utilities.

There are obviously other Lisps written in Go already, including Joker, a Clojure interpreter and linter. But I wasn't particularly interested with implementing a particular Lisp dialect; rather, I wanted to try to implement a language core that one could extend in a variety of different directions, including implementing the language in itself. Other possible directions include graphics programming, text-based games, and scripting. The working name of this Lisp is l1 ("el-one"), hinting at a possible series of small, experimental Lisps.

Features

Here is a summary of what's implemented & planned:

l1 has	doesn't have	will have	might get
integers (essentially unlimited size)	keywords	macros	curses
comments (`;; ....`)	maps	syntax quote	graphics
atoms	strings	reader macros (`, ', …)	subprocess / shells
lists	namespaces	REPL / editor integration	big floats
4 special forms: `cond`, `def`, `lambda`, `quote`	exceptions	`let` (as a macro)	`error` equiv.
16 built-in functions	loops	`defun` / `defn` (as a macro)	tail call optimization
recursion
closures			byte code compilation/interpretation

Performance

The current implementation is on GitHub. Its speed surprises me a bit.

Consider the following program in l1, which computes a factorial:

;; fact.l1: Return the factorial of `n`:
(def fact
     (lambda (n)
       (cond ((eq 0 n) 1)
             (t (* n (fact (- n 1)))))))

(print (fact 100))

outputting

933262154439441526816992388562667004907159682643816214685929638
952175999932299156089414639761565182862536979208272237582511852
10916864000000000000000000000000

Its equivalent in Clojure (or its nimble alternative implementations, Babashka or Joker) can be written as follows:

;; fact.clj: Return the factorial of `n`:
(def fact
  (fn [n]
    (cond (= 0 n) 1
          :else (*' n (fact (- n 1))))))

(println (fact 100))

… and in my Python Scheme implementation as

;; fact.scm
(define (fact n)
  (if (= n 1)
      1
      (* n (fact (- n 1)))))

(display (fact 100))

I also did the equivalent for Common Lisp. I used Oatmeal to make a skeleton Lisp project and used that to create an executable program called fact.

The execution times break down thusly:

Language	Running It	Execution Time (ms)
Clojure	`clojure -M fact.clj`	1729
Babashka	`bb fact.clj`	200
`smallscheme`	`scheme.py fact.scm`	120
Common Lisp	`./fact`	68
Joker	`joker fact.clj`	59
`l1`	`l1 fact.l1`	8

This is a very limited benchmark! However, it does give a flavor for start-up times. Note that l1 is a tree-walking interpreter; I wonder what speed might be possible if implemented with a byte code interpreter.

Lexing

I started the journey by writing the lexer. A post on Hacker News led me to this video where Rob Pike, of the core Go language team, describes an elegant design for a lexer used in the Go templating library. I was able to extract the relevant bits into a fairly general-purpose library that I then used for this Lisp with relatively little code (of course, one shouldn't expect too much code when lexing a Lisp, since there is not much syntax).

Fun With Atoms and Lists

You may have noticed the lack of character strings in the feature table, above. Many interesting Lisp programs don't use traditional strings (arrays of characters encoded as bytes), and I am curious to see what can be done strictly without them, though I could see adding them at some point.

Without strings, one will probably want to manipulate atoms in various ways. As a start, l1 introduces the notion of "splitting" (creating a list from an atom or number) and "fusing" (joining such a list back together again):

> (split (quote greenspun))
(g r e e n s p u n)
> (split 1395871)
(1 3 9 5 8 7 1)
> (fuse (quote (1 2 3 4 5)))
12345
> (randigits 10)
(6 1 5 4 4 8 8 3 0 1)
> (randigits 10)
(2 7 5 4 7 9 1 6 6 1)
> (fuse (randigits 1000))
797522288353215977146173184650097900747324790200919947108552266
896559408664559755127840959435738695979408193086120089317097735
733247509700258597497415541421859295990446300938230591278544826
160148998280910399112793807480556810085222871423786728939605143
312291343715625109027024093962060686621901049553518883581864852
832439160531395519838325036388642484613231265974048363263732041
737675913066537856875008449087672272329301144164887429770199070
521721230755123684155760268379043481645391533460833091119801604
531684362848585264816725347753593965869286499060052823295397069
598147167103689429912992818647290966641807288375144076084638850
885562168375457674623070776900707693203757775691854059277861315
130019383408298242102129643369889134587749005021251080452606062
217504938911721968545635428643266561957454859338694605115003758
001930119736513921576952435852918640253473323143920762789645830
700672642264667728965761815048634636071828415705273836146286590
4892309215088593977646507232497245814663081971549675531
>

Testing

As with previous Lisps I've worked on, most of the tests were represented in Go code as strings containing Lisp code and the expressions they evaluate to. Here are some examples, taken more or less at random.

        ...
        // `split` function
        {Cases(S("(split)", "", "expects a single argument"))},
        {Cases(S("(split 1)", "(1)", OK))},
        {Cases(S("(split -1)", "(-1)", OK))},
        {Cases(S("(split -321)", "(-3 2 1)", OK))},
        {Cases(S("(split (quote a))", "(a)", OK))},
        {Cases(S("(split (quote (a b c)))", "", "expects an atom or a number"))},
        {ECases(S("(split (quote greenspun))", "(g r e e n s p u n)", OK))},
        {ECases(S("(split (* 12345 67890))", "(8 3 8 1 0 2 0 5 0)", OK))},
        {ECases(S("(len (split (* 99999 99999 99999)))", "15", OK))},
        {ECases(S("(split (quote greenspun))", "(g r e e n s p u n)", OK))},
        {ECases(S("(split (* 12345 67890))", "(8 3 8 1 0 2 0 5 0)", OK))},
        {Cases(S("(fuse (quote (1 2)))", "12", OK))},
        {ECases(S("(+ 2 (fuse (quote (1 2 3))))", "125", OK))},
        {ECases(S("(fuse (split 1295807125987))", "1295807125987", OK))},
        {ECases(S("(len (randigits 10))", "10", OK))},
        {ECases(S("((lambda (x) (+ 1 x)) 1)", "2", OK))},
        {ECases(
                S("(def incrementer (lambda (n) (lambda (x) (+ x n))))", "<lambda(n)>", OK),
                S("(def inc (incrementer 1))", "<lambda(x)>", OK),
                S("(inc 5)", "6", OK),
                S("(def add2 (incrementer 2))", "<lambda(x)>", OK),
                S("(add2 5)", "7", OK),
        )},
        ...

Here Cases is a utility function that create a local environment (a possibly nested set of variable bindings) and evaluates one or more expressions in that environment; ECases (E for Exemplary) is the same, but saves its test cases in an examples.txt file which stores a reduced set of illustrative cases, e.g. for adding to the README. S is a function which takes an expression to evaluate, the result that should obtain, or an error message fragment which is expected if the expression should fail.

This seems to be a good way to test a new language implementation, though I plan to implement some generative / "fuzzing" tests as well, since I doubt all the bugs have been found yet.

Further Work

Macros are next. I'm pretty sure I know how I want to do them, and for me the power of macros is the whole point of Lisp, or at least a big part of the point.

A Final Note on Performance, and Conclusion

To return to the performance discussion, above: a reasonable person might object that l1 is so lean on features, that it's not at all surprising that it starts fast. Clojure, on the other hand, leverages the JVM, which has been tuned for decades for speed in long-running processes, and also provides a thoughtful and comprehensive language design that does many things no hobby language could easily support.

That being said, one thing I am excited about with this project is this: pretty much anything I can do in Go (which is very many things) can be added to my Lisp without too much effort. Conversely, to add features to a Common Lisp or Clojure implementation would be very difficult indeed. Building a small language core that you can extend in a variety of directions is a fun, if not necessarily entirely practical, way to go.

I met a speaker at LambdaJam 2015 who used his own Lisp for music performance. It was not garbage-collected, for real-time performance.

From Elegance to Speed

Wed, 25 Sep 2019 00:00:00 +0000

Making Things Fast

A while back, we used a toy particle physics trigger to explore concepts in lazy evaluation and functional programming in Clojure. We came up with the following function which selected groups of eight events clustered close together in time:

(defn smt-8 [times]
  (->> times
       (partition 8 1)
       (map (juxt identity
                  (comp (partial apply -)
                        (juxt last first))))
       (filter (comp (partial > 1000) second))))

Though elegant, this function processed events at a rate of 250 kHz, which I found slightly disappointing, performance-wise.

I have been interested in writing faster programs lately, which is one of the reasons I've been learning Common Lisp. Common Lisp lets you create extremely high-level programming constructs, but also lets you get closer to the metal when you need to. In this post we'll look at some ways for improving performance and compare our results to the original 250 kHz event rate.

First Steps

The first step is to translate our code as is to Common Lisp. We're going to punt entirely on the question of laziness for this post, though we might take it up in the future.

Here's my first attempt. We have to start with our source of random times. In Clojure, we had:

(def times (iterate #(+ % (rand-int 1000)) 0))

Since I'm going to benchmark performance, we'll need to set a limit to the number of events (times) we'll process. So, in Common Lisp:

(defun time-sequence (n)
  (loop repeat n
     for y = 0 then (+ y (random 1000))
     collect y))

where n is the number of event times to process. Here and throughout what follows, I'm using the loop macro, which is a Swiss army knife for iteration with a somewhat strange, non-Lispy, but readable syntax.

We also want to set our random state so as to guarantee different results each time we execute our simulation:

(setf *random-state* (make-random-state t))

My first attempt in Common Lisp looked like this benchmarking snippet:

(timing
  (length
    (->> 1000000
         time-sequence
         (partition-n 8 1)
         (mapcar (juxt #'car (compose #'car #'last) #'identity))
         (mapcar #'(lambda (l) `(,(- (cadr l) (car l)) ,(caddr l))))
         (remove-if-not #'(lambda (l) (< (car l) 1000))))))

This is less lovely than the Clojure, partly because of all the hash-quoting of functions (Common Lisp is a Lisp-2, and Clojure is a Lisp 1). ->> is from the arrow-macros library, and compose is from the cl-utilities library. The other Clojure-ish functions, partition-n (a replacement for Clojure's partition, whose name collides with an entirely different function in Common Lisp) and juxt are from cl-oju, a small library I've been writing for those still-frequent times when I want a Clojure function or idiom in Common Lisp. (The library ignores laziness for now, since I haven't needed it yet.)

timing is a macro I adapted from the Common Lisp Cookbook, which captures both the results of a computation and the elapsed CPU time used (not wall clock time) in msec:

(defmacro timing (&body forms)
  (let ((run1 (gensym))
	(run2 (gensym))
	(result (gensym)))
    `(let ((,run1 (get-internal-run-time))
	   (,result (progn ,@forms))
	   (,run2 (get-internal-run-time)))
       `(duration ,(- ,run2 ,run1) msec... result ,,result))))

For now, result is simply the number of eight-fold time clusters occuring within 1000 units of time. The execution time was roughly a second for the first few times I ran this:

 '(DURATION 1045 MSEC... RESULT 235)
 '(DURATION 1554 MSEC... RESULT 201)
 '(DURATION 827 MSEC... RESULT 164)

This already processed events at 1 MHz, roughly 4x faster than the Clojure speed of 250kHz.

Later, however, when I revisited the code, I noticed it ran significantly faster:

 '(DURATION 435 MSEC... RESULT 193)
 '(DURATION 279 MSEC... RESULT 189)
 '(DURATION 205 MSEC... RESULT 177)
 '(DURATION 601 MSEC... RESULT 180)

This is 2.6 Mhz, 10x the Clojure code speed. It took me a while to figure out that I had changed the SBCL heap size in order to handle larger arrays for testing (this is one area where laziness would help!). The larger heap size was making the code run faster.

It should be said at this point that I made no attempt in my original blog post to optimize the Clojure code for performance. Nevertheless, I think it's interesting that my first attempt to write the same algorithm in Common Lisp, using roughly the same idioms, performed so much faster.

Interlude

Looking back over my career, it seems I have been moving gradually from fast, lower-level languages to slower, more expressive, high-level languages: FORTRAN and C, to Perl and Python, then to Clojure (which is faster than Python for most long-running programs, while being arguably more expressive¹). I've enjoyed this trajectory, because higher level languages let you implement so much, so quickly.

My feelings on this are shifting, however. I wrote some code a decade ago for a system that has been running constantly for a decade on thousands of devices. My code is not particularly CPU efficient and, therefore, not as energy-efficient as it could be. As the planet warms, I think it's important to try and find small efficiencies where we can… not necessarily under the assumption that it will make a massive difference, but because it's just the right thing to do as conscientious professionals. (The fact that the sins of my past are executing on processors running on top of and within Antarctic ice adds an extra bit of irony here).

Common Lisp is particularly interesting to me of late because, having been designed largely for Good Old-Fashioned AI, it allows you to work at a very high level (you can, for example, add Prolog-like functionality to a Lisp in a few hundred lines of code ² – but, you can generate very fast (i.e., more energy efficient) code as well, as we shall see shortly.

Improvements

The next thing I tried was to separate the event generation from the trigger calculation and to use a vector for the input times, instead of a list:

(defun time-sequence-vec (n)
  (make-array `(,n)
              :initial-contents
              (loop repeat n
                 for y = 0 then (+ y (random 1000))
                 collect y)))

(defparameter array-size (* 100 1000 1000))

(defparameter *s* (time-sequence-vec array-size))

I also make the test sample much larger to improve the run durations and statistics.

Both Graham³ and Norvig⁴ emphasize tuning your algorithm first before turning to other optimizations. In our case the problem is O(n), which is the best time complexity we can hope for, but our use of function composition, while elegant, could be turned into something more efficient. To improve the algorithm proper, I perform the entire computation in a single loop statement. I also change the output slightly to show the index for matching 8-fold events, start and end time, and event duration:

(defun second-try ()
  (loop for x below (- array-size 8)
     if (< (- (elt *s* (+ x 8))
              (elt *s* x))
           1000)
     collect `(index ,x
                     startval ,(elt *s* x)
                     endval   ,(elt *s* (+ x 8))
                     diff     ,(- (elt *s* (+ x 8))
                                  (elt *s* x)))))
(take 4 (second-try))
;;=>
'((INDEX 69087 STARTVAL 34504666 ENDVAL 34505615 DIFF 949)
  (INDEX 153990 STARTVAL 76991815 ENDVAL 76992630 DIFF 815)
  (INDEX 237070 STARTVAL 118617491 ENDVAL 118618358 DIFF 867)
  (INDEX 294951 STARTVAL 147513479 ENDVAL 147514346 DIFF 867))

;; Time it, a few times:
(timing (length (second-try)))
;;=>
'(DURATION 2158 MSEC... RESULT 2427)
;;=>
'(DURATION 2167 MSEC... RESULT 2427)
;;=>
'(DURATION 2162 MSEC... RESULT 2427)
;;=>

Note that the starting and ending time values increase monotonically, and the reported time differences (DIFF) are under 1000, as desired⁵.

The new trigger function improves the processing rate to 46.2 MHz. At this point I'm starting to wonder what the assembly language looks like:

(disassemble 'second-try)
;;=>
; disassembly for SECOND-TRY
; Size: 1061 bytes. Origin: #x228FE0B0                        ; SECOND-TRY
; 0B0:       498B7510         MOV RSI, [R13+16]               ; thread.binding-stack-pointer
; 0B4:       488975F8         MOV [RBP-8], RSI
; 0B8:       488B3591FFFFFF   MOV RSI, [RIP-111]              ; 'ARRAY-SIZE
; 0BF:       8B56F5           MOV EDX, [RSI-11]
; 0C2:       4A8B142A         MOV RDX, [RDX+R13]
; 0C6:       83FA61           CMP EDX, 97
; 0C9:       480F4456F9       CMOVEQ RDX, [RSI-7]
; 0CE:       83FA51           CMP EDX, 81
; 0D1:       0F84BF030000     JEQ L17
; 0D7:       BF10000000       MOV EDI, 16
; 0DC:       FF1425B000B021   CALL QWORD PTR [#x21B000B0]     ; GENERIC--

;; .................. many lines omitted ................

; 4B6: L24:  CC18             BREAK 24                        ; UNBOUND-SYMBOL-ERROR
; 4B8:       27               BYTE #X27                       ; '*S*
; 4B9: L25:  6880000000       PUSH 128
; 4BE:       FF14252800B021   CALL QWORD PTR [#x21B00028]     ; ALLOC-TRAMP-R11
; 4C5:       E9FFFEFFFF       JMP L12
; 4CA: L26:  6A10             PUSH 16
; 4CC:       FF14252800B021   CALL QWORD PTR [#x21B00028]     ; ALLOC-TRAMP-R11
; 4D3:       EB94             JMP L14

(258 lines in total.)

My assembler-fu is not strong, but I notice the GENERIC-- and we do not need generic functions: there are many opportunities here to introduce type declarations.

After declaring to the compiler we want more speed and less type-checking ((declaim (optimize (speed 3) (debug 0) (safety 0)))), let's declare our array to be type fixnum (which on SBCL is an integer from -4611686018427387904 to 4611686018427387903).

(defparameter *s* (make-array array-size
                              :element-type 'fixnum))
;; Initialize array:
(loop
   for y = 0 then (+ y (random 1000))
   for i below array-size
   do (setf (elt *s* i) y))

(timing
  (length
   (loop for x below (- array-size 8)
      if (< (- (elt *s* (+ x 8))
               (elt *s* x))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1964 MSEC... RESULT 2435)
;;=>
'(DURATION 1939 MSEC... RESULT 2435)
;;=>
'(DURATION 2010 MSEC... RESULT 2435)

… maybe a slight improvement. Let's now type-hint where we can throughout the tight loop:

(timing
  (length
   (loop for x below (- array-size 8)
      if (< (- (the fixnum (elt *s* (the fixnum (+ x 8))))
               (the fixnum (elt *s* (the fixnum x))))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1740 MSEC... RESULT 2435)
;;=>
'(DURATION 1738 MSEC... RESULT 2435)
;;=>
'(DURATION 1753 MSEC... RESULT 2435)

A 10% or so improvement… not too bad.

At this point I'm tempted to look at the macroexpansion of my loop expression and see what it's doing. For simplicity, I collect just the starting time index for times satisfying the trigger:

(macroexpand-1
 '(loop for x below (- array-size 8)
     if (< (- (the fixnum (elt *s* (the fixnum (+ x 8))))
            (the fixnum (elt *s* (the fixnum x))))
         1000)
     collect x))
;;=>
'(block nil
  (let ((#:loop-limit-711 (- array-size 8)) (x 0))
    (declare (type (and number real) x)
             (type (and number real) #:loop-limit-711))
    (sb-loop::with-loop-list-collection-head (#:loop-list-head-712
                                              #:loop-list-tail-713)
      (tagbody
       sb-loop::next-loop
         (when (>= x #:loop-limit-711) (go sb-loop::end-loop))
         (if (<
              (- (the fixnum (elt *s* (the fixnum (+ x 8))))
                 (the fixnum (elt *s* (the fixnum x))))
              1000)
             (sb-loop::loop-collect-rplacd
              (#:loop-list-head-712 #:loop-list-tail-713) (list x)))
         (sb-loop::loop-desetq x (1+ x))
         (go sb-loop::next-loop)
       sb-loop::end-loop
         (return-from nil
           (sb-loop::loop-collect-answer #:loop-list-head-712))))))

I notice that x and :loop-limit-711 have an overly-general type declaration. I'm not sure how to type-hint x, but I can type-hint the upper bound:

(timing
  (length
   (loop for x below (- (the fixnum array-size) 8)
      if (< (- (the fixnum (elt *s* (+ (the fixnum x) 8)))
               (the fixnum (elt *s* (the fixnum x))))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1460 MSEC... RESULT 2513)
;;=>
'(DURATION 1455 MSEC... RESULT 2513)
;;=>
'(DURATION 1486 MSEC... RESULT 2513)

Another modest but significant improvement.

At this point I reached for the SBCL profiler to see what I could learn from it. Most of the processing time is spent in array lookups, which I don't think we can avoid.

Let's look at the assembler again. Since our trigger matches are rare, most of our time is going to be spent in finding the events, rather than in collecting the detailed results for the matches. If we switch back to using collect x to avoid the extra assembly associated with building `(index ,x startval ...):

(defun trigger ()
(loop for x below (- (the fixnum array-size) 8)
     if (< (- (the fixnum (elt *s* (+ (the fixnum x) 8)))
              (the fixnum (elt *s* (the fixnum x))))
           1000)
     collect x))
(disassemble 'trigger)
;;=>
;; disassembly for TRIGGER
;; Size: 448 bytes. Origin: #x22610EA0                         ; TRIGGER
;; 0EA0:       498B4510         MOV RAX, [R13+16]              ; thread.binding-stack-pointer
;; 0EA4:       488945F8         MOV [RBP-8], RAX
;; 0EA8:       488B05B1FFFFFF   MOV RAX, [RIP-79]              ; 'ARRAY-SIZE
;; 0EAF:       8B70F5           MOV ESI, [RAX-11]
;; 0EB2:       4A8B342E         MOV RSI, [RSI+R13]
;; 0EB6:       83FE61           CMP ESI, 97
;; 0EB9:       480F4470F9       CMOVEQ RSI, [RAX-7]
;; 0EBE:       83FE51           CMP ESI, 81
;; 0EC1:       0F8477010000     JEQ L11
;; 0EC7:       40F6C601         TEST SIL, 1
;; 0ECB:       0F8568010000     JNE L10
;; 0ED1:       48D1FE           SAR RSI, 1
;; 0ED4:       4883EE08         SUB RSI, 8
;; 0ED8:       488975F0         MOV [RBP-16], RSI
;; 0EDC:       31F6             XOR ESI, ESI
;; 0EDE:       49896D28         MOV [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 0EE2:       4D8B5D68         MOV R11, [R13+104]             ; thread.alloc-region
;; 0EE6:       498D4310         LEA RAX, [R11+16]
;; 0EEA:       493B4570         CMP RAX, [R13+112]
;; 0EEE:       0F874D010000     JNBE L12
;; 0EF4:       49894568         MOV [R13+104], RAX             ; thread.alloc-region
;; 0EF8: L0:   498D4307         LEA RAX, [R11+7]
;; 0EFC:       49316D28         XOR [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 0F00:       7402             JEQ L1
;; 0F02:       CC09             BREAK 9                        ; pending interrupt trap
;; 0F04: L1:   C740F917001020   MOV DWORD PTR [RAX-7], #x20100017  ; NIL
;; 0F0B:       C7400117001020   MOV DWORD PTR [RAX+1], #x20100017  ; NIL
;; 0F12:       488945E8         MOV [RBP-24], RAX
;; 0F16:       488945E0         MOV [RBP-32], RAX
;; 0F1A:       E9BB000000       JMP L4
;; 0F1F:       90               NOP
;; 0F20: L2:   8B142524144A20   MOV EDX, [#x204A1424]          ; tls_index: *S*
;; 0F27:       4A8B142A         MOV RDX, [RDX+R13]
;; 0F2B:       83FA61           CMP EDX, 97
;; 0F2E:       480F44142528144A20 CMOVEQ RDX, [#x204A1428]     ; *S*
;; 0F37:       83FA51           CMP EDX, 81
;; 0F3A:       0F840F010000     JEQ L13
;; 0F40:       488D7E10         LEA RDI, [RSI+16]
;; 0F44:       4883EC10         SUB RSP, 16
;; 0F48:       488975D8         MOV [RBP-40], RSI
;; 0F4C:       B904000000       MOV ECX, 4
;; 0F51:       48892C24         MOV [RSP], RBP
;; 0F55:       488BEC           MOV RBP, RSP
;; 0F58:       B862934F22       MOV EAX, #x224F9362            ; #<FDEFN ELT>
;; 0F5D:       FFD0             CALL RAX
;; 0F5F:       488B75D8         MOV RSI, [RBP-40]
;; 0F63:       F6C201           TEST DL, 1
;; 0F66:       0F85CA000000     JNE L9
;; 0F6C:       4C8BC2           MOV R8, RDX
;; 0F6F:       49D1F8           SAR R8, 1
;; 0F72:       4C8945D0         MOV [RBP-48], R8
;; 0F76:       8B142524144A20   MOV EDX, [#x204A1424]          ; tls_index: *S*
;; 0F7D:       4A8B142A         MOV RDX, [RDX+R13]
;; 0F81:       83FA61           CMP EDX, 97
;; 0F84:       480F44142528144A20 CMOVEQ RDX, [#x204A1428]     ; *S*
;; 0F8D:       83FA51           CMP EDX, 81
;; 0F90:       0F84BC000000     JEQ L14
;; 0F96:       488BFE           MOV RDI, RSI
;; 0F99:       4883EC10         SUB RSP, 16
;; 0F9D:       488975D8         MOV [RBP-40], RSI
;; 0FA1:       B904000000       MOV ECX, 4
;; 0FA6:       48892C24         MOV [RSP], RBP
;; 0FAA:       488BEC           MOV RBP, RSP
;; 0FAD:       B862934F22       MOV EAX, #x224F9362            ; #<FDEFN ELT>
;; 0FB2:       FFD0             CALL RAX
;; 0FB4:       4C8B45D0         MOV R8, [RBP-48]
;; 0FB8:       488B75D8         MOV RSI, [RBP-40]
;; 0FBC:       F6C201           TEST DL, 1
;; 0FBF:       7572             JNE L8
;; 0FC1:       488BC2           MOV RAX, RDX
;; 0FC4:       48D1F8           SAR RAX, 1
;; 0FC7:       498BD0           MOV RDX, R8
;; 0FCA:       4829C2           SUB RDX, RAX
;; 0FCD:       4881FAE8030000   CMP RDX, 1000
;; 0FD4:       7C22             JL L5
;; 0FD6: L3:   4883C602         ADD RSI, 2
;; 0FDA: L4:   488BC6           MOV RAX, RSI
;; 0FDD:       48D1F8           SAR RAX, 1
;; 0FE0:       483B45F0         CMP RAX, [RBP-16]
;; 0FE4:       0F8C36FFFFFF     JL L2
;; 0FEA:       488B45E8         MOV RAX, [RBP-24]
;; 0FEE:       488B5001         MOV RDX, [RAX+1]
;; 0FF2:       488BE5           MOV RSP, RBP
;; 0FF5:       F8               CLC
;; 0FF6:       5D               POP RBP
;; 0FF7:       C3               RET
;; 0FF8: L5:   4C8B45E0         MOV R8, [RBP-32]
;; 0FFC:       49896D28         MOV [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 1000:       4D8B5D68         MOV R11, [R13+104]             ; thread.alloc-region
;; 1004:       498D4310         LEA RAX, [R11+16]
;; 1008:       493B4570         CMP RAX, [R13+112]
;; 100C:       7747             JNBE L15
;; 100E:       49894568         MOV [R13+104], RAX             ; thread.alloc-region
;; 1012: L6:   498D4307         LEA RAX, [R11+7]
;; 1016:       49316D28         XOR [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 101A:       7402             JEQ L7
;; 101C:       CC09             BREAK 9                        ; pending interrupt trap
;; 101E: L7:   488970F9         MOV [RAX-7], RSI
;; 1022:       C7400117001020   MOV DWORD PTR [RAX+1], #x20100017  ; NIL
;; 1029:       488945E0         MOV [RBP-32], RAX
;; 102D:       49894001         MOV [R8+1], RAX
;; 1031:       EBA3             JMP L3
;; 1033: L8:   CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 1035:       08               BYTE #X08                      ; RDX
;; 1036: L9:   CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 1038:       08               BYTE #X08                      ; RDX
;; 1039: L10:  CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 103B:       18               BYTE #X18                      ; RSI
;; 103C:       CC10             BREAK 16                       ; Invalid argument count trap
;; 103E: L11:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1040:       00               BYTE #X00                      ; RAX
;; 1041: L12:  6A10             PUSH 16
;; 1043:       FF14252800B021   CALL QWORD PTR [#x21B00028]    ; ALLOC-TRAMP-R11
;; 104A:       E9A9FEFFFF       JMP L0
;; 104F: L13:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1051:       27               BYTE #X27                      ; '*S*
;; 1052: L14:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1054:       27               BYTE #X27                      ; '*S*
;; 1055: L15:  6A10             PUSH 16
;; 1057:       FF14252800B021   CALL QWORD PTR [#x21B00028]    ; ALLOC-TRAMP-R11
;; 105E:       EBB2             JMP L6

… this is almost a manageable amount of assembler, though I don't claim to understand all of it. If I needed more speed, though, I could dig into the assembler as a way to try and ferret out even more efficiency.

Summary

We went from our initial POC in very functional-looking Clojure, running at 250 kHz, to optimized Common Lisp running at 100,000,000 / 1467 msec = 68.2 MHz, which is nearly 300x faster.

Checking our intuition about how fast this really is: my processor is 2.7 GHz, and our processing rate is 68.2 MHz, meaning we're spending about 40 clock ticks per event, which doesn't strike me as too bad.

In the end, we made the following changes to our initial attempt:

Separated out the initial event time-sequence generation as a separate step;
Rewrote the algorithm as a single loop;
declaim'ed optimizations to the compiler;
Type-hinted the input array, the variables used in the time difference calculation, and the loop upper bound.

We also touched on profiling (which didn't provide much benefit this time, but can be very useful generally), and showed how to inspect the generated assembler code.

Conclusion

Common Lisp can be made pretty fast! I should write a C-language version of this toy problem to compare. The final code is not nearly as beautiful as the original, but without too much effort we got much faster code. I'll conclude with this line from ANSI Common Lisp: "Lisp is really two languages: a language for writing fast programs and a language for writing programs fast." It seems quite powerful to me to have both of these on hand.

Reasonable people can disagree on the relative expressiveness of languages; for me, Lisps win on expressiveness because I can bend the language to my will using macros in a way that I cannot with non-Lisps.

See Paul Graham, On Lisp p. vii

Paul Graham, ANSI Common Lisp

⁴

Peter Norvig, Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp

⁵

The original Clojure code reported the entire sequence of eight times for each triggered event, but we omit that level of detail for clarity. Since the triggered clusters are rare, this doesn't significantly impact the timing measurements.

Common Lisp How-Tos

Sun, 15 Sep 2019 00:00:00 +0000

Intro

I've been writing this post as I go, and you may still see changes if you check back from time to time.

This post is for writing about things I'm figuring out, and for keeping notes on what I want to figure out next, as I get ramped up on Common Lisp. When I'm done there will hopefully be stuff here other people can use.

There's a lot to love and admire about Common Lisp, but the language has some pointy bits that can at times make Clojure look friendly. With so many things to sort out I'm going to try to keep a few notes going forward just to keep it all organized. Caveat emptor, YMMV, USE AT YOUR OWN RISK, etc., etc.

Lessons Learned So Far

How Do I Set Up Common Lisp on a Mac?

I made a repo to automate how I set up a fresh SBCL install on a Mac, including Quicklisp. This gets me from zero to a working Emacs REPL system fairly quickly.

Note that I also set a LISP_HOME environment variable in my .bash_profile which I use in the examples which follow.

How Do I Get Environment Variables from Lisp?

From the Common Lisp Cookbook:

(uiop:getenv "HOME")

How Do I Quickly Create a New Project?

(There is a nice writeup on how Lisp projects are organized.)

First, it seems it's best to pick a standard place to put your Lisp projects. I have a deep directory tree on my filesystem so would prefer to set that path as an environment variable; this makes the setup easy to port to another computer.

I made this bash script, lisplib:

#!/bin/bash

projname=${1?}
sbcl --non-interactive  \
     --disable-debugger \
     --eval '(ql:quickload "cl-project")' \
     --eval '(ql:quickload "cl-utilities")' \
     --eval '(let* ((projname "'$projname'")
                    (home (sb-unix::posix-getenv "LISP_HOME"))
                    (projpath (concatenate '"'"'string home "/" projname)))
               (cl-project:make-project (pathname projpath)))'

open -a /Applications/Emacs.app $LISP_HOME/$projname/src/main.lisp

(Omit the newlines inside the last single-quoted expression.)

When I, say, lisplib foo, it creates the project using make-project and opens the editor on the main source file.

How Do I Add Dependencies to One Of My Projects?

Add it to the .asd file in the :depends-on list.

How Do I Get My Local Projects Loadable By Other Projects?

I did this once:

$ cd ~/quicklisp
$ mv local-projects /tmp  # Mine was basically empty...
$ ln -s $LISP_HOME local-projects

How Do I Get Dependencies Loaded

put your project in the directory symlinked above;
jack into your REPL;
update depends-on in the project's .asd file as you add dependencies;
(ql:quickload <current-package>) to pick up its dependencies.
Optionally, to get the correct namespacing for the package you're working on, use slime-repl-set-package (C-c M-p) to set the default package to the one you're working on (not 100% sure what effect this has yet)

How Do I Build Binaries?

I use to Xach Beane's buildapp, built like so:

git clone git@github.com:xach/buildapp.git
cd buildapp
make
make install

A small working build script using buildapp can be found here.

How Do I Get My Project Into Quicklisp?

Make a pull request with Xach Beane here.

How Do I Use `Curses` (or equivalent)?

After some research I decided to follow in Steve Losh's footsteps and use cl-charms (tutorial here). This cost me multiple days due to some outdated Homebrew packages, but worked out in the end after updating those and reinstalling my Lisp setup.

How Do I Get Help?

Aside from Stack Overflow, the #lisp channel on IRC was helpful to me for the above issue.

Future Projects

Finish Weeds (redo of my blog software in CL)
Port namejen to CL
Write a Roguelike

Implementing Scheme in Python

Wed, 28 Aug 2019 00:00:00 +0000

Background

Lately I've been a bit of a Lisp kick again, reading On Lisp and Practical Common Lisp, working problems and re-casting my Clojure-based blog software into Common Lisp to see how that goes. The urge to do these things caught me somewhat by surprise: in the past few years I haven't done much programming outside of my day job because I've been focused more on painting and drawing. It seems, though, that these things are cyclical and right now I'm in the grips of Lisp mania again.

This cycle goes back to the mid-1980s when I was a college student studying physics and learning how to program. It's hard to say how I got the Lisp bug initially. I remember reading Douglas Hofstadter's explanation of recursive Lisp functions; hanging out near the AI workstations in the basement of the Madison Academic Computing Center where I worked (the sans-serif Lisp program printouts looked way better than anybody else's output); and reading about AI and Lisp on Usenet. At any rate, I managed to pick up a copy of PC-Lisp, a port of Franz Lisp to IBM PCs which apparently is still a thing after all these years. Somewhat astonishingly, the interpreter ran on my underpowered IBM PCjr, loading from floppy disk in half a minute or so. I remember how even fairly small programs would pause for several seconds while the garbage collector did its thing. This interpreter, along with a Lisp textbook I picked up somewhere, was enough to get me started learning about atoms, lists, quote, car and cdr, symbolic computing, and simple recursive algorithms.

I also wrote a small s-expression parser in Turbo Pascal on the same PCjr as a way of graphing arbitrary math functions of two variables. I had no compiler class but my roommate taught me enough about finite state machines, lexers, parsers, etc. to be dangerous… enough knowledge to get my grapher up and running. (I wish I still had some of these old programs, but they vanished when I "lent" the computer and all my diskettes to a friend; they are probably landfill somewhere.)

This all subsided for several years while I got sucked into physics, then painting, then capoeira, then physics, then painting, then software engineering in C, Perl, and Python. In the early aughts I got interested in Paul Graham's writing, which led to a few skirmishes with Common Lisp every couple of years, and ultimately to me getting totally hooked on Clojure starting in 2011.

I intend to write more about what makes Lisps in general compelling for me¹ and why I've returned to it over and over throughout my career. In the mean time….

Scheming in Python

Happily, my Lisp mania has crested coincidentally with Chicago Python teacher David Beazley announcing another class on The Structure and Interpretation of Computer Programs, a classic MIT computing text based on a dialect of Lisp called Scheme, and I managed to get on the list before it filled up. Since it's been awhile since I've written much Python, I thought I'd warm up for the class by trying to write a small Scheme implementation in Python. My hope was to refresh my memory of the basics of Scheme, improve my slightly-dusty Python, and have fun.

Prior Art

Researching this post (after most of the code was done), I noticed that there are a ton of small Lisps written in Python (including two I worked on the better part of a decade ago). One that stuck out in my memory was Peter Norvig's hundred-odd-line Lispy, which is particularly striking in its elegance (in style, it reminds me of the Lisp programs in PAIP).

The Stages

My plan for writing the interpreter was to follow the actual REPL stages: Read, Eval, and Print. Read would involve parsing input expressions and turning them into data structures to be evaluated; Eval would be a (presumably recursive) step whereby, for function calls, arguments would be Eval'ed first, with the results handed to the appropriate code implementing the actual function, implemented either in Python (for a few primitive functions) or, eventually, through user-defined functions. An exception to the argument evaluation rule would have to be made for a few special forms, which would have to be handled on a case-by-case basis.

Parsing

My first approach for the parser was to use PLY, which I've used before. Having been spoiled by the excellent Instaparse library for Clojure, upon returning to PLY I found the library's interface relatively clumsy and abstruse.

Then I thought I'd try a regex to parse everything, since Scheme's base syntax and grammar are so simple. The following regex actually worked well for awhile:

pat = """^\s*\((?P<list>.*)\)$   # Something beginning/ending with parens, or
         |
         (?P<bool>(\#t|\#f))\s*  # A boolean true/false value, or
         |                       # An atom starting w/ a letter or arithmetic op:
         (?P<atom>[a-zA-Z\+\-\*\/]+[a-zA-Z0-9\+\-\*\/]*)\s*
         |
         (?P<num>[0-9]+)\s*"""   # Or, a positive natural number.

(I had not gotten to negative or floating point numbers at this stage of the project.)

Expressions of the form (+ 1 2 (+ 1 1 1)), for example, parsed correctly. However, the regex didn't handle lists that nested in other than the last position, for example, (+ (* 2 3) 4). It might be possible to do the whole problem with regular expressions, but at this point I decided to look for a nice modern parser library that let me write down the grammer in EBNF notation and simply returned a parse tree when given a valid expression. I found Lark, which is roughly as featureful as Instaparse and worked well. Here's the Lark grammar as it currently stands:

start  : _exprs
_exprs : _e* _e
_e     : ATOM
       | _num
       | BOOL
       | list
TRUE   : "#t"
FALSE  : "#f"
BOOL   : TRUE | FALSE
list   : "(" _exprs? ")"
INT    : /[-+]?[0-9]+/
ATOM   : /[a-zA-Z]+[a-zA-Z0-9\-\?]*/
       | /[\*\/\=\>\<]/
       | /[\-\+](?![0-9])/
FLOAT  : /[-+]?[0-9]+\.[0-9]*/
_num   : INT | FLOAT
%import common.WS
%ignore WS

Note that while libraries like Lark and Instaparse are a nice way to turn out a powerful parser quickly, Norvig's lis.py just splits the inputs on open/close parens and uses them to increase/decrease nesting level as they appear. The approach to parsing numbers is also elegant: try to parse it as an integer; if that fails, as a float; otherwise, it's a symbol (what my grammar calls ATOM). I suspect that using a "real" parser makes it easier to add more syntax to your language if needed, though.

Data Model

Once Lark parses an expression, I convert to my own internal representation before evaluation (function convert_ast). I did not use objects to represent lists, atoms, etc., but just simple tuples and lists:

Data Type	Example
Atom	`('atom', 'foo')`
Boolean	`('bool', True)`
Int	`('int', 123)`
Float	`('float', 3.14)`
List	`('list', [('atom', 'define'), ('atom', 'x'), ('int', 3)])`

It might be more idiomatic Python to use objects, but I'm more used to thinking in terms of nested structures of base data types (lists, tuples, dictionaries), which have the advantage of being easily printable while debugging.

Here's an example expression, parsed with Lark and then converted into my internal representation:

Scheme raw source

(define (abs x)
  (if (< x 0)
      (- x)
      x))

Lark output

Tree(start,
     [Tree(list,
           [Token(ATOM, 'define'),
            Tree(list,
                 [Token(ATOM, 'abs'),
                  Token(ATOM, 'x')]),
            Tree(list, [Token(ATOM, 'if'),
                        Tree(list,
                             [Token(ATOM, '<'),
                              Token(ATOM, 'x'),
                              Token(INT, '0')]),
                        Tree(list,
                             [Token(ATOM, '-'),
                              Token(ATOM, 'x')]),
                        Token(ATOM, 'x')])])])

Internal representation used in the interpreter

[('list', [('atom', 'define'),
           ('list', [('atom', 'abs'),
                     ('atom', 'x')]),
           ('list', [('atom', 'if'),
                     ('list', [('atom', '<'),
                               ('atom', 'x'),
                               ('int', 0)]),
                     ('list', [('atom', '-'),
                               ('atom', 'x')]),
                     ('atom', 'x')])])]

Evaluation

Evaluation turned out, not surprisingly, to be where most of the work is; specifically, the individual cases for special forms. These were:

quote
cond / if (I implemented these separately but, had I implemented macros, I might implement one in terms of the other)
define
lambda
or
and

Aside from these, there was code for normal function application rules, with a slight variance between internally-defined functions and functioned defined by the user with define.

I did not, as Paul Graham did, attempt to find the minimal subset of language elements needed to bootstrap a complete Lisp, though that is certainly an interesting exercise. My goal was to get my tests green in a straightforward way, as described below.

Printing

The Print stage needed to convert the internal representation of the evaluated result to a string, both for a user running a REPL, and for my unit tests. This turned out to be simple enough that I can put the entire function here:

def printable_value(ast):
    k, v = ast
    if k == 'int' or k == 'float':
        return str(v)
    if k == 'bool':
        return {True: "#t",
                False: "#f"}.get(v)
    if k == 'intproc':
        return "Internal procedure '%s'" % v
    if k == 'atom':
        return v
    if k == 'list':
        return '(' + ' '.join([printable_value(x)
                               for x in v]) + ')'
    if k == 'nop':
        return ''
    if k == 'fn':
        (fn_name, _, _) = v
        if fn_name == 'lambda':
            return "Anonymous-function"
        return "Function-'%s'" % str(fn_name)
    raise Exception('Unprintable ast "%s"' % str(ast))

The primitive data types (int, float, bool and atom) are obvious enough; intproc and fn are for internally-defined and user-defined functions, respectively; finally, nop is for when you don't want any value shown to the user (i.e., after one enters a define expression, there is no output printed before the next prompt):

scheme> (define a 3)
scheme>

Testing Approach

I used a fairly strict test-driven development workflow for this project: write just enough new test code to fail; write just enough production code to get the test to pass; then refactor. This workflow generally serves me well for harder problems where I'm unlikely to see the best way to do things from the outset and I expect to make many improvements: I know the tests will have my back when I make changes. This turned out to be the right approach, since I had to pivot several times.

I didn't just use tests drive the code forward, but let SICP drive the tests forward as well. When I needed the next thing to work on, all I had to do was turn forward a few pages in the book, and the page count gave me a measure of my progress.

A few of the tests, with the corresponding page numbers:

    # Adapted from SICP p. 8:
    t("(define pi 3.14159)")
    t("(define radius 10)")
    t("(* pi (* radius radius))", "314.159")
    ...
    # p. 19:
    t("(define x 7)")
    t("(and (> x 5) (< x 10))", "#t")
    ...
    # p. 37
    t("""(define (fib n)
           (cond ((= n 0) 0)
                 ((= n 1) 1)
                 (else (+ (fib (- n 1))
                          (fib (- n 2))))))""")
    t("(fib 5)", "5")

Here t (a very short function name since it's used so many times) is a testing function which evaluates the code supplied in the first argument, using the current "environment" (scope). If a second argument (a string) is supplied, it also asserts that the code in the first argument evaluates to the second.

The Code Comes To Life

One of my favorite things about this project was when I started writing nontrivial tests which already passed, because enough of the language had been implemented already. This happened more often as the project progressed, and it made me feel like I'd implemented a "real" language, an oddly compelling feeling.

The code, in its entirety, can be found here.

Next Steps

Though I have already gone far past my goals for the project, there are a few directions which I could take this.

Implement more of Scheme

For example, I don't have strings yet in my implementation!

Extend it Further

Implement Python interop and shell features. This could be genuinely fun and useful, but there's already a fairly full-featured Lisp skin for Python called Hy.

Go Low Level

Do it again, but in a faster, lower-level language. Python is relatively slow compared to Clojure for long-running programs, and slower than Common Lisp or C for program startup. I've been wanting to dust off my C chops, and this would be an excuse to do so, though C is a bit primitive compared to, say, Rust. (Go is not to my taste so I wouldn't pick it for a fun project like this one.) So, either an excuse to learn Rust or to improve my C chops. Which leads me to the final question: would I rather implement Lisps, or program in them? Up until now it has been the latter, but after the past few weeks I can definitely taste the appeal of the former.

Thoughts Upon Revisiting Python

This was the biggest Python project I've done in five years, so it was interesting to check my experience of the language after some time away in Clojure-land. I was a big Python fan for many years but my enthusiasm for the language has definitely cooled. This is partly due to the performance comparison I just mentioned, but the two other things I really missed were REPL integration and structural editing.

In Lisps such as Clojure and Common Lisp I can send expressions to the REPL directly from Emacs and see the results instantly. It's difficult to describe how helpful this short feedback cycle is until you've experienced it. Writing the interpreter, I got sort of close by using conttest plus nosetests: my unit tests ran every time I saved a source file, and if I wanted to inspect some object I could add a print or two temporarily to the code I was working on. But this is definitely more clumsy than a Lisp REPL directly integrated with my editor.

Structural editing (via Paredit) is another thing that is hard to explain the utility of but which is hard to live without once you're used to it. The basic idea is that you are no longer editing strings of text, but the actual tree of nested lists comprising your code - killing, splitting, joining, absorbing ("slurping") into and expelling ("barfing") from those lists. Working with the code at the tree level is a powerful way to edit your code, and watching this video might help may give a hint of it. Once you're used to the affordances of Paredit or its cousins, it feels clunky and primitive to go back to editing plain old text.

Other value propositions of Clojure, such as macros, immutable data structures and concurrency, weren't really an issue for this project, though I would certainly miss them for bigger projects.

Conclusion

Implementing your own Lisp is surprisingly do-able (and fun!). Python is certainly a fine tool for the job (though if you want to skip straight to programming in a modern Lisp for your day job, please contact me).

Postscript

While digging through my digital and mental archives for Lisp things I remembered something I hadn't seen in a long time: the use of closing square brackets to terminate a deeply-nested collection of s-expressions (i.e. ] instead of )))))))). Did I imagine it? While not necessarily a good idea, it's an interesting one. Then, while reading through Gabriel and Steele's Evolution of Lisp this morning, I came across the following snippet:

"One of the minor, but interesting, syntactic extensions that Interlisp made was the introduction of the superparenthesis, or superbracket. If a right square bracket ] is encountered during a read operation, it balances all outstanding open left parentheses, or back to the last outstanding left square bracket [."

DEFINEQ((FACTORIAL
  (LAMBDA (N)
   (COND [(ZEROP N) 1]
         (T (TIMES N (FACTORIAL (SUB1 N]

Bingo! I wasn't imagining it. This suggests those workstations in the basement of MACC at UW-Madison were probably running Interlisp.

So compelling that I left an exciting science project partly in order to do Lisp (Clojure) programming full time.

Code and Data

Mon, 04 Jun 2018 00:00:00 +0000

Introduction

This article was adapted from discussions we had at the weekly Clojure study group at OpinionLab, in June, 2018.

Clojure is a Lisp. Code is data in Lisp, and data can also be code. Lisp code consists of lists of symbols (or atoms, as they are called in other Lisps), other data primitives such as numbers and keywords, or (nested) lists of the same.¹

The first item in an expression (list) is the name of an operation, usually a function; the rest of the items are the arguments. When lists are nested, the inner expressions are evaluated first: in (+ 1 (* 2 3)), the multiplication happens before the addition.

Somewhat less commonly, instead of function names, the first item in the list may be the name of a macro or so-called "special form." In these cases, the exact method of evaluation depends on the implementation of the macro or form. For example, in (dotimes [_ 10] (println 'hi)), the println is evaluated 10 times; if dotimes were a function, println would be evaluated exactly once.

To prevent evaluation of an expression, one can quote it:

(quote (+ 1 (* 2 3)))
;;=>
(+ 1 (* 2 3))

The single quote (') character is an abbreviation for quote:

'(+ 1 (* 2 3))
;;=>
(+ 1 (* 2 3))

The REPL will evaluate expressions unless you quote them; if you decide later you want to evaluate an expression, you can use eval:

(def expr '(+ 1 (* 2 3)))
(eval expr)
;;=>
7

Quoting and evaluation are inverse operations:

(eval (eval (eval (eval ''''(+ 1 (* 2 3))))))
;;=>
7

Here are some examples you should try at the REPL, to help build an understanding of these two operations:

(+ 1 1)
'+
+
'(+ 1 1)
(cons '+ '(1 1))
(eval (cons '+ '(1 1)))
(first '(+ - * /))
(cons (first '(+ - * /)) '(1 1))
(eval (cons (first '(+ - * /)) (rest '(not-a-number 1 1))))
(->> '(not-a-number 1 1)
     rest
     (cons (first '(+ - * /)))
     eval)

Try to understand each of these as best you can, and try some variations. The last expression is the same as the one prior to it, just rewritten using the thread-last (->>) macro.

Arithmetic of the Fittest

The close relationship of code and data suggests making new code on the fly to solve problems, which is a concept used in genetic algorithms. Here is an example inspired by GAs. Let's write a function which finds expressions involving the four arithmetic operators that return the number 2. Examples:

(+ 1 1)
(- 4 2)
(* 1 2)
(/ 2 1)

First, let's make our expression generator (type or copy-paste into your REPL):

(defn oper [] (rand-nth '(* / - +)))
(defn digit [] (rand-int 10))

(defn numlist []
  (let [numnums (rand-int 10)]
    (repeatedly numnums digit)))

(defn expr-fn []
  (cons (oper) (numlist)))

Trying these out:

(oper)    ;;=> *
(oper)    ;;=> /
(numlist) ;;=> (6)
(numlist) ;;=> (6 9 3 4 4 2 4 3 2)

(expr-fn) ;;=> (/ 6 1 3 7 0 3 0 9)
(expr-fn) ;;=> (- 5 7 2)

Our expression generator selects an arithmetic operator at random, and tacks it on to the beginning of a random list of 0-9 digits.

Evaluating some example functions:

(eval (expr-fn)) ;;=> 0
(eval (expr-fn)) ;;=> 41
(eval (expr-fn)) ;;=> 7/1152
(eval (expr-fn)) ;;=> java.lang.ArithmeticException stacktrace!?!?!

This last one shows a problem: some of our expressions will not be valid mathematically (for example, (/ 1 0)). Let's catch any exceptions:

(defn eval-with-catch-exception [expr]
  (try
    (eval expr)
    (catch Throwable t
      (-> t .getClass .getSimpleName symbol))))

This function attempts to evaluate expr and, if the code throws an exception, simply returns the exception name as a symbol.

We are now in a position to try our code generator out:

(dotimes [_ 10]
  (let [e (expr-fn)]
    (println e (eval-with-catch-exception e))))

which gives

(/ 9 8 7 0 0 7 2 2 6) ArithmeticException
(+ 1 3 3 0 7 7 6) 27
(+ 4 2) 6
(- 4 1 2 1 0 0 0) 0
(/ 5 7 0 7) ArithmeticException
(* 0 4 6) 0
(* 8 6 7 2 4 1) 2688
(* 7 4 6 9 8 6) 72576
(*) 1
(- 7) -7

(* called with no arguments returns 1; + with no arguments returns 0. This may or may not make intuitive sense to you, depending on how much math you've had in school.)

Let's generate some that return the desired result, 2:

(distinct
 (for [_ (range 1000)
       :let [f (expr-fn)
             result (eval-with-catch-exception f)
             fitness-fn #(= result 2)]
       :when (fitness-fn)]
   f))
;;=>
((+ 2)
 (- 9 1 1 1 4)
 (- 2 0)
 (/ 4 2)
 (/ 2 1)
 (* 2))

(Note that (+ n) and (* n) both just return n.)

Macros

Macros are an advanced topic, but they are relevant because they allow (among other things) selective evaluation of arguments. Arguments to a macro can be evaluated zero, one, or many times; this makes macros especially helpful for implementing control flow: loop, for, while etc. are all macros.

Macros rely on code-as-data: they essentially functions which generate code, which is then evaluated. Most or all modern Lisps have macros, the equivalent of which are very rare in non-Lisps. These macros play a large role in making Lisp so expressive.

Exercises

Exercise 1: Creating and evaluating code on the fly in this manner is not unique to Lisp, but Lisp provides a very natural way to do so. Consider how to do the above examples in JavaScript, Ruby, or other mainstream languages. What, if any, are the differences between Clojure/Lisp and those languages?

Exercise 2: Set a different arithmetic goal (other than 2) and generate expressions which satisfy it.

Exercise 3: Adapt your code generator to include nested expressions, so that some of the numbers are generated expressions in their own right. Example: (+ 1 4) -> (+ 1 (* 2 2)).

Exercise 4: Instead of generating expressions, adapt expr-fn to return functions of no arguments, and adapt the rest of the code to call and evaluate those functions. I.e., (+ 1 2 3) becomes (fn [] (+ 1 2 3)).

Exercise 5 (advanced): Both the generated functions and the "fitness function" fitness-fn can be made more complex; for example, in addition to the operators + - * / and the numbers 0-9, you could add an "argument" variable x and change the fitness function to expect expressions that return the square of x; some acceptable outputs would then be (* x x) and (* 1 x x 1).

Change your expression generator and fitness functions to implement the squaring function.

Exercise 6 (advanced): try adding other Clojure core functions and data types to the list of possible operators and arguments; try other fitness functions as well.

Exercise 7 (more advanced): develop a way of combining expressions and evaluating the fitness of the resulting expression. To build on our previous example, if you have the two lists (+ 1 1) and (* 2 3), you could choose terms at random from each list, e.g. (* 1 3); you must decide how to handle lists with different lengths. The fitness function could just be fitness-fn from our example, or a more ambitious one². If you want to include nested expressions, then you have to decide how to combine those.

Rather than success or failure, the fitness function could return a number which reflects how well or poorly the expression performed. A strategy from genetic algorithms is to generate many expressions, evaluate these, take the best performers, and then generate new expressions by combining these. Implement this.

(oodles of delicious atoms)

Wed, 30 May 2018 00:00:00 +0000

A Recursive Feast

In 1983, Douglas R. Hofstadter, author of "Gödel, Escher, Bach: An Eternal Golden Braid" wrote a series of articles on Lisp in Scientific American (later republished in the book Metamagical Themas, which is where I encountered them for the first time). While much of the writing focused on the nature of recursion in programming, the third and last article in the series provided an entertaining example of blurring the line between code and data, which is typical for Lisp (though somewhat less common in Clojure, except for the occasional macro).

In the example, a set of acronyms are presented; each acronym can be "expanded" into a phrase which itself may contain other acronyms. The program generates random expansions of these acronyms, "bottoming out" (hitting the base case) at random, though with increasing probability of termination for more deeply-nested expressions.

What follows is my translation of the Lisp code in his articles into Clojure, with a few extra details and explanations. A few interesting edge cases have to be accounted for to accommodate the differences between the languages.

If you're interested in running the code, it's on GitHub.

Cooking It Up

First, a small macro helps us keep things tidy: defpasta makes a function that simply returns the list supplied as its argument, avoiding the need for the argument vector or the quote. Note that just one small macro provides enough "syntactic sugar" to make the examples more readable. We also attach metadata for convenience in testing (see below).

(defmacro defpasta [name_ & expr]
  `(defn ~(with-meta name_ {:pasta true}) []
     '~@expr))

Next are the actual acronym definitions. These are the definitions Hofstadter uses, with commas treated differently. The commas are important in his output, but commas are whitespace in Clojure… so where the symbols are used with commas in compound statements, we append _CO to tell the program to supply a comma in the output. (Example: SAUCE, becomes SAUCE_CO.)

We cannot, unfortunately, add commas after lists in this way, but his example is inconsistent in this regard (three examples have commas after lists, but the example expansion later in the article does not).

Hostadter presents the acronyms in capital letters, though Common Lisp atoms (analogous to Clojure symbols) are case-insensitive; we follow that convention here and use it to indicate terms which may be expanded… that is to say, the COFFEE in the expansion of SAUCE can itself be expanded to a phrase which includes ESPRESSO, and so on.

(defpasta TOMATOES (TOMATOES on MACARONI (and TOMATOES only)
                             exquisitely SPICED))
(defpasta MACARONI (MACARONI and CHEESE (a REPAST of Naples_CO Italy)))
(defpasta REPAST (rather extraordinary PASTA and SAUCE_CO typical))
(defpasta CHEESE (cheddar_CO havarti_CO, emmentaler
                                (especially SHARP emmenthaler)))
(defpasta SHARP (strong_CO hearty_CO and rather pungent))
(defpasta SPICED (sweetly pickled in CHEESE ENDIVE dressing))
(defpasta ENDIVE (egg NOODLES_CO dipped in vinegar eggnog))
(defpasta NOODLES (NOODLES (oodles of delicious LINGUINI) elegantly served))
(defpasta LINGUINI (LAMBCHOPS (including NOODLES)
                                  gotten usually in Northern Italy))
(defpasta PASTA (PASTA and SAUCE (that's ALL!)))
(defpasta ALL! (a lucious lunch))
(defpasta SAUCE (SHAD and unusual COFFEE (excellente!)))
(defpasta SHAD (SPAGHETTI_CO heated al dente))
(defpasta SPAGHETTI (standard PASTA_CO always good_CO hot particularly
                              (twist_CO then ingest)))
(defpasta COFFEE (choice of fine flavors, particularly ESPRESSO))
(defpasta ESPRESSO (excellent_CO strong_CO powerful_CO rich ESPRESSO_CO
                                 suppressing sleep outrageously))
(defpasta BASTA! (belly all stuffed (tummy ache!)))
(defpasta LAMBCHOPS (LASAGNE and meatballs_CO casually heaped onto PASTA SAUCE))
(defpasta LASAGNE (LINGUINI and SAUCE and GARLIC (NOODLES everywhere!)))
(defpasta RHUBARB (RAVIOLI_CO heated under butter and RHUBARB (BASTA!)))
(defpasta RAVIOLI (RIGATONI and vongole in oil_CO lavishly introduced))
(defpasta RIGATONI (rich Italian GNOCCHI and TOMATOES (or NOODLES instead)))
(defpasta GNOCCHI (GARLIC NOODLES over crisp CHEESE_CO heated immediately))
(defpasta GARLIC (green and red LASAGNE in CHEESE))

Hofstadter leaves the implementation of acronym? undefined, since it is so implementation-specific. In our case, something is an acronym if it's a symbol and its name is equal to an upper case version of itself.

(defn acronym? [x]
  (and (symbol? x)
       (= (name x) (.toUpperCase (name x)))))

We also want to be able to remove commas from a symbol and evaluate the result as a function. This is also very implementation-specific; in our case, it relies on Clojure's machinery for converting symbols to strings and vice-versa, and on the subtle differences between symbols, vars, and functions:

(defn strip-comma-and-eval [sym]
  (let [base-symbol-name (-> sym
                             name
                             (clojure.string/replace #"_CO$" "")
                             symbol)]
    ((-> *ns*
          ns-map
          (get (symbol base-symbol-name))
          var-get))))

(strip-comma-and-eval 'RHUBARB)
;;=>
(RAVIOLI_CO heated under butter and RHUBARB (BASTA!))

lower reduces the probability by some amount to encourage our recursion to bottom out (and thereby avoid stack overflows). Hofstadter multiplies probabilities by 0.8, but I simply square the probability so that higher probabilities decrease more slowly than lower ones as the recursion progresses:

(defn lower [x] (* x x))

A significant portion of Hofstadter's article relates to the recursive expand function, so I won't go into details here. This version is similar to his; it adds a bottoming-out clause to accommodate the base case of an empty phrase (since () and nil are not the same in Clojure, as they are in Common Lisp). Also we use strip-comma-and-eval instead of plain eval to handle the comma-ed symbols.

(defn expand [phrase probability]
  (cond
    (symbol? phrase) phrase
    (empty? phrase) phrase

    (acronym? (first phrase))
    (if (< (rand) probability)
      (concat
       (expand (strip-comma-and-eval (first phrase)) (lower probability))
       (expand (rest phrase) probability))
      (cons (first phrase) (expand (rest phrase) probability)))

    :else (cons (expand (first phrase) (lower probability))
                (expand (rest phrase) (lower probability)))))

Ultimately we want to make the output similar to what Hofstadter shows in the book. The normalize function does this, relying on two helper functions, whose names are self-explanatory:

(defn lower-case-symbol [x]
  (if-not (symbol? x)
    x
    (-> x name .toLowerCase symbol)))

(defn get-commas-back [x]
  (if-not (symbol? x)
    x
    (-> x
        name
        (clojure.string/replace #"_CO$" ",")
        symbol)))

When we're done expanding our lists of food (and our bellies), we prepare our data for output to the user: turn _CO suffixes into actual commas (using the fact that (symbol "x,") yields a valid symbol, if one that would be hard to read back at the REPL); and, lower-case all symbols to adhere to the style of the output shown in Metamagical Themas:

(defn normalize [expr]
  (->> expr
       (clojure.walk/postwalk get-commas-back)
       (clojure.walk/postwalk lower-case-symbol)))

Dinner Is Served

Now for the actual function we'll use to generate our expansions. Setting a large probability will ensure longer examples:

(defn dinner [expr]
  (normalize (expand expr 0.9999)))

We should test our code to make sure it doesn't bomb out on any of our acronyms. Here we find all the acronyms based on the metadata added by the defpasta macro, ensuring each runs to completion:

(let [nsm (ns-map *ns*)]
  (doseq [[k v] nsm :when (:pasta (meta v))]
    (dotimes [_ 10]
      (dinner (list k)))))

If, like me, you use Emacs and CIDER and want long REPL examples, you have to set `*print-length*` to something bigger than its default:

(set! *print-length* 1000)

Now to run it! Are you hungry yet?

(dinner '(LINGUINI))
;;=>
(lasagne and meatballs, casually heaped onto pasta sauce (including
noodles) gotten usually in northern italy and standard pasta, always
good, hot particularly (twist, then ingest) heated al dente and
unusual coffee (excellente!) and green and red lasagne in cheese
(noodles (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served everywhere!) and meatballs,
casually heaped onto pasta shad and unusual coffee (excellente!)
(including noodles (oodles of delicious linguini) elegantly served
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious lasagne and meatballs, casually heaped
onto pasta sauce (including noodles) gotten usually in northern italy)
elegantly served) gotten usually in northern italy and standard pasta
and shad and unusual coffee (excellente!) (that's all!) and shad and
unusual coffee (excellente!) (that's all!) always good, hot
particularly (twist, then ingest) heated al dente and unusual choice
of fine flavors particularly espresso (excellente!) and green and red
lasagne in cheese (noodles (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served everywhere!) and meatballs, casually heaped onto
pasta and sauce (that's all!) and sauce (that's all!) and shad and
unusual coffee (excellente!) (that's all!) sauce (including noodles
(oodles of delicious linguini) elegantly served (oodles of delicious
lambchops (including noodles) gotten usually in northern italy)
elegantly served (oodles of delicious lambchops (including noodles)
gotten usually in northern italy) elegantly served) gotten usually in
northern italy and standard pasta and shad and unusual coffee
(excellente!) (that's all!) and standard pasta, always good, hot
particularly (twist, then ingest) heated al dente and unusual coffee
(excellente!) (that's all!) and spaghetti, heated al dente and unusual
coffee (excellente!) (that's a lucious lunch) and standard pasta,
always good, hot particularly (twist, then ingest) heated al dente and
unusual coffee (excellente!) (that's a lucious lunch) always good, hot
particularly (twist, then ingest) heated al dente and unusual choice
of fine flavors particularly espresso (excellente!) and green and red
linguini and sauce and garlic (noodles everywhere!) and meatballs,
casually heaped onto pasta sauce (including noodles) gotten usually in
northern italy and spaghetti, heated al dente and unusual coffee
(excellente!) and green and red lasagne in cheese (noodles (oodles of
delicious linguini) elegantly served everywhere!) in cheese (noodles
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious lambchops
(including noodles) gotten usually in northern italy) elegantly served
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served everywhere!) and meatballs, casually heaped
onto pasta and shad and unusual coffee (excellente!) (that's a lucious
lunch) and sauce (that's all!) and spaghetti, heated al dente and
unusual coffee (excellente!) (that's a lucious lunch) and standard
pasta and sauce (that's all!) always good, hot particularly (twist,
then ingest) heated al dente and unusual coffee (excellente!) (that's
a lucious lunch) shad and unusual choice of fine flavors particularly
espresso (excellente!) (including noodles (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious lasagne and
meatballs, casually heaped onto pasta sauce (including noodles) gotten
usually in northern italy) elegantly served (oodles of delicious
lambchops (including noodles) gotten usually in northern italy)
elegantly served (oodles of delicious linguini and sauce and garlic
(noodles everywhere!) and meatballs, casually heaped onto pasta sauce
(including noodles (oodles of delicious linguini) elegantly served)
gotten usually in northern italy) elegantly served (oodles of
delicious lambchops (including noodles (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served
(oodles of delicious lasagne and meatballs, casually heaped onto pasta
sauce (including noodles) gotten usually in northern italy and shad
and unusual coffee (excellente!) and garlic (noodles everywhere!) and
meatballs, casually heaped onto pasta sauce (including noodles (oodles
of delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served
(oodles of delicious linguini and sauce and garlic (noodles
everywhere!) and meatballs, casually heaped onto pasta sauce
(including noodles) gotten usually in northern italy and spaghetti,
heated al dente and unusual coffee (excellente!) and garlic (noodles
(oodles of delicious linguini) elegantly served everywhere!) and
meatballs, casually heaped onto pasta sauce (including noodles (oodles
of delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served)
gotten usually in northern italy)

Man, I'm stuffed!

A talk based on this code was given as in instructional / fun exercise at Clojurepalooza at OpinionLab in May of 2018.

Lisp on John Jacobsen

Lisp Testing Fun

Fun

To The Metal... Compiling Your Own Language(s)

The Problem In General

Small Is Beautiful

An Approach

Enter Babashka

Building A Compiling Calculator

Lisp

Alternatives and Future Directions

Conclusion

Adding Tail Call Optimization to A Lisp Written in Go

Motivation

The Optimization

The Approach

Implementation

Conclusion

Tests by Example in Clojure and Common Lisp

(Yet Another) Lisp In Go

Features

Performance

Lexing

Fun With Atoms and Lists

Testing

Further Work

A Final Note on Performance, and Conclusion

From Elegance to Speed

Making Things Fast

First Steps

Interlude

Improvements

Summary

Conclusion

Common Lisp How-Tos

Intro

Lessons Learned So Far

How Do I Set Up Common Lisp on a Mac?

How Do I Get Environment Variables from Lisp?

How Do I Quickly Create a New Project?

How Do I Add Dependencies to One Of My Projects?

How Do I Get My Local Projects Loadable By Other Projects?

How Do I Get Dependencies Loaded

How Do I Build Binaries?

How Do I Get My Project Into Quicklisp?

How Do I Use Curses (or equivalent)?

How Do I Get Help?

Future Projects

Implementing Scheme in Python

Background

Scheming in Python

Prior Art

The Stages

Parsing

Data Model

Scheme raw source

Lark output

Internal representation used in the interpreter

Evaluation

Printing

Testing Approach

The Code Comes To Life

Next Steps

Implement more of Scheme

Extend it Further

Go Low Level

Thoughts Upon Revisiting Python

Conclusion

Postscript

Code and Data

Introduction

Arithmetic of the Fittest

Macros

Exercises

See Also

(oodles of delicious atoms)

A Recursive Feast

Cooking It Up

Dinner Is Served

How Do I Use `Curses` (or equivalent)?