John Jacobsen

"Sites of Arrival" Group Show

Thu, 17 Jul 2025 00:00:00 +0000

Sites of Arrival

A group show of (mostly) figurative art at the Palette and Chisel Academy of Fine Arts:

Location and Hours

Palette and Chisel Academy of Fine Arts
August 8-17, 2025

Opening August 8, 6-9 PM

1012 N. Dearborn St.
Chicago, IL 60610

Office phone: 312 642 4400
Office email: fineart1012@sbcglobal.net

Works in the Show

Noli Appropinquare, oil on canvas, 40"x28"

Atop Taygetus, oil on panel, 12"x9"

Evening Canalscape, oil on canvas, 12"x8"

Wall Town Denizens, gouache on paper, 12"x9"

Road Embrace, gouache on paper, 10"x7"

Pink Butte, oil on panel, 10"x10"

Grey Figure Study, gouache and ink on paper, 10"x7"

Four Women, oil on four panels, approx. 14"x7"

Figures and Dog, gouache and ink on paper, 12"x9"

Woman, Seated, at Night, oil on wood, 12"x9.5"

Female Figures, graphite on paper, 14"x11"

Seated and Standing Figures, graphite on paper, 14"x11"

Seated Figures, graphite on paper, 14"x11"

Artist's Statement

I have been showing works done from life at the Palette and Chisel for several years. Such studies are very helpful for training the eye and are entirely consistent with the P&C’s founding mission: to provide a place to draw, paint and sculpt from the model, and to show the resulting artwork.

At the same time I have long maintained a practice of drawing and painting from imagination, sometimes deriving finished work in the studio from life studies. Lessons learned from painstaking work from life can blossom in the studio into more complex, multi-figure compositions, wide-open spaces, and otherworldly color palettes.

Some of the work shown here was begun decades earlier and tinkered with slowly over the years, each piece going back on the shelf for long periods of time, and then coming out when inspiration strikes or the solution to some unresolved problem suddenly becomes clear. Bodo’s suggestion to join this show gave me the welcome nudge to complete some of these long-evolving pieces.

I view each drawing or painting as a unique experiment in solving certain problems simultaneously. The bedrock requirement of an artwork is that it makes sense at the level of design, while also obeying the laws of perspective, light, and anatomy, as well as the artist’s own interior laws, which are only revealed through the process of painting itself.

I am suspicious of paintings where the outcome seems to have been decided in advance. Each of these pieces is as much a surprise to me as it may be to the viewer. Their logic is their own, though I have tried to fill in the blanks. As one of my art heroes, Gregory Gillespie, once said, “Paintings make sense if I paint them beautiful enough.” Pretty much sums it up, though he set a high bar in that regard.

Biography

John Jacobsen was born in Madison, Wisconsin in the previous millennium. He studied art and physics at the University of Wisconsin–Madison and received his Ph.D. in 1996. In addition to his work as an artist, Jacobsen has had a long career as a physicist and software engineer, traveling extensively to such places as CERN (Geneva), Hawaii, Europe, and the Amundsen-Scott South Pole Station in Antarctica, which he has visited ten times in his work on the AMANDA and IceCube astrophysical experiments.

After living and exhibiting in the San Francisco Bay Area from 1997-2002, he moved to Chicago, where he currently lives.

Jacobsen’s artwork is influenced by his travels, as well as his artistic idols, which include Gregory Gillespie, Odd Nerdrum, Andrew Wyeth, and American “Golden Age” illustrators. His teachers have included notable printmakers Jack Damer and David H. Becker at UW-Madison, and the painters Steven Assael and Andrew S. Conklin at the Palette and Chisel Academy of Fine Arts in Chicago.

Thanks to Bodo Stolczenberger for helping to arrange this show.

For news and updates

Say hi on email, or check me on Instagram (eigenhombre).

Lisp Testing Fun

Fri, 11 Apr 2025 00:00:00 +0000

I've been making some progress with steelcut, my Common Lisp template generator (does every Lisper write one of these as a rite of passage?), and I keep running into interesting problems which have been fun to solve.

Whenever my code starts to accumulate any sort of complexity, I try to err on the side of writing more tests, even for "recreational" projects such as this one. The tests help make it safe to make changes, while also documenting the expected functionality. But test code can be fairly repetitive, often with many small variations and lots of boilerplate, in the midst of which it can be hard to see the forest for the trees.

Such was the case for steelcut tests. I found myself remembering Paul Graham's line, "Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough – often that I'm generating by hand the expansions of some macro that I need to write."

In my current non-Lisp programming work, I most often miss macros when writing test code. But in this case, the macro I (apparently) needed to write is here. Very roughly speaking, with-setup creates a temporary directory, runs the template generator with a given list of requested features, and allows the user to check properties of the generated files.

The macro grew and shrank as I DRYed out and simplified the test code. I realized many of my tests were checking the generated dependencies for the projects (from the .ASD file), so it would be helpful if the macro would make those available to the test code in the form of a function the user could name.

This wound up being a sweet bit of sugar for my tests, but not every test needed such a helper function. Stubborn "unused variable" warnings ensued, for those tests which don't use the deps symbol bound by the macro. I went back and forth with ChatGPT on this one (it made several wrong suggestions) until I realized I could give the variable an explicit _ name and change the behavior based on the name. This is something I've seen in other languages and was tickled I could get it so easily here.

I find the resulting tests pretty easy to read:

(test needed-files-created
  (with-setup appname "testingapp" appdir _ +default-features+
    (loop for file in '("Makefile"
                        "Dockerfile"
                        ".github/workflows/build.yml"
                        "build.sh"
                        "test.sh"
                        "src/main.lisp"
                        "src/package.lisp"
                        "test/test.lisp"
                        "test/package.lisp")
          do
             (is (uiop:file-exists-p (join/ appdir file))))))

(test cmd-feature-adds-dependency-and-example
  (with-setup appname "testingapp" appdir deps +default-features+
    (let ((main-text (slurp (join/ appdir "src/main.lisp"))))
      ;; :cmd is not there:
      (is (not (find :cmd (deps))))
      ;; It doesn't contain the example function:
      (is (not (has-cmd-example-p main-text)))))

  ;; Add :cmd to features, should get the new dependency:
  (with-setup appname "test2" appdir deps (cons :cmd +default-features+)
    (let ((main-text (slurp (join/ appdir "src/main.lisp"))))
      ;; :cmd is now there:
      (is (find :cmd (deps)))
      ;; It contains the example function:
      (is (has-cmd-example-p main-text)))))

N.B.: A keyword argument would also work, obviously. For a macro in production code, I would probably go that route instead.

Fun

I've also been thinking about a recent post on Hacker News, and the ensuing discussion, in which both the author and commenters point out that programming Lisp is fun. While I must acknowledge that Lisp has been a life-long attraction/distraction/diversion, I'm noticing that the way it is fun for me is evolving somewhat.

First, I used to find Common Lisp pretty painful because of all its sharp edges – the language is so powerful and flexible, you can shoot yourself in the foot pretty much any way you like, and it's not as easy as some newer languages to get started with. But now, with LLMs, it's much easier to get help when you get into the weeds. Sure, "the corpus of training data was smaller for Lisp than for mainstream languages," blah, blah. But often even the wrong answers from ChatGPT point me in the right direction (some of this is probably just rubberducking). When one's own ignorance is less of an obstacle, the pointy bits become less intimidating.

Second, there are some really good, and fun, books on Lisp. PAIP, Let Over Lambda, Practical Common Lisp, Land of Lisp (plus video!). Lisp has some pretty mind-bending things in it, and I'm enjoying taking the time to dig into some of these books again, understanding things better than the first time I swung by.

Third, I'm regularly stunned by how fast Lisp is. All the tests I've been writing run effectively instantly at the REPL on my M1 Macbook Air. And now that I'm starting to get good enough at the language that it is getting out of my way, that speed is more noticeable, and, well, fun.

Finally, I am intrigued by the history of computing, of which Lisp has been an important part. This is worth saying more about in a future post, but for the moment I find a stable, highly-performant, fun language that has intrigued people for decades to be more of interest than the bleeding edge. (I'm not hating on Rust. Rust is nice. But I don't find it fun, at least not yet.)

Beyond fun, I'm also enjoying "escaping the hamster wheel of backwards incompatibility" for awhile. While so many things around us crumble and whirl, and new forms of AI scare the pants out of us as much as they intrigue, older tools and traditions start to feel more like comfortable tools that weigh down one's hands satisfyingly and invite calm creation.

To The Metal... Compiling Your Own Language(s)

Tue, 06 Aug 2024 00:00:00 +0000

Like many programmers, I have programming hobbies. One of these is implementing new languages. My most recent language project, l1, was a Lisp dialect whose primary data types are symbols and arbitrarily-large integers.

I've been happy with l1, but it is interpreted; since I was actively working on it last (2022), I've been wondering about the best way to generate compiled standalone executable programs, written in l1 or any other language.

The Problem In General

Execution models for programming languages take three basic approaches, listed in increasing order of speed:

Tree-walking interpreter: Programs are read and parsed into ASTs in memory, then executed step-by-step by an interpreter. This is the approach l1 uses.
Bytecode VM: Programs are compiled into a sort of abstract machine language, simpler than the physical processor's, and executed by a virtual machine (VM). Java and Python work this way.
Machine code generation: The code is directly compiled into machine language and executed on the user's hardware. C and C++ programs work this way.

Languages using Option 2 often add just-in-time compilation to machine code, for extra performance. Option 3 is typically fastest, but is sometimes skipped in introductory compiler classes and tutorials. For example, Robert Nystrom's excellent Crafting Interpreters book devotes the first section to implementing a tree-walking interpreter implementation in Java and the second half to a compiler and bytecode VM written in C, with minimal coverage of how to target physical hardware. And the (also excellent) class on compiler writing that I took from David Beazley, in its first incarnation, stopped at the point of generating of so-called intermediate representation (IR) output (though students in the current iteration of the class do compile to native code, using LLVM).

Compiling to machine code is tricky because CPUs are inherently complex. Real hardware is intricate, cumbersome, and unintuitive if you're primarily accustomed to high-level languages. Additionally, there are numerous significant variants to consider (e.g., CPU/GPU, ARM/Intel, 32-bit/64-bit architectures).

But targeting machine code rather than interpreters or bytecode VMs is appealing, not just because it is an interesting challenge, but also because the resulting artifacts are small, stand-alone, and typically very fast. While running Python, Ruby, and Java programs require the appropriate infrastructure to be in place on the target machine at all times, Go, Rust, and C programs (among others) benefit from targeting the physical hardware: their programs tend to be smaller, and can be deployed to identical computers simply by copying the executable file, needing to deploy the interpreter, extra libraries, etc. to the target machine(s).

Small Is Beautiful

As a programmer who came up during the dawn of personal computers, I have some nostalgia for an era when programs or even entire operating systems fit on a few-hundred-kB floppy disk. Much existing software feels bloated to me, though some widespread tools are still lean and fast. For illustration purposes, here are the physical sizes of some of the venerable command-line Unix programs I use on a daily basis (this is on MacOS):

Program	Size (kB)
`wc`	100
`cat`	116
`df`	116
`more`	360

These were chosen more or less at random from my bash history and are representative of old-school Unix utilities. For comparison, iMovie on my Mac is 2.8 GB, several thousand times larger than the largest of these. Of course, the comparison is somewhat ludicrous - iMovie does many amazing things… but I use all the above programs hundreds or thousands of times more often than I do iMovie, so it's good that that they are compact and run quickly. In a time of increasingly bloated software stacks, I find myself especially drawn to simple tools with small footprints.

An Approach

If targeting physical hardware is hard, what tools can we use to make the job easier?

I recently started learning about LLVM, a modular set of compiler tools which "can be used to develop a frontend for any programming language and a backend for any instruction set architecture" (Wikipedia). LLVM has been used heavily in the Rust toolchain and in Apple's developer tools.

The "modular" adjective is critical here: LLVM is separated into front-end, back-end and optimizing parts thanks to a shared "intermediate representation" (IR) – a sort of portable assembly language which represents simple computation steps in a machine-independent but low-level manner.

The LLVM IR takes a little getting used to but, with a little practice, is reasonably easy to read, and, more importantly, to generate.

As an example, consider the following simple C program, three.c, which stores the number 3 in a variable and uses it as its exit code. We will use clang, the LLVM C/C++/Obj-C/… compiler for the LLVM ecosystem:

$ cat three.c
int x = 3;

int main() {
  return x;
}
$ clang three.c -o three
$ ./three; echo $?
3

One can easily view, and possibly even understand, the assembler output for such a simple program:

$ clang -O3 -S three.c -o three.s
$ cat -n three.s
     1		.section	__TEXT,__text,regular,pure_instructions
     2		.build_version macos, 14, 0	sdk_version 14, 4
     3		.globl	_main                           ; -- Begin function main
     4		.p2align	2
     5	_main:                                  ; @main
     6		.cfi_startproc
     7	; %bb.0:
     8	Lloh0:
     9		adrp	x8, _x@PAGE
    10	Lloh1:
    11		ldr	w0, [x8, _x@PAGEOFF]
    12		ret
    13		.loh AdrpLdr	Lloh0, Lloh1
    14		.cfi_endproc
    15	                                        ; -- End function
    16		.section	__DATA,__data
    17		.globl	_x                              ; @x
    18		.p2align	2, 0x0
    19	_x:
    20		.long	3                               ; 0x3
    21
    22	.subsections_via_symbols

In comparison, here is the LLVM IR for the same program:

$ clang -S -emit-llvm three.c -o three.ll
$ cat -n three.ll
     1	; ModuleID = 'three.c'
     2	source_filename = "three.c"
     3	target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
     4	target triple = "arm64-apple-macosx14.0.0"
     5
     6	@x = global i32 3, align 4
     7
     8	; Function Attrs: noinline nounwind optnone ssp uwtable(sync)
     9	define i32 @main() #0 {
    10	  %1 = alloca i32, align 4
    11	  store i32 0, ptr %1, align 4
    12	  %2 = load i32, ptr @x, align 4
    13	  ret i32 %2
    14	}
    15
    16	attributes #0 = { noinline nounwind optnone ssp ;; .... very long list...
    17
    18	!llvm.module.flags = !{!0, !1, !2, !3, !4}
    19	!llvm.ident = !{!5}
    20
    21	!0 = !{i32 2, !"SDK Version", [2 x i32] [i32 14, i32 4]}
    22	!1 = !{i32 1, !"wchar_size", i32 4}
    23	!2 = !{i32 8, !"PIC Level", i32 2}
    24	!3 = !{i32 7, !"uwtable", i32 1}
    25	!4 = !{i32 7, !"frame-pointer", i32 1}
    26	!5 = !{!"Apple clang version 15.0.0 (clang-1500.3.9.4)"}

There is a fair amount of stuff here, but a lot of it looks suspiciously like metadata we don't really care about for our experiments going forward. The, uh, main region of interest is from lines 9-14 – notice that the function definition itself looks a little more readable than the assembly language version, but slightly lower-level than the original C program.

You can turn the IR into a runnable program:

$ clang -O3 three.ll -o three
$ ./three; echo $?
3

The approach I explore here is to generate LLVM IR "by fair means or foul." Here, let's just edit our IR down to something more minimal and see how it goes. I suspect the store of 0 in "register" %1 is gratuitous, so let's try to remove it, along with all the metadata:

$ cat 3.ll
target triple = "x86_64-apple-macosx14.0.0"

@x = global i32 3, align 4

define i32 @main() {
  %1 = load i32, ptr @x, align 4
  ret i32 %1
}
$ clang -O3 3.ll -o 3
$ ./3; echo $?
3

This is frankly not much more complicated than the C code, and it shows a helpful strategy at work:

Step 1: To understand how to accomplish something in LLVM IR, write the corresponding C program and use clang to generate the IR, being alert for possible "extra stuff" like we saw in the example.

Step 2. Try to generate, and test, working programs from the IR you write or adapt, making adjustments as desired.

There is another, optional step as well:

Step 3. Use, or write, language "bindings" to drive LLVM generation from the language of your choice. This is the step we will consider next.

Enter Babashka

While one can write LLVM IR directly, as we have seen, we are interested in compiling other languages (possibly higher-level ones of our own invention), so we will want to generate IR somehow. For this project I chose Babashka, an implementation of the Clojure programming language I have found ideal for small projects where both start-up speed and expressiveness are important.

(I assume some familiarity with Lisp and Clojure in this post; for those just getting started, Clojure for the Brave and True is a good introduction.)

The repo https://github.com/eigenhombre/llbb contains the source files discussed here. The bulk of the code in this repo is in llir.bb, a source file which provides alternative definitions in Clojure for common LLVM idioms. Some of these are trivial translations:

(def m1-target "arm64-apple-macosx14.0.0")

(defn target [t] (format "target triple = \"%s\"" t))

… whereas other expressions leverage the power of Clojure to a greater degree. For example, this section defines translations used to represent arithmetic operations:

(defn arithm [op typ a b]
  (format "%s %s %s, %s"
          (name? op)
          (name? typ)
          (sigil a)
          (sigil b)))

(defn add [typ a b] (arithm :add typ a b))
(defn sub [typ a b] (arithm :sub typ a b))
(defn mul [typ a b] (arithm :mul typ a b))
(defn div [typ a b] (arithm :sdiv typ a b))

(comment
  (div :i32 :a :b)
  ;;=>
  "sdiv i32 %a, %b"

  (add :i8 :x 1)
  ;;=>
  "add i8 %x, 1")

You can see this approach at work by representing the C program discussed earlier:

(module
 (assign-global :i :i32 3)
 (def-fn :i32 :main []
   (assign :retval (load :i32 :i))
   (ret :i32 :retval)))

which evaluates to

target triple = "arm64-apple-macosx14.0.0"

@i = global i32 3

define i32 @main() nounwind {
  %retval = load i32, i32* @i, align 4
  ret i32 %retval
}

I use here a slightly different but equivalent pointer syntax for the load expression than output by clang in the example above.

Two very small helper functions allow me to test out small programs quickly:

(require '[babashka.process :as sh])

(defn sh
  "
  Use `bash` to run command(s) `s`, capturing both stdout/stderr
  as a concatenated string.  Throw an exception if the exit code
  is nonzero.
  "
  [s]
  (let [{:keys [out err]}
        (sh/shell {:out :string, :err :string}
                  (format "bash -c '%s'" s))]
    (str/join "\n" (remove empty? [out err]))))

and

(require '[babashka.fs :as fs])

(defn compile-to
  "
  Save IR `body` to a temporary file and compile it, writing the
  resulting binary to `progname` in the current working directory.
  "
  [progname body]
  (let [ll-file
        (str (fs/create-temp-file {:prefix "llbb-", :suffix ".ll"}))]
    (spit ll-file body)
    (sh (format "clang -O3 %s -o %s"
                ll-file
                progname))))

These two together allow me to test small programs out quickly at the REPL. Some examples follow, obtained by running the C compiler for equivalent programs, generating and inspecting the LLVM IR, and translating them into new Clojure bindings as cleanly as possible.

Minimum viable program: just return zero:

(compile-to "smallest-prog"
            (module
             (def-fn :i32 :main []
               (ret :i32 0))))

(sh "./smallest-prog; echo -n $?")
;;=>
"0"

Argument count: return, as the exit code, the number of arguments, including the program name:

;; Argument counting: return number of arguments as an exit code:
(compile-to "argcount"
            (module
             (def-fn :i32 :main [[:i32 :arg0]
                                 [:ptr :arg1_unused]]
               (assign :retptr (alloca :i32))
               (store :i32 :arg0 :ptr :retptr)
               (assign :retval (load :i32 :retptr))
               (ret :i32 :retval))))

(sh "./argcount; echo -n $?") ;;=> "1"
(sh "./argcount 1 2 3; echo -n $?") ;;=> "4"

Hello, world:

(let [msg "Hello, World."
      n (inc (count msg))] ;; Includes string terminator
  (compile-to "hello"
              (module
               (external-fn :i32 :puts :i8*)
               (def-global-const-str :message msg)
               (def-fn :i32 :main []
                 (assign :as_ptr
                         (gep (fixedarray n :i8)
                              (star (fixedarray n :i8))
                              (sigil :message)
                              [:i64 0]
                              [:i64 0]))
                 (call :i32 :puts [:i8* :as_ptr])
                 (ret :i32 0)))))

(sh "./hello") ;;=> "Hello, World.\n"

This is the first program one typically writes in a new programming language. Note that we use here idioms (external-fn, call) to define and invoke an external function from the C standard library.

Let's see how big the resulting program is:

(sh "ls -l hello")
;;=>
"-rwxr-xr-x  1 jacobsen  staff  33432 Aug 14 21:09 hello\n"

At this point I want to pause to reconsider one of the points of this exercise, which is to produce small programs. Here are the rough executable sizes for a "Hello, World" example program in various languages that I use frequently:

Language	Size	Relative Size
Common Lisp	38 MB	1151
Clojure	3.4 MB	103
Go	1.9 MB	58
C	33 kB	1
LLVM IR	33 kB	1

I threw Clojure in there for comparison even though, unlike the other examples, the resulting überjar also requires a Java bytecode VM in order to run. The programs generated from C and from LLVM IR are equivalent; this is not surprising, given that I used the C program to guide my writing and translation of the LLVM IR.

Building A Compiling Calculator

We are now ready to implement a "compiler" for something approaching a useful language, namely, greatly reduced subset of Forth. Forth is a stack-based language created in the 1970s and still in use today, especially for small embedded systems.

LLVM will handle the parts commonly known as "compiler backend" tasks, and Babashka will provide our "frontend," namely breaking the text into tokens and parsing them. This task is made easy for us, because Forth is syntactically quite simple, and Babashka relatively powerful. Here are the language rules we will adopt:

Program tokens are separated by whitespace.
Non-numeric tokens are math operators.
Only integer operands are allowed.
Comments begin with \\.

Forth expressions typically place the arguments first, and the operator last (so-called "reverse-Polish notation"). Here is an example program which does some math and prints the result:

(def example "

2 2 +  \\ 4
5 *    \\ multiply by five to get 20
2 /    \\ divide by 2 -> 10
-1 +   \\ add -1 -> 9
8 -    \\ subtract 8 -> 1
.      \\ prints '1'

")

The code in forth.bb handles the parser, whose goal is to consume the raw program text and generate an abstract syntax tree (in our case, just a list) of operations to translate into IR:

(defn strip-comments
  "
  Remove parts of lines beginning with backslash
  "
  [s]
  (str/replace s #"(?sm)^(.*?)\\.*?$" "$1"))

(defn tokenize
  "
  Split `s` on any kind of whitespace
  "
  [s]
  (remove empty? (str/split s #"\s+")))

(defrecord node
    [typ val] ;; A node has a type and a value
  Object
  (toString [this]
    (format "[%s %s]" (:typ this) (:val this))))

;; Allowed operations
(def opmap {"+" :add
            "-" :sub
            "/" :div
            "*" :mul
            "." :dot
            "drop" :drop})

(defn ast
  "
  Convert a list of tokens into an \"abstract syntax tree\",
  which in our Forth is just a list of type/value pairs.
  "
  [tokens]
  (for [t tokens
        :let [op (get opmap t)]]
    (cond
      ;; Integers (possibly negative)
      (re-matches #"^\-?\d+$" t)
      (node. :num (Integer. t))

      ;; Operations
      op (node. :op op)

      :else (node. :invalid :invalid))))

Running this on our example,

(->> example
     strip-comments
     tokenize
     ast
     (map str))

;;=>
("[:num 2]"
 "[:num 2]"
 "[:op :add]"
 "[:num 5]"
 "[:op :mul]"
 "[:num 2]"
 "[:op :div]"
 "[:num -1]"
 "[:op :add]"
 "[:num 8]"
 "[:op :sub]"
 "[:op :dot]")

The remainder of forth.bb essentially just implements the needed operators, as well as the required stack and the reference to the printf C library function. It is perhaps a bit lengthy to go through here in its entirety, so I will share one example where Babashka helps:

(defn def-arithmetic-op [nam op-fn]
  (def-fn :void nam []
    (assign :sp (call :i32 :get_stack_cnt))
    (if-lt :i32 :sp 2
           (els)  ;; NOP - not enough on stack
           (els
            (assign :value2
                    (call :i32 :pop))
            (assign :value1
                    (call :i32 :pop))
            (assign :result (op-fn :i32 :value1 :value2))
            (call :void :push [:i32 :result])))
    (ret :void)))

This makes LLVM IR which does the following, in pseudo-code:

get current stack position; ensure at least two entries (else return)
pop the operands off the stack
apply the arithmetic operator to the operands
put the result on the stack

Implementing the four arithmetic operators is then as simple as invoking

;; ...
(def-arithmetic-op :mul mul)
(def-arithmetic-op :add add)
(def-arithmetic-op :sub sub)
(def-arithmetic-op :div div)
;; ...

when generating the IR.

Aside from the general-purpose LLVM IR code (llir.bb), the Forth implementation is under two hundred lines. It includes the invocation to clang to compile the temporary IR file to make the runnable program. Here's an example compilation session:

$ cat example.fs

       \\ initial state          stack: []
3      \\ put 3 on stack.        stack: [3]
99     \\ put 99 on stack.       stack: [3, 99]
drop   \\ discard top item.      stack: [3]
drop   \\ discard top item.      stack: []
2 2    \\ put 2 on stack, twice: stack: [2, 2]
+      \\ 2 + 2 = 4.             stack: [4]
5 *    \\ multiply 4 * 5.        stack: [20]
2 /    \\ divide by 2.           stack: [10]
-1 +   \\ add -1                 stack: [9]
8 -    \\ subtract 8 -> 1        stack: [1]
.      \\ prints '1'             stack: [1]
drop   \\ removes 1.             stack: []

$ ./forth.bb example.fs
$ ./example
1

The resulting program is fast, as expected…

$ time ./example
1

real	0m0.007s
user	0m0.002s
sys	0m0.003s

… and small:

$ ls -l ./example
-rwxr-xr-x  1 jacobsen  staff  8952 Aug 16 09:52 ./example

Let's review what we've done: we have implemented a small subset of Forth, writing a compiler front-end in Babashka/Clojure to translate source programs into LLVM IR, and using clang to turn the IR into compact binaries. The resulting programs are small and fast.

Sensible next steps would be to implement more of Forth's stack operators, and maybe start to implement the : (colon) operator, Forth's mechanism for defining new symbols and functions.

Lisp

Instead, let's implement a different variant of our arithmetic calculating language, using Lisp syntax (S-expressions). Consider our first Forth example:

2 2 +
5 *
2 /
-1 +
8 -
.

In Lisp, this looks like:

$ cat example.lisp
(print (- (+ -1
             (/ (* 5
                   (+ 2 2))
                2))
          8))

Rather than coming last, as they did before ("postfix" notation), our operators come first ("prefix" notation). The order of operations is determined by parentheses, as opposed to using stack as we did for our Forth implementation.

Here Babashka helps us tremendously because such parenthesized prefix expressions are valid EDN data, which the Clojure function clojure.edn/read-string can parse for us. But we need to convert the resulting nested list into the "SSA" (single static assignment) expressions LLVM understands. This is relatively straightforward with a recursive function which expands leaves of the tree and stores the results as intermediate values:

(defn to-ssa [expr bindings]
  (if (not (coll? expr))
    expr
    (let [[op & args] expr
          result (gensym "r")
          args (doall
                (for [arg args]
                  (if-not (coll? arg)
                    arg
                    (to-ssa arg bindings))))]
      (swap! bindings conj (concat [result op] args))
      result)))

(defn convert-to-ssa [expr]
  (let [bindings (atom [])]
    (to-ssa expr bindings)
    @bindings))

We use gensym here to get a unique variable name for each assignment, and doall to force the evaluation of the lazy for expansion of the argument terms. The result:

(->> "example.lisp"
     slurp
     (edn/read-string)
     convert-to-ssa)
;;=>
[(r623 + 2 2)
 (r622 * 5 r623)
 (r621 / r622 2)
 (r620 + -1 r621)
 (r619 - r620 8)
 (r618 print r619)]

The next step will be to actually write out the corresponding LLVM IR. The rest of lisp.bb is satisfyingly compact. Operators (we have five, but more can easily be added), are just a map of symbols to tiny bits of LLVM code:

(def ops
  {'* #(mul :i32 %1 %2)
   '+ #(add :i32 %1 %2)
   '/ #(div :i32 %1 %2)
   '- #(sub :i32 %1 %2)
   'print
   #(call "i32 (i8*, ...)"
          :printf
          [:i8* :as_ptr]
          [:i32 (sigil %1)])})

Similar to our Forth implementation, but even more compact, the main Babashka function, after a brief setup for printf, generates a series of SSA instructions.

(defn main [[path]]
  (when path
    (let [assignments (->> path
                           slurp
                           edn/read-string
                           convert-to-ssa)
          outfile (->> path
                       fs/file-name
                       fs/split-ext
                       first)
          ir (module
              (external-fn :i32 :printf :i8*, :...)
              (def-global-const-str :fmt_str "%d\n")
              (def-fn :i32 :main []
                (assign :as_ptr
                        (gep (fixedarray 4 :i8)
                             (star (fixedarray 4 :i8))
                             (sigil :fmt_str)
                             [:i64 0]
                             [:i64 0]))

                ;; Interpolate SSA instructions / operator invocations:
                (apply els
                       (for [[reg op & args] assignments
                             :let [op-fn (ops op)]]
                         (if-not op-fn
                           (throw (ex-info "bad operator" {:op op}))
                           (assign reg (apply op-fn args)))))
                (ret :i32 0)))]
      (compile-to outfile ir))))

(main *command-line-args*)

Putting these parts together (see lisp.bb on GitHub), we have:

$ ./lisp.bb example.lisp
$ ./example
1

It, too, is small and fast:

$ time ./example
1

real	0m0.006s
user	0m0.001s
sys	0m0.003s
$ ls -al ./example
-rwxr-xr-x  1 jacobsen  staff  33432 Aug 16 20:52 ./example

To say this is a "working Lisp compiler" at this point would be grandiose (we still need functions, lists and other collection types, eval, macros, …) but we have developed an excellent foundation to build upon.

To summarize, the strategy we have taken is as follows:

Use a high level language (in our case, Babashka/Clojure) to parse input and translate into LLVM IR;
When needed, write and generate small C programs to understand the equivalent IR to generate.
Compile the IR to small, fast binaries using clang.

Alternatives and Future Directions

I should note that C itself has long been used as an intermediate language, and we could have used it here instead of LLVM IR; I don't have a strong sense of the tradeoffs involved yet, but wanted to take the opportunity to learn more about LLVM for this project.

LLVM is interesting to me because of the modularity of its toolchain; it also provides a JIT compiler which allows one to build and execute code at run-time. We didn't investigate tooling for that here (it needs deeper LLVM language bindings than the homegrown Babashka code I used), but it could provide a way to do run-time compilation similar to what SBCL (a Common Lisp implementation which can compile functions at run-time) does.

Here are some directions I'm considering going forward:

Try interfacing with external libraries, e.g. a bignum library;
Implement more Forth functionality;
Implement more Lisp, possibly including a significant subset of l1;
Try a JIT-based approach, possibly using Rust as the host language.

Conclusion

Whenever possible, I want to make small, fast programs, and I like playing with and creating small programming languages. LLVM provides a fascinating set of tools and techniques for doing so, and using Babashka to make small front-ends for IR generation turns out to be surprisingly effective, at least for simple languages.

Wrangling Half a Thousand Dreams

Sat, 27 Apr 2024 00:00:00 +0000

On Monday, February 4th, 1985, at the tender age of eighteen, I wrote down in the back of my journal two meagre fragments of the previous night's dreams:

I’m at Fred’s and the cab is 45 minutes late.

Leaping over an empty oil drum on a highway.

This is the first of several hundred dream entries I've accumulated over nearly forty years. Though I'm not entirely sure what provided the impulse at the time to start writing them down, I've always been fascinated by dreams, and can remember a few childhood dreams as clearly as anything else that happened when I was growing up.

Until recently, I hadn't given much thought to the practice of dream journaling, or even re-read many of the entries, which have been scattered throughout a dozen or so handwritten notebooks and electronic files. This year, however, my interest in dream journaling peaked, which spurred me to start collecting all the entries into a single e-book that I could read at leisure on the various devices in my life.

The collating and formatting is now largely complete, and, while the combined journal is a bit too personal to post publicly in its entirety, I would like at least to share a few insights from the project.

(In-)consistency

As of this morning, there are 470 dated entries. Each entry, typically written first thing in the morning, captures at least one dream from the previous night (unless you wake up in between, it's hard to know when one dream ends, and another begins). Though the numbers average to about one entry per month during my adult life so far, my actual rate of dream journaling has been highly variable, and has tended to track my rate of daytime journaling. Here are the number of dreams per year, as of today:

1985:   68 ....................................................................
1986:   59 ...........................................................
1987:   10 ..........
1988:    7 .......
1989:    9 .........
1990:   37 .....................................
1991:    7 .......
1992:   23 .......................
1993:    4 ....
1994:    1 .
1996:    2 ..
1998:    2 ..
2000:   10 ..........
2001:    2 ..
2002:   34 ..................................
2003:    3 ...
2004:    3 ...
2005:    4 ....
2006:    1 .
2011:    4 ....
2017:    2 ..
2019:   68 ....................................................................
2021:    2 ..
2022:   22 ......................
2023:   29 .............................
2024:   57 .........................................................
TOTAL: 470

I will say more about how the statistics were generated in a moment. Each dot indicates a dated entry of one or more dreams from the previous night. Some years I averaged an entry or two a week; other times I went for years with no entries. That the data would be this variable piqued my curiosity.

Naïvely, I might expect to have written more during times of crisis or big change in my life, and that is approximately true. For example, there are peaks during my first year in college (1985-1986), the year I lived abroad in Switzerland (1990), and the year I had various personal crises and moved away from the Bay Area (2002). 2019 was a year of some tumult as well. But my interest in dreaming has waxed and waned over time, and interest, while hard to track, certainly drives the rate as well.

It is also well-known that remembering and writing down dreams one morning makes it easier to do so the next morning, so I'd expect the journals to clump somewhat in time. However, given the sheer number of entries, it surprises me a little to have gone literally years with no entries at all. There are very few entries in the interval from 2003 through 2018, and a few multi-year gaps. For any "missed" year, if you assume 3-6 dreams per night (a very rough, possibly bogus average I found online), over a thousand dreams completely slipped through the net. (By the same measure, my total average dream recall since reaching adulthood has been less than one percent.)

The handwritten journals have some nice features compared to digital files. A few of them have small drawings or thumbnail sketches of a particular moment or image from a dream. They also show the variability in my handwriting over the years (which has never been great), and their proximity to "normal" diary entries help show what else was going on at the time. (I typically would write the diary entries forward from the front page of the journal, and dream journal entries back-to-front from the last page, starting a new journal book when the two met in the middle.)

Obviously, I'll continue to hold onto the physical journals. But the combined digital journal has two major advantages: it is much easier to read, and all the dreams are in one place. These two together make reading the journal as a whole much easier. This, in turn, made repeated images and common themes more apparent.

Dream Content

It is a hallmark of dreams that they seem to defy characterization, lurching from image to image in surprising ways, but I find some dream topics cropping up consistently over the years, whereas others seem to have come and gone. Common topics over the years have included:

Moving through big, open landscapes, especially at night;
Making art;
Fighting, with people or animals;
Family and friends;
Teachers, mostly physics and art teachers I’ve studied with;
Physics, and technology in general;
Nuclear war/power/devices;
Storms;
Sex;
Trains;
Travel, especially Paris, Greece, New Zealand, and the South Pole.

Some of these (sex, family) are probably common for most people; others (nuclear devices, trains, Antarctica) maybe a little less so.

I also tried to spot changes over time. Common topics from earlier years, that are now less common, included: flying or falling, flooding or shallow water, war / soldiers / cops / authority figures, and being in school. More common topics these days include being in art workshops; traveling through disused, dark interior spaces; and programming or other computer work.

I have always resisted thinking about dreams as something demanding interpretation, but the fact is they do come from somewhere, and are probably informed by the events in our lives. Reading through all the repeated variations, I get the sense of my mind working through things, though in exactly what way is obviously a mystery.

My point in thinking about evolution of dream content is not so much to drive specific insights about my life, but rather to take a step back and experience a broader sense of dreaming as a part of life… to feel what it is like to be "one who dreams."

Making the Ebook

I won't weigh this post down with too much technical detail, but present a few details here in case anyone finds it helpful; others can obviously skip this section.

When I started organizing things this year, the largest dream journal I had was a single Emacs Org Mode file dreams.org. I used this as the nucleus of the project, copying or typing all the content from the other sources, sorted by increasing date. Here are the first few lines:

#+TITLE: Dreams
#+DATE: <2017-08-29 Tue>
#+OPTIONS: toc:nil num:nil

* 1985

** <1985-02-04 Mon>

I'm at Fred's and the cab is 45 minutes late.

Leaping over an empty oil drum on a highway.

** <1985-02-05 Tue>

[...]

The plain text format is readable on its own, and editable in any editor, though obviously Emacs has some advantages for this format. Without any extra work, having the content in Org Mode immediately allowed me to view the entire document as plain text, as HTML in the browser, or as a PDF, thanks to Org Mode's export functionality.

During the long process of typing older dreams into this document (doing only a few per day to keep the project from taking over my life), I started looking for ways to turn the dream document into something I could view on my phone, Kindle, iPad, Kobo reader, etc. I found I preferred reading old dream content on an e-ink display such as the Kindle or Kobo rather than on a backlit screen, so an e-book format such as EPUB seemed a natural next step.

I described my initial approach in an earlier blog post. The basic idea was to tweak the Org Mode formatting slightly to make it equivalent to Markdown and then use Calibre's ebook-convert program to turn that output into an EPUB file. However, that program ran more slowly than I liked, and I found myself wishing I could customize the output (especially the table of contents). I also was curious what exactly went into all those e-books I've been reading over the years – what, exactly, did they look like, under the hood?

So, I decided to skip the middleman and write my own Org Mode to EPUB converter. The result of that work is up on GitHub. While some of the details are out of scope for this article, I will say two things about the approach I took. First, I was very happy with my choice of Babashka for the implementation. Babashka is fast and expressive, and I did not need any external libraries, relying only on the very standard zip program as the only external dependency. My gratitude to Michiel Borkent for Babashka continues undiminished.

(Ironically, while looking up Babashka tricks in Daniel Higgenbotham's excellent Babashka Babooka, I discovered he also uses a dream journal as his example Babashka project.)

The other approach worth mentioning is the strategy I used to produce a working EPUB. This turned out to be more difficult than I thought, as EPUB is a rich and complex format with a number of historical variants. I spent a good amount of time with Google, ChatGPT, and Wikipedia to try to get a simple example to load in both Calibre and Apple Books; when this did not succeed, I switched gears and picked a random e-book from Standard Ebooks out of my collection, gradually stripping out all the actual text content and extraneous data out (zipping, importing and visually checking the files after each iteration) until I had the barest minimum set of files, with two "Lorem ipsum…" style chapters and a table of contents, which worked in both Calibre and Books.

Then I automated the production of that minimum example using Babashka, and posted the results. This provided something I could build on to make my final dream e-book implementation, which has the following features:

Cover art;
Introduction and Collophon sections;
Organization by month and year, with table of contents entries for each;
Correct conversion of the following formatting elements from Org Mode into the EPUB HTML:
1. Italic, boldface, ~~strikethrough~~, underline, code
2. Sections and subsections (used to organize the dates as well as subsections for each day, as needed);
3. Correctly-formatted beginning and ending quotes and apostrophes;
4. French guillemets (a few of the dreams are written down in French or have French dialogue).

The script uses an intermediate Markdown representation for the content before converting to HTML, since the transformations are simple and I have the idea I might eventually support Markdown in addition to, or instead of, Org Mode. The formatting requirements are minimal enough to be easily supported by a sequence of regular expressions to convert from one format to another.

The script's output is the EPUB file itself, along with the statistics display I posted above.

Common Words

A special mode of the program (invoked with the -w command-line option) prints out the 300 most common words in the dreams:

above across actually again against almost along am a... another
apartment are area around arrive art ask asks away beautiful bed been
before behind being below between big bike bit black blue boat body
both building buildings bunch can't car cars city class close clothes
comes coming corner country couple dad dark d... different does
doesn't dog doing don't door down drawing dream drive driving each
e... either else end everyone family feel feeling few fight fighting
figure finally find fish floor flying food f... french friend friends
front game gets getting girl glass goes going gone gray great green
ground group guy guys had hair hand hanging hard has having he's head
hear help here high hill home house huge i'll i'm i've inside is it's
j... j... keep kind lake large last later least leave left light
little long looking looks lot lots love made madison makes making man
many maybe meet men mom more mostly move moving much myself name near
need next night off office old older open others outside painting
paper paris part party past path perhaps person p... p... phone
physics piece place plane playing point possibly probably pull put
ready real realize really red remember restaurant ride right road room
rooms run running same says school seems series several she's short
should show showing side similar sit sitting slightly small somehow
someone something somewhat somewhere sort space spot stairs standing
start starts still stop strange street stuff suddenly sure swimming
table taken takes taking talk talking tell there's thing things
thinking though three through together too top towards town train
trees truck try trying turn under very wait wake walk walking wall
walls wants was water we're where while white whole why window

(In this list, the most common 100 English words have been subtracted out to help distinguish the content from any other text.)

The words that stand out for me, aside from first names (which I elide here using ...), are nouns which indicate settings, actions or activities, and commonly occurring objects from the dreams. I suppose this list, or something like it, could be considered a sort of "dreamer's fingerprint." It would be interesting to compare with someone else's output (another English speaker, perhaps). If you're a programmer and have a dream journal of your own, here is a link to the relevant code that you could adapt to generate the list. (Or, of course, you could convert your journal to Org Mode and use the software I wrote "out of the box.")

Parting Thoughts

Digging into this material, both the combined journal itself, and automating the text conversion, has been engaging enough to consume most of my limited free time over the past month or so. I was surprised at how fun it was to read through the collected dreams, though I suspect the material would be much less entertaining for anyone who didn't know me well. The writing itself tends to be more compact and faster moving than my other writing. In this regard it sets an example I want to emulate in the rest of my writing.

The work also provoked contemplations about memory, a topic I tend to think more about as I get older. One mysterious aspect of dreams is that, for the most part, we forget them in the morning. They disappear completely unless we make a record of them, giving them that kick of energy they need to turn from virtual particles into real ones. In re-reading the dream entries, I was struck by the fact that, not only did I have no memory of most of the dreams I'd collected, but for the most part I had no memory at all of writing them down, either. It was strange to confront a whole tranche of my life of which I had basically no memory, having only the "evidence at hand" to work with. In this regard, I felt at times a bit like a scientist studying someone else's dreams.

The collection of dreams is packed with myriad images, whether strange, disturbing, or funny, or all three… each of them somehow connected with my life. As an artist who sometimes struggles to figure out what to draw or paint, having a trove of material to work with feels exciting and a little daunting. Regardless of what any of it means, the material is more "mine" than anything inspired by other artists, or films, books, etc. It's up to me to see what, if anything, I can actually do with it.

Programming Languages and Art Media

Sun, 24 Mar 2024 00:00:00 +0000

What computer programming languages "go with" (feel like) what art mediums? Here is my first crack at the question.

Language	Medium
Clojure	graphite
Babashka	charcoal
Common Lisp	dip pens with india ink
Emacs Lisp	ball point
C	engraving
C++	etching
Perl	watercolor
Go	gouache
Python	oil painting
Java	lithography
Sh	woodcut
Bash	linoleum cut
Rust	Procreate on the iPad
_	_
(Copilot)	collage

I've tried all of these except for the engraving, which I'd like to learn at some point.

eBook of Dreams

Tue, 13 Feb 2024 00:00:00 +0000

Since at least the 1980s I've kept a (very intermittent) dream journal in various forms, including

dedicated paper notebooks
notes or drawings in sketchbooks
fragments of entries in daily Org Mode diary pages
individual iCloud Notes, one per dream
a single, dedicated Org Mode file.

Lately I thought it might be interesting to review these and possibly use them as ideas for artworks, and I've found last format to be most amenable to later review (e.g., by exporting to a Web page or a nicely-formatted PDF).

As an avid e-book reader, I wondered yesterday, How hard might it be to turn such a dream journal into an e-book, for reading on one of my e-paper devices? (I bounce back and forth between a Kindle and a Kobo reader).

It turns out this is very easy and maps quite naturally into my existing "book workflow," which is, very briefly:

acquire (legally) one or more e-books
import into Calibre
plug in the e-reader device (I always keep WiFi off on these devices to avoid the vendor pushing up unwanted updates)
use Calibre to send the book to the device.

Now, my dream journal is in this format:

#+TITLE: Dreams
#+DATE: <2017-08-29 Tue>
#+OPTIONS: toc:nil num:nil

* <2017-08-29 Tue>

It was after hours in the lower, windowless floors of some science
building [....]

I wanted to:

Strip off the front matter;
Convert the dates to chapter headers;
Add a cover image, taken more or less at random from my library of roughly 10,000 artistic reference photos and images.

For the cover, it was enough to simply pick one, crop it to roughly the dimensions of my e-reader, and put it in the same directory as my dream journal file.

With these requirements in mind, to save time on the regex wrangling, I had a short conversation with ChatGPT, and an even shorter session with my code editor, resulting in this small bash script:

#!/bin/bash

# Define paths
DREAMFILE="/Users/myuser/org/dreams.org"
EPUB_OUTPUT="$HOME/Desktop/dreams.epub"
COVER_IMAGE="/Users/myuser/org/dreams-cover.png"

# Remove leading front matter and convert Markdown to temporary file
sed -E -e '1,/^$/d' -e 's/^\* <([0-9]{4}-[0-9]{2}-[0-9]{2} [A-Za-z]{3})>$/# \1/' "$DREAMFILE" > /tmp/dreams_tmp.md

# Import temporary Markdown file into Calibre
ebook-convert /tmp/dreams_tmp.md "$EPUB_OUTPUT" --output-profile=tablet --cover="$COVER_IMAGE"

# Clean up temporary file
rm /tmp/dreams_tmp.md

I'm very happy with the results. All that remains is to consolidate some of the older dreams into the single Org Mode file I mentioned; then I can update my eBook of Dreams periodically… and peer back into the dream world at will.

Making A Tiny E-Paper Status Display for the Raspberry Pi Zero

Thu, 25 Jan 2024 00:00:00 +0000

I've gotten interested in Raspberry Pis again, having recently picked up a Pi Zero 2 W. Something about these cheap little boards that are fully-functional Linux computers, "as powerful" as the VAXes I originally learned Unix on, appeals to the young hacker in me who fell in love with computers in the 1980s and is still able to be amazed by their potential. I've also been trying to get more comfortable doing hardware projects, tinkering with an Arduino kit and brushing up on the electronics I learned in college.

For my first "finished" project I wanted to work with an e-paper (also known as e-ink) display. These displays, used in most eBook readers such as the Kindle, Nook, Kobo, etc., are great for low-power applications, where information updates relatively infrequently, where an external light source is available, and where monochrome output is suitable. Personally, I prefer no backlight and monochrome output, frequently setting my iPhone screen to greyscale as a way of reducing its stranglehold on my attention.

I like the idea of building devices which truly fade into the background rather than being sources of interruption. Devices that don't tell you things directly, but rather invite you to look their way when you care to. I thought an enjoyable first project might be a little computer that just quietly tells you about itself and doesn't do much else. Tangentially, this relates to a movement known as "calm technology" which advocates for presenting information less invasively, often in subliminal ways ("under the threshold of consciousness").

This post records what I needed to do to build such a small e-paper-based device. Since there were a lot of details I'm likely to forget down the road, I used Literate DevOps in Org Mode from this document to be able to run and record (and re-run) most of the steps while editing the document itself (see the linked blog post for an explanation).

Supply List

I used the following:

Vilros Raspberry Pi Zero 2 W Basic Starter Kit
2.13 inch e-paper HAT from Waveshare
128 GB MicroSD card (way overkill)
Solder and soldering iron

I highly recommmend the Vilros kit - it was beautifully packaged and had everything I needed. I did have to solder the 40-pin GPIO connector onto the Pi Zero.

Pi Zero and Waveshare hat (underside)

Total cost was about $80, not counting the new soldering iron I had to purchase after my old Radio Shack soldering iron shorted out, though I could have saved money by buying a smaller SD card.

The Waveshare product is reasonably well-documented, though they recommend some steps I think are best avoided or changed. The software setup I used is shown next. (The rest of this post is somewhat technical – skip to the end if you just want to see the pictures.)

Dependencies

In honor of particle astrophysics, I gave the computer the name pion. It runs Raspbian, which is a Debian variant.

Virtualenv-based setup

I prefer not to install Python dependencies into system directories, because it's better to isolate sets of dependencies across different applications; the venv module creates virtual environments ("virtualenvs") which do this. Also, Waveshare's guide recommends numpy, which is a heavyweight dependency I prefer to avoid unless I need it, and as far as I can tell, we do not. My dependency installation steps were as follows (here and below, the command that was issued has a grey background, and the output has a tan-colored background):

# On pion:
sudo apt-get update -qq
sudo apt-get install -yqq python3-pip
python -m venv venv
source venv/bin/activate
pip install -U -q setuptools
pip install -q spidev
pip install -q gpiozero
pip install -q pillow

Selecting previously unselected package python3-pip.
(Reading database ... [...progress info elided...]
Preparing to unpack .../python3-pip_23.0.1+dfsg-1+rpt1_all.deb ...
Unpacking python3-pip (23.0.1+dfsg-1+rpt1) ...
Setting up python3-pip (23.0.1+dfsg-1+rpt1) ...
Processing triggers for man-db (2.11.2-2) ...

Running the Demo

Once the hardware was attached and the dependencies loaded, I had some hope of interacting with the device. The next step was to check out the Waveshare e-paper repository from GitHub…

git clone -q https://github.com/waveshare/e-Paper.git

… and then to try to run their demo script:

cd e-Paper/RaspberryPi_JetsonNano/python/examples
source ~/venv/bin/activate && python epd_2in13_V3_test.py

/home/rpi/venv/lib/python3.11/site-packages/gpiozero/devices.py:295: PinFactoryFallback: Falling back from lgpio: No module named 'lgpio'
  warnings.warn(
/home/rpi/venv/lib/python3.11/site-packages/gpiozero/devices.py:295: PinFactoryFallback: Falling back from rpigpio: No module named 'RPi'
  warnings.warn(
/home/rpi/venv/lib/python3.11/site-packages/gpiozero/devices.py:295: PinFactoryFallback: Falling back from pigpio: No module named 'pigpio'
  warnings.warn(
/home/rpi/venv/lib/python3.11/site-packages/gpiozero/devices.py:292: NativePinFactoryFallback: Falling back to the experimental pin factory NativeFactory because no other pin factory could be loaded. For best results, install RPi.GPIO or pigpio. See https://gpiozero.readthedocs.io/en/stable/api_pins.html for more information.
  warnings.warn(NativePinFactoryFallback(native_fallback_message))
INFO:root:epd2in13_V3 Demo
INFO:root:init and Clear
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
INFO:root:1.Drawing on the image...
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy
[... lots more debugging info here...]
DEBUG:waveshare_epd.epd2in13_V3:e-Paper busy release
INFO:root:Goto Sleep...
DEBUG:waveshare_epd.epdconfig:spi end
DEBUG:waveshare_epd.epdconfig:close 5V, Module enters 0 power consumption ...

When I did this I saw their series of changes on the small screen attached to the RPi Zero. Exciting! (The warnings in the output seem to have to do w/ how the Python code wants to "speak GPIO" and does not seem to negatively effect the outcome.)

Rolling Our Own

I wanted to make my own display, and simply load the Waveshare driver code as a Python package. Unfortunately, I was unable to get Waveshare's (undocumented) setup.py working with my virtualenv; the quick-and-dirty workaround was to update sys.path to find their library, but use the virtualenv for the other dependencies installed earlier.

As a proof of concept, I wrote a script to write the device IP address to the screen:

# On pion:
cat <<EOF > /tmp/ip-demo.py
from PIL import Image, ImageDraw, ImageFont
import os
import socket
import sys
import warnings

SRCDIR = "/home/rpi/e-Paper/RaspberryPi_JetsonNano/python"

sys.path.append(os.path.join(SRCDIR, "lib"))

# Suppress GPIO library warning:
with warnings.catch_warnings(action="ignore"):
    from waveshare_epd import epd2in13_V4

def get_ip_address():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.connect(('8.8.8.8', 80))
    local_ip_address = s.getsockname()[0]
    s.close()
    return local_ip_address

picdir = os.path.join(SRCDIR, 'pic')
font16 = ImageFont.truetype(os.path.join(picdir, 'Font.ttc'), 20)
font24 = ImageFont.truetype(os.path.join(picdir, 'Font.ttc'), 24)
epd = epd2in13_V4.EPD()
epd.init()
image = Image.new('1', (epd.height, epd.width), 127)
draw = ImageDraw.Draw(image)
draw.text((10, 68), 'pion ip address', font = font16, fill = 0)
draw.text((10, 90), f'{get_ip_address()}', font = font24, fill = 0)
epd.display(epd.getbuffer(image))
epd.sleep()
print("OK")
EOF
source venv/bin/activate && python /tmp/ip-demo.py

OK

And here's what it looked like on the Pi:

Prototyping a Better Display

At this point I felt ready to make a more informative display panel. For this project I chose to display various information easily accessible to the Pi:

hostname
On-board temperature
IP address
WiFi signal strength
Uptime, including CPU busy fraction
"Disk" (storage) size and usage fraction

This meant reading online and running lots of little experiments to try things out. The RPi Zero W 2 is powerful enough to edit and run code on, but I'd rather use my editors on my laptop. I did need to explore the commands on the Pi needed to read out the quantities I was interested in… here are some of the details. (This section relies heavily on the the Literate Devops setup mentioned above; commands were all launched on the RPi from this document, with the results inserted directly inline.)

Temperature

On Raspbian, vcgencmd will tell you tons of things about your Pi. Here we use it to read out the CPU temperature:

# On pion:
/usr/bin/vcgencmd measure_temp

temp=38.1'C

Memory

free is fairly common across Linux variants:

# On pion:
free

               total        used        free      shared  buff/cache   available
Mem:          436980      109240      213468         948      166464      327740
Swap:         102396           0      102396

It uses /proc/meminfo under the hood:

# On pion:
cat /proc/meminfo

MemTotal:         436980 kB
MemFree:          111828 kB
MemAvailable:     332116 kB
Buffers:           31068 kB
Cached:           220648 kB
SwapCached:         1184 kB
Active:           103120 kB
Inactive:         159140 kB
Active(anon):      10676 kB
Inactive(anon):       32 kB
Active(file):      92444 kB
Inactive(file):   159108 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        102396 kB
SwapFree:          91900 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               800 kB
Writeback:             0 kB
AnonPages:         10328 kB
Mapped:            29968 kB
Shmem:               164 kB
KReclaimable:      27584 kB
Slab:              43208 kB
SReclaimable:      27584 kB
SUnreclaim:        15624 kB
KernelStack:        1040 kB
PageTables:         1088 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      320884 kB
Committed_AS:     168456 kB
VmallocTotal:    1622016 kB
VmallocUsed:        7056 kB
VmallocChunk:          0 kB
Percpu:              416 kB
CmaTotal:         262144 kB
CmaFree:           86804 kB

Lots of interesting things here to try and understand someday… maybe.

WiFi

Searching around on online turned up two ways of getting WiFi signal strength:

# On pion:
 iwconfig wlan0

wlan0     IEEE 802.11  ESSID:"CornellCroft"
          Mode:Managed  Frequency:2.437 GHz  Access Point: F4:92:BF:7F:55:E4
          Bit Rate=72.2 Mb/s   Tx-Power=31 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=64/70  Signal level=-46 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:1  Invalid misc:0   Missed beacon:0

# On pion:
cat /proc/net/wireless

Inter-| sta-|   Quality        |   Discarded packets               | Missed | WE
 face | tus | link level noise |  nwid  crypt   frag  retry   misc | beacon | 22
 wlan0: 0000   61.  -49.  -256        0      0      0      1      0        0

I used the former.

Disk Space

df is at least 37 years old and works on almost anything Unix-like:

# On pion:
df -k

Filesystem     1K-blocks    Used Available Use% Mounted on
udev               81736       0     81736   0% /dev
tmpfs              43700     932     42768   3% /run
/dev/mmcblk0p2 122364296 4306628 111824324   4% /
tmpfs             218488       0    218488   0% /dev/shm
tmpfs               5120       8      5112   1% /run/lock
/dev/mmcblk0p1    522232   95702    426530  19% /boot/firmware
tmpfs              43696       0     43696   0% /run/user/1000

I only wanted the "root" filesystem, since it takes up almost all the space.

# On pion:
df -k | egrep '/$'

/dev/mmcblk0p2 122364296 4306624 111824328   4% /

I probably didn't need to spring for 128 GB of storage on my MicroSD card.

CPU Load and Uptime

I found two methods:

# On pion:
uptime

 16:21:16 up  1:15,  2 users,  load average: 0.08, 0.02, 0.01

# On pion:
cat /proc/uptime

4544.69 18031.09

What is this proc file actually telling us? RedHat says,

The first value represents the total number of seconds the system has been up. The second value is the sum of how much time each core has spent idle, in seconds. Consequently, the second value may be greater than the overall system uptime on systems with multiple cores.

This implies that there are four cores on the board (something one can confirm by looking at /proc/cpuinfo); I actually didn't know there were four cores when I bought the unit.

Which suggests an interesting statistic, "Average CPU load since boot": (4544 * 4 - 18031) / 4544 * 4 = 0.8%. I added that to my dashboard.

What else is there?

There are always interesting things to be found in the proc filesystem, which provides an easily-accessible file-like interface to many aspects of the Linux kernel.

# On pion:
ls /proc

1     1485  272  4    48   72	asound	     kallsyms	    stat
10    1486  275  40   49   73	buddyinfo    key-users	    swaps
1012  1487  28	 41   490  813	bus	     keys	    sys
1070  15    29	 415  497  822	cgroups      kmsg	    sysrq-trigger
1071  16    293  416  5    918	cmdline      kpagecgroup    sysvipc
11    164   294  42   50   939	consoles     kpagecount     thread-self
1155  167   3	 421  503  94	cpu	     kpageflags     timer_list
1158  168   315  422  53   941	cpuinfo      latency_stats  tty
1159  17    317  423  54   942	crypto	     loadavg	    uptime
12    18    319  424  577  945	device-tree  locks	    version
1243  183   32	 426  587  947	devices      meminfo	    vmallocinfo
13    184   321  43   59   948	diskstats    misc	    vmstat
1327  19    322  430  590  95	driver	     modules	    zoneinfo
133   2     33	 433  6    951	execdomains  mounts
14    22    35	 434  60   952	fb	     net
140   228   36	 435  61   96	filesystems  pagetypeinfo
1411  23    374  436  62   967	fs	     partitions
1412  235   379  44   63   972	interrupts   schedstat
1418  24    38	 444  64   973	iomem	     self
1419  253   387  446  65   974	ioports      slabinfo
1484  27    39	 450  651  985	irq	     softirqs

Digging into these some other time might suggest more ideas for our display. (Incidentally, I remember from my time writing Linux device drivers that it is not that hard to add /proc entries to the kernel – making a new proc file can be a good first kernel project.)

Building the Mockup

With the above output snippets in hand I could go about building a mockup to play around with locally on my laptop, then adapt to the Pi by wiring in the actual shell commands in the appropriate places.

First, I created a new Python project for local use. The only extra library needed was pillow, for image manipulation functions of the kind used by the Waveshare demo.

I called my e-paper display paperproto:

# Locally, on my laptop:
cd /Users/jacobsen/Programming/Python
mkdir -p paperproto
cd paperproto
python -m venv venv
pip install pillow

The resulting code is up on GitHub. I won't go through the entire program, though it is based on the Python example I showed above. There are, perhaps, two interesting features. First, the elements in the display are organized in a simple table which makes it easy to play with the layout. Here is the top-level table:

fields = [
    [None, get_hostname(), font24, [0, 0]],
    [None, get_ip_address(), font14, [0, 40]],
    ["WiFi", get_wifi_strength(), font14, [120, 40]],
    [None, get_datetime(), font14, [0, 60]],
    ["Mem", get_mem(), font14, [120, 60]],
    ["Disk", get_disk(), font14, [0, 80]],
    [None, get_temp(), font14, [120, 80]],
    ["Up", get_uptime(), font14, [0, 100]],
]

The first field is an optional prefix for the information; the next, an invocation of the relevant function to give the info in question; followed by the font/size for the information. The last column consists of $x$, $y$ location. By playing with these fields, the design could be quickly changed.

Second, each of the functions shown has a "mockup" or local version, a fixed string which was copied from the exploratory output shown above; and a "real" or deployed version which executes on the Raspberry Pi when it's running "in production." Here is an example of one of the functions implemented in this dual manner:

def get_uptime():
    if is_pi:
        uptimestr = subprocess.check_output("cat /proc/uptime", shell=True).decode(
            "utf-8"
        )
    else:
        uptimestr = """
        4544.69 18031.09
        """
    match = re.search(r"(\d+\.\d+)\s+(\d+\.\d+)", uptimestr)
    if not match:
        return "NO UPTIME"
    total_seconds = float(match.group(1))
    idle_cores = float(match.group(2))
    up_days = float(total_seconds / 86400)
    active_percent = float((1 - idle_cores / (4 * total_seconds)) * 100)
    return f"{up_days:.2f}d, active {active_percent:.2f}%"

This separation of mockup or "virtual hardware," which can run anywhere, and the "real" or target hardware, is something I have had good success with in more serious embedded work. The result in this case is that I could iterate very quickly on my mockup, and produce something like the following, without taking the trouble to copy things over to the RPi:

(jjair is the hostname of my laptop, where the mockup is running.)

Deployment

Getting it to work on the Pi was as simple as copying the file over…

# Locally, on my laptop:
scp /Users/jacobsen/Programming/Python/paperproto/proto.py pion:

… and making a small wrapper script to activate the virtualenv and run the Python code:

# On pion:
cat <<EOF > proto
#!/bin/bash
cd /home/rpi
source venv/bin/activate
python proto.py
EOF
chmod +x proto

Running "In Production"

Finally we get to see the results!

# On pion:
./proto

None pion 0 0
None 192.168.0.69 0 40
WiFi 70/70 -29 dBm 120 40
None 2024-01-24 14:23 0 60
Mem 25% 120 60
Disk 4G/122G 4% 0 80
None 38.6 C 120 80
Up 2.97d, active 0.57% 0 100

Running It Periodically

We would like our display to regularly update and to survive reboots. One option would be to create a long-running program which loops forever, sleeping and then waking up to make its update. And you could make a systemd service out of that, etc. But cron does this sort of periodic thing wonderfully, and our wrapper script makes this easy:

# On pion:
cat <<EOF > /tmp/cron
*/10 * * * * /home/rpi/proto >> /home/rpi/proto.out 2>&1
EOF
crontab /tmp/cron && rm /tmp/cron

With the Pi on my desk and the cron job installed, I can see the screen refreshing every ten minutes. Job complete!

Future Directions

Here are some things I'd like to try:

Try a bigger display
Look into wall-mounted displays
Display remote info (weather, crime, financial, politics, …)
Add sensors, e.g. air quality or room temperature, and make a display for those
Make it battery powered! (But power consumption of Pis is somewhat high for that.)
Build a case for it.

One final thing I might play with is partial updates to the display. Waveshare supports a partial update mode where the entire screen need not be redrawn when a region is changed. Such updates are a little more visually unobtrusive – as it is, the whole display flashes a few times when it updates.

I've enjoyed having this little device on my desk this week. I can ssh into it, poke around at the command line, shut it down and unplug it, throw it in my backpack (wrapped in an ESD bag: no case yet!), show it to friends. When unplugged, the information on the display stays unchanged until you plug it in again, another nice e-paper feature. Otherwise it just sits there quietly, refreshing every ten minutes, drawing a small amount of power, and forming a tiny part of my physical and digital world.

In Praise of Small Shell Scripts

Thu, 18 Jan 2024 00:00:00 +0000

Over the decades I've relied heavily on bash aliases and functions to automate common tasks, resulting in a .bash_profile that's over a thousand lines long. But lately I find myself writing many small scripts instead.

These tiny scripts, which live in a directory on my $PATH, have several advantages:

They compose nicely, in line with the Unix philosophy.
They are reasonably portable. If it works for me on Linux, it'll often work on my Mac, and vice versa.
They start quickly and are often as fast as something you'd write in Python, Ruby, etc.
Their file creation / modification times are distinct and provide a history of what you worked on, when; by contrast, everything in .bash_profile is muddied together.
They run in subprocesses without needing to get your .bash_profile involved (helpful for Org Babel or any situation where you need to run your script from another program).

Even something as trivial as an alias might get its own shell script, if I can give it a name that's easier to remember than the original program name or combination of arguments. Bash is an awkward language for doing complex things, but expressive enough for simple tasks. And ChatGPT or other AI tools can often write simple scripts correctly for you with little or no modification.

Here are some examples of small scripts I've been using recently:

ghjobs : Show status of my active CI jobs on GitHub (wrapper around gh tool); avoids having to go to the Web site, usable inside watch, etc.
inprog / ondeck : Show GitHub issues tagged as "in progress" or "on deck," across all my "work" repositories;
(several programs) : Run one of many different build variants on the embedded system I'm working on (build for Mac, build for target hardware, build with fast tests, build all variants prior to pushing code, etc.);
(several programs) : Select random figurative references for inspiration or to draw from (there are many variations of this one);
bashscript : Create a new bash script in the appropriate place (on $PATH).

This last one is simple and helpful enough to show here:

#!/bin/bash

script_name=$1

cat << EOF > ~/bin/$script_name
#!/bin/bash

echo "OK"
EOF

chmod +x ~/bin/$script_name

Bash is boring and old (I still have csh scripts from more than 30 years ago; csh is a close cousin of bash). It's also everywhere – on my Mac, inside my Docker containers, on my Raspberry Pi's, on my client's servers, and on the cloud VMs I work on.

The "many small scripts" approach is complementary to how I use make, which I wrote about previously.

Finally, here's the small script I wrote yesterday to hopefully get me writing more often:

#!/bin/bash

BLOGHOME=${BLOGHOME:-$HOME/Docs/websites/johnj22}

cd $BLOGHOME
pwd

echo -n "Title for post: "; read title
postdate=$(date '+%Y-%m-%d')
echo $title $postdate

# Turn the title lower case/kebab case, remove extra characters.  Thanks, ChatGPT!
title_kebab=$(echo "$title" |
              tr '[:upper:]' '[:lower:]' |
              tr -s '[:space:]' '-' |
              sed 's/[^a-z0-9-]//g; s/^-//; s/-$//')

echo -n "$title" | pbcopy

file="content/posts/$title_kebab"".org"

if [ ! -e $file ]; then
    cat <<EOF > $file
---
title: $title
date: $postdate
draft: true
tags:
---
# Org Mode stuff in case I export the document on its own without using Hugo:
#+OPTIONS: toc:nil num:nil

EOF

fi

ae $file  # `ae` is another small script: loads the resulting file in Emacs.

Typing "newpost" asks me for a title and pops me in my editor with all the boilerplate needed to get me on my way. Nice and easy.

2024 Blogging Intentions

Wed, 17 Jan 2024 00:00:00 +0000

I can't believe I haven't written anything here in over a year.

I increasingly want to move my experimental, art, hobby, and professional content away from platforms that exist to extract value from me and my friends. I'd also like to be less precious with this space, to treat it in the casual, experimental fashion I would treat Instagram or Facebook. So I'm going to be playing around with posting shorter posts, unfinished things, and worrying less about writing complete articles (like I did at times while traveling to the Pole).

This is partly inspired by recent efforts along somewhat similar lines by my friend Stephen Starkey.

Emacs, Continued

A few years ago I started using the static site generator Hugo for this site. I think that's still going reasonably well.

I use Emacs less than I used to, using VSCode for most programming tasks recently (because Copilot). But there are three things I still use Emacs for habitually:

Writing prose

Plain text is king, completely platform agnostic. And though I'm not sure why, I find it's actually nice to use a different platform for writing than for writing large amounts of production code.

Git operations

Magit is such a huge time saver. I typically commit and push code several, or dozens, of times a day – with Magit, these operations are streamlined to the point where they are essentially frictionless.

"Literate Devops"

Very often I want to record what I'm doing on some system in a reproducible way without necessarily going through the process of coding it up in a full-fledged infrastructure-as-code system like Ansible, Terraform, etc. Often the main thing is to record my intentions (see "Writing prose," above) and record the command-line invocation required. Org Mode and Org Babel are great for this. I also rely on yasnippet to make this frictionless. Here, for example, is the snippet I use to create a Bash command:

#+BEGIN_SRC bash :exports both :results pp replace :eval never-export :prologue "exec 2>&1" :epilogue "true"
# Locally, on my laptop:
... actual shell command goes here ...
#+END_SRC

This snippet is bound to the abbreviation begsh: when I type that prefix plus TAB I get the above Org-Babel snippet inserted directly at point. Then, to actually execute the command, typing C-c C-c within the code block runs it and displays the results in the document I'm writing.

Here's a slightly more complex snippet for executing code remotely on my Raspberry Pi 4:

#+BEGIN_SRC bash :async :exports both :results pp replace :eval never-export :prologue "exec 2>&1" :epilogue "true" :dir /ssh:4pi:
... actual shell command goes here ...
#+END_SRC

In this case, C-c C-c actually runs the command remotely, on my Pi, and injects the results directly into the document, as before. The :async option allows me to continue editing elsewhere and wait for the results to be inserted when the command completes.

Future Work

It would be nice to make adding images easier. I still haven't solved the "post an image quickly or insert it into a document without friction" problem (one of the things Instagram accomplishes… along with a lot of extra stuff I wish they wouldn't accomplish). Ideally adding images would be a drag-and-drop kind of operation, and would handle any needed Bash, Emacs or Hugo operations automagically.

Summary

I'm trying to use a set of simple tools to make blogging frictionless and less precious… without indenturing myself to companies to whom I am the product.

Practices for Software Projects

Tue, 06 Sep 2022 00:00:00 +0000

What follows is a selection of practices which I find helpful when working with software projects, whether in public or in private, in solo or in collaborative work. Many of these practices are quite standard, but some are, at best, unevenly practiced by either commercial or academic software teams. Although they are especially important for collaborative work, I like to follow these practices on my own projects, both as a matter of professional habit, and because I find them beneficial. (This post is essentially an expanded version of a gist which I made a few months ago, and which I have been endeavoring to adhere to in my open source work ever since.)

Checklist

I’ll start with a checklist I use, then go into details for each item. I try to check these items for each repository/project I work on.

Are there unit tests?
- Are they maintained?
- Are they run often by developers…
- … and by builds that are triggered by every commit pushed to the group repository?
- Is test coverage sufficient, to the extent agreed by the team?
Is there a README for the project? Does it show:
- What the project is for?
- How to build it and run tests?
- An example snippet showing it in action?
- Links to further documentation, if any?
- A license / copyright information?
Are builds generally repeatable? Does firing the same build several times for a recent commit yield the same result each time?
Is an issue tracker being used to track bugs and planned work?
Is there any relevant documentation that can be auto-generated?
Is the software versioned?
- Does your application / library know what version it is?
- Is the process of updating the version and releasing the code automated?
Are the following steps fully automated and executable via Make?
- Running unit tests (make test … can be broken into fast and slow targets)
- Running code style checks and/or automated reformatting (make lint)
- Releasing the software, including building any needed artifacts (make release)
- Running all the tests in Docker, to ensure consistency with the automated build (make docker)

Automated Tests

Let’s start with the most important item. Writing and continuously running automated tests is always my first line of defense against defects. To quote Michael Feathers,

Code without tests is bad code. It doesn’t matter how well written it is; it doesn’t matter how pretty or object-oriented or well-encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them, we don’t really know if our code is getting better or worse.

In what follows, “production code” means the code that actually does what your program is trying to do.

There’s a lot to be said about how to write tests, about the difference between unit tests and integration tests, about test frameworks, mocks, fakes, stubs, and so on. These discussions shade into ones about architectural patterns, object orientation, organization of software into layers, etc. To my view, these are all secondary points. The main point is, is our software safe to change? Only the presence, or absence, of tests, and the extent to which our tests cover our production code determine the answer to this question.

Test-Driven Development

When starting to write tests, a common question is, What should I test? This question is usually easiest to answer before you’ve written the code you want to test. In that case, the the answer is:

Write the tests that force you to write code that implements the functionality you want.

In other words, don’t write production code that a failing test hasn’t forced you to write. This is the key to test-driven development (TDD).

Note that I wrote “a failing test,” above, rather than “failing tests”: the process is best done in tiny increments, as follows:

Add a small increment of test code, which makes the tests fail (“red”).
Add a small increment of production code, which makes the tests pass (“green”).
Refactor as needed to keep the design of the code clean and simple (repeat 3 in small steps, as needed, before going back to step 1).

Ideally, each step in the “red…green…refactor…” cycle is as small as possible. It should be the “red” step which actually pushes you towards the functionality you want – that failing test is a statement about where you want the production code to go.

If you find yourself writing multiple tests at once, or a more complex test than is strictly needed at the moment, then stop and try to break the problem into smaller steps.

If you only write production code in response to a failing unit test, then you know that code has both a reason to exist, and test coverage for that code. See Kent Beck’s Test-Driven Development By Example for a more detailed explanation of this process, or try it for yourself. I found when I adapted this pattern, about 14 years ago, my productivity, the reliability of my software, and my enjoyment of writing it, all increased noticeably.

Probing With Tests

On the other hand, maybe you’ve inherited some code from somebody else. Or maybe you’ve written a bunch of untested code as a prototype, and you want to consider it production code. In these situations, I like to poke at the code in question using tests. These exploratory tests can be a way to build understanding about how the code works, and to document your findings. While reading the code under study, I may find an area that looks like a defect: in that case, I write tests to try to show the defect (or prove its absence). Some of these more speculative tests may not live on permanently in my test suite, but they usually do.

Importantly, whenever a defect is found, I always try to write a test which detects that defect before I fix it, to guard against the defect ever recurring.

Tradeoffs

TDD affects development speed – for very exploratory work, such as building prototypes, it can be slower, though I find it usually helps velocity as projects grow in complexity. It definitely, in my experience, helps add reliability.

I like the analogy of rock climbing. If I’m scrambling over a short wall, I don’t need a rope. Summiting a tall mountain face definitely calls for serious protection. Bouldering close to the ground, not so much, but be careful that your “bouldering” code doesn’t sneak into your “summiting” code.

There are times when strict TDD is less critical, and times when I find it especially important. Obviously, for exploratory coding on a prototype, TDD can be overkill. Lispers armed with REPLs tend to be especially comfortable with exploratory coding, but it is important to shore up our REPL-driven snippets with tests, lest we lose the ability to make changes safely. (A favorite technique for this situation is to comment out bits of code and see if your existing tests fail, then TDD your way back to bringing the code back in.)

One place where I find TDD especially helpful is when the programming is more difficult than usual. With TDD, I handle the simple cases first, and gradually add sophistication to my production code as I add tests. As Robert Martin puts it: As the tests get more specific, the production code becomes more general.

If your software is hard to test, step back and take a look at the design. A difficult testing story is often a symptom of an overly-coupled design. Picking apart the ball of yarn can be difficult, and demands experience, but there are established techniques for doing so. The book Working Effectively with Legacy Code, by Michael Feathers, treats this challenge in detail.

While orienting your code around testability can help somewhat with the design of that code (by encouraging less coupling, etc.), the practice of TDD by itself will not guarantee a good, understandable, maintainable design, as Rich Hickey explains in his talk Simple Made Easy (which I highly recommend). The wrong tests can actually slow your ability to make design changes – one needs to consider design throughout the process, and be willing to rework both the tests and the production code as needed if the design needs improving.

Speed

A key point about all these tests: they must be fast. Fast, as in running the entire test suite takes less than a couple seconds. When you start relying on tests as your safety net, you can only move as fast as that net lets you move. If the tests take 10 seconds, a minute, 10 minutes to run, progress will be slow. I often run my tests multiple times a minute, literally every time I save a file in the directory I’m working in. Or, if I am writing in a dialect of Lisp such as Clojure or Common Lisp, I run the entire test suite with a couple of keystrokes and expect instant visual feedback directly in my editor. The good news is that most of the tests you write can be very fast. Slow tests can be a design smell – or at least a sign that you’re testing at too high a level, through too many dependencies. It can take time to do the needed refactoring to make your tests fast, but it is worth the investment. Organizing your software in layers and using mocking, dependency injection, or polymorphic dispatch¹ to implement simple test harnesses for individual layers can be helpful here.

Some tests are intrinsically slower. Examples include tests through a database, or generative / fuzzing tests that programmatically generate many examples and make assertions about program behavior while running against those examples. These tests can run separately, such as after every commit, or prior to a release. Invest in separating out slow tests when needed.

Clean Tests

All code in your repository has a maintenance cost, including tests. I try to keep test code as clean as production code. Because tests can be repetitive, often with subtle variation between test cases, it is important to use the abstractions available in your programming language to make the test as clean and as readable as possible. Squeezing out repetitive idioms is one area where high-level languages such as Python, Clojure, or Lisps have an advantage. But clean tests should be a goal regardless of language.

Tests by example, also known as table-driven tests, are an excellent way to get clean tests. While Clojure’s are macro (trivially portable to other Lisps) gives the cleanest implementation I know of, table tests are also common in Go and probably in several other languages as well. It is worth discovering if your language offers this construction, or, if not, perhaps creating it, if possible.

Test Frameworks

Though some communities (Java, Python) gravitate towards the xUnit style of test frameworks, I find them to be overkill for most small projects, and for some large ones. Test frameworks try to address common areas of repetition in tests, but the abstractions your project needs in order to write clean tests may or may not be a natural fit for an existing, heavyweight framework. The job of your test framework is to make sure that every assertion in your test suite is run, and that the build fails if any of the assertions fail. Good examples of lightweight test frameworks include clojure.test for Clojure and 1AM for Common Lisp (perhaps with the addition of my are macro). Go’s testing package is another.

In some cases, a few test functions, a trivial test runner, and judicious use of assert are all you need.

Test Coverage

Tools are available for most languages which will generate test coverage reports. I am not a fan of strictly defined coverage targets which fail test suites if coverage is below some fixed amount. Nevertheless, I think that inspecting code coverage from time to time can be helpful in identifying areas that need better tests.

Discuss with your team what fraction of coverage you want to shoot for in your project. Some projects made up of mostly pure functions can get to 100% easily, whereas others, perhaps heavy on error handling, UI code, or integration with other services which are hard to mock, settle at something less, perhaps 70%-80%.

Monitor your repositories for slippage in coverage, and use coverage reports to inform your work as you write code.

Style Checks

It is my experience working with a variety of junior and senior engineers that standardizing coding styles leads to more readable code and higher quality overall. Therefore, I like to run “linters” at the same time I run my tests. Linters are tools which check code style according to agreed-upon guidelines and which raise an error when the standard isn’t met. Examples of linters are lint in C, pycodestyle (formerly pep8) in Python, and kibit and bikeshed in Clojure.

In addition to or in place of a linter, an automated code formatter, such as gofmt for the Go programming language, or clang-format for C, can reformat source code to adhere to community style rules. Since they fix style violations automatically, they have the benefit of not slowing down your workflow or miring developers in endless formatting discussions during code review, etc. The Go story is especially successful in this regard, with the result that formatting discussions are largely absent in the Go community.

Most code formatters and style checkers are configurable. These options should be stored in a configuration inside the repository. As the team’s preferences and needs evolve, the configuration options can be discussed and updated via pull requests or according to your team’s code review procedures.

Automated Builds

Once the testing / linting rhythm is well established, automated builds shore up the practice further. Every project should have a primary build which is triggered on a build machine, known commonly as a Continuous Integration (CI) server, whenever anybody pushes to the master branch. This build runs the automated tests for the project, to ensure that no broken code gets shipped, or, if it does, that it can be noticed and fixed as soon as possible.

Rarely is the code I’m writing destined to run on the machine I’m writing it on. Achieving fidelity between development and production environments is frequently a challenge, though it is easier for some languages than for others. Regardless of the language or target platform, I’ll try to achieve as much parity as possible, generally using Docker for builds. The Dockerfiles for these projects tend to be simple: a few lines to install any needed dependencies, a line to copy the files into the container, and a line to run the build tasks using make (more on that below).

Since I host my repositories on GitHub, I use GitHub Actions for my builds, generally running them within Docker except for highly standardized builds such as for Go programs.²

The history of the build success or failure can be easily seen on the GitHub Actions page for that build. If the history is not full of mostly green builds, your developers are probably not running their tests locally often enough, or there may a discrepancy between build and development environments that needs addressing.

I make sure that the README has a build badge which shows the most recent build status. These badges may seem frivolous, but I feel they do give some insight into how well a project is maintained. Any build failures also trigger an email to the committer, and optionally to any people “watching” the repository on GitHub.

Repeatability

Builds should be repeatable. That is to say, if a build fails for particular commit, it should always fail, and if it succeeds for a particular commit, it should always succeed. Any other behavior makes collaboration and troubleshooting harder (“it worked when I tried it / on my machine”) and adds noise which slows the team down. Tests that fail only sometimes are usually a sign of race conditions, particularly insidious bugs which tend to be hard to identify and troubleshoot but which generally boil down to defects in the tests in question and/or in the production code.

To reiterate: unpredictable builds should be viewed as bugs, and should not linger unaddressed.

READMEs

I like every repository to have a README file, written in Markdown³. The file explains:

what the project is for;
how to acquire, build and test it;
gives an example snippet showing the project in action;
provides links to further documentation, and
has a license, along with a possible legal disclaimer (important for open source projects where you may receive less legal cover, and financial benefit, than for software written for the benefit of an employer).

The example snippet shown in the README is especially important. It not only gives a brief snapshot of your software in action, but it also gives a feel for what it is like to use it and what it actually does. Examples can be found here and here.

Whether for open source or closed/private work, stating license terms clearly can be helpful. GitHub makes this especially easy now. For closed-source, restricted repos I add a simple reminder in place of a license: “© <year> MyOrganization. All rights reserved.” For open source projects, I generally choose a fairly permissive license, such as the one from MIT.

On a personal note, for my own projects, I try to give every repository its own unique artwork or photograph which appears at the top of the README, an image ideally at least tangentially related in to the project (this can be a challenge for a software project). I find that it adds to my enjoyment while working on the software, and helps me keep track of what repo I’m looking at at any given time. It also ties my creative practices (painting, drawing, photography) with my work in software development, areas of my life which otherwise rarely overlap.

Versions and Releases

Your project should have versioned releases. The process of making releases should be entirely automated. Typically, I make a release by tagging the software in Git, and pushing the tags to GitHub. Depending on the language and target platform, this could involve other steps, like making a tarball, triggering a build on GitHub which makes the release files available for download, or starting a deployment process to a staging environment.

Your program, Web service, app, or library should know and be able to report what version it currently is.

The release process should be scripted and have the following steps:

Make sure no code is uncommitted in the local directory
Get the most recent version, e.g. v0.0.99
Determine the new version, e.g. v0.0.100
Update the local software so that it can report the new version when asked
Commit the updated local software, e.g. git commit -am "Release v0.0.100"
Tag the new software, e.g. git tag v0.0.100
Push the new tag, e.g. git push --tags
Increment the local software version to indicate its “tainted” (off-version) state, based on the new tagged version, e.g. v0.0.100-dirty.
Commit the local software again, e.g. git commit -am "Taint post v0.0.100 release"

An example release script can be seen here.

Any released artifacts (tarballs, jar files, etc.) should reflect the release number, so if you’re generating release artifacts locally, it should be done after Step 4, and before Step 8, above. Otherwise, do it on your CI server based on the tag you push in Step 7.

This automation can take some effort up front to set up. But it removes a major source of resistance to doing releases, and allows for tighter feedback loops. An important component to this practice is to keep the software releasable at all times – use feature flags or other mechanism to “hide” new features under development from users or otherwise indicate their preliminary nature. If software is always releasable, and the automated tests are trustworthy, it will be safer to quickly deploy bug fixes quickly when things go wrong, and easier to track down bugs in production if they ever occur.

If you can package your software using the target platform’s packaging system (Homebrew, apt, Clojars, PyPI, etc.), it goes a long way to making installation easier, particularly when onboarding new developers. This is typically easier for open source projects, but can be helpful for private repositories as well (an example is private Maven repositories for the Clojure and Java ecosystems).

A side note on semantic versioning: there are many passionate opinions about how to do semver correctly. If you follow my open source repositories, you’ll see mostly versions like “v0.0.123” rather than “v1.10.1”. I personally believe that semver is overrated – in particular, breaking changes (so-called MAJOR revisions) are both hostile to users and usually avoidable … and should therefore be extremely rare. This talk by Rich Hickey explains the viewpoint far better than I can.

My advice is to avoid breaking changes whenever possible. One strategy for this is to version your APIs, so that older and newer versions coexist together. Another is to simply call an existing project done, and releasing a completely new project with extensive changes, rather than breaking the old one for existing users.

Issue Tracking

Your organization probably already does issue tracking, either through JIRA or some similar system. If you don’t, consider using GitHub Issues to track bugs and future work. I use these for my open source projects, both to encourage other people to file issues, and to give visibility into my to-do list. When I make a commit, I tag the issue number in the commit. For example,

git commit -am "Fix horrible race condition.  Fixes #66."

GitHub automatically cross links these commits with the issues themselves, and in the case of the above example actually closes the ticket automagically. JIRA can be configured to do the much same. This kind of tracking can make it easier to understand what changes were made when, and for what reason. In some kinds of environments (e.g., highly regulated ones), this kind of change-tracking is a firm requirement.

Other Helpful Practices

Using a Common Build Tool

I frequently work in Python, C, Clojure, Go, and Common Lisp. Each of these languages has its own toolchain, which may be more or less standard for the language, but I find that using make to automate most common tasks makes it very easy for me to remember how to carry them out.

Here are some make targets I write for most projects:

make test runs all the unit tests locally.
make lint does any style checks.
make on its own runs any tests and linting / style checking (this typically matches what runs on the build server after every push).
make release does everything needed to cause a release to be created.
make docker runs tests and linting steps inside a Docker container.

Though make is an old tool, with a strange, if powerful, syntax, it is very fast, and relatively simple to use for the above use cases. (I find many modern build tools to be slower and more awkward to use than Make for common operational tasks.) The practice of using Make in this way can be helpful in larger organizations, where several different languages may be in use, and where developers may switch from project to project with some regularity. Some of my repos which use this pattern are l1 (Go), smallscheme (Python), oatmeal (Clojure), cl-oju (Common Lisp), lexutil (Go) and hbook (Common Lisp).

Getting “Invisible Feedback”

This trick relates to the “fast feedback” requirement mentioned earlier and applies primarily to local development. If you can get your computer to speak out loud, you can run your builds in another terminal window and not even look at that window most of the time. For example, on my Mac, I have the following running much of the day:

conttest 'make && say ok || say fail'

Here conttest is a continuous testing tool I wrote, which runs the supplied argument, a shell command, every time the current directory is updated. When I save a file I’m editing, I instantly hear the result. If my tests are doing what I think they should be doing in the moment, I can simply hear the results as I go, without looking away from the code. Once again, this little genie is especially helpful when my tests are fast.

Auto-Generating Documentation

Look for opportunities to automate your documentation. Examples include API documentation generated from the source code, literate programs, and automating updates to READMEs based on program output (for example, see how the L1 README is updated).

Summary

Although these practices require some up-front investment, I find that they increase my efficiency overall, especially when I’m working with other collaborators. Good, clean operational practices like these can cut back on common sources of pain when working on long-running software projects.

Acknowledgments

My views have been influenced by discussions with former colleagues at OppFi and OpinionLab over the past eight years. Timothy Coleman wrote the original version of the bumpver release script linked above.

Examples include using protocols in Clojure, generics in Common Lisp, and interfaces in Go. ↩︎
I prefer to run even Clojure builds in Docker, since I’ve been bitten by subtle differences between my local and the target deployment environments). ↩︎
I actually prefer Org Mode for my own writing, but GitHub’s Markdown support is better than its support for Org Mode. ↩︎

Adding Tail Call Optimization to A Lisp Written in Go

Mon, 08 Aug 2022 00:00:00 +0000

The last few days have been devoted to improving l1, the homegrown lisp I wrote about earlier this year. A number of changes have landed in the last week:

Implemented let, defn and sugar for quote;
Figured out basic REPL integration with Emacs;
Added numeric comparison operators;
Reviewed my Minimum Viable Repo checklist for this project;
Fixed four bugs.

I also implemented the bulk of the automated tests in the language itself. This was a decisive step forward in both ease of creating new tests and confidence that the language was approaching something usable.

The work I'm happiest with, though, because it taught me the most, was implementing tail call optimization (TCO) in the language, which the rest of this post will be about.

Motivation

The need for some form of TCO became clear as I started to write more small programs in l1. Perhaps the simplest example is one that sums all the natural numbers up to $n$:

(defn sum-to-acc (n acc)
  (cond ((zero? n) acc)
        (t (sum-to-acc (- n 1) (+ n acc)))))

(defn sum-to (n)
  (sum-to-acc n 0))

Calling sum-to for small $n$ worked fine:

(sum-to 100)
;;=>
5050

However, larger $n$ blew up spectacularly:

(sum-to (* 1000 1000))
;;=>
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x14020500360 stack=[0x14020500000, 0x14040500000]
fatal error: stack overflow

runtime stack:
runtime.throw({0x10289aa2b?, 0x10294ddc0?})
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/panic.go:992 +0x50
runtime.newstack()
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/stack.go:1101 +0x46c
runtime.morestack()
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/runtime/asm_arm64.s:314 +0x70

goroutine 1 [running]:
strings.(*Reader).ReadByte(0x1401c2820e0?)
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/strings/reader.go:66 +0x98 fp=0x14020500360 sp=0x14020500360 pc=0x102883348
math/big.nat.scan({0x0, 0x1401c2820e0?, 0x0}, {0x1028dc7c8, 0x1401c2820e0}, 0xa, 0x0)
	/opt/homebrew/Cellar/go/1.18.3/libexec/src/math/big/natconv.go:126 +0x80 fp=0x14020500430 sp=0x14020500360 pc=0x10288b1e0
;; ...

This happens, of course, because sum-to-acc calls itself a million times, each time storing a copy of its local bindings on the stack, which eventually consumes all the space on the stack.

Getting simple recursive functions like this to work for large $n$ is especially important because l1 doesn't have loops (yet)!

The Optimization

The solution is hinted at already in my test case. Note that I did not write sum-to as a single recursive function, as follows:

(defn sum-to-notail (n)
  (cond ((zero? n) 0)
        (t (+ n (sum-to-notail (- n 1))))))

While this function looks slightly simpler, it is harder for a compiler or interpreter to optimize. The difference is that, whereas sum-to-notail does some work after calling itself (by adding n to the result), sum-to-acc calls itself from the tail position; that is, the function returns immediately after calling itself.

People have long realized that function calls from the tail position can be replaced by updating the return address and then jumping directly to the the new function without adding new information to the stack. This is something I had heard about for years and used in various "functional" languages, without ever really implementing myself (and therefore fully understanding). It's an easy thing to take for granted without knowing anything about how it's actually implemented under the hood. The failure of sum-to-acc and similar recursive functions, described above, meant I would have to learn.

Two very different blog posts were helpful to me in pointing the way forward: Adding tail call optimization to a Lisp interpreter in Ruby, and How Tail Call Optimization Works. The posts focus on very different languages (Ruby vs. C / assembler), but they each revolve around what are effectively GOTO statements. I'm old enough to remember BASIC and the pernicious GOTO statement leading to "spaghetti code." I doubt I've ever used a GOTO statement in production code, whose use in modern programming languages fell out of favor in the aftermath of Dijkstra's famous Go To Statement Considered Harmful paper. But the ability to transfer control to another part of your program without invoking a function call is key to the optimization.

The Approach

Since the strategy is general, let's lose all the parentheses for a moment and rewrite sum-to-acc in language-agnostic pseudo-code:

function sum-to-acc(n sum)
   if n == 0, then return sum
   return sum-to-acc(n - 1, n + sum)

In most languages (without TCO), when this function is called, the values of n and sum, as well as the return address, will be put on the stack, whose evolution looks something like the following.¹:

 first invocation: [n=5, sum=0,  ret=sum-to:...]

second invocation: [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 third invocation: [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

fourth invocation: [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 fifth invocation: [n=1, sum=14, ret=sum-to-acc:...]
                   [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

 sixth invocation: [n=0, sum=15, ret=sum-to-acc:...]
                   [n=1, sum=14, ret=sum-to-acc:...]
                   [n=2, sum=12, ret=sum-to-acc:...]
                   [n=3, sum=9,  ret=sum-to-acc:...]
                   [n=4, sum=5,  ret=sum-to-acc:...]
                   [n=5, sum=0,  ret=sum-to:...]

At the sixth invocation, our terminating condition is reached, and 15 is returned, with all the pending stack frames popped off the stack.

With TCO, the implementation looks more like the following:

function sum-to-acc(n sum)
TOP:
   if n == 0, then return sum
   n = n - 1
   sum = sum + n
   GOTO TOP

as a result, the evolution of the stack looks as follows:

 first invocation: [n=5, sum=0, ret=sum-to:...]

second invocation: [n=4, sum=5, ret=sum-to:...]

 third invocation: [n=3, sum=9, ret=sum-to:...]

fourth invocation: [n=2, sum=12, ret=sum-to:...]

 fifth invocation: [n=1, sum=14, ret=sum-to:...]

 sixth invocation: [n=0, sum=15, ret=sum-to:...]

All those extra stack frames are gone: recursion has turned into a form of iteration.

Implementing TCO, then, has two ingredients:

Replace the values of the current arguments with their new values directly.
Jump straight to the next call of the function without adding to the stack;

This low-level, imperative optimization makes high-level, functional, recursive implementations efficient.

Implementation

In thinking about the implementation for l1, I was pleased to learn that Go actually has a goto statement. However, my implementation was poorly set up to use it.

Early in the implementation of l1, I noticed that each data type (numbers, atoms, and lists) had its own evaluation rules, so it made sense to make use of Go's features supporting polymorphism, namely interfaces and receivers. I had a Sexpr interface which looked like the following:

type Sexpr interface {
	String() string
	Eval(*env) (Sexpr, error)  // <--------
	Equal(Sexpr) bool
}

Numbers and atoms, for example, had fairly simple Eval implementations. For example,

func (a Atom) Eval(e *env) (Sexpr, error) {
	if a.s == "t" {
		return a, nil
	}
	ret, ok := e.Lookup(a.s)
	if ok {
		return ret, nil
	}
	ret, ok = builtins[a.s]
	if ok {
		return ret, nil
	}
	return nil, fmt.Errorf("unknown symbol: %s", a.s)
}

And, of course, numbers eval to themselves:

func (n Number) Eval(e *env) (Sexpr, error) {
	return n, nil
}

Lists, as you would expect, were more complicated – evaluating a list expression needs to handle special forms², user-defined functions, and built-in functions. Following the classic Structure and Interpretation of Computer Programs, I separated the core logic for function application into separate Eval and Apply phases. And to prevent the Eval for lists from getting too large, I broke out the evaluation rules for different cases (e.g. for let and cond special forms and for function application) into their own functions.

In other words, I had evaluation logic spread over ten functions in five files. Sadly, the need to jump back to the beginning of an evaluation rather than recursively calling Eval again meant that several of those nicely broken out functions had to be brought together into a single function, because goto does not support jumping from one function to another. (C has setjmp and longjmp, which effectively do this, but I would want to upgrade my IQ by a few points before applying them in this situation.)

There were actually three cases where I was performing an evaluation step right before returning, and the goto pattern could be used:

When evaluating code in the tail position of a user-defined function;
When evaluating code in the last expression in a let block;
When evaluating code in the chosen branch of a cond clause.

I wound up with code which with looks like the following. Several steps are indicated only with comments. Note the tiny, easy-to-miss top: label at the very beginning:

// lisp.go
//
func eval(expr Sexpr, e *env) (Sexpr, error) {
top:
	switch t := expr.(type) {
	case Atom:
		return evAtom(t, e)
	case Number:
		return expr, nil
	// ...
	case *ConsCell:
		if t == Nil {
			return Nil, nil
		}
		// special forms:
		if carAtom, ok := t.car.(Atom); ok {
			switch {
			case carAtom.s == "quote":
				return t.cdr.(*ConsCell).car, nil
			case carAtom.s == "cond":
				pairList := t.cdr.(*ConsCell)
				if pairList == Nil {
					return Nil, nil
				}
				for {
					if pairList == Nil {
						return Nil, nil
					}
					pair := pairList.car.(*ConsCell)
					ev, err := eval(pair.car, e)
					if err != nil {
						return nil, err
					}
					if ev == Nil {
						pairList = pairList.cdr.(*ConsCell)
						continue
					}
					expr = pair.cdr.(*ConsCell).car
					goto top
				}
			// ...

The code so far shows the evaluation for atoms, numbers, and cond statements. cond does not introduce any new bindings, but when the first truthy condition is encountered, it evaluates the next argument as its final act. So the code above simply replaces the expression to be evaluated, expr, with the expression from the matching clause, and then restarts the evaluation via goto, without the overhead of a separate function call.

The code for let is somewhat similar:

			case carAtom.s == "let":
				args := t.cdr.(*ConsCell)
				if args == Nil {
					return nil, fmt.Errorf("let requires a binding list")
				}
				// ... code to set up let bindings ...
				body := args.cdr.(*ConsCell)
				var ret Sexpr = Nil
				for {
					var err error
					if body == Nil {
						return ret, nil
					}
					// Implement TCO for `let`:
					if body.cdr == Nil {
						expr = body.car
						e = &newEnv
						goto top
					}
					ret, err = eval(body.car, &newEnv)
					if err != nil {
						return nil, err
					}
					body = body.cdr.(*ConsCell)
				}

The for loop invokes a new eval for each expression in the body of the let, except for the last one: when the last expression is reached, (the cdr is Nil), the last eval is done by jumping to the beginning of the function, once it has updated its environment to point to include the new bindings.

The last use of this pattern is in function invocation proper, which looks similar:

			// (... code to set up new environment based on passed arguments ...)
			var ret Sexpr = Nil
			for {
				if lambda.body == Nil {
					return ret, nil
				}
				// TCO:
				if lambda.body.cdr == Nil {
					expr = lambda.body.car
					e = &newEnv
					goto top
				}
				ret, err = eval(lambda.body.car, &newEnv)
				if err != nil {
					return nil, err
				}
				lambda.body = lambda.body.cdr.(*ConsCell)
			}

I've skipped various parts of eval that aren't relevant for TCO optimization – if you're interested, you can check out the code yourself.

To be clear, what we are optimizing is all tail calls, not just recursive ones – though the recursive ones were the primary objective due to the stack overflows reported above.

The end result is that sum-to now can complete for large values of $n$:

(sum-to (* 1000 1000))
;;=>
500000500000

Incidentally, a variant of our test case failed before I added the TCO optimization to let shown above; this now works, as well:

(defn sum-to-acc-with-let (n acc)
  (let ((_ 1))
    (cond ((zero? n) acc)
          (t (sum-to-acc-with-let (- n 1) (+ n acc))))))

(defn sum-to-with-let (n) (sum-to-acc-with-let n 0))

(sum-to-with-let (* 1000 1000))
;;=>
500000500000

Conclusion

Getting tail-call optimization to work was very satisfying… though the eval implementation is certainly more complex than before. (Ah, optimization!)

To ensure TCO continues to work, variants of sum-to with and without let are run on every build, along with a few other short example programs.

After implementing TCO in my own code, I can appreciate and understand the optimization better when I see it in the wild. I fully expect to use the pattern again when implementing future lisps (yes, I hope there will be more).

Note that this is a somewhat abstract representation: the details are language-specific. The ret=sum-to:... notation means that when the function returns, control will pass back to where it left off inside the sum-to function.

A special form is one that does not follow the normal evaluation rule for functions – it may evaluate its arguments once, many times, or not at all. (I am glossing over macros for the time being; l1 does not have them yet.)

Tests by Example in Clojure and Common Lisp

Wed, 13 Jul 2022 00:00:00 +0000

Histogram made by a 1980's version of the author.

While playing around with some dice-rolling simulations recently in Common Lisp (a topic for some future post, perhaps), I looked around for a way of making histograms. Finding none close to hand (it being Common Lisp), it wasn't too hard to "roll my own" library (it being… Common Lisp) to create text-based histograms in the tradition of libraries from CERN that I used as a fledgling physicist in the late 1980s.

Since the project was just for fun, I drove most of the code forward just by REPLing. But I am trying to follow the same habits in my "for fun" GitHub repositories as I would for paid work. This means, among other things, unit tests (in fact, for critical work I still prefer to use TDD when I can).

So, in fleshing the tests out after the fact, I found myself writing somewhat repetitive code that looked like this:

(test hist-values-test
  (let ((h (histogram '(1 2) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(1 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 1) 2)))
    (is (= 2 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 2 (hist-max h)))
    (is (equalp #(2 1) (hist-bin-heights h)))
    (is (equalp #(1 2) (hist-bin-xs h))))
  (let ((h (histogram '(1 2 2 3) 3)))
    (is (= 3 (hist-count h)))
    (is (= 1 (hist-min h)))
    (is (= 3 (hist-max h)))
    (is (equalp #(1 2 1) (hist-bin-heights h)))
    (is (equalp #(1 2 3) (hist-bin-xs h)))))

This is using the minimalist 1AM testing library, which I like for its simplicity. It is also similar to the standard clojure.test library I'm used to in Clojure. It lacks, however, one important feature of clojure.test, namely the are macro. The macro allows one to represent tests as a table of examples:

;; Clojure code:
(deftest example
  (are
    [nums       prod]         ;; [1]

    (= prod (reduce * nums))  ;; [2]

    []          1             ;; [3]
    [1]         1
    [0]         0
    [0 1]       0
    [1 1]       1
    [2 2]       4
    [2 3]       6
    [-1 -1]     1
    [2 2 2]     8
    [1000 1000] 1000000))

An invocation of are has three parts: a vector of symbols to be bound to values from the examples to follow [1]; the expression to be evaluated for each example [2]; and the actual examples [3], with one example per line being the usual, but optional, convention. (In the code above I have added extra spaces to make clear that nums goes with the first column of data, and prod goes with the second column.)

If at first you didn't understand that code being tested ([2]) was related to multiplication, you'd probably be able to figure it out just by looking at the examples. This is the primary advantage of are, namely that examples often provide as good or better documentation than a written description of functionality does.

Missing such a compact testing idiom for my unit tests, I found myself wondering how hard it would be to port it to Common Lisp. The answer, as you might expect, was: Not that hard. As is often the case, there is not much actual Clojure code behind the function in question, and the corresponding Common Lisp code is similarly compact, as we shall see.

If you look at the actual function definition in clojure.test, most of the macro proper is error checking; the actual work is delegated to a completely different namespace, clojure.template (in what follows, I will ignore the error checking):

;; Clojure code:
(defmacro are [argv expr & args]
  ;; ... error checking
  `(clojure.template/do-template ~argv (is ~expr) ~@args)
  ;; ... error checking
)

A few macro expansions shows what's happening:

;; Clojure code:
(macroexpand-1
 '(are [nums prod]
    (= prod (reduce * nums))
    []      1
    [1]     1
    [2 2 2] 8))

gives

(clojure.template/do-template
 [nums prod]
 (clojure.test/is (= prod (reduce * nums)))
 []      1
 [1]     1
 [2 2 2] 8)

do-template is also a macro. Simply running macroexpand-1 again shows what it does:

;; Clojure code:
(macroexpand-1
 '(clojure.template/do-template
   [nums prod]
   (clojure.test/is (= prod (reduce * nums)))
   []      1
   [1]     1
   [2 2 2] 8))

gives

(do (clojure.test/is (= 1 (reduce * [])))
    (clojure.test/is (= 1 (reduce * [1])))
    (clojure.test/is (= 8 (reduce * [2 2 2]))))

The expanded code is exactly equivalent to what you would write if you didn't have the are macro. This is accomplished by simply repeating the code being invoked, but substituting the actual example values for nums and prod.

In order to do accomplish the same in Common Lisp, we take a bottom-up approach. First, we need to split our test cases into groups corresponding to the number of columns in our argv vector (in this case, 2):

(let ((cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (partition-n 2 2 cases))
;;=>
'((#()      1)
  (#(1)     1)
  (#(2 2 2) 8))

The output looks very similar to cases, but each pair has been grouped into its own list. Here I use partition-n from cl-oju, a Common Lisp library I wrote specifically to take advantage of sequence idioms from the Clojure core library; partition-n is a translation of Clojure's partition function. (The equivalent code in clojure.template also uses partition for the same purpose.)

We now need to flesh out those cases by making use of each example in the expression being tested, whose Common Lisp translation is

(is (= prod (reduce #'* nums)))

For example, if nums is #(2 2 2), then prod should be 8.

In other words, we want our test cases to be:

(is (= 1 (reduce #'* #())))
(is (= 1 (reduce #'* #(1))))
(is (= 8 (reduce #'* #(2 2 2)))))

The replacement of symbols with their actual values in the Clojure code is accomplished with a tree-walking function called postwalk-replace. We can do it in Common Lisp with the subst function, which works on linear or nested lists:

(subst 3 'b '(* (+ b b) (/ b (- b b))))
;;=>
'(* (+ 3 3) (/ 3 (- 3 3)))

But because we have multiple bindings or example arguments we have to convert, we need to do it repeatedly:

(defun apply-bindings (bindings case expr)
  (loop with ret = expr
        for b in bindings
        for c in case
        do (setf ret (subst c b ret))
        finally (return ret)))

(apply-bindings '(x y) '(2 3) '(+ x y))
;;=>
'(+ 2 3)

Now we have everything we need to expand our test cases:

(let ((argv '(nums prod))
      (expr '(is (= prod (reduce #'* nums))))
      (c (length argv))
      (cases '(#()      1
               #(1)     1
               #(2 2 2) 8)))
  (loop for case in (partition-n c c cases)
        collect (apply-bindings argv case expr)))
;;=>
'((IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

All the macro has to do is wrap this in a progn:

(defmacro are (argv expr &rest cases)
  "
  Analog of clojure.test/are.  Apply multiple assertions
  based on some set up code and variations of one or more
  variable bindings.
  "
  (let ((c (length argv)))
    `(progn ,@(loop for case in (partition-n c c cases)
                    collect (apply-bindings argv case expr)))))

giving, for our example:

(macroexpand-1
 '(are (nums prod)
   (is (= prod (reduce #'* nums)))
   #()      1
   #(1)     1
   #(2 2 2) 8))
;;=>
'(PROGN
  (IS (= 1 (REDUCE #'* #())))
  (IS (= 1 (REDUCE #'* #(1))))
  (IS (= 8 (REDUCE #'* #(2 2 2)))))

This is equivalent to the macroexpansion of Clojure's are that we saw above. I leave as an exercise for the reader a further exploration of clojure.template/apply-template and its use in the do-template macro; and the addition of error checking such as making sure the number example values is a multiple of the number of binding arguments.

One rough edge that I glossed over: in my first attempt, my macro couldn't find apply-bindings at compile time until I wrapped that function's definition as follows:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defun apply-bindings (...) ...))

This is apparently needed for functions used by macros defined in the same file; it's not an issue for functions defined in other files. Since Clojure has a single-pass compiler, this sort of ceremony is not needed for Clojure macros.

I expect to use this macro again, so will probably spin it out into a small stand-alone library or incorporate it into a thin wrapper around 1AM. Here it is in action in the hbook library where our discussion started:

(test hist-values-test
  (are (h c mn mx heights xs)
       (progn
         (is (= c (hist-count h)))
         (is (= mn (hist-min h)))
         (is (= mx (hist-max h)))
         (is (equalp heights (hist-bin-heights h)))
         (is (equalp xs (hist-bin-xs h))))

       (histogram '(1 2)     2)   2 1 2 #(1 1)   #(1 2)
       (histogram '(1 2 1)   2)   2 1 2 #(2 1)   #(1 2)
       (histogram '(1 2 2 3) 3)   3 1 3 #(1 2 1) #(1 2 3)))

The code which are makes possible is usually denser and can be harder to parse initially than more repetitive test code would, as is clear from this example. However, I find the brevity of are tests extremely helpful, especially when comparing the test examples to each another. Such examples are especially helpful when calling out edge cases where behavior changes. I have given such tables to stakeholders many times, in order to make sure we are aligned on what the behavior should be. (Even if the stakeholder doesn't care about the details of your implementation or your test code per se, they probably will care about the specific examples you provide).

Parenthetically, here is the histogramming library in action, showing the distribution of the sum of 1000 six-sided dice:

(defun d () (1+ (random 6)))
(defun dn (n) (loop repeat n sum (d)))

(princ (hbook (loop repeat 100000 collect (dn 1000))
              80
              30))
;;=>

      4962                                       X  X
      4791                                       X  X
      4620                                     X X  X X
      4449                                     X X  X X
      4278                                     X XXXXXX X
      4106                                     X XXXXXX X
      3935                                   X XXXXXXXXXX
      3764                                   XXXXXXXXXXXX
      3593                                   XXXXXXXXXXXX
      3422                                   XXXXXXXXXXXX
      3251                                   XXXXXXXXXXXXX X
      3080                                  XXXXXXXXXXXXXXXX
      2909                                X XXXXXXXXXXXXXXXX
      2738                                XXXXXXXXXXXXXXXXXX
      2566                                XXXXXXXXXXXXXXXXXX
      2395                                XXXXXXXXXXXXXXXXXXXX
      2224                               XXXXXXXXXXXXXXXXXXXXX
      2053                              XXXXXXXXXXXXXXXXXXXXXX
      1882                              XXXXXXXXXXXXXXXXXXXXXX
      1711                              XXXXXXXXXXXXXXXXXXXXXXX
      1540                             XXXXXXXXXXXXXXXXXXXXXXXXX
      1369                             XXXXXXXXXXXXXXXXXXXXXXXXXX
      1198                           XXXXXXXXXXXXXXXXXXXXXXXXXXXX
      1027                           XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       856                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       684                         XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       513                       XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       342                       XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
       171                    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
         0 X   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX    X

                                     111222233344544544443332211111
                            11223568933623981977003313702313558540076333111
                      11244625381526456400925471219232518023527888048930961854221 1
           1   11235564827658185382948873231241519783970927498485883199710549330480234    1

           33333333333333333333333333333333333333333333333333333333333333333333333333333333
           22222222233333333333333334444444444444445555555555555555666666666666666677777777
           45566788900122344556778990112334456678890011233455677899001223445667889901123345
           40639628518417306395285184073062952851740739629528417406396295184173063962851841
           ................................................................................
           04826059371504826159371604827159372604827159382604837159382604937159482604937150
           02457912468913578024579134680135780246791346802357902467913568023579124689135680

(Yet Another) Lisp In Go

Sun, 27 Mar 2022 00:00:00 +0000

For the past month I've been spending some free time writing another Lisp implementation, this time in Go. My previous attempt was a subset of Scheme made in preparation for a class on the classic Structure and Interpretation of Computer Programs. That project was great fun and I learned a lot.

I started learning Go late last year and began thinking about writing another Lisp. Go has garbage collection, a feature that has long been an essential part of most Lisps¹. Getting GC essentially for free would save a lot of time and effort.

Go is also extremely fast. For applications where startup performance is not an issue, I'm pretty happy with Clojure most of the time. But working with Common Lisp in recent years, and Go more recently, has spoiled me and made me hungry for more performance, especially for short-running command-line utilities.

There are obviously other Lisps written in Go already, including Joker, a Clojure interpreter and linter. But I wasn't particularly interested with implementing a particular Lisp dialect; rather, I wanted to try to implement a language core that one could extend in a variety of different directions, including implementing the language in itself. Other possible directions include graphics programming, text-based games, and scripting. The working name of this Lisp is l1 ("el-one"), hinting at a possible series of small, experimental Lisps.

Features

Here is a summary of what's implemented & planned:

l1 has	doesn't have	will have	might get
integers (essentially unlimited size)	keywords	macros	curses
comments (`;; ....`)	maps	syntax quote	graphics
atoms	strings	reader macros (`, ', …)	subprocess / shells
lists	namespaces	REPL / editor integration	big floats
4 special forms: `cond`, `def`, `lambda`, `quote`	exceptions	`let` (as a macro)	`error` equiv.
16 built-in functions	loops	`defun` / `defn` (as a macro)	tail call optimization
recursion
closures			byte code compilation/interpretation

Performance

The current implementation is on GitHub. Its speed surprises me a bit.

Consider the following program in l1, which computes a factorial:

;; fact.l1: Return the factorial of `n`:
(def fact
     (lambda (n)
       (cond ((eq 0 n) 1)
             (t (* n (fact (- n 1)))))))

(print (fact 100))

outputting

933262154439441526816992388562667004907159682643816214685929638
952175999932299156089414639761565182862536979208272237582511852
10916864000000000000000000000000

Its equivalent in Clojure (or its nimble alternative implementations, Babashka or Joker) can be written as follows:

;; fact.clj: Return the factorial of `n`:
(def fact
  (fn [n]
    (cond (= 0 n) 1
          :else (*' n (fact (- n 1))))))

(println (fact 100))

… and in my Python Scheme implementation as

;; fact.scm
(define (fact n)
  (if (= n 1)
      1
      (* n (fact (- n 1)))))

(display (fact 100))

I also did the equivalent for Common Lisp. I used Oatmeal to make a skeleton Lisp project and used that to create an executable program called fact.

The execution times break down thusly:

Language	Running It	Execution Time (ms)
Clojure	`clojure -M fact.clj`	1729
Babashka	`bb fact.clj`	200
`smallscheme`	`scheme.py fact.scm`	120
Common Lisp	`./fact`	68
Joker	`joker fact.clj`	59
`l1`	`l1 fact.l1`	8

This is a very limited benchmark! However, it does give a flavor for start-up times. Note that l1 is a tree-walking interpreter; I wonder what speed might be possible if implemented with a byte code interpreter.

Lexing

I started the journey by writing the lexer. A post on Hacker News led me to this video where Rob Pike, of the core Go language team, describes an elegant design for a lexer used in the Go templating library. I was able to extract the relevant bits into a fairly general-purpose library that I then used for this Lisp with relatively little code (of course, one shouldn't expect too much code when lexing a Lisp, since there is not much syntax).

Fun With Atoms and Lists

You may have noticed the lack of character strings in the feature table, above. Many interesting Lisp programs don't use traditional strings (arrays of characters encoded as bytes), and I am curious to see what can be done strictly without them, though I could see adding them at some point.

Without strings, one will probably want to manipulate atoms in various ways. As a start, l1 introduces the notion of "splitting" (creating a list from an atom or number) and "fusing" (joining such a list back together again):

> (split (quote greenspun))
(g r e e n s p u n)
> (split 1395871)
(1 3 9 5 8 7 1)
> (fuse (quote (1 2 3 4 5)))
12345
> (randigits 10)
(6 1 5 4 4 8 8 3 0 1)
> (randigits 10)
(2 7 5 4 7 9 1 6 6 1)
> (fuse (randigits 1000))
797522288353215977146173184650097900747324790200919947108552266
896559408664559755127840959435738695979408193086120089317097735
733247509700258597497415541421859295990446300938230591278544826
160148998280910399112793807480556810085222871423786728939605143
312291343715625109027024093962060686621901049553518883581864852
832439160531395519838325036388642484613231265974048363263732041
737675913066537856875008449087672272329301144164887429770199070
521721230755123684155760268379043481645391533460833091119801604
531684362848585264816725347753593965869286499060052823295397069
598147167103689429912992818647290966641807288375144076084638850
885562168375457674623070776900707693203757775691854059277861315
130019383408298242102129643369889134587749005021251080452606062
217504938911721968545635428643266561957454859338694605115003758
001930119736513921576952435852918640253473323143920762789645830
700672642264667728965761815048634636071828415705273836146286590
4892309215088593977646507232497245814663081971549675531
>

Testing

As with previous Lisps I've worked on, most of the tests were represented in Go code as strings containing Lisp code and the expressions they evaluate to. Here are some examples, taken more or less at random.

        ...
        // `split` function
        {Cases(S("(split)", "", "expects a single argument"))},
        {Cases(S("(split 1)", "(1)", OK))},
        {Cases(S("(split -1)", "(-1)", OK))},
        {Cases(S("(split -321)", "(-3 2 1)", OK))},
        {Cases(S("(split (quote a))", "(a)", OK))},
        {Cases(S("(split (quote (a b c)))", "", "expects an atom or a number"))},
        {ECases(S("(split (quote greenspun))", "(g r e e n s p u n)", OK))},
        {ECases(S("(split (* 12345 67890))", "(8 3 8 1 0 2 0 5 0)", OK))},
        {ECases(S("(len (split (* 99999 99999 99999)))", "15", OK))},
        {ECases(S("(split (quote greenspun))", "(g r e e n s p u n)", OK))},
        {ECases(S("(split (* 12345 67890))", "(8 3 8 1 0 2 0 5 0)", OK))},
        {Cases(S("(fuse (quote (1 2)))", "12", OK))},
        {ECases(S("(+ 2 (fuse (quote (1 2 3))))", "125", OK))},
        {ECases(S("(fuse (split 1295807125987))", "1295807125987", OK))},
        {ECases(S("(len (randigits 10))", "10", OK))},
        {ECases(S("((lambda (x) (+ 1 x)) 1)", "2", OK))},
        {ECases(
                S("(def incrementer (lambda (n) (lambda (x) (+ x n))))", "<lambda(n)>", OK),
                S("(def inc (incrementer 1))", "<lambda(x)>", OK),
                S("(inc 5)", "6", OK),
                S("(def add2 (incrementer 2))", "<lambda(x)>", OK),
                S("(add2 5)", "7", OK),
        )},
        ...

Here Cases is a utility function that create a local environment (a possibly nested set of variable bindings) and evaluates one or more expressions in that environment; ECases (E for Exemplary) is the same, but saves its test cases in an examples.txt file which stores a reduced set of illustrative cases, e.g. for adding to the README. S is a function which takes an expression to evaluate, the result that should obtain, or an error message fragment which is expected if the expression should fail.

This seems to be a good way to test a new language implementation, though I plan to implement some generative / "fuzzing" tests as well, since I doubt all the bugs have been found yet.

Further Work

Macros are next. I'm pretty sure I know how I want to do them, and for me the power of macros is the whole point of Lisp, or at least a big part of the point.

A Final Note on Performance, and Conclusion

To return to the performance discussion, above: a reasonable person might object that l1 is so lean on features, that it's not at all surprising that it starts fast. Clojure, on the other hand, leverages the JVM, which has been tuned for decades for speed in long-running processes, and also provides a thoughtful and comprehensive language design that does many things no hobby language could easily support.

That being said, one thing I am excited about with this project is this: pretty much anything I can do in Go (which is very many things) can be added to my Lisp without too much effort. Conversely, to add features to a Common Lisp or Clojure implementation would be very difficult indeed. Building a small language core that you can extend in a variety of directions is a fun, if not necessarily entirely practical, way to go.

I met a speaker at LambdaJam 2015 who used his own Lisp for music performance. It was not garbage-collected, for real-time performance.

Migrating, Again

Thu, 03 Feb 2022 21:33:17 -0600

For the fifth or sixth time I am performing a major migration of this site from one implementation to another.

I'd like to do more writing, but the blogging sofware I wrote a few years ago has been slowing me down, literally: it takes several seconds to generate the whole site (something I do repeatedly when I am making changes or additions to the site). I want to be able to post more freely, in smaller chunks, with frequent edits – to some extent, to replicate the ease with which I can post on social media.

I also want to break down the walls between my artistic and my software development work. I've long tried to keep them separate somehow and it no longer feels right, or even honest, to do this.

So: how to reduce friction in posting, and marry art and tech-related posts in a single site?

Past Lives

Previously, this site was written in:

Platform	Post Format	Static?	Era
Pure HTML	HTML	Y	1994
Various hand-rolled Perl scripts	HTML	Y	2000
Blosxom	text	Y	2005
Jekyll	Markdown	Y
Hand-built Django / Python app	text / custom	N	2009
Hand-built Clojure app (1)	S-expressions	Y	2015
Hand-built Clojure app (2)	Org Mode	Y	2016
???			now

One common thread: I've had good results with "static" Web sites: generate content locally at home, then sync that content to the cloud. The resulting Web sites load fast and are easy to cache.

Requirements

What I want from the next iteration:

FAST: less than 3 seconds to regenerate the entire site. Ideally, live-update browser pages when the site regenerates.
A smooth path for integrating images of artworks in progress, and technical blogging, on the same platform.
Handle math expressions.
Handle image galleries.
A little wabi-sabi.
Bonus if I can write my content in Org Mode instead of Markdown.

Hugo

I've been diving deep into Hugo because it checks most of those boxes: the site you're currently looking at was generated with Hugo¹. (Currently it takes less than a tenth of a second to generate the entire site.)

Hugo does have a learning curve compared to other systems, which makes sense given the richness of its feature set. I found some helpful hints on this site for working with Org Mode in Hugo.

So far, I've been happy with Hugo, though it's a little strange to use someone else's software for this site!

Any wabi-sabi, always hard to come by on a computer, probably has to emerge from the visual design and, maybe, the writing.

About

Wed, 02 Feb 2022 20:17:03 -0600

See my Now Page for what I'm up to these days.

From Elegance to Speed

Wed, 25 Sep 2019 00:00:00 +0000

Making Things Fast

A while back, we used a toy particle physics trigger to explore concepts in lazy evaluation and functional programming in Clojure. We came up with the following function which selected groups of eight events clustered close together in time:

(defn smt-8 [times]
  (->> times
       (partition 8 1)
       (map (juxt identity
                  (comp (partial apply -)
                        (juxt last first))))
       (filter (comp (partial > 1000) second))))

Though elegant, this function processed events at a rate of 250 kHz, which I found slightly disappointing, performance-wise.

I have been interested in writing faster programs lately, which is one of the reasons I've been learning Common Lisp. Common Lisp lets you create extremely high-level programming constructs, but also lets you get closer to the metal when you need to. In this post we'll look at some ways for improving performance and compare our results to the original 250 kHz event rate.

First Steps

The first step is to translate our code as is to Common Lisp. We're going to punt entirely on the question of laziness for this post, though we might take it up in the future.

Here's my first attempt. We have to start with our source of random times. In Clojure, we had:

(def times (iterate #(+ % (rand-int 1000)) 0))

Since I'm going to benchmark performance, we'll need to set a limit to the number of events (times) we'll process. So, in Common Lisp:

(defun time-sequence (n)
  (loop repeat n
     for y = 0 then (+ y (random 1000))
     collect y))

where n is the number of event times to process. Here and throughout what follows, I'm using the loop macro, which is a Swiss army knife for iteration with a somewhat strange, non-Lispy, but readable syntax.

We also want to set our random state so as to guarantee different results each time we execute our simulation:

(setf *random-state* (make-random-state t))

My first attempt in Common Lisp looked like this benchmarking snippet:

(timing
  (length
    (->> 1000000
         time-sequence
         (partition-n 8 1)
         (mapcar (juxt #'car (compose #'car #'last) #'identity))
         (mapcar #'(lambda (l) `(,(- (cadr l) (car l)) ,(caddr l))))
         (remove-if-not #'(lambda (l) (< (car l) 1000))))))

This is less lovely than the Clojure, partly because of all the hash-quoting of functions (Common Lisp is a Lisp-2, and Clojure is a Lisp 1). ->> is from the arrow-macros library, and compose is from the cl-utilities library. The other Clojure-ish functions, partition-n (a replacement for Clojure's partition, whose name collides with an entirely different function in Common Lisp) and juxt are from cl-oju, a small library I've been writing for those still-frequent times when I want a Clojure function or idiom in Common Lisp. (The library ignores laziness for now, since I haven't needed it yet.)

timing is a macro I adapted from the Common Lisp Cookbook, which captures both the results of a computation and the elapsed CPU time used (not wall clock time) in msec:

(defmacro timing (&body forms)
  (let ((run1 (gensym))
	(run2 (gensym))
	(result (gensym)))
    `(let ((,run1 (get-internal-run-time))
	   (,result (progn ,@forms))
	   (,run2 (get-internal-run-time)))
       `(duration ,(- ,run2 ,run1) msec... result ,,result))))

For now, result is simply the number of eight-fold time clusters occuring within 1000 units of time. The execution time was roughly a second for the first few times I ran this:

 '(DURATION 1045 MSEC... RESULT 235)
 '(DURATION 1554 MSEC... RESULT 201)
 '(DURATION 827 MSEC... RESULT 164)

This already processed events at 1 MHz, roughly 4x faster than the Clojure speed of 250kHz.

Later, however, when I revisited the code, I noticed it ran significantly faster:

 '(DURATION 435 MSEC... RESULT 193)
 '(DURATION 279 MSEC... RESULT 189)
 '(DURATION 205 MSEC... RESULT 177)
 '(DURATION 601 MSEC... RESULT 180)

This is 2.6 Mhz, 10x the Clojure code speed. It took me a while to figure out that I had changed the SBCL heap size in order to handle larger arrays for testing (this is one area where laziness would help!). The larger heap size was making the code run faster.

It should be said at this point that I made no attempt in my original blog post to optimize the Clojure code for performance. Nevertheless, I think it's interesting that my first attempt to write the same algorithm in Common Lisp, using roughly the same idioms, performed so much faster.

Interlude

Looking back over my career, it seems I have been moving gradually from fast, lower-level languages to slower, more expressive, high-level languages: FORTRAN and C, to Perl and Python, then to Clojure (which is faster than Python for most long-running programs, while being arguably more expressive¹). I've enjoyed this trajectory, because higher level languages let you implement so much, so quickly.

My feelings on this are shifting, however. I wrote some code a decade ago for a system that has been running constantly for a decade on thousands of devices. My code is not particularly CPU efficient and, therefore, not as energy-efficient as it could be. As the planet warms, I think it's important to try and find small efficiencies where we can… not necessarily under the assumption that it will make a massive difference, but because it's just the right thing to do as conscientious professionals. (The fact that the sins of my past are executing on processors running on top of and within Antarctic ice adds an extra bit of irony here).

Common Lisp is particularly interesting to me of late because, having been designed largely for Good Old-Fashioned AI, it allows you to work at a very high level (you can, for example, add Prolog-like functionality to a Lisp in a few hundred lines of code ² – but, you can generate very fast (i.e., more energy efficient) code as well, as we shall see shortly.

Improvements

The next thing I tried was to separate the event generation from the trigger calculation and to use a vector for the input times, instead of a list:

(defun time-sequence-vec (n)
  (make-array `(,n)
              :initial-contents
              (loop repeat n
                 for y = 0 then (+ y (random 1000))
                 collect y)))

(defparameter array-size (* 100 1000 1000))

(defparameter *s* (time-sequence-vec array-size))

I also make the test sample much larger to improve the run durations and statistics.

Both Graham³ and Norvig⁴ emphasize tuning your algorithm first before turning to other optimizations. In our case the problem is O(n), which is the best time complexity we can hope for, but our use of function composition, while elegant, could be turned into something more efficient. To improve the algorithm proper, I perform the entire computation in a single loop statement. I also change the output slightly to show the index for matching 8-fold events, start and end time, and event duration:

(defun second-try ()
  (loop for x below (- array-size 8)
     if (< (- (elt *s* (+ x 8))
              (elt *s* x))
           1000)
     collect `(index ,x
                     startval ,(elt *s* x)
                     endval   ,(elt *s* (+ x 8))
                     diff     ,(- (elt *s* (+ x 8))
                                  (elt *s* x)))))
(take 4 (second-try))
;;=>
'((INDEX 69087 STARTVAL 34504666 ENDVAL 34505615 DIFF 949)
  (INDEX 153990 STARTVAL 76991815 ENDVAL 76992630 DIFF 815)
  (INDEX 237070 STARTVAL 118617491 ENDVAL 118618358 DIFF 867)
  (INDEX 294951 STARTVAL 147513479 ENDVAL 147514346 DIFF 867))

;; Time it, a few times:
(timing (length (second-try)))
;;=>
'(DURATION 2158 MSEC... RESULT 2427)
;;=>
'(DURATION 2167 MSEC... RESULT 2427)
;;=>
'(DURATION 2162 MSEC... RESULT 2427)
;;=>

Note that the starting and ending time values increase monotonically, and the reported time differences (DIFF) are under 1000, as desired⁵.

The new trigger function improves the processing rate to 46.2 MHz. At this point I'm starting to wonder what the assembly language looks like:

(disassemble 'second-try)
;;=>
; disassembly for SECOND-TRY
; Size: 1061 bytes. Origin: #x228FE0B0                        ; SECOND-TRY
; 0B0:       498B7510         MOV RSI, [R13+16]               ; thread.binding-stack-pointer
; 0B4:       488975F8         MOV [RBP-8], RSI
; 0B8:       488B3591FFFFFF   MOV RSI, [RIP-111]              ; 'ARRAY-SIZE
; 0BF:       8B56F5           MOV EDX, [RSI-11]
; 0C2:       4A8B142A         MOV RDX, [RDX+R13]
; 0C6:       83FA61           CMP EDX, 97
; 0C9:       480F4456F9       CMOVEQ RDX, [RSI-7]
; 0CE:       83FA51           CMP EDX, 81
; 0D1:       0F84BF030000     JEQ L17
; 0D7:       BF10000000       MOV EDI, 16
; 0DC:       FF1425B000B021   CALL QWORD PTR [#x21B000B0]     ; GENERIC--

;; .................. many lines omitted ................

; 4B6: L24:  CC18             BREAK 24                        ; UNBOUND-SYMBOL-ERROR
; 4B8:       27               BYTE #X27                       ; '*S*
; 4B9: L25:  6880000000       PUSH 128
; 4BE:       FF14252800B021   CALL QWORD PTR [#x21B00028]     ; ALLOC-TRAMP-R11
; 4C5:       E9FFFEFFFF       JMP L12
; 4CA: L26:  6A10             PUSH 16
; 4CC:       FF14252800B021   CALL QWORD PTR [#x21B00028]     ; ALLOC-TRAMP-R11
; 4D3:       EB94             JMP L14

(258 lines in total.)

My assembler-fu is not strong, but I notice the GENERIC-- and we do not need generic functions: there are many opportunities here to introduce type declarations.

After declaring to the compiler we want more speed and less type-checking ((declaim (optimize (speed 3) (debug 0) (safety 0)))), let's declare our array to be type fixnum (which on SBCL is an integer from -4611686018427387904 to 4611686018427387903).

(defparameter *s* (make-array array-size
                              :element-type 'fixnum))
;; Initialize array:
(loop
   for y = 0 then (+ y (random 1000))
   for i below array-size
   do (setf (elt *s* i) y))

(timing
  (length
   (loop for x below (- array-size 8)
      if (< (- (elt *s* (+ x 8))
               (elt *s* x))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1964 MSEC... RESULT 2435)
;;=>
'(DURATION 1939 MSEC... RESULT 2435)
;;=>
'(DURATION 2010 MSEC... RESULT 2435)

… maybe a slight improvement. Let's now type-hint where we can throughout the tight loop:

(timing
  (length
   (loop for x below (- array-size 8)
      if (< (- (the fixnum (elt *s* (the fixnum (+ x 8))))
               (the fixnum (elt *s* (the fixnum x))))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1740 MSEC... RESULT 2435)
;;=>
'(DURATION 1738 MSEC... RESULT 2435)
;;=>
'(DURATION 1753 MSEC... RESULT 2435)

A 10% or so improvement… not too bad.

At this point I'm tempted to look at the macroexpansion of my loop expression and see what it's doing. For simplicity, I collect just the starting time index for times satisfying the trigger:

(macroexpand-1
 '(loop for x below (- array-size 8)
     if (< (- (the fixnum (elt *s* (the fixnum (+ x 8))))
            (the fixnum (elt *s* (the fixnum x))))
         1000)
     collect x))
;;=>
'(block nil
  (let ((#:loop-limit-711 (- array-size 8)) (x 0))
    (declare (type (and number real) x)
             (type (and number real) #:loop-limit-711))
    (sb-loop::with-loop-list-collection-head (#:loop-list-head-712
                                              #:loop-list-tail-713)
      (tagbody
       sb-loop::next-loop
         (when (>= x #:loop-limit-711) (go sb-loop::end-loop))
         (if (<
              (- (the fixnum (elt *s* (the fixnum (+ x 8))))
                 (the fixnum (elt *s* (the fixnum x))))
              1000)
             (sb-loop::loop-collect-rplacd
              (#:loop-list-head-712 #:loop-list-tail-713) (list x)))
         (sb-loop::loop-desetq x (1+ x))
         (go sb-loop::next-loop)
       sb-loop::end-loop
         (return-from nil
           (sb-loop::loop-collect-answer #:loop-list-head-712))))))

I notice that x and :loop-limit-711 have an overly-general type declaration. I'm not sure how to type-hint x, but I can type-hint the upper bound:

(timing
  (length
   (loop for x below (- (the fixnum array-size) 8)
      if (< (- (the fixnum (elt *s* (+ (the fixnum x) 8)))
               (the fixnum (elt *s* (the fixnum x))))
            1000)
      collect `(index ,x
                      startval ,(elt *s* x)
                      endval   ,(elt *s* (+ x 8))
                      diff     ,(- (elt *s* (+ x 8))
                                   (elt *s* x))))))
;;=>
'(DURATION 1460 MSEC... RESULT 2513)
;;=>
'(DURATION 1455 MSEC... RESULT 2513)
;;=>
'(DURATION 1486 MSEC... RESULT 2513)

Another modest but significant improvement.

At this point I reached for the SBCL profiler to see what I could learn from it. Most of the processing time is spent in array lookups, which I don't think we can avoid.

Let's look at the assembler again. Since our trigger matches are rare, most of our time is going to be spent in finding the events, rather than in collecting the detailed results for the matches. If we switch back to using collect x to avoid the extra assembly associated with building `(index ,x startval ...):

(defun trigger ()
(loop for x below (- (the fixnum array-size) 8)
     if (< (- (the fixnum (elt *s* (+ (the fixnum x) 8)))
              (the fixnum (elt *s* (the fixnum x))))
           1000)
     collect x))
(disassemble 'trigger)
;;=>
;; disassembly for TRIGGER
;; Size: 448 bytes. Origin: #x22610EA0                         ; TRIGGER
;; 0EA0:       498B4510         MOV RAX, [R13+16]              ; thread.binding-stack-pointer
;; 0EA4:       488945F8         MOV [RBP-8], RAX
;; 0EA8:       488B05B1FFFFFF   MOV RAX, [RIP-79]              ; 'ARRAY-SIZE
;; 0EAF:       8B70F5           MOV ESI, [RAX-11]
;; 0EB2:       4A8B342E         MOV RSI, [RSI+R13]
;; 0EB6:       83FE61           CMP ESI, 97
;; 0EB9:       480F4470F9       CMOVEQ RSI, [RAX-7]
;; 0EBE:       83FE51           CMP ESI, 81
;; 0EC1:       0F8477010000     JEQ L11
;; 0EC7:       40F6C601         TEST SIL, 1
;; 0ECB:       0F8568010000     JNE L10
;; 0ED1:       48D1FE           SAR RSI, 1
;; 0ED4:       4883EE08         SUB RSI, 8
;; 0ED8:       488975F0         MOV [RBP-16], RSI
;; 0EDC:       31F6             XOR ESI, ESI
;; 0EDE:       49896D28         MOV [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 0EE2:       4D8B5D68         MOV R11, [R13+104]             ; thread.alloc-region
;; 0EE6:       498D4310         LEA RAX, [R11+16]
;; 0EEA:       493B4570         CMP RAX, [R13+112]
;; 0EEE:       0F874D010000     JNBE L12
;; 0EF4:       49894568         MOV [R13+104], RAX             ; thread.alloc-region
;; 0EF8: L0:   498D4307         LEA RAX, [R11+7]
;; 0EFC:       49316D28         XOR [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 0F00:       7402             JEQ L1
;; 0F02:       CC09             BREAK 9                        ; pending interrupt trap
;; 0F04: L1:   C740F917001020   MOV DWORD PTR [RAX-7], #x20100017  ; NIL
;; 0F0B:       C7400117001020   MOV DWORD PTR [RAX+1], #x20100017  ; NIL
;; 0F12:       488945E8         MOV [RBP-24], RAX
;; 0F16:       488945E0         MOV [RBP-32], RAX
;; 0F1A:       E9BB000000       JMP L4
;; 0F1F:       90               NOP
;; 0F20: L2:   8B142524144A20   MOV EDX, [#x204A1424]          ; tls_index: *S*
;; 0F27:       4A8B142A         MOV RDX, [RDX+R13]
;; 0F2B:       83FA61           CMP EDX, 97
;; 0F2E:       480F44142528144A20 CMOVEQ RDX, [#x204A1428]     ; *S*
;; 0F37:       83FA51           CMP EDX, 81
;; 0F3A:       0F840F010000     JEQ L13
;; 0F40:       488D7E10         LEA RDI, [RSI+16]
;; 0F44:       4883EC10         SUB RSP, 16
;; 0F48:       488975D8         MOV [RBP-40], RSI
;; 0F4C:       B904000000       MOV ECX, 4
;; 0F51:       48892C24         MOV [RSP], RBP
;; 0F55:       488BEC           MOV RBP, RSP
;; 0F58:       B862934F22       MOV EAX, #x224F9362            ; #<FDEFN ELT>
;; 0F5D:       FFD0             CALL RAX
;; 0F5F:       488B75D8         MOV RSI, [RBP-40]
;; 0F63:       F6C201           TEST DL, 1
;; 0F66:       0F85CA000000     JNE L9
;; 0F6C:       4C8BC2           MOV R8, RDX
;; 0F6F:       49D1F8           SAR R8, 1
;; 0F72:       4C8945D0         MOV [RBP-48], R8
;; 0F76:       8B142524144A20   MOV EDX, [#x204A1424]          ; tls_index: *S*
;; 0F7D:       4A8B142A         MOV RDX, [RDX+R13]
;; 0F81:       83FA61           CMP EDX, 97
;; 0F84:       480F44142528144A20 CMOVEQ RDX, [#x204A1428]     ; *S*
;; 0F8D:       83FA51           CMP EDX, 81
;; 0F90:       0F84BC000000     JEQ L14
;; 0F96:       488BFE           MOV RDI, RSI
;; 0F99:       4883EC10         SUB RSP, 16
;; 0F9D:       488975D8         MOV [RBP-40], RSI
;; 0FA1:       B904000000       MOV ECX, 4
;; 0FA6:       48892C24         MOV [RSP], RBP
;; 0FAA:       488BEC           MOV RBP, RSP
;; 0FAD:       B862934F22       MOV EAX, #x224F9362            ; #<FDEFN ELT>
;; 0FB2:       FFD0             CALL RAX
;; 0FB4:       4C8B45D0         MOV R8, [RBP-48]
;; 0FB8:       488B75D8         MOV RSI, [RBP-40]
;; 0FBC:       F6C201           TEST DL, 1
;; 0FBF:       7572             JNE L8
;; 0FC1:       488BC2           MOV RAX, RDX
;; 0FC4:       48D1F8           SAR RAX, 1
;; 0FC7:       498BD0           MOV RDX, R8
;; 0FCA:       4829C2           SUB RDX, RAX
;; 0FCD:       4881FAE8030000   CMP RDX, 1000
;; 0FD4:       7C22             JL L5
;; 0FD6: L3:   4883C602         ADD RSI, 2
;; 0FDA: L4:   488BC6           MOV RAX, RSI
;; 0FDD:       48D1F8           SAR RAX, 1
;; 0FE0:       483B45F0         CMP RAX, [RBP-16]
;; 0FE4:       0F8C36FFFFFF     JL L2
;; 0FEA:       488B45E8         MOV RAX, [RBP-24]
;; 0FEE:       488B5001         MOV RDX, [RAX+1]
;; 0FF2:       488BE5           MOV RSP, RBP
;; 0FF5:       F8               CLC
;; 0FF6:       5D               POP RBP
;; 0FF7:       C3               RET
;; 0FF8: L5:   4C8B45E0         MOV R8, [RBP-32]
;; 0FFC:       49896D28         MOV [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 1000:       4D8B5D68         MOV R11, [R13+104]             ; thread.alloc-region
;; 1004:       498D4310         LEA RAX, [R11+16]
;; 1008:       493B4570         CMP RAX, [R13+112]
;; 100C:       7747             JNBE L15
;; 100E:       49894568         MOV [R13+104], RAX             ; thread.alloc-region
;; 1012: L6:   498D4307         LEA RAX, [R11+7]
;; 1016:       49316D28         XOR [R13+40], RBP              ; thread.pseudo-atomic-bits
;; 101A:       7402             JEQ L7
;; 101C:       CC09             BREAK 9                        ; pending interrupt trap
;; 101E: L7:   488970F9         MOV [RAX-7], RSI
;; 1022:       C7400117001020   MOV DWORD PTR [RAX+1], #x20100017  ; NIL
;; 1029:       488945E0         MOV [RBP-32], RAX
;; 102D:       49894001         MOV [R8+1], RAX
;; 1031:       EBA3             JMP L3
;; 1033: L8:   CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 1035:       08               BYTE #X08                      ; RDX
;; 1036: L9:   CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 1038:       08               BYTE #X08                      ; RDX
;; 1039: L10:  CC48             BREAK 72                       ; OBJECT-NOT-FIXNUM-ERROR
;; 103B:       18               BYTE #X18                      ; RSI
;; 103C:       CC10             BREAK 16                       ; Invalid argument count trap
;; 103E: L11:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1040:       00               BYTE #X00                      ; RAX
;; 1041: L12:  6A10             PUSH 16
;; 1043:       FF14252800B021   CALL QWORD PTR [#x21B00028]    ; ALLOC-TRAMP-R11
;; 104A:       E9A9FEFFFF       JMP L0
;; 104F: L13:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1051:       27               BYTE #X27                      ; '*S*
;; 1052: L14:  CC18             BREAK 24                       ; UNBOUND-SYMBOL-ERROR
;; 1054:       27               BYTE #X27                      ; '*S*
;; 1055: L15:  6A10             PUSH 16
;; 1057:       FF14252800B021   CALL QWORD PTR [#x21B00028]    ; ALLOC-TRAMP-R11
;; 105E:       EBB2             JMP L6

… this is almost a manageable amount of assembler, though I don't claim to understand all of it. If I needed more speed, though, I could dig into the assembler as a way to try and ferret out even more efficiency.

Summary

We went from our initial POC in very functional-looking Clojure, running at 250 kHz, to optimized Common Lisp running at 100,000,000 / 1467 msec = 68.2 MHz, which is nearly 300x faster.

Checking our intuition about how fast this really is: my processor is 2.7 GHz, and our processing rate is 68.2 MHz, meaning we're spending about 40 clock ticks per event, which doesn't strike me as too bad.

In the end, we made the following changes to our initial attempt:

Separated out the initial event time-sequence generation as a separate step;
Rewrote the algorithm as a single loop;
declaim'ed optimizations to the compiler;
Type-hinted the input array, the variables used in the time difference calculation, and the loop upper bound.

We also touched on profiling (which didn't provide much benefit this time, but can be very useful generally), and showed how to inspect the generated assembler code.

Conclusion

Common Lisp can be made pretty fast! I should write a C-language version of this toy problem to compare. The final code is not nearly as beautiful as the original, but without too much effort we got much faster code. I'll conclude with this line from ANSI Common Lisp: "Lisp is really two languages: a language for writing fast programs and a language for writing programs fast." It seems quite powerful to me to have both of these on hand.

Reasonable people can disagree on the relative expressiveness of languages; for me, Lisps win on expressiveness because I can bend the language to my will using macros in a way that I cannot with non-Lisps.

See Paul Graham, On Lisp p. vii

Paul Graham, ANSI Common Lisp

⁴

Peter Norvig, Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp

⁵

The original Clojure code reported the entire sequence of eight times for each triggered event, but we omit that level of detail for clarity. Since the triggered clusters are rare, this doesn't significantly impact the timing measurements.

Common Lisp How-Tos

Sun, 15 Sep 2019 00:00:00 +0000

Intro

I've been writing this post as I go, and you may still see changes if you check back from time to time.

This post is for writing about things I'm figuring out, and for keeping notes on what I want to figure out next, as I get ramped up on Common Lisp. When I'm done there will hopefully be stuff here other people can use.

There's a lot to love and admire about Common Lisp, but the language has some pointy bits that can at times make Clojure look friendly. With so many things to sort out I'm going to try to keep a few notes going forward just to keep it all organized. Caveat emptor, YMMV, USE AT YOUR OWN RISK, etc., etc.

Lessons Learned So Far

How Do I Set Up Common Lisp on a Mac?

I made a repo to automate how I set up a fresh SBCL install on a Mac, including Quicklisp. This gets me from zero to a working Emacs REPL system fairly quickly.

Note that I also set a LISP_HOME environment variable in my .bash_profile which I use in the examples which follow.

How Do I Get Environment Variables from Lisp?

From the Common Lisp Cookbook:

(uiop:getenv "HOME")

How Do I Quickly Create a New Project?

(There is a nice writeup on how Lisp projects are organized.)

First, it seems it's best to pick a standard place to put your Lisp projects. I have a deep directory tree on my filesystem so would prefer to set that path as an environment variable; this makes the setup easy to port to another computer.

I made this bash script, lisplib:

#!/bin/bash

projname=${1?}
sbcl --non-interactive  \
     --disable-debugger \
     --eval '(ql:quickload "cl-project")' \
     --eval '(ql:quickload "cl-utilities")' \
     --eval '(let* ((projname "'$projname'")
                    (home (sb-unix::posix-getenv "LISP_HOME"))
                    (projpath (concatenate '"'"'string home "/" projname)))
               (cl-project:make-project (pathname projpath)))'

open -a /Applications/Emacs.app $LISP_HOME/$projname/src/main.lisp

(Omit the newlines inside the last single-quoted expression.)

When I, say, lisplib foo, it creates the project using make-project and opens the editor on the main source file.

How Do I Add Dependencies to One Of My Projects?

Add it to the .asd file in the :depends-on list.

How Do I Get My Local Projects Loadable By Other Projects?

I did this once:

$ cd ~/quicklisp
$ mv local-projects /tmp  # Mine was basically empty...
$ ln -s $LISP_HOME local-projects

How Do I Get Dependencies Loaded

put your project in the directory symlinked above;
jack into your REPL;
update depends-on in the project's .asd file as you add dependencies;
(ql:quickload <current-package>) to pick up its dependencies.
Optionally, to get the correct namespacing for the package you're working on, use slime-repl-set-package (C-c M-p) to set the default package to the one you're working on (not 100% sure what effect this has yet)

How Do I Build Binaries?

I use to Xach Beane's buildapp, built like so:

git clone git@github.com:xach/buildapp.git
cd buildapp
make
make install

A small working build script using buildapp can be found here.

How Do I Get My Project Into Quicklisp?

Make a pull request with Xach Beane here.

How Do I Use `Curses` (or equivalent)?

After some research I decided to follow in Steve Losh's footsteps and use cl-charms (tutorial here). This cost me multiple days due to some outdated Homebrew packages, but worked out in the end after updating those and reinstalling my Lisp setup.

How Do I Get Help?

Aside from Stack Overflow, the #lisp channel on IRC was helpful to me for the above issue.

Future Projects

Finish Weeds (redo of my blog software in CL)
Port namejen to CL
Write a Roguelike

A Daily Journal in Org Mode

Mon, 02 Sep 2019 00:00:00 +0000

In a comment on yesterday's Hacker News thread about journaling, I sketched out my daily journaling practice of late. This post contains the full details on this procedure for those who are interested in using Org Mode for daily journaling.

The Basics

Every day gets its own Org Mode file. What goes in is not specified, but things I've included so far include:

To do lists
Things I probably won't do but would like to
Previous night's dreams
Free-form journaling
Notes on projects in progress
Small snippets of beautiful code I've written or seen that day
Summary of the day's events
Quotes or poems

The main thing is to write something every day and keep some kind of record about what's happening in my life.

So far I've kept each day to a single page. At the end of the day, this page gets exported as a two-column PDF (via Org's LaTeX export), printed out and kept in a binder on my desk. I review the binder occasionally to get a feel for how the days have been going.

If I continue on this path for a few more months, I'll have a hundred or so entries; after a year, of course, hundreds. At that point I'll probably concatentate the PDFs into a single PDF and turn it into a bound book, perhaps via Lulu.

Explanation

Org Mode

For programming and writing, I use a 40+ year old text editor called Emacs, which is matchless in its programmability. If you're a programmer (or want to be one) and you want complete control over your editing experience, Emacs might be for you. Org Mode is a set of functionality that has been added to Emacs which allows for easy outlining, table creation and editing, export into a variety of formats (e.g. HTML and PDF), and a variety of other features for making pretty documents that are also readable as plain text. If you're familiar with the Markdown text format, Org Mode is like Markdown, but on steroids. (Every post on this site is generated from an Org Mode file.)

Org Mode files can have header information at the top of the file to define how the file looks when it's exported. Since I'm picky about how the printout looks, I've tweaked this to my liking as follows:

#+TITLE: [...today's date...]
#+OPTIONS: toc:nil num:nil author:nil date:nil
#+STARTUP: align
#+HTML_HEAD: <link rel="stylesheet" type="text/css href="styles.css" />
#+LaTeX_CLASS: article
#+LaTeX_CLASS_OPTIONS: [9pt,twocolumn,portrait]
#+LATEX_HEADER: \usepackage[margin=0.5in]{geometry}
#+LATEX_HEADER: \usepackage{enumitem}

styles.css is a style sheet I have which overrides the default styles for Org Mode-generated HTML, in case I want to view the file in a Web browser, which runs faster than the LaTeX/PDF export. The LaTeX-related items specify the multi-column layout, along with smaller text and narrower margins than the default.

Automating It

At first, I created and edited the files manually, copying and pasting the above snippet into a new file each morning, and setting the title in the header using org-time-stamp-inactive (C-c !). Eventually I automated all that using the following Emacs Lisp (added to $HOME/.emacs.d/init.el):

(defun open-journal-file ()
  (let* ((today (format-time-string "%Y-%m-%d"))
         (path (concat (getenv "HOME") "/path/to/my/journal/" today ".org"))
         (hdr-list (list (concat "#+TITLE: [" today "]")
                         "#+OPTIONS: toc:nil num:nil author:nil date:nil"
                         "#+STARTUP: align"
                         "#+HTML_HEAD: <link rel=\"stylesheet\" type=\"text/css\" href=\"styles.css\" />"
                         "#+LaTeX_CLASS: article"
                         "#+LaTeX_CLASS_OPTIONS: [9pt,twocolumn,portrait]"
                         "#+LATEX_HEADER: \\usepackage[margin=0.5in]{geometry}"
                         "#+LATEX_HEADER: \\usepackage{enumitem}"))
         (hdr (apply 'concat
                     (mapcar (lambda (s) (concat s "\n"))
                             hdr-list)))
         (has-hdr (lambda ()
                    (save-excursion
                      (goto-char (point-min))
                      (search-forward "#+TITLE" nil t)))))
    (message (concat "opening " path " ..."))
    (find-file path)
    (unless (funcall has-hdr)
      (save-excursion
        (goto-char (point-min))
        (insert hdr)))
    (message "Enjoy your journaling!")))

Then, to bind it to the key combination Control-o 1:

(global-set-key "\C-o1"
                (lambda ()
                  (interactive)
                  (open-journal-file)))

Now that this is set up, with just a keystroke or two I have the day's journal entry formatted exactly the way I want (because of the HTML_HEAD and LATEX_* options). If the file was already created, the shortcut puts me back in the file to make updates.

To Do Lists

The way I organize my tasks for the day ("organize" is too heavyweight a word - it's a very light touch) goes something like this:

| pt exercises          |
| coffee                |
| work on =smallscheme= |
| breakfast             | <-- stuff I did
|                       |
| train to work         | <-- stuff I'm doing now
| work on blog post     |
|                       |
| demo prep             | <-- stuff I think I'll do
| pairing               |
| meeting about xyz     |
| demo/standup/retro    |
| bike home             |
| dinner                |
| dishes                |
| meditate              |
| bed                   |

The Org Mode table editing shortcuts let me reorder, add or eliminate items quickly. Anything I don't get to by the end of the day either gets a ~~strikethrough~~ so I can remember that I didn't get to it, or just removed. The point of the todo table is not simply to make sure I do certain things, but to record the trajectory of the day.

Why This Approach?

For many years I've been keeping to-do lists and writing journals using various strategies, including:

the Things app on laptop and phone;
a single, massive Org Mode file; and
a pen and a bound paper journal.

I love the physicality of a paper journal, but I find handwritten journals harder to read than a typed manuscript. (The handwritten journals also have a way of turning into sketchbooks.) The printed journal gives me a bit more physicality and much better readability.

I like many things about Org Mode, but find the TODO states too ugly and heavyweight for a day's tasks. Generally, if I can't finish something in a single day, I don't want to be pestered about it (and I know what I care about longer-term). I really do need to be reminded of it I'll set a reminder in my calendar or my phone. (Most of the things I have to worry about at my day job are in our JIRA ticketing system or explicitly on my calendar.) Finally, something about the single, massive Org file gets to be a little oppressive. A separate file for every day has a certain cleanness to it.

I've enjoyed going back and rereading the entries. Whereas meditation builds awareness of the present moment, journaling and reading the journals gives a sense of the wider arc of time, inviting contemplation about how I'm spending my days, and shedding light on themes and feelings that persist over time.

The other comments in the Hacker News thread are worth a read if you're into this sort of thing.

Implementing Scheme in Python

Wed, 28 Aug 2019 00:00:00 +0000

Background

Lately I've been a bit of a Lisp kick again, reading On Lisp and Practical Common Lisp, working problems and re-casting my Clojure-based blog software into Common Lisp to see how that goes. The urge to do these things caught me somewhat by surprise: in the past few years I haven't done much programming outside of my day job because I've been focused more on painting and drawing. It seems, though, that these things are cyclical and right now I'm in the grips of Lisp mania again.

This cycle goes back to the mid-1980s when I was a college student studying physics and learning how to program. It's hard to say how I got the Lisp bug initially. I remember reading Douglas Hofstadter's explanation of recursive Lisp functions; hanging out near the AI workstations in the basement of the Madison Academic Computing Center where I worked (the sans-serif Lisp program printouts looked way better than anybody else's output); and reading about AI and Lisp on Usenet. At any rate, I managed to pick up a copy of PC-Lisp, a port of Franz Lisp to IBM PCs which apparently is still a thing after all these years. Somewhat astonishingly, the interpreter ran on my underpowered IBM PCjr, loading from floppy disk in half a minute or so. I remember how even fairly small programs would pause for several seconds while the garbage collector did its thing. This interpreter, along with a Lisp textbook I picked up somewhere, was enough to get me started learning about atoms, lists, quote, car and cdr, symbolic computing, and simple recursive algorithms.

I also wrote a small s-expression parser in Turbo Pascal on the same PCjr as a way of graphing arbitrary math functions of two variables. I had no compiler class but my roommate taught me enough about finite state machines, lexers, parsers, etc. to be dangerous… enough knowledge to get my grapher up and running. (I wish I still had some of these old programs, but they vanished when I "lent" the computer and all my diskettes to a friend; they are probably landfill somewhere.)

This all subsided for several years while I got sucked into physics, then painting, then capoeira, then physics, then painting, then software engineering in C, Perl, and Python. In the early aughts I got interested in Paul Graham's writing, which led to a few skirmishes with Common Lisp every couple of years, and ultimately to me getting totally hooked on Clojure starting in 2011.

I intend to write more about what makes Lisps in general compelling for me¹ and why I've returned to it over and over throughout my career. In the mean time….

Scheming in Python

Happily, my Lisp mania has crested coincidentally with Chicago Python teacher David Beazley announcing another class on The Structure and Interpretation of Computer Programs, a classic MIT computing text based on a dialect of Lisp called Scheme, and I managed to get on the list before it filled up. Since it's been awhile since I've written much Python, I thought I'd warm up for the class by trying to write a small Scheme implementation in Python. My hope was to refresh my memory of the basics of Scheme, improve my slightly-dusty Python, and have fun.

Prior Art

Researching this post (after most of the code was done), I noticed that there are a ton of small Lisps written in Python (including two I worked on the better part of a decade ago). One that stuck out in my memory was Peter Norvig's hundred-odd-line Lispy, which is particularly striking in its elegance (in style, it reminds me of the Lisp programs in PAIP).

The Stages

My plan for writing the interpreter was to follow the actual REPL stages: Read, Eval, and Print. Read would involve parsing input expressions and turning them into data structures to be evaluated; Eval would be a (presumably recursive) step whereby, for function calls, arguments would be Eval'ed first, with the results handed to the appropriate code implementing the actual function, implemented either in Python (for a few primitive functions) or, eventually, through user-defined functions. An exception to the argument evaluation rule would have to be made for a few special forms, which would have to be handled on a case-by-case basis.

Parsing

My first approach for the parser was to use PLY, which I've used before. Having been spoiled by the excellent Instaparse library for Clojure, upon returning to PLY I found the library's interface relatively clumsy and abstruse.

Then I thought I'd try a regex to parse everything, since Scheme's base syntax and grammar are so simple. The following regex actually worked well for awhile:

pat = """^\s*\((?P<list>.*)\)$   # Something beginning/ending with parens, or
         |
         (?P<bool>(\#t|\#f))\s*  # A boolean true/false value, or
         |                       # An atom starting w/ a letter or arithmetic op:
         (?P<atom>[a-zA-Z\+\-\*\/]+[a-zA-Z0-9\+\-\*\/]*)\s*
         |
         (?P<num>[0-9]+)\s*"""   # Or, a positive natural number.

(I had not gotten to negative or floating point numbers at this stage of the project.)

Expressions of the form (+ 1 2 (+ 1 1 1)), for example, parsed correctly. However, the regex didn't handle lists that nested in other than the last position, for example, (+ (* 2 3) 4). It might be possible to do the whole problem with regular expressions, but at this point I decided to look for a nice modern parser library that let me write down the grammer in EBNF notation and simply returned a parse tree when given a valid expression. I found Lark, which is roughly as featureful as Instaparse and worked well. Here's the Lark grammar as it currently stands:

start  : _exprs
_exprs : _e* _e
_e     : ATOM
       | _num
       | BOOL
       | list
TRUE   : "#t"
FALSE  : "#f"
BOOL   : TRUE | FALSE
list   : "(" _exprs? ")"
INT    : /[-+]?[0-9]+/
ATOM   : /[a-zA-Z]+[a-zA-Z0-9\-\?]*/
       | /[\*\/\=\>\<]/
       | /[\-\+](?![0-9])/
FLOAT  : /[-+]?[0-9]+\.[0-9]*/
_num   : INT | FLOAT
%import common.WS
%ignore WS

Note that while libraries like Lark and Instaparse are a nice way to turn out a powerful parser quickly, Norvig's lis.py just splits the inputs on open/close parens and uses them to increase/decrease nesting level as they appear. The approach to parsing numbers is also elegant: try to parse it as an integer; if that fails, as a float; otherwise, it's a symbol (what my grammar calls ATOM). I suspect that using a "real" parser makes it easier to add more syntax to your language if needed, though.

Data Model

Once Lark parses an expression, I convert to my own internal representation before evaluation (function convert_ast). I did not use objects to represent lists, atoms, etc., but just simple tuples and lists:

Data Type	Example
Atom	`('atom', 'foo')`
Boolean	`('bool', True)`
Int	`('int', 123)`
Float	`('float', 3.14)`
List	`('list', [('atom', 'define'), ('atom', 'x'), ('int', 3)])`

It might be more idiomatic Python to use objects, but I'm more used to thinking in terms of nested structures of base data types (lists, tuples, dictionaries), which have the advantage of being easily printable while debugging.

Here's an example expression, parsed with Lark and then converted into my internal representation:

Scheme raw source

(define (abs x)
  (if (< x 0)
      (- x)
      x))

Lark output

Tree(start,
     [Tree(list,
           [Token(ATOM, 'define'),
            Tree(list,
                 [Token(ATOM, 'abs'),
                  Token(ATOM, 'x')]),
            Tree(list, [Token(ATOM, 'if'),
                        Tree(list,
                             [Token(ATOM, '<'),
                              Token(ATOM, 'x'),
                              Token(INT, '0')]),
                        Tree(list,
                             [Token(ATOM, '-'),
                              Token(ATOM, 'x')]),
                        Token(ATOM, 'x')])])])

Internal representation used in the interpreter

[('list', [('atom', 'define'),
           ('list', [('atom', 'abs'),
                     ('atom', 'x')]),
           ('list', [('atom', 'if'),
                     ('list', [('atom', '<'),
                               ('atom', 'x'),
                               ('int', 0)]),
                     ('list', [('atom', '-'),
                               ('atom', 'x')]),
                     ('atom', 'x')])])]

Evaluation

Evaluation turned out, not surprisingly, to be where most of the work is; specifically, the individual cases for special forms. These were:

quote
cond / if (I implemented these separately but, had I implemented macros, I might implement one in terms of the other)
define
lambda
or
and

Aside from these, there was code for normal function application rules, with a slight variance between internally-defined functions and functioned defined by the user with define.

I did not, as Paul Graham did, attempt to find the minimal subset of language elements needed to bootstrap a complete Lisp, though that is certainly an interesting exercise. My goal was to get my tests green in a straightforward way, as described below.

Printing

The Print stage needed to convert the internal representation of the evaluated result to a string, both for a user running a REPL, and for my unit tests. This turned out to be simple enough that I can put the entire function here:

def printable_value(ast):
    k, v = ast
    if k == 'int' or k == 'float':
        return str(v)
    if k == 'bool':
        return {True: "#t",
                False: "#f"}.get(v)
    if k == 'intproc':
        return "Internal procedure '%s'" % v
    if k == 'atom':
        return v
    if k == 'list':
        return '(' + ' '.join([printable_value(x)
                               for x in v]) + ')'
    if k == 'nop':
        return ''
    if k == 'fn':
        (fn_name, _, _) = v
        if fn_name == 'lambda':
            return "Anonymous-function"
        return "Function-'%s'" % str(fn_name)
    raise Exception('Unprintable ast "%s"' % str(ast))

The primitive data types (int, float, bool and atom) are obvious enough; intproc and fn are for internally-defined and user-defined functions, respectively; finally, nop is for when you don't want any value shown to the user (i.e., after one enters a define expression, there is no output printed before the next prompt):

scheme> (define a 3)
scheme>

Testing Approach

I used a fairly strict test-driven development workflow for this project: write just enough new test code to fail; write just enough production code to get the test to pass; then refactor. This workflow generally serves me well for harder problems where I'm unlikely to see the best way to do things from the outset and I expect to make many improvements: I know the tests will have my back when I make changes. This turned out to be the right approach, since I had to pivot several times.

I didn't just use tests drive the code forward, but let SICP drive the tests forward as well. When I needed the next thing to work on, all I had to do was turn forward a few pages in the book, and the page count gave me a measure of my progress.

A few of the tests, with the corresponding page numbers:

    # Adapted from SICP p. 8:
    t("(define pi 3.14159)")
    t("(define radius 10)")
    t("(* pi (* radius radius))", "314.159")
    ...
    # p. 19:
    t("(define x 7)")
    t("(and (> x 5) (< x 10))", "#t")
    ...
    # p. 37
    t("""(define (fib n)
           (cond ((= n 0) 0)
                 ((= n 1) 1)
                 (else (+ (fib (- n 1))
                          (fib (- n 2))))))""")
    t("(fib 5)", "5")

Here t (a very short function name since it's used so many times) is a testing function which evaluates the code supplied in the first argument, using the current "environment" (scope). If a second argument (a string) is supplied, it also asserts that the code in the first argument evaluates to the second.

The Code Comes To Life

One of my favorite things about this project was when I started writing nontrivial tests which already passed, because enough of the language had been implemented already. This happened more often as the project progressed, and it made me feel like I'd implemented a "real" language, an oddly compelling feeling.

The code, in its entirety, can be found here.

Next Steps

Though I have already gone far past my goals for the project, there are a few directions which I could take this.

Implement more of Scheme

For example, I don't have strings yet in my implementation!

Extend it Further

Implement Python interop and shell features. This could be genuinely fun and useful, but there's already a fairly full-featured Lisp skin for Python called Hy.

Go Low Level

Do it again, but in a faster, lower-level language. Python is relatively slow compared to Clojure for long-running programs, and slower than Common Lisp or C for program startup. I've been wanting to dust off my C chops, and this would be an excuse to do so, though C is a bit primitive compared to, say, Rust. (Go is not to my taste so I wouldn't pick it for a fun project like this one.) So, either an excuse to learn Rust or to improve my C chops. Which leads me to the final question: would I rather implement Lisps, or program in them? Up until now it has been the latter, but after the past few weeks I can definitely taste the appeal of the former.

Thoughts Upon Revisiting Python

This was the biggest Python project I've done in five years, so it was interesting to check my experience of the language after some time away in Clojure-land. I was a big Python fan for many years but my enthusiasm for the language has definitely cooled. This is partly due to the performance comparison I just mentioned, but the two other things I really missed were REPL integration and structural editing.

In Lisps such as Clojure and Common Lisp I can send expressions to the REPL directly from Emacs and see the results instantly. It's difficult to describe how helpful this short feedback cycle is until you've experienced it. Writing the interpreter, I got sort of close by using conttest plus nosetests: my unit tests ran every time I saved a source file, and if I wanted to inspect some object I could add a print or two temporarily to the code I was working on. But this is definitely more clumsy than a Lisp REPL directly integrated with my editor.

Structural editing (via Paredit) is another thing that is hard to explain the utility of but which is hard to live without once you're used to it. The basic idea is that you are no longer editing strings of text, but the actual tree of nested lists comprising your code - killing, splitting, joining, absorbing ("slurping") into and expelling ("barfing") from those lists. Working with the code at the tree level is a powerful way to edit your code, and watching this video might help may give a hint of it. Once you're used to the affordances of Paredit or its cousins, it feels clunky and primitive to go back to editing plain old text.

Other value propositions of Clojure, such as macros, immutable data structures and concurrency, weren't really an issue for this project, though I would certainly miss them for bigger projects.

Conclusion

Implementing your own Lisp is surprisingly do-able (and fun!). Python is certainly a fine tool for the job (though if you want to skip straight to programming in a modern Lisp for your day job, please contact me).

Postscript

While digging through my digital and mental archives for Lisp things I remembered something I hadn't seen in a long time: the use of closing square brackets to terminate a deeply-nested collection of s-expressions (i.e. ] instead of )))))))). Did I imagine it? While not necessarily a good idea, it's an interesting one. Then, while reading through Gabriel and Steele's Evolution of Lisp this morning, I came across the following snippet:

"One of the minor, but interesting, syntactic extensions that Interlisp made was the introduction of the superparenthesis, or superbracket. If a right square bracket ] is encountered during a read operation, it balances all outstanding open left parentheses, or back to the last outstanding left square bracket [."

DEFINEQ((FACTORIAL
  (LAMBDA (N)
   (COND [(ZEROP N) 1]
         (T (TIMES N (FACTORIAL (SUB1 N]

Bingo! I wasn't imagining it. This suggests those workstations in the basement of MACC at UW-Madison were probably running Interlisp.

So compelling that I left an exciting science project partly in order to do Lisp (Clojure) programming full time.

Show at NUPOC

Wed, 10 Oct 2018 00:00:00 +0000

I am pleased to announce:

Of the Flesh but not the World...

A show of nine works in a small gallery space at the Feinberg School of Medicine at the Northwestern University Prosthetics-Orthotics Center (NUPOC).

The works include four paintings made from life at NUPOC over the course of the last few months. There are also paintings and drawings done from life at the Palette and Chisel Academy of Fine Art.

Plagiocephaly Helmet Casts, oil on panel, 12" x 9"

Location and Hours

Feinberg School of Medicine
Northwestern University Prosthetics-Orthotics Center (NUPOC)
680 North Lake Shore Drive, Suite 1100
Chicago, IL 60611
312-503-5700

Dates: November 2 through the end of 2018.

Hours: Monday-Friday, 9 AM - 4:30 PM

A quick Google will show a number of parking options nearby. The building is about four blocks from Navy Pier; the closest El stop is Chicago Ave., on the Red Line.

Portrait of J., oil on panel, 11.5" x5.5"

Artist's Statement

"Of the flesh but not the world? How peculiar. How can that be? It sounds like maths, only more practical somehow." – Thomas Pynchon, Against the Day

For the past six months I have had the privilege of painting from life roughly once per week at the Northwestern University Prosthetics-Orthotics Center (NUPOC). NUPOC carries out research and training in the design, manufacture, and application of orthotic and prosthetic devices. Aside from representing a fascinating synthesis of engineering and medicine, it is a visually rich environment hosting hundreds of artifacts, all of which relate in some way to the human form.

These artifacts appeal to both the engineer and the artist in me. The anatomical models, common in both medical schools and art studios, are not only helpful objects of study (a figurative artist ought to know how the body is put together), but beautiful objects of contemplation in their own right. In addition, NUPOC's collection of historical and current prostheses and orthoses, ranging from the crude to the intricate, is visually fascinating… any of these objects could be the cornerstone of a striking still life painting.

In addition to the four paintings done at NUPOC, there are figurative paintings and drawings based on live models. They represent a subset of many studies carried out over the past few years at the Palette and Chisel Academy of Fine Arts, a century-old Chicago institution where one can find people drawing and painting from life nearly every day of the year. Working from life and studying anatomy are centuries-old strategies for turning paint into convincing illusions - to create bodies that are "of the flesh, but not the world."

From the standpoint of engineering, I find the devices at NUPOC intriguing for their mix of high- and low-tech. 3D printers and advanced polymers contrast with mechanical joints, hooks, levers and cables. Automated mass-production of some components exists in conjunction with highly customized fittings of others, to adapt to every individual's unique morphology and requirements. High-tech is not necessarily better tech: advanced prostheses with multiple digits controlled by nerve induction may be less appropriate than a more straightforward, affordable device that can be maintained by its wearer "in the field."

More generally, the artifacts represent a literal meeting of the human organism, with its animal origins, and the engineered world. Limbs lost, or never present, can be augmented in capability, at times far past their "baseline" functionality. What could be more human than to build devices to mold or extend the function of our bodies challenged by trauma, age, genetics, or other vicissitudes? What are the limits of our flesh, assisted or no? These are interesting questions for a figurative artist to ponder.

Many thanks to the staff at NUPOC for their assistance and hospitality.

NUPOC Leg Model ("Legcorché"), oil on linen, 7"x28"

Gift to NUPOC with RJ Garrick

For news and updates

Say hi on email, or check me on Instagram (eigenhombre).

Color Notations

Wed, 19 Sep 2018 00:00:00 +0000

I have been experimenting with notations for color mixtures, so I can make notes about colors on the go without actually carrying a whole set of oil paints everywhere.

Here's an example from a very quick sketch during tonight's commute home:

Though it may look black and white, there are a dozen colors captured in this quick sketch. In this post, I briefly describe the notation.

The letters are based on my oil painting palette of late: usually,

name	provider	pigment number	code
cremnitz white	RGH	PW 1	w
nickel (titanium) yellow	Williamsburg	PY 53	n
winsor (primary) yellow	Holbein	PY 74	y
raw sienna	Gamblin	PBr 7	s
transparent earth yellow	Gamblin	PY 42	t
venetian red	Gamblin	PR 101	v
cadmium red light	Gamblin	PR 108	c
pyrrole red	Holbein	PR 254	p
brown pink	Williamsburg	PR 101	-
spanish earth	Williamsburg	PR 102	a
quinacradone violet	Gamblin	PV 19	-
dioxazine purple	Gamblin	PV 23	-
cobalt blue	Gamblin	PB 28	o
(real) manganese blue	Vasari	PB 33	m
prussian blue	Gamblin	PB 27	r
ivory black	Williamsburg	PBk 9	b
viridian	Gamblin	PG 18	i
terre verte	Gamblin	PY 43, PG 18, PBk 9	e
raw umber	Williamsburg	PBr 7	u

Since there are fewer than two dozen colors, a single letter can be assigned to each pigment (the rightmost column shows my assignations as of this morning). I'd like to whittle the number of colors down even further, but I'm hooked on the various properties of each of these. Almost all of them are pure pigments… the only mixture is terre verte, which I love for its transparency, mass tone and utility in mixing cool midtones.

Occasionally I'll throw in another pigment for special occasions, or start out with just a few for underpaintings, but usually I lay most of these out for any given painting session.

The notation, then, is very similar to that used for writing chemical compositions, which you might remember from high school chemistry: $\mathrm{H_2O}$ is water, and $\mathrm{C_{18}H_{32}O_2}$ is linoleic acid, one of the components of linseed oil.

For paint mixtures, we apply a similar rule: one letter per "element" (paint color/pigment), with an optional numerical suffix to indicate a correspondingly higher proportion (for simplicity we eliminate the subscripting of the numbers).

For example, king's blue, which is typically white plus cobalt blue, is, in my system,

w2o

… i.e., roughly two parts Cremnitz (lead) white per part of cobalt blue.

Not every color I might use is assigned a letter. In this case, just write the pigment code in parentheses. For example, for an exact match for commercial king's blue, I would use titanium white (PW-6) instead of the more transparent lead white; in this case, king's blue is

(PW6)2o

Now let's talk through the sketch at the top, location by location. The head is mostly backlit (contre-jour) with light from the window and some interior light inside the train.

Hair on top of head, b4uw: four parts black, plus raw umber and white.
Fairly flat area of color on forehead, eta: terre verte plus transparent earth yellow plus Spanish earth.
Left upper highlight, w4(PV19) - mostly white with a bit of quinacradone violet.
Forward/top plane of nose, w(PV19)s: white, quinacradone violet, raw sienna.

For the ear, we use a variant of the notation which has light, mid tone, and dark values, stacked (in parentheses):

w(PV19)

… which means,

Light: venetian red + white
Mid tone: pyrrole red + nickel titanium yellow
Dark: quinacradone violet + white

For the lips I could see, roughly, light and dark values:

w2(PV19)

w1a1(PV19)1 [the 1's are superfluous]

i.e., two portions white per quinacradone violet for light, and white, Spanish earth and quinacradone violet for the dark.

Are you serious about all this?

This may seem like a lot of work, or hard to remember. The notation is highly personal, though – your palette and your notation will be different, and you will remember yours better than mine. I do find it interesting that in just a few minutes I was able to make a study recording a dozen color decisions with just paper and pencil. One could work up a simple portrait on this basis alone (especially if the sketch were a little more complete).

This will be hard to do unless you've painted enough with your palette to have some knowledge about how your colors behave when mixed with each other. It is a good way to practice thinking about color when out in the world and away from the easel. Like preparing for drawing from memory, it's something you can do just about anywhere.

This is a work in progress: since I used quinacradone violet multiple times in my sketch, I am hereby officially dedicating the letter q to that pigment. Maybe Williamsburg brown pink can be k (originally reserved for my chalk putty medium, which maybe I don't need a letter for). Dioxazine purple can be d. As my palette evolves (as they tend to), I expect to swap around my letter assignations … and will have to remember to write down a key now and then so I can still "decode" my notations in years to come.

Code and Data

Mon, 04 Jun 2018 00:00:00 +0000

Introduction

This article was adapted from discussions we had at the weekly Clojure study group at OpinionLab, in June, 2018.

Clojure is a Lisp. Code is data in Lisp, and data can also be code. Lisp code consists of lists of symbols (or atoms, as they are called in other Lisps), other data primitives such as numbers and keywords, or (nested) lists of the same.¹

The first item in an expression (list) is the name of an operation, usually a function; the rest of the items are the arguments. When lists are nested, the inner expressions are evaluated first: in (+ 1 (* 2 3)), the multiplication happens before the addition.

Somewhat less commonly, instead of function names, the first item in the list may be the name of a macro or so-called "special form." In these cases, the exact method of evaluation depends on the implementation of the macro or form. For example, in (dotimes [_ 10] (println 'hi)), the println is evaluated 10 times; if dotimes were a function, println would be evaluated exactly once.

To prevent evaluation of an expression, one can quote it:

(quote (+ 1 (* 2 3)))
;;=>
(+ 1 (* 2 3))

The single quote (') character is an abbreviation for quote:

'(+ 1 (* 2 3))
;;=>
(+ 1 (* 2 3))

The REPL will evaluate expressions unless you quote them; if you decide later you want to evaluate an expression, you can use eval:

(def expr '(+ 1 (* 2 3)))
(eval expr)
;;=>
7

Quoting and evaluation are inverse operations:

(eval (eval (eval (eval ''''(+ 1 (* 2 3))))))
;;=>
7

Here are some examples you should try at the REPL, to help build an understanding of these two operations:

(+ 1 1)
'+
+
'(+ 1 1)
(cons '+ '(1 1))
(eval (cons '+ '(1 1)))
(first '(+ - * /))
(cons (first '(+ - * /)) '(1 1))
(eval (cons (first '(+ - * /)) (rest '(not-a-number 1 1))))
(->> '(not-a-number 1 1)
     rest
     (cons (first '(+ - * /)))
     eval)

Try to understand each of these as best you can, and try some variations. The last expression is the same as the one prior to it, just rewritten using the thread-last (->>) macro.

Arithmetic of the Fittest

The close relationship of code and data suggests making new code on the fly to solve problems, which is a concept used in genetic algorithms. Here is an example inspired by GAs. Let's write a function which finds expressions involving the four arithmetic operators that return the number 2. Examples:

(+ 1 1)
(- 4 2)
(* 1 2)
(/ 2 1)

First, let's make our expression generator (type or copy-paste into your REPL):

(defn oper [] (rand-nth '(* / - +)))
(defn digit [] (rand-int 10))

(defn numlist []
  (let [numnums (rand-int 10)]
    (repeatedly numnums digit)))

(defn expr-fn []
  (cons (oper) (numlist)))

Trying these out:

(oper)    ;;=> *
(oper)    ;;=> /
(numlist) ;;=> (6)
(numlist) ;;=> (6 9 3 4 4 2 4 3 2)

(expr-fn) ;;=> (/ 6 1 3 7 0 3 0 9)
(expr-fn) ;;=> (- 5 7 2)

Our expression generator selects an arithmetic operator at random, and tacks it on to the beginning of a random list of 0-9 digits.

Evaluating some example functions:

(eval (expr-fn)) ;;=> 0
(eval (expr-fn)) ;;=> 41
(eval (expr-fn)) ;;=> 7/1152
(eval (expr-fn)) ;;=> java.lang.ArithmeticException stacktrace!?!?!

This last one shows a problem: some of our expressions will not be valid mathematically (for example, (/ 1 0)). Let's catch any exceptions:

(defn eval-with-catch-exception [expr]
  (try
    (eval expr)
    (catch Throwable t
      (-> t .getClass .getSimpleName symbol))))

This function attempts to evaluate expr and, if the code throws an exception, simply returns the exception name as a symbol.

We are now in a position to try our code generator out:

(dotimes [_ 10]
  (let [e (expr-fn)]
    (println e (eval-with-catch-exception e))))

which gives

(/ 9 8 7 0 0 7 2 2 6) ArithmeticException
(+ 1 3 3 0 7 7 6) 27
(+ 4 2) 6
(- 4 1 2 1 0 0 0) 0
(/ 5 7 0 7) ArithmeticException
(* 0 4 6) 0
(* 8 6 7 2 4 1) 2688
(* 7 4 6 9 8 6) 72576
(*) 1
(- 7) -7

(* called with no arguments returns 1; + with no arguments returns 0. This may or may not make intuitive sense to you, depending on how much math you've had in school.)

Let's generate some that return the desired result, 2:

(distinct
 (for [_ (range 1000)
       :let [f (expr-fn)
             result (eval-with-catch-exception f)
             fitness-fn #(= result 2)]
       :when (fitness-fn)]
   f))
;;=>
((+ 2)
 (- 9 1 1 1 4)
 (- 2 0)
 (/ 4 2)
 (/ 2 1)
 (* 2))

(Note that (+ n) and (* n) both just return n.)

Macros

Macros are an advanced topic, but they are relevant because they allow (among other things) selective evaluation of arguments. Arguments to a macro can be evaluated zero, one, or many times; this makes macros especially helpful for implementing control flow: loop, for, while etc. are all macros.

Macros rely on code-as-data: they essentially functions which generate code, which is then evaluated. Most or all modern Lisps have macros, the equivalent of which are very rare in non-Lisps. These macros play a large role in making Lisp so expressive.

Exercises

Exercise 1: Creating and evaluating code on the fly in this manner is not unique to Lisp, but Lisp provides a very natural way to do so. Consider how to do the above examples in JavaScript, Ruby, or other mainstream languages. What, if any, are the differences between Clojure/Lisp and those languages?

Exercise 2: Set a different arithmetic goal (other than 2) and generate expressions which satisfy it.

Exercise 3: Adapt your code generator to include nested expressions, so that some of the numbers are generated expressions in their own right. Example: (+ 1 4) -> (+ 1 (* 2 2)).

Exercise 4: Instead of generating expressions, adapt expr-fn to return functions of no arguments, and adapt the rest of the code to call and evaluate those functions. I.e., (+ 1 2 3) becomes (fn [] (+ 1 2 3)).

Exercise 5 (advanced): Both the generated functions and the "fitness function" fitness-fn can be made more complex; for example, in addition to the operators + - * / and the numbers 0-9, you could add an "argument" variable x and change the fitness function to expect expressions that return the square of x; some acceptable outputs would then be (* x x) and (* 1 x x 1).

Change your expression generator and fitness functions to implement the squaring function.

Exercise 6 (advanced): try adding other Clojure core functions and data types to the list of possible operators and arguments; try other fitness functions as well.

Exercise 7 (more advanced): develop a way of combining expressions and evaluating the fitness of the resulting expression. To build on our previous example, if you have the two lists (+ 1 1) and (* 2 3), you could choose terms at random from each list, e.g. (* 1 3); you must decide how to handle lists with different lengths. The fitness function could just be fitness-fn from our example, or a more ambitious one². If you want to include nested expressions, then you have to decide how to combine those.

Rather than success or failure, the fitness function could return a number which reflects how well or poorly the expression performed. A strategy from genetic algorithms is to generate many expressions, evaluate these, take the best performers, and then generate new expressions by combining these. Implement this.

(oodles of delicious atoms)

Wed, 30 May 2018 00:00:00 +0000

A Recursive Feast

In 1983, Douglas R. Hofstadter, author of "Gödel, Escher, Bach: An Eternal Golden Braid" wrote a series of articles on Lisp in Scientific American (later republished in the book Metamagical Themas, which is where I encountered them for the first time). While much of the writing focused on the nature of recursion in programming, the third and last article in the series provided an entertaining example of blurring the line between code and data, which is typical for Lisp (though somewhat less common in Clojure, except for the occasional macro).

In the example, a set of acronyms are presented; each acronym can be "expanded" into a phrase which itself may contain other acronyms. The program generates random expansions of these acronyms, "bottoming out" (hitting the base case) at random, though with increasing probability of termination for more deeply-nested expressions.

What follows is my translation of the Lisp code in his articles into Clojure, with a few extra details and explanations. A few interesting edge cases have to be accounted for to accommodate the differences between the languages.

If you're interested in running the code, it's on GitHub.

Cooking It Up

First, a small macro helps us keep things tidy: defpasta makes a function that simply returns the list supplied as its argument, avoiding the need for the argument vector or the quote. Note that just one small macro provides enough "syntactic sugar" to make the examples more readable. We also attach metadata for convenience in testing (see below).

(defmacro defpasta [name_ & expr]
  `(defn ~(with-meta name_ {:pasta true}) []
     '~@expr))

Next are the actual acronym definitions. These are the definitions Hofstadter uses, with commas treated differently. The commas are important in his output, but commas are whitespace in Clojure… so where the symbols are used with commas in compound statements, we append _CO to tell the program to supply a comma in the output. (Example: SAUCE, becomes SAUCE_CO.)

We cannot, unfortunately, add commas after lists in this way, but his example is inconsistent in this regard (three examples have commas after lists, but the example expansion later in the article does not).

Hostadter presents the acronyms in capital letters, though Common Lisp atoms (analogous to Clojure symbols) are case-insensitive; we follow that convention here and use it to indicate terms which may be expanded… that is to say, the COFFEE in the expansion of SAUCE can itself be expanded to a phrase which includes ESPRESSO, and so on.

(defpasta TOMATOES (TOMATOES on MACARONI (and TOMATOES only)
                             exquisitely SPICED))
(defpasta MACARONI (MACARONI and CHEESE (a REPAST of Naples_CO Italy)))
(defpasta REPAST (rather extraordinary PASTA and SAUCE_CO typical))
(defpasta CHEESE (cheddar_CO havarti_CO, emmentaler
                                (especially SHARP emmenthaler)))
(defpasta SHARP (strong_CO hearty_CO and rather pungent))
(defpasta SPICED (sweetly pickled in CHEESE ENDIVE dressing))
(defpasta ENDIVE (egg NOODLES_CO dipped in vinegar eggnog))
(defpasta NOODLES (NOODLES (oodles of delicious LINGUINI) elegantly served))
(defpasta LINGUINI (LAMBCHOPS (including NOODLES)
                                  gotten usually in Northern Italy))
(defpasta PASTA (PASTA and SAUCE (that's ALL!)))
(defpasta ALL! (a lucious lunch))
(defpasta SAUCE (SHAD and unusual COFFEE (excellente!)))
(defpasta SHAD (SPAGHETTI_CO heated al dente))
(defpasta SPAGHETTI (standard PASTA_CO always good_CO hot particularly
                              (twist_CO then ingest)))
(defpasta COFFEE (choice of fine flavors, particularly ESPRESSO))
(defpasta ESPRESSO (excellent_CO strong_CO powerful_CO rich ESPRESSO_CO
                                 suppressing sleep outrageously))
(defpasta BASTA! (belly all stuffed (tummy ache!)))
(defpasta LAMBCHOPS (LASAGNE and meatballs_CO casually heaped onto PASTA SAUCE))
(defpasta LASAGNE (LINGUINI and SAUCE and GARLIC (NOODLES everywhere!)))
(defpasta RHUBARB (RAVIOLI_CO heated under butter and RHUBARB (BASTA!)))
(defpasta RAVIOLI (RIGATONI and vongole in oil_CO lavishly introduced))
(defpasta RIGATONI (rich Italian GNOCCHI and TOMATOES (or NOODLES instead)))
(defpasta GNOCCHI (GARLIC NOODLES over crisp CHEESE_CO heated immediately))
(defpasta GARLIC (green and red LASAGNE in CHEESE))

Hofstadter leaves the implementation of acronym? undefined, since it is so implementation-specific. In our case, something is an acronym if it's a symbol and its name is equal to an upper case version of itself.

(defn acronym? [x]
  (and (symbol? x)
       (= (name x) (.toUpperCase (name x)))))

We also want to be able to remove commas from a symbol and evaluate the result as a function. This is also very implementation-specific; in our case, it relies on Clojure's machinery for converting symbols to strings and vice-versa, and on the subtle differences between symbols, vars, and functions:

(defn strip-comma-and-eval [sym]
  (let [base-symbol-name (-> sym
                             name
                             (clojure.string/replace #"_CO$" "")
                             symbol)]
    ((-> *ns*
          ns-map
          (get (symbol base-symbol-name))
          var-get))))

(strip-comma-and-eval 'RHUBARB)
;;=>
(RAVIOLI_CO heated under butter and RHUBARB (BASTA!))

lower reduces the probability by some amount to encourage our recursion to bottom out (and thereby avoid stack overflows). Hofstadter multiplies probabilities by 0.8, but I simply square the probability so that higher probabilities decrease more slowly than lower ones as the recursion progresses:

(defn lower [x] (* x x))

A significant portion of Hofstadter's article relates to the recursive expand function, so I won't go into details here. This version is similar to his; it adds a bottoming-out clause to accommodate the base case of an empty phrase (since () and nil are not the same in Clojure, as they are in Common Lisp). Also we use strip-comma-and-eval instead of plain eval to handle the comma-ed symbols.

(defn expand [phrase probability]
  (cond
    (symbol? phrase) phrase
    (empty? phrase) phrase

    (acronym? (first phrase))
    (if (< (rand) probability)
      (concat
       (expand (strip-comma-and-eval (first phrase)) (lower probability))
       (expand (rest phrase) probability))
      (cons (first phrase) (expand (rest phrase) probability)))

    :else (cons (expand (first phrase) (lower probability))
                (expand (rest phrase) (lower probability)))))

Ultimately we want to make the output similar to what Hofstadter shows in the book. The normalize function does this, relying on two helper functions, whose names are self-explanatory:

(defn lower-case-symbol [x]
  (if-not (symbol? x)
    x
    (-> x name .toLowerCase symbol)))

(defn get-commas-back [x]
  (if-not (symbol? x)
    x
    (-> x
        name
        (clojure.string/replace #"_CO$" ",")
        symbol)))

When we're done expanding our lists of food (and our bellies), we prepare our data for output to the user: turn _CO suffixes into actual commas (using the fact that (symbol "x,") yields a valid symbol, if one that would be hard to read back at the REPL); and, lower-case all symbols to adhere to the style of the output shown in Metamagical Themas:

(defn normalize [expr]
  (->> expr
       (clojure.walk/postwalk get-commas-back)
       (clojure.walk/postwalk lower-case-symbol)))

Dinner Is Served

Now for the actual function we'll use to generate our expansions. Setting a large probability will ensure longer examples:

(defn dinner [expr]
  (normalize (expand expr 0.9999)))

We should test our code to make sure it doesn't bomb out on any of our acronyms. Here we find all the acronyms based on the metadata added by the defpasta macro, ensuring each runs to completion:

(let [nsm (ns-map *ns*)]
  (doseq [[k v] nsm :when (:pasta (meta v))]
    (dotimes [_ 10]
      (dinner (list k)))))

If, like me, you use Emacs and CIDER and want long REPL examples, you have to set `*print-length*` to something bigger than its default:

(set! *print-length* 1000)

Now to run it! Are you hungry yet?

(dinner '(LINGUINI))
;;=>
(lasagne and meatballs, casually heaped onto pasta sauce (including
noodles) gotten usually in northern italy and standard pasta, always
good, hot particularly (twist, then ingest) heated al dente and
unusual coffee (excellente!) and green and red lasagne in cheese
(noodles (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served everywhere!) and meatballs,
casually heaped onto pasta shad and unusual coffee (excellente!)
(including noodles (oodles of delicious linguini) elegantly served
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious lasagne and meatballs, casually heaped
onto pasta sauce (including noodles) gotten usually in northern italy)
elegantly served) gotten usually in northern italy and standard pasta
and shad and unusual coffee (excellente!) (that's all!) and shad and
unusual coffee (excellente!) (that's all!) always good, hot
particularly (twist, then ingest) heated al dente and unusual choice
of fine flavors particularly espresso (excellente!) and green and red
lasagne in cheese (noodles (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served everywhere!) and meatballs, casually heaped onto
pasta and sauce (that's all!) and sauce (that's all!) and shad and
unusual coffee (excellente!) (that's all!) sauce (including noodles
(oodles of delicious linguini) elegantly served (oodles of delicious
lambchops (including noodles) gotten usually in northern italy)
elegantly served (oodles of delicious lambchops (including noodles)
gotten usually in northern italy) elegantly served) gotten usually in
northern italy and standard pasta and shad and unusual coffee
(excellente!) (that's all!) and standard pasta, always good, hot
particularly (twist, then ingest) heated al dente and unusual coffee
(excellente!) (that's all!) and spaghetti, heated al dente and unusual
coffee (excellente!) (that's a lucious lunch) and standard pasta,
always good, hot particularly (twist, then ingest) heated al dente and
unusual coffee (excellente!) (that's a lucious lunch) always good, hot
particularly (twist, then ingest) heated al dente and unusual choice
of fine flavors particularly espresso (excellente!) and green and red
linguini and sauce and garlic (noodles everywhere!) and meatballs,
casually heaped onto pasta sauce (including noodles) gotten usually in
northern italy and spaghetti, heated al dente and unusual coffee
(excellente!) and green and red lasagne in cheese (noodles (oodles of
delicious linguini) elegantly served everywhere!) in cheese (noodles
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious lambchops
(including noodles) gotten usually in northern italy) elegantly served
(oodles of delicious linguini) elegantly served (oodles of delicious
linguini) elegantly served everywhere!) and meatballs, casually heaped
onto pasta and shad and unusual coffee (excellente!) (that's a lucious
lunch) and sauce (that's all!) and spaghetti, heated al dente and
unusual coffee (excellente!) (that's a lucious lunch) and standard
pasta and sauce (that's all!) always good, hot particularly (twist,
then ingest) heated al dente and unusual coffee (excellente!) (that's
a lucious lunch) shad and unusual choice of fine flavors particularly
espresso (excellente!) (including noodles (oodles of delicious
linguini) elegantly served (oodles of delicious linguini) elegantly
served (oodles of delicious linguini) elegantly served (oodles of
delicious linguini) elegantly served (oodles of delicious lasagne and
meatballs, casually heaped onto pasta sauce (including noodles) gotten
usually in northern italy) elegantly served (oodles of delicious
lambchops (including noodles) gotten usually in northern italy)
elegantly served (oodles of delicious linguini and sauce and garlic
(noodles everywhere!) and meatballs, casually heaped onto pasta sauce
(including noodles (oodles of delicious linguini) elegantly served)
gotten usually in northern italy) elegantly served (oodles of
delicious lambchops (including noodles (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served
(oodles of delicious lasagne and meatballs, casually heaped onto pasta
sauce (including noodles) gotten usually in northern italy and shad
and unusual coffee (excellente!) and garlic (noodles everywhere!) and
meatballs, casually heaped onto pasta sauce (including noodles (oodles
of delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served
(oodles of delicious linguini and sauce and garlic (noodles
everywhere!) and meatballs, casually heaped onto pasta sauce
(including noodles) gotten usually in northern italy and spaghetti,
heated al dente and unusual coffee (excellente!) and garlic (noodles
(oodles of delicious linguini) elegantly served everywhere!) and
meatballs, casually heaped onto pasta sauce (including noodles (oodles
of delicious linguini) elegantly served (oodles of delicious linguini)
elegantly served) gotten usually in northern italy) elegantly served)
gotten usually in northern italy)

Man, I'm stuffed!

A talk based on this code was given as in instructional / fun exercise at Clojurepalooza at OpinionLab in May of 2018.

Painting and Time

Tue, 29 May 2018 00:00:00 +0000

Painting is quite old and hard to fool. –Tad Spurgeon

In Living Craft, Spurgeon calls out time as the first "material" used to make paintings. He speaks of "extra-logical," emotional time, time experienced by the mind of the painter, rather than purely physical time measured by clocks. His remarks frame an invitation to contemplate how one experiences time in the studio and what time means to painters (and, by extension, viewers).

There is no question that paintings take time to make, sometimes a lot of time… from minutes to hundreds or thousands of hours.¹ Much contemporary activity seems to be oriented around saving time, but a single painting can take the kind of time jobs take, or marriages, or kids². To me, this slowness is part of painting's appeal, one of the reasons that it remains relevant today (as an antidote, at least). Becoming a painter has meant, for me, learning how (and why) to spend that kind of time at the easel… on many paintings at once, switching them as drying times and inspiration require, or on just one at a time.

Paintings connect the present moment of painting (or viewing) with various points in the past and future. This moment, this tip of this brush containing this much of this color and this consistency of paint with this precise location on this surface, and this breath, this experience of the image reaching the eye and brain, all relate to:

[ $-$ (0-8) hours ] the moments just prior to this one; the wet paint just put down, to be blended into, left alone, rubbed out, scraped off.
[ + (0-8) hours ] the moments about to come, in this session; future marks that will be blended into this one; other manipulations about to come, including removal if the current experiment is judged a failure.
[ $-$ (days, months, years) ] previous paint layers. Ground, underpainting, impasto… existing layers are like sedimentary rocks whose geology tells the story of what the mind and hand were doing during earlier sessions, recording course corrections, adjustment of values and edges, and the evolution of emotion, content and intent. The strata laid down previously are always in the process of being covered or exposed by the current brush strokes, knife application, scraping, sanding. These layers also show things (generally, but not always, mistakes) that you couldn't see previously, but that suddenly become, sometimes painfully, clear.
[ + (days, months, years) ] the painting as it slowly dries, and paint layers to come. Thoughts of translucency and opacity, of optical mixing, of glazing and scumbling. Where the painting might go. Anticipation of coming successes and failures.
[ $-$ (days, months, years) ] past drawings or other studies you made for this painting. Other sketches or paintings you made which led you down your current path.
[ + years ] future paintings you will (hopefully) make. The evolution of your thinking, themes, urges, longings; mistakes you will no longer make; new territories that will open up, that you can't even see yet.
[ $-$ (years, decades) ] teachings you've received from teachers and mentors.
[ + (years, decades) ] paintings to be made by students you taught / may teach or people you've inspired / may inspire.
[ $-$ (years, centuries) ] paintings made in the past that you've seen / liked / hated. All the Great Masters; others, in your personal pantheon of artists. The first artworks you ever saw, or made.
[ $-$ millennia ] the earliest paintings made. Cave paintings, paintings from ancient civilization.
[ $-$ ($10^0$ - $10^2$) years ] the production, storage, transportation and cultural context of the raw materials that find their way into your paintings: panels, stretcher bars, linen, linseed oil, pigment, brush fibers, bugs, hair, etc. I have panels prepared by my grandfather and recovered from his woodpile after he died; a handmade cherry-wood palette I got off Instagram that I cherish; and a few kilos of powdered manganese blue (PB-33) that I stockpiled because nobody is producing the pigment or paint anymore.
[ $-$ ($10^9$ - $10^{10}$) years ] the history of the substances (atoms/molecules) composing those materials: the annual growth and harvest of flax from fields in Europe; slow reactions involving iron in the earth's crust; the early history of the planet's formation; formation of heavy elements such as cadmium, cobalt, and lead in supernova nucleosynthesis; the creation of lighter elements occurring the early universe after the Big Bang.
[ + (days, years, centuries) ] the fate of your work on the walls of someone's house / palace / fortress / vault / museum / bomb shelter (that is, if anyone keeps it around). Happenstance viewings in various circumstances by future generations.
[ + ($10^{??}$ years) ] the atoms from your paintings, after they are burned / rotted / crushed / vaporized /….

Every moment of painting is in dialogue with all these other moments and time scales. This conversation of material and mind is far greater than any one person, but it is available while you work, or look, right now.

:Ὁ βίος βραχύς,

ἡ δὲ τέχνη μακρή,

ὁ δὲ καιρὸς ὀξύς,

ἡ δὲ πεῖρα σφαλερή,

ἡ δὲ κρίσις χαλεπή.

Vita brevis, ars longa, occasio praeceps, experimentum periculosum, iudicium difficile.

(Life is short, and art long, opportunity fleeting, experimentation perilous, and judgment difficult.³)

They also take time to get to, and to digest once you get a chance to see them.

For an example, see Ivan Albright's painting, That Which I Should Have Done I Did Not Do (The Door), which took ten years to complete.

https://en.wikipedia.org/wiki/Ars_longa,_vita_brevis

Contact Me

Sat, 10 Mar 2018 00:00:00 +0000

Physical

John Jacobsen
NPX Designs, LLC.
5341 S. Cornell Ave #3
Chicago, IL 60615

Logical

john@johnj.com

Instagram: @eigenhombre

Mastodon: @eigenhombre@mastodon.art

GitHub: eigenhombre

Numeric

+1 312 436 1522 (office/mobile/home)

Learning Muscular Anatomy

Thu, 22 Feb 2018 00:00:00 +0000

Skeleton and muscle overlay study (underlying photo by Pixiduc)

A knowledge of the human form is vital for figurative artists. A detailed understanding of anatomy is especially helpful when working from life or constructing the figure from imagination.

Various approaches have been taken to learning artistic anatomy. While I'd rather not rob graveyards and do my own dissections, there are, fortunately, excellent anatomy books[1] available for the artist.

I have found the muscles harder to learn than the skeleton, so I developed the method outlined here, which involves reading, data entry, and memorization. Other methods include drawing muscles on tracing paper over figure drawings or photographs, or sculpting muscles atop a pre-formed skeleton. This practice supplements, but does not replace, those other methods.

A few key facts can be helpful to learn, for each of the important muscles (< 100 of them, roughly):

Muscle name
Origin of muscle (attachment point, stationary during contraction)
Insertion of muscle (attachment point, moves during contraction)
Function: what the muscle does
What the muscle overlaps
What the muscle is overlapped by.

These last two points can be helpful in constructing a mental model of the muscular forms and of the entire form they help to create.

I use the Anki flash card system as a tool for memorization. Anki uses a carefully designed model to set the timing of when you see individual cards, showing you ones you got wrong sooner, and gradually showing them less and less often as you learn.

For all the muscles you want to study, you can enter these key facts into a table, and then export the table into flashcards. The original Org file can be kept in Git under source control and shared with others, and presents a convenient overview of the entire corpus of data. Merely entering the data in the first place starts the memorization, and then each table field can be paired with the muscle name both "forwards" and "backwards" (fact->muscle and muscle->fact) as separate flashcards. Thus, for, say, 100 muscle names and the associated five facts per muscle, one has 100 x 5 x 2 = 1,000 flashcards.

Example table in Org Mode

There are Anki decks for muscular anatomy already available, but they tend to be large in size and have a medical rather than artistic context; also, finding and summarizing the information required to create the cards helps me learn.

I wrote a tiny Clojure program (GitHub repository here) to facilitate exporting the Org data into Anki, which accepts tab-separated lines. I run it whenever I add new muscles into my table.

Since Anki synchronizes with my phone, I can review whenever I have a short break. If I enter a few muscles a week, the data entry should roughly keep up with my memorization practice, and will then have all the relevant muscles memorized in well under a year with minimal daily effort… perhaps five minutes per day.

Sample program output

$ lein run
Function of Sartorius	flexes, weakly abducts, rotates upper leg; flexes and medially \
rotates lower leg
Muscle w/ function flexes, weakly abducts, rotates upper leg; flexes and medially \
rotates lower leg	Sartorius
Origin of Sartorius	Just below tip of ASIS
Muscle with origin 'Just below tip of ASIS'	Sartorius
Insertion of Sartorius	Pes anserinus, on tibia
Muscle with insertion 'Pes anserinus, on tibia'	Sartorius
Sartorius overlaps...	Gracilis, Rectus Femoris, Vastus Medialis, Adductor Longus
Gracilis, Rectus Femoris, Vastus Medialis, Adductor Longus are overlapped by...	Sartorius
Function of Deltoid	flex arm forward, abducts outward, pulls backward
Muscle w/ function flex arm forward, abducts outward, pulls backward	Deltoid
Origin of Deltoid	lateral 1/3 of clavicle, acromion process, spine of scapula
...

… and a sample flash card (top portion only is shown initially, with the answer below shown after a tap or keypress):

Example Anki flash card

Workflow

Clone the GitHub repository and cd to the new directory;
Edit resources/muscle-structure.org;
Export Org file to CSV (org-table-export) in the same directory;
Run the program and save the output: lein run > cards.txt;
Import cards.txt (removing any previously-imported entries first) into Anki;
Memorize away!
Repeat as new muscles are added.

Conclusion

After a few weeks of minimal effort (a lot less than my previous year-long project), I've added half a dozen or so muscles so far and have them pretty well down. Since the overlapping forms are included, one starts to flesh out one's three-dimensional understanding of the figure from the outset. (I recommend starting with the largest muscles first if you're not as familiar with them – e.g. pectoralis, gluteus maximum, rectus abdominis, etc.). This method is a bit personal, Emacs, Org Mode, Clojure, etc. not being standard artists' tools, but could be adapted to e.g. Excel or CSV files, or other programming languages, with minimal effort.

[1] See, for example, Human Anatomy for Artists: The Elements of Form by Elliot Goldfinger.

Bio

Fri, 24 Nov 2017 00:00:00 +0000

I grew up¹[some old pictures of me]]] in Madison, Wisconsin and studied art and physics at the University of Wisconsin — Madison, receiving my doctorate in physics there in 1996 under Prof. Francis Halzen. I also worked for a year at CERN in Geneva, Switzerland, under Prof. Sau Lan Wu on the ALEPH experiment.

My doctoral work and postdoc focused on the AMANDA neutrino detector constructed at the Geographic South Pole, and the subsequent, much larger IceCube Neutrino Observatory. My physics career is described in more depth in my Curriculum Vitae; the IceCube project is described quite engagingly and at length by Mark Bowen in his book, The Telescope in the Ice: Inventing a New Astronomy at the South Pole.

While in graduate school I also studied painting, drawing and printmaking in the UW-Madison graduate art program, mostly with Jack Damer, Richard Long and David H. Becker.

In the 21st century I have been a software engineer and painter. I travelled to the Geographic South Pole ten times for AMANDA and IceCube, going at first as a physicist, and then as a software consultant (blog posts here). Some of my early paintings were described in William Fox's book, Terra Antarctica: Looking Into the Emptiest Continent.

Since 2015 I have maintained a regular studio practice focusing on figurative art based on observation and imagination. This effort has involved some re-training, mostly in the form of workshops at the Palette and Chisel Academy of Fine Art with Andrew S. Conklin, Stephen Assael, and Larry Paulsen. My exhibition history is here, while recent work and work in progress are shown on my Instagram and Tumblr feeds.

Just for fun, here are [[./more-pix-of-me.html

Reflections on a Year of Daily Memory Drawings

Mon, 18 Sep 2017 00:00:00 +0000

365 Drawings

The sketchbooks shown below are the result of doing one drawing from memory every day for a year, a project I finished this month.

A year's worth of Daily Compositions

This drawing practice was based on the Daily Composition, which, as I explained last January, is an exercise from The Natural Way to Draw, by Kimon Nicolaïdes (published posthumously in 1941). The requirements for this exercise are as follows:

See things and events that interest you throughout the day;
Remember them, and
Capture them on paper, in the form of a single image …
… Once per day.

Each of these is challenging in its own way, and each reinforces the other. Maintaining the practice over a full year taught me a number of lessons.

Example drawing: 54th and Cornell

Lesson 1: The limits of visual memory

I’d struggled with this exercise now and again over the years, and in taking it up again I was immediately reminded just how little I remember of what I see throughout the day.

The mind is able to recognize, and, to a greater or lesser extent, to respond mentally and emotionally to a stunning variety of objects and occurrences in the world around us. This happens automatically and without effort. But this sort of ordinary seeing is not the same as being able to recall and to visually reproduce the things we "see."

To demonstrate this to yourself, try to draw your spouse's face, or a friend's, from memory. This is a person you would have no trouble recognizing, even at some distance, yet you don't really have access to how your brain stores that information. Now try to draw your car, or bicycle, or the front of your building. These are things you know well, yet you probably cannot describe them visually in much detail.

Doing this exercise faithfully made this fact painfully obvious. Each day, out of the millions of details that passed in front of my eyes, I was left grasping only a handful of facts to work with at day's end. (The number of memories was small, but I was surprised that it was never zero.)

Contrast this with the capabilities of a camera: it faithfully records the details it is exposed to, millions of bits of data per shot. But the camera, a mere machine, cannot recognize or react emotionally to those details¹.

Adams St. Bridge

Lesson 2: Trying to remember amplifies seeing

It would be easy to think of the Daily Composition exercise as a way of learning how to do a camera's job: you are exposed to scenes, and later you record them on paper. I did find myself, in fact, taking mental "pictures" most days, of a person or group doing something, in some specific place. I would see a person getting into a cab, or construction workers huddled over a hole, or (one day) a unicycle rider cresting a small hill, and I would think, "that's it!" and try to remember, for that evening's drawing, everything I could about the moment I'd just seen.

During the course of the year, I didn't actually get that much better in terms of the sheer number of details I could recall. A half dozen "facts" (the size and shape of a doorway; the size and number of windows in a storefront; the particular shape of a woman's boots), along with a rough gesture for and/or placement of the key figures, were about as I could expect for any given day's drawing.

A much bigger change occurred in my ability to notice things, and my emotional response to them was amplified. Knowing that I needed material for a drawing at the end of the day simply opened my eyes more. It was like going on vacation somewhere totally new. I saw things in my neighborhood, and on my commute, that I had passed by without noticing for years: the shapes and placement of signs, light poles, fire hydrants; the construction and weathering of steel supports for elevated trains; the multitudes of ornamental details on buildings. The endless variety of people's faces… commuters, shopkeepers, construction workers, homeless men and women; the smiles, grim stares (especially at commute times) or lines of care they bore; their postures and gestures (all too often, in relationship to their phones); the cut and fit of their clothing, coats, boots or shoes. The endless repetitive movements of pigeons, and the surprising variety of other birds (sparrows, hawks, cranes, herons, geese, ducks, swallows, and several migratory bird species I couldn't identify). The printed word, in steel or brass relief, to be seen everywhere in streets and sidewalks, indicating lines of survey, or the origin or builders of utility covers or nearby buildings. All the endless varieties of trees, contours of trunks, branching patterns, leaves, and all the grasses, flowers and weeds of the city… these things came to my notice, were revealed fresh as though I had previously had some sort of screen over my vision that had fallen away.

Part of seeing these things became caring about them. In that regard I became less, not more, like a camera.

Promontory Point

Lesson 3: Emotion aids recall

If I saw something I reacted to emotionally, it was usually a lot easier to recall later. This seems obvious, but to live the connection between emotion and memory as part of one's daily routine for a year had a cumulative effect. This is what I meant when I said in my last post that I felt a stronger connection to Chicago and my surroundings. I suspect that building these sorts of connections to daily life is one of the main reason Nicolaïdes recommends the exercise as a daily practice.

South Loop Mural and Smoker

Lesson 4: Other tricks for remembering

Aside from focusing on things that give an emotional charge, here are some more tricks I used to remember things I saw:

Practice drawing or painting something while you look at it. This includes making movements in the air as if you were drawing, building a little actual muscle memory (and inviting curious glances from passersby).
Re-remember things. If I saw something I wanted to "save up" for the evening's drawing, then whenever I saw something else that looked interesting, I would look carefully at it, but then I would use the whole noticing-something-noteworthy experience as a reminder to remember the other thing I was trying to remember also. In other words, every new thing I noticed became a reminder to remember the other thing. Occasionally, I would build a list of things to remember and would then go through the list items one-by-one whenever this occurred.
De-distract and focus. Of course, sometimes the above tricks got to be too much; at those times, I'd try to minimize my visual input by looking down at the sidewalk if safe to do so (allowing my peripheral vision or looking up when needed, in order to keep from getting hit by buses, etc.), just making the effort to hold the memories in my mind while I walked or ran.

Weed whacking on the Lakefront

Lesson 5: Invention and knowledge help fill in details

Given a relative paucity of remembered facts, how do you complete your drawing? Just as our mind fills in details about what we see without our being consciously aware of it, one can invent the elements of a scene that elude our grasp when we try to recall them. For this, a knowledge of perspective, of human anatomy and clothing, of the construction of builings and plants and trees, and especially of light and shadow are indispensible for adding extra life and believability to a composition, no matter how quickly executed. For example, I would rarely remember a specific shadow, but if I knew where I was and what time it was, I could place the shadows based on where the sun would have been at the time. Used skillfully, invention can also amplify whatever emotional response led one to record the scene in one's mind to begin with.

I think I got slightly better at "filling in the gaps" this way over the year. The exercise does seem to strengthen the connection between memory, knowledge, and imagination.

Dearborn subway entrance

Lesson 6: Drawing can distort memory; revisiting a place helps

I often noticed that, after drawing something that occurred in a specific place, my memory would be strongly affected by whatever inventions (or mistakes) I came up with to compensate for my ignorance of the drawing's setting. For this reason, I sometimes disliked looking at the drawings after the fact because they were so clearly wrong and they interfered with my "real" memory of where the event occurred. An easy remedy for this was just to go back to the place in question and look around. One sees many more details the second time around, and it's always amusing and humbling to realize how wrong you got it the first time.

On a related note, I find that taking and reviewing photographs shapes my memory as well. I remember things differently, and more emotionally, having drawn them, either from life or from memory, than I do when I've photographed them. I sometimes limit my photography when I travel for this reason (although I still take a ton of pictures).

North Avenue Apple Store

Lesson 7: Don't make it a big deal

To do the exercise three hundred and sixty five days in a row did require commitment, since I have a busy schedule and was generally doing it at the end of the day when my willpower budget and my ability to focus were all but spent.

Nicolaïdes makes a point of not making a big deal of these drawings, setting a fifteen minute time limit for each. I broke this rule many times, especially at the beginning when I would try to recall as much as possible. Eventually, however, I settled for just jotting down a few things I remembered seeing (sometimes starting with just remembering where I was and what I did, which could be hard enough), and turning the most compelling memory into a small composition. Even on days when I was the most busy and distracted I could typically remember at least one thing to put down on paper. And I found there was only a loose correlation between how much effort I put into remembering things throughout any given day, and how much I liked the drawing at the end of that day. Some days I was excited about the results, and some days I was just happy to put the drawing away afterward. Part of what's good about doing something like this is you can fail over and over again, and your daily commitment to the practice means that any specific failure is no big deal.²

Cyclists kissing at The Point

Lesson 8: More can be better

At least half the drawings are so bad, I find them pretty painful to look at now. Many of them show an interesting tidbit or idea worth developing further. And a small fraction, less than 10%, I enjoy looking at and find appealing on their own terms. Some of these are shown here.

The few successes illustrate an important rule that I still struggle with: with art making, sometimes quantity wins. I prefer to work for a long time on paintings, but it is nice to complete something every day and, occasionally, to have it succeed. No matter who you are, I suspect if you make a drawing a day for a whole year, at least a few of them will suprise and please you.

Randolph and Wabash

Lesson 9: It's about composition

Composition is about choosing what goes in a picture and arranging those things in a pleasing design which communicates one's ideas or intentions. One reason I took up this practice is that I find inventing compositions (e.g., in the studio) more challenging than finding them directly in front of me. Ultimately, memory and rendering are only part of the practice… the rest (perhaps the biggest part) is design. Nearly four hundred drawings later I still find it challenging, but solving a specific design problem every night has made the whole problem of composition seem tractable in a way it didn't before. This, I imagine, was another reason Nicolaïdes emphasized this exercise.

Planting Bulbs

Moving forward

It will be helpful if you keep this up for a year, and it will be twice as helpful if you keep it up for two years. The really serious student will make a quick composition every day for the rest of his life despite everything else he has to do. –Nicolaïdes

In later chapters of The Natural Way to Draw, variations on the Daily Composition are introduced, including a greater amount of invention, and revisiting the same scene over several days in an extended composition. For myself, I wanted the challenge of working primarily from memory for a year, but have increasingly been craving more time for both invention and for working from life. I intend to continue regular compositional practice, but more as a natural and playful part of my creative rhythm, rather than as a tightly-defined daily practice. I don't know where it will lead, but look forward to finding out.

More Drawings

Here are a few more drawings from the year.

Shambhala Center

54th and Cornell

Van Buren

Truck, alley off Van Buren

March for Science, Columbus Drive

Hipster cyclist crossing Metra tracks

Stephen Assael class demo

Printers' Row Book Fair

55th St.

Cement mixer, Lake Shore Drive under McCormick Place

63rd St. Beach

Checking phone, near Oak St.

Some machine learning systems have gotten pretty good at recognizing specific objects in a digital image; thankfully, none of these has emotions yet.

The ability to fail and to start anew, over and over again, is the most important creative skill that I know of.

Repainting

Sun, 14 May 2017 00:00:00 +0000

Painting in Progress - from imagination

As I return to the painting on my easel, reworking both the major figures and probably everything else as well, I am put to mind of a lesson by a former teacher, Carol Pylant. She told me to be paint over the edges of my foreground figures when working on backgrounds, so the backgrounds don’t get too “stiff.” Of course, this applies not just to edges. “If you painted it once, you can paint it again,” she said.

There are some variants to this last phrase which come to mind:

If you painted it once, and are afraid you won’t be able to paint it as well or better, you probably need more practice.
If you painted it once, and had fun doing so, you can have fun doing it again.

I think a major source of strength in a drawing or a painting (or a computer program, for that matter) is the ability of letting go of work completed so that better work can emerge.

This was vividly reinforced for me when I watched Steven Assael paint for two days in February. He built up beautiful passages again and again and destroyed them… mostly. Traces remained, in the layers of marks and paint and texture. Same thing, last month, watching him draw the figure, and encouraging students to make radical changes even in the last few minutes of a pose… destroying what was there to open a path for something much better to emerge. As time goes on I can see that opening that path over and over again is absolutely critical for successful work. Destruction is often a prerequisite for successful creation.

Daily Memory Drawings

Tue, 03 Jan 2017 00:00:00 +0000

Riverside Plaza Memory Sketch

Every day for the past three months I have drawn some scene or event that I saw that day, purely from memory.

Usually this is something I saw on the way to work, or, if I'm home that day, something I saw when out for a run or a walk somewhere in the neighborhood.

Since I usually make the drawing at the end of the day, several hours will have passed by the time I get around to recording what I saw. To keep this purely about memory (with bits of invention as needed), I don't allow myself any notes or photo references. As a result, while going about my day, I need to work extra hard to have a chance to remember the things I see. Sometimes I'll stop on the street and spend extra time looking at something, perhaps making little drawing gestures in the air to build a little muscle memory about the shapes, and no doubt drawing a stare or two from passersby. Other times if I've seen something transitory while walking I'll look down at the sidewalk and go over the scene in my mind with minimal distraction from new visual inputs.

Throughout the day, if I think of it, I'll try to recall one or two things I've seen to increase the chance of recall later.

When I sit down to draw, I first make a list of all the things I can remember seeing, sometimes accompanied by small thumbnail sketches.

Details remembered from Walk in Hyde Park

Then I choose one thing as the subject of a larger, 15- to 20-minute drawing, usually 6"x8".

Van Buren Barber Shop

A Capella Band at Metra Entrance

While the drawings are usually not particularly good (and definitely not "finished" by my usual standards), doing this exercise religiously for awhile has had a few noticeable effects:

The exercise forces me to get out of the house at least once per day, no matter my schedule or the weather.
The drawings seem to be getting a little better, and a little more complex, on average.
I can remember more at the end of the day, often enough to fill two pages of notes and drawings.
Things I see throughout the day are more vivid and can pack more emotional punch. I feel a stronger connection to my surroundings, the city of Chicago in particular.
Imagining or inventing things, coming up with ideas for drawings or compositions, seems easier.

I have been appreciating the last two items especially.

I don't have a great memory, so this exercise sometimes feels impossible, and involves a fair amount of staring off into space. Like writing down dreams, however, once I get going, I tend to remember more than I thought I would.

This activity is, with a minor variation or two, the "Daily Composition" exercise proposed by Kimon Nicolaïdes in The Natural Way to Draw. Without giving much explanation, he asserts,

No other exercise in the book is more important than this one. These compositions do not have to be right. They can be all wrong. The important thing is to do them — three hundred and sixty-five of them between today and this date next year. It will be helpful if you keep this up for a year, and it will be twice as helpful if you keep it up for two years. The really serious student will make a quick composition every day for the rest of his life despite everything else he has to do.

I first read about this exercise in the 1980s. How I wish I had followed that advice consistently since then! Nevertheless, when I go back and look at old sketchbooks and discover my occasional Daily Compositions, no matter how bad the drawings, I do feel the strings of memory vibrating strongly.

I think this exercise is hard to do consistently because, at first, there isn't much to work with… nothing to copy from. A newcomer to the exercise will struggle when all he or she has to work with is a few wispy clouds of memory and spotty knowledge of lighting, anatomy, weather, architecture, costume, etc. But working on one's weak spots is a great way to improve. In particular, when I go back to a spot I've tried to draw, I notice many things I missed the first time. It's fun to re-draw those things, armed with the extra knowledge.

It's also nice to have a fairly clear idea of what to draw at least once per day, without worrying about making it anything complicated or clever. (Generally one of the day's memories stands out clearly as the best subject for a drawing.)

All art considerations aside, this exercises exposes the vast extent of my ignorance of the simplest things I encounter every day. The process of learning to notice my environment more feels like a healthy (albeit upstream) direction to take in a world where constant distraction by one's smartphone has become the norm.

Since starting this exercise I've looked around for other references to working from memory. It seems that the practice was more common in the 19th Century. Degas, in particular, and Whistler used memory extensively in their work. More on this, perhaps, in later posts. In any case, it seems to me that the practice is not given its due in contemporary teaching and discourse (I don't remember any discussion of it in art school).

What things that you see every day are worthy of your repeated attention and study… or even memorization?

Van Buren Memory Sketch

Questions to Ask

Thu, 25 Feb 2016 00:00:00 +0000

Questions to Ask

… of a painting in progress, with strategies for recovery.

Diagrammatic landscape sketch made "en plein air" at the Geographic South Pole

I’ve been learning a lot lately by studying books, blogs, and doing studies from life. It’s made me reflect upon my past strategies (or lack thereof) for making artworks.

These are roughly in order of importance, in that if the answer to a previous question is, “no,” it’s probably not worth spending too much time on the current question.

Is the concept, content or intent clear in your mind, and is it what you want?

If not, go back to the drawing board:

make studies from life;
generate ideas in sketchbooks;
look to other artists’ work for inspiration;
look to previous successes and failures to see what the next step is.
Consider destruction, deferral, or radical transformation of the current piece.

Does the overall design make sense?

Large Shapes and Edges
Composition
Perspective
Arrangement of objects in the illusionistic space of the piece

If not, fix at the level of drawing (shapes/outlines/values). Also ask, Is there anything that could be removed to make the work stronger?

Do the values communicate the right forms well?

If not, fix at the level of drawing (values). Check:

Light sources and the nature of any reflections (specular vs. diffuse)
Accuracy / illusion of form
Level of detail

Do the colors work in harmony with each other and with the other concerns in the painting?

If not, repaint one or more areas, maintaining value but shifting hue.

Do the marks and surface quality relate to the intent/energy/mood? Are they interesting?

No matter how “objective” the intent, at some level all paintings and drawings are abstract. They should be beautiful at any distance or level of “magnification.”

If not, re-work those areas, using strategies of sanding, scraping, erasure, or starting over.

Macro-writing Macros

Wed, 25 Nov 2015 00:00:00 +0000

… in which we explore the power of macros, and macro-writing macros, to DRY out repetitive code.

I’ve been writing Clojure code full time for nearly two years now. I have a pretty good feel for the language, its virtues and its faults. Mostly, I appreciate its virtues (though I still wish the REPL started faster).

For me one of the language’s attractions has always been that it’s a Lisp — a “homoiconic” language, i.e., one defined in terms of its own data structures. Homoiconicity has one primary virtue, which is that it makes metaprogramming more powerful and straightforward than it is in non-homoiconic languages (arguably at some cost to readability).

In Lisp, this metaprogramming is accomplished with macros, which are functions that transform your code during a separate stage of compilation. In other words, you write little programs to change your programs before they execute. In effect, you extend the compiler itself.

I run a Clojure study group at work and find that it can be hard to explain the utility (or appeal) of this to newcomers to Lisp. This is partly because macros do things you can’t easily do in other languages, and because the things you want to do tend to relate to abstractions latent in a particular codebase.

While playing around with 3d rendering in Quil, I recently came across a use case that reminded me of the following quote by Paul Graham:

The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I’m using abstractions that aren’t powerful enough— often that I’m generating by hand the expansions of some macro that I need to write¹.

In Quil, there are multiple situations in which one needs to create a temporary context to carry out a series of operations, restoring the original state afterwards:

Save current style with push-style; change style and draw stuff; restore previous style with pop-style.
Start shape with begin-shape; draw vertices; end-shape to end.
Save current position/rotation with push-matrix; translate / rotate and draw stuff; restore old position/rotation with pop-matrix.

Here’s an example:

(push-matrix)
(try
  (push-style)
  (try
    (fill 255)
    (no-stroke)
    (translate [10 10 10])
    (begin-shape)
    (try
      (vertex x1 y1 0)
      (vertex x2 y2 0)
      (vertex x2 y2 h)
      (vertex x1 y1 h)
      (vertex x1 y1 0)
      (finally
        (end-shape)))
    (finally
      (pop-style)))
  (finally
    (pop-matrix)))

The (try ... (finally ...)) constructions may not be strictly needed for a Quil drawing, but it’s a good habit to guarantee that stateful context changes are undone, even if problems occur.

In a complex Quil drawing the idioms for saving style, translation state, and denoting shapes appear often enough that one hungers for a more compact way of representing each. Here’s one way to do it:

(defmacro with-style [& body]
  `(do
     (push-style)
     (try
        ~@body
       (finally
         (pop-style)))))

(defmacro with-matrix [& body]
  `(do
     (push-matrix)
     (try
        ~@body
       (finally
         (pop-matrix)))))

(defmacro with-shape [& body]
  `(do
     (begin-shape)
     (try
        ~@body
       (finally
         (end-shape)))))

The original code then becomes more compact and easier to read:

(with-matrix
  (with-style
    (fill 255)
    (no-stroke)
    (translate [10 10 10])
    (with-shape
      (vertex x1 y1 0)
      (vertex x2 y2 0)
      (vertex x2 y2 h)
      (vertex x1 y1 h)
      (vertex x1 y1 0))))

In this example code, the contexts with-matrix, etc. appear so often that the resulting savings in lines of code and mental overhead for the reader is substantial.

However, the astute reader will realize that the macro definitions themselves are pretty repetitive—in fact, they look almost identical except for the setup and teardown details (this kind of “context manager” pattern is common enough that Python has its own language construct for it).

I generally reach for macros when I have a pattern that occurs with obvious repetition that’s not easy to abstract out using just pure functions. Control abstractions such as loops or exception handling are common examples. (I find this situation occurs especially frequently when writing test code).

In any case, the solution for our repetitive macros could be something like:

(defmacro defcontext
  [nom setup teardown]
  `(defmacro ~(symbol (str “with-” nom))
     [~'& body#]
     `(do
        ~'~setup
        (try
          ~@body#
          (finally
            ~'~teardown)))))

Yikes! I have to admit I had to write a lot of macros, and also refer to this helpful page for reference, before I could write (and grok) this macro.

With defcontext in hand, our repetitive macro code just becomes:

(defcontext style (push-style) (pop-style))
(defcontext shape (begin-shape) (end-shape))
(defcontext matrix (push-matrix) (pop-matrix))

These are exactly equivalent to the three context macros (with-*) defined above.

With a little effort, it’s actually not too hard to construct such a nested macro. It’s largely a matter of writing out the code you want to generate, and then writing the code that generates it, testing with macroexpand-1 at the REPL as you go. This page by A. Malloy has a lot of helpful remarks, including this cautionary note: “Think twice before trying to nest macros: it’s usually the wrong answer.” In this case, I actually think it’s the right answer, because the pattern of a context with setup and teardown is so common that I know I’ll reuse this macro for many other things—we have effectively added one of my favorite Python features to Clojure in just a few lines of code[fn::To be even more like Python’s context managers, defcontext would want to enable the user to bind some local state resulting from the setup phase of execution (“ with x() as y: ” idiom); examples include file descriptors or database connections. This is left as an exercise for the reader.]

There’s a saying in the Clojure community: data > functions > macros. I’m a big believer in this. Clojure’s powerful built-in abstractions for wrangling data in all its forms make it the language I prefer above all others these days. But occasionally that means wrangling the data that is the code itself, thereby reaping the benefits in power, brevity and expressiveness.

Image generated by the Quil code used for this example.

Paul Graham, "Revenge of the Nerds": http://www.paulgraham.com/icad.html

Time Limits

Sat, 25 Apr 2015 00:00:00 +0000

Picking up the art blogging again after a long hiatus.

Recently I’ve been thinking about constraints a lot, and finding out how helpful they are. I think they are especially helpful for artists.

For example, since I started my new job last year, I am in the habit of taking photos on the way to or from work. I use the following constraints:

Use only the iPhone with no extra lenses (digital filters/manipulation OK, but only on the iPhone itself).
Picture must be taken on the way to or from work, or near the office.
Post the ones I like on Facebook, to this album ("Views from the Commute").

The pictures are not always awesome, but the practice itself is most satisfying. It’s a way to stay creative and visually-focused despite the demands of a “real” job. And I find the instant feedback I get from friends and family on Facebook more satisfying than many a gallery show has been.

Image from "Views from the Commute"

Another way I’ve been exploring constraints in the last few years is by painting outside. This practice has a number of constraints more or less built in: you can’t carry too much stuff around; it’s only light out for so long; and you typically paint the stuff you find in front of you (rather than stuff you make up).

Lately I’ve been redoing my Web sites and thinking about some old art projects as I sift through blog posts. I really enjoyed working with SketchUp awhile back and have been itching to return to that. So I’ve been wondering what sort of constraints might be helpful, similar to the ones I get “for free” by painting outside.

I decided on the following formula:

Four hour limit per painting:
- Hour one: make an image using SketchUp
- Hour two: transfer to panel and work on the drawing
- Third hour: underpainting
- Fourth hour: top layers / final painting.
Fifteen minute break in between hours
Use a Clojure program I wrote to help with the gridding of the drawing.

First stage, after one hour: exported from SketchUp and gridded

After second hour: drawing

After third hour: underpainting

Final image

Detail 1

Detail 2

Lessons:

Constraints help. I ordinarily have trouble finishing paintings. Though there were a few points when I thought things were totally off the rails and the work was going to be a disaster, overall I had a lot more focus (and fun!) than I often do while painting.
I originally planned hour three (underpainting) to be in acrylics; but I found I wanted to just jump in with oils, and it worked fairly well. Being flexible about the plan in this regard was good.
Having painting supports ready in advance was helpful; NOT having a concept or content queued up (for the SketchUp part) was also good, I think, though I might experiment with brainstorming content as a (timed) stage in the process.
SketchUp is a total rabbit hole. I could have burned the entire time just fooling around in SketchUp alone. Warning: dragons dwell there!

Lazy Physics

Thu, 12 Feb 2015 00:00:00 +0000

… in which we explore lazy sequences and common functional idioms in Clojure via the example of looking for (nearly) coincident clusters of times in a series.

A fundamental technical problem in experimental particle physics is how to distinguish the signatures of particles from instrumental noise.

Imagine a tree full of hundreds of sparrows, each nesting on a branch, each chirping away occasionally. Suddenly, for a brief moment, they all start chirping vigorously (maybe a hawk flew past). A clustering of chirps in time is the signal that something has happened! The analogous situation occurs in instruments consisting of many similar detector elements, each generating some amount of random noise that, on its own, is indistinguishable from any evidence left by particles, but which, taken together, signals that, again, something has happened —a muon, an electron, a neutrino has left a sudden spume of electronic evidence in your instrument, waiting to be read out and distinguished from the endless noise.

This process of separating the noise from the signal is known in physics as triggering and is typically done through some combination of spatial or time clustering; in many cases, time is the simplest to handle and the first “line of defense” against being overrun by too much data. (It is often impractical to consume all the data generated by all the elements —data reduction is the name of the game at most stages of these experiments.)

This data is typically generated continously ad infinitum, and must therefore be processed differently than, say, a single file on disk. Such infinite sequences of data are an excellent fit for the functional pattern known as laziness, in which, rather than chewing up all your RAM and/or hard disk space, data is consumed and transformed only as needed / as available. This kind of processing is baked into Clojure at many levels and throughout its library of core functions, dozens of which can be combined (“composed”) to serve an endless variety of data transformations. (This style of data wrangling is also available in Python via generators and functional libraries such as Toolz.)

Prompted by a recent question on the topic from a physicist and former colleague, I got to thinking about the classic problem of triggering, and realized that the time series trigger provides a nice showcase for Clojure’s core library and for processing lazy sequences. The rest of this post will describe a simple trigger, essentially what particle astrophysicists I know call a “simple majority trigger”; or a “simple multiplicity trigger” (depending on whom you talk to).

Now for some Clojure code. (A small amount of familiarity with Clojure’s simple syntax is recommended for maximum understanding of what follows.) We will build up our understanding through a series of successively more complex code snippets. The exposition follows closely what one might do in the Clojure REPL, building up successively more complete examples. In each case, we use take to limit what would otherwise be infinite sequences of data (so that our examples can terminate without keeping us waiting forever…).

First we create a sorted, infinite series of ever-increasing times (in, say, nsec):

(def times (iterate #(+ % (rand-int 1000)) 0))
;; Caution: infinite sequence...

(take 30 times)

;;=>
(0 955 1559 2063 2735 2858 3542 4067 4366 5246 5430 6168 7127 7932
 8268 8929 9426 9918 10436 10850 11680 12367 12569 13343 14155 14420
 15062 15171 15663 16355)

times is an infinite (but “unrealized”) series, constructed by iterating the anonymous function #(+ % (rand-int 1000)) which adds a random integer from 0 to 999 to its argument (starting with zero). The fact that it is infinite does not prevent us from defining it or (gingerly) interrogating it via take¹.

Now, the way we’ll look for excesses is to look for groupings of hits (say, eight of them) whose first and last hit times are within 1 microsecond (1000 nsec) of each other. To start, there is a handy function called partition which groups a series in blocks of fixed length:

(take 10 (partition 8 times))

;;=>
((0 955 1559 2063 2735 2858 3542 4067)
 (4366 5246 5430 6168 7127 7932 8268 8929)
 (9426 9918 10436 10850 11680 12367 12569 13343)
 (14155 14420 15062 15171 15663 16355 16700 16947)
 (17919 17949 18575 18607 18849 19597 20410 20680)
 (20737 21289 21315 21323 21426 21637 22422 23000)
 (23477 24351 24426 25106 25861 26568 27511 28332)
 (29071 29831 29957 30761 31073 31914 32591 33187)
 (33878 34739 34842 35674 36444 36960 36983 37400)
 (37587 38012 38969 39131 39317 40135 40587 40759))

We’ll rewrite this using Clojure’s thread-last macro ->>, which is a very helpful tool for rewriting nested expressions as a more readable pipeline of successive function applications:

(->> times
     (partition 8)
     (take 10))

;;=>
((0 955 1559 2063 2735 2858 3542 4067)
 (4366 5246 5430 6168 7127 7932 8268 8929)
 ...same as above...)

However, this isn’t quite what we want, because it won’t find clusters of times close together who don’t happen to begin on our partition boundaries. To fix this, we use the optional step argument to partition:

(->> times
     (partition 8 1)
     (take 10))

;;=>
((0 955 1559 2063 2735 2858 3542 4067)
 (955 1559 2063 2735 2858 3542 4067 4366)
 (1559 2063 2735 2858 3542 4067 4366 5246)
 (2063 2735 2858 3542 4067 4366 5246 5430)
 (2735 2858 3542 4067 4366 5246 5430 6168)
 (2858 3542 4067 4366 5246 5430 6168 7127)
 (3542 4067 4366 5246 5430 6168 7127 7932)
 (4067 4366 5246 5430 6168 7127 7932 8268)
 (4366 5246 5430 6168 7127 7932 8268 8929)
 (5246 5430 6168 7127 7932 8268 8929 9426))

This is getting closer to what we want—if you look carefully, you’ll see that each row consists of the previous one shifted by one element. The next step is to grab (via map) the first and last times of each group, using juxt to apply both first and last to each subsequence….

(->> times
     (partition 8 1)
     (map (juxt last first))
     (take 10))

;;=>
([4067 0]
 [4366 955]
 [5246 1559]
 [5430 2063]
 [6168 2735]
 [7127 2858]
 [7932 3542]
 [8268 4067]
 [8929 4366]
 [9426 5246])

… and turn these into time differences:

(->> times
     (partition 8 1)
     (map (comp (partial apply -) (juxt last first)))
     (take 10))

;;=>
(4067 3411 3687 3367 3433 4269 4390 4201 4563 4180)

Note that so far these time differences are all > 1000. comp, above, turns a collection of multiple functions into a new function which is the composition of these functions, applied successively one after the other (right-to-left). partial turns a function of multiple arguments into a function of fewer arguments, by binding one or more of the arguments in a new function. For example,

((partial + 2) 3)

;;=>
5

((comp (partial apply -) (juxt last first)) [3 10])

;;=>
7

Recall that we only want events whose times are close to each other; say, whose duration is under a maximum limit of 1000 nsec. In general, to select only the elements of a sequence which satisfy a filter function, we use filter:

(->> times
     (partition 8 1)
     (map (comp (partial apply -) (juxt last first)))
     (filter (partial > 1000))
     (take 10))

;;=>
(960 942 827 763 597 682 997 836 986 966)

((partial > 1000) is a function of one argument which returns true if that argument is strictly less than 1000.)

Great! We now have total “durations”; for subsequences of 8 times, where the total durations are less than 1000 nsec.

But this is not actually that helpful. It would be better if we could get both the total durations and the actual subsequences satisfying the requirement (the analog of this in a real physics experiment would be returning the actual hit data falling inside the trigger window).

To do this, juxt once again comes to the rescue, by allowing us to juxt-apose the original data alongside the total duration to show both together….

(->> times
     (partition 8 1)
     (map (juxt identity (comp (partial apply -) (juxt last first))))
     (take 10))

;;=>
([(0 309 410 562 979 1423 2180 3159) 3159]
 [(309 410 562 979 1423 2180 3159 3585) 3276]
 [(410 562 979 1423 2180 3159 3585 4325) 3915]
 [(562 979 1423 2180 3159 3585 4325 4573) 4011]
 [(979 1423 2180 3159 3585 4325 4573 5074) 4095]
 [(1423 2180 3159 3585 4325 4573 5074 5942) 4519]
 [(2180 3159 3585 4325 4573 5074 5942 6599) 4419]
 [(3159 3585 4325 4573 5074 5942 6599 7458) 4299]
 [(3585 4325 4573 5074 5942 6599 7458 8128) 4543]
 [(4325 4573 5074 5942 6599 7458 8128 8439) 4114])

… and adapt our filter slightly to apply our filter only to the time rather than the original data:

(->> times
     (partition 8 1)
     (map (juxt identity (comp (partial apply -) (juxt last first))))
     (filter (comp (partial > 1000) second))
     (take 3))

;;=>
([(1577315 1577322 1577514 1577570 1577793 1577817 1577870 1578151)
  836]
 [(3119967 3120203 3120416 3120469 3120471 3120620 3120715 3120937)
  970]
 [(6752453 6752483 6752522 6752918 6752966 6753008 6753026 6753262)
  809])

Finally, to turn this into a function for later use, use defn and remove take:

(defn smt-8 [times]
  (->> times
       (partition 8 1)
       (map (juxt identity (comp (partial apply -) (juxt last first))))
       (filter (comp (partial > 1000) second))))

smt-8 consumes one, potentially infinite sequence and outputs another, “smaller” (but also potentially infinite) lazy sequence of time-clusters-plus-durations, in the form shown above.

Some contemplation will suggest many variants; for example, one in which some number of hits outside the trigger “window” are also included in the output. This is left as an exercise for the advanced reader.

A “real” physics trigger would have to deal with many other details: each hit, in addition to its time, would likely have an amplitude, a sensor ID, and other data associated with it. Also, the data may not be perfectly sorted, some sensors may drop out of the data stream, etc. But in some sense this prototypical time clustering algorithm is one of the fundamental building blocks of experimental high energy physics and astrophysics and was used (in some variant) in every experiment I worked on over a 25+ year period. The representation above is certainly one of the most succinct, and shows off the power and elegance of the language, its core library, and lazy sequences. (It is also reasonably fast for such a simple algorithm; smt-8 consumes input times at a rate of about 250 kHz. This is not, however, fast enough for an instrument like IceCube, whose 5160 sensors each count at a rate of roughly 300 Hz, for a total rate of 1.5 MHz. A future post may look at ways to get better performance.)

To model a Poisson process — one in which any given event time is independent of the future or past times — one would normally choose an exponential rather than a uniformly flat distribution of time differences, but this is not important for our discussion, so, in the interest of simplicity, we’ll go with what we have here.

Fun with Instaparse

Tue, 12 Nov 2013 00:00:00 +0000

One of my favorite talks from this month’s excellent Clojure/conj was Carin Meier's presentation, which combined storytelling, live coding, philosophy, the history of computing, and flying robotic drones. She used the relatively new Instaparse library to create her own language to explore something called “Speech Acts” (which I won’t go into here, but do catch the video of her talk when it goes up).

My university work was in physics (and art) rather than CS, but I have long been interested in the implementation of programming languages, even going so far as to write a simple parser for Lisp-style math expressions in Pascal many years ago. Last year I had the opportunity to take the first “write a compiler in Python” class offered by David Beazley here in Chicago, in which we implemented a subset of the Go language. His PLY library is a great way to get started with implementing language parsers in Python, and the relative ease of doing so, compared with classic C implementations described in the infamous Dragon Book, inspired me to do some further experimentation of my own.

With this background, and inspired by Carin’s talk, I have been waiting for an opportunity to try out Instaparse, which is getting great press in the Clojure world. Instaparse takes a grammar as input (in the form of a string), and gives you a parser in the language specified by that grammar. It will also let you specify rules for transforming the resulting tree into something your Clojure program can use more directly (for example, by converting data types or removing unneeded elements from the parse tree).

When the need arose this weekend to read in Python configuration files into a Clojure program, I decided the time was ripe. I also wanted to document the journey using some form of literate programming. A library called Marginalia (Michael Fogus et. al.) made this pretty easy.

The results are on GitHub.

My impressions, after doing this project in just a few hours, are that (1) literate programming is great fun; and (2) Instaparse sets a new standard for power and expressiveness when converting structured text into abstract syntax trees. If you have a DSL or some other text-based, formal language you want to parse, and you are either literate in Clojure or interested in becoming so, Instaparse would be a great tool to check out.

Nucleotide Repetition Lengths

Sun, 03 Nov 2013 00:00:00 +0000

After a long hiatus, let’s continue our foray into amateur genomics with Clojure, in which we continue to encounter elegant expressions of Clojure’s power as well as a few additional wrinkles in the Clojure ecosystem (in this case, Incanter).

In our last post we upgraded our genome decoder and used it to show frequencies of the four nucleotides, illustrating Chargaff’s Rule.

Another simple question to be asked of the data is, What is the distribution of nucleotide repetition lengths? In other words, if we represent the number of times any nucleotide is repeated as a histogram, then AAGGCATTTT would yield two entries for 1, two entries for 2, zero for 3, and one for 4. What does this distribution look like for the entire human or yeast genome?

As a “null hypothesis”, consider the likelihood of getting $N$ A’s when drawing at random from A, G, C or T. The probability for the first selection to be A is 25%; for the first two draws to be AA the probability is $P(AA) = {1 \over {4^2}} = 0.625$, and for $N$ consecutive draws, $P(A \times N) = 4^{-N}$.

You can easily simulate (“Monto Carlo”) such a situation using a purely random sequence:

(defn randgenome []
  (repeatedly #(rand-nth [:A :G :C :T])))

This function yields a sequence like (:A :A :C :A :T :G :G :T :G :A :C :C ...).

To convert this into repetition lengths, we can first divide the sequence up into repeating elements using partition-by and the identity function:

(partition-by identity (randgenome))
;;=> ((:A :A) (:C) (:A) (:T) (:G :G) (:T) (:G) (:A) (:C :C) ...)

Extracting the lengths is then trivial: just map count over the sequences:

(defn get-lengths [s]
  (->> s
      (partition-by identity)
      (map count)))

(get-lengths (randgenome))
;;=> (2 1 1 1 2 1 1 1 2 ...)

(Note: running these functions unmodified will hang your REPL, since randgenome yields an infinite sequence! Wrap them with take if needed.)

Histogramming the data

The plots in my previous posts were made by the ~~simple~~ easy expedient of copying data into a Numbers spreadsheet on my Mac. We’d like to eventually do some more ambitious plotting, however, so let’s investigate a Clojure-based solution.

Incanter provides a broad set of Clojure-based utilities for data analysis, including matrices and linear algebra operations, statistical distributions, and so on. Built on top of JFreeChart, it implements several different kinds of plots, including histograms, and it should be an obvious candidate for displaying our length distribution histogram.

Unfortunately, because of two issues I discovered, Incanter will not work for our purposes. To make a long story short, it does not handle empty (zero) entries in histograms properly when displaying on a logarithmic scale, and it generates out-of-memory errors when given more than a million or so entries. (Coming from high energy physics, where such histograms are extremely common, these both seem like serious flaws to me, which I hope will be remedied in the next Incanter release.)

I wound up using the JFreeChart API directly, though this was very time consuming, as the API has approximately the same complexity as the entire US space program or the Large Hadron Collider. (In what follows below I elide details about false starts, descriptions of hair pulling, and expletives.)

First, I needed to convert the lengths from get-lengths into a histogram with arbitrarily many entries:

(defn make-hist
  "
  Convert seq of input xs into a histogram of nbins bins, from xmin to
  xmax.  Discard overflows or underflows
  "
  [xmin xmax nbins xs]
  (let [;; "base" histogram (zeros):
        zero-map (into (sorted-map)
                       (map (fn [x] [x 0]) (range nbins)))
        ;; get actual bin values for every input in xs:
        xbins (map #(int (* nbins (/ (- % xmin)
                                     (- xmax xmin))))
                   xs)
        ;; strip out undeflows & overflows:
        no-overflows (->> xbins
                          (remove #(< % 0))
                          (remove #(>= % nbins)))]
    ;; yield histogram as array of [ibin, height] pairs:
    (into [] (reduce #(update-in %1 [%2] inc) zero-map no-overflows))))

Supplying the minimum and maximum value ahead of time simplifies the histogramming code substantially and allows us to choose the same range for multiple plots, as will be shown below.

Armed with this transformation, the generation of the histogram plot is as follows:

(ns jenome.graphs
  (:import [org.jfree.data.xy XYSeriesCollection XYSeries]
           [org.jfree.chart ChartFrame JFreeChart]
           [org.jfree.chart.plot XYPlot]
           [org.jfree.chart.axis NumberAxis]
           [org.jfree.chart.renderer.xy XYBarRenderer StandardXYBarPainter]
           [org.jfree.chart.renderer.category]))

(defn trim-zeros 
  "
  Convert zeros (or negatives) to small positive values to allow for
  graphing on log scale
  "
  [vals]
  (map (fn [[x y]] [x (if (> y 0) y 0.0001)]) vals))

(defn draw-hist
  "
  Draw histogram of bins as generated by make-hist
  "
  [x-label values]
  (let [renderer (XYBarRenderer.)
        painter (StandardXYBarPainter.)
        series (XYSeries. [])
        blue (java.awt.Color. 0x3b 0x6c 0x9d)
        coll (XYSeriesCollection. series)
        y-axis (org.jfree.chart.axis.LogarithmicAxis. "Entries")
        plot (XYPlot. coll (NumberAxis. x-label) y-axis renderer)
        panel (JFreeChart. plot)
        frame (ChartFrame. "Histogram" panel)]
    (doto plot
      (.setBackgroundAlpha 0.0)
      (.setRangeGridlinesVisible false)
      (.setDomainGridlinesVisible false))
    (doto renderer
      (.setBarPainter painter)
      (.setPaint blue)
      (.setDrawBarOutline true)
      (.setOutlinePaint blue)
      (.setOutlineStroke (java.awt.BasicStroke. 1))
      (.setShadowVisible false))
    (doseq [[x y] values]
      (.add series (+ x 0.5) y))
    (.setLowerBound y-axis 0.5)
    (.setVisible (.getLegend panel) false)
    (doto frame
      (.setSize 800 250)
      (.setVisible true))))

I will omit a detailed explanation of how draw-hist works because I’m still not a JFreeChart expert (though if I do many more blog posts on this topic I may be forced to become one, however reluctantly). The principal difference with the Incanter implementation of histograms is that we provide our own set of bin heights and simply plot those. This follows a better separation of concerns anyways: our fairly simple binning function remains separate from the visual presentation of the bin positions and heights (we could, for example, add the counting of overflows and underflows – this is left as an exercise to the reader).

(trim-zeros exists to transform bin values of zero to small positive numbers to avoid taking the logarithm of zero, which is undefined.)

Armed with these tools, we can now make our first distribution:

(->> (randgenome)
     (take 1000000)
     get-lengths
     (make-hist 0.5 60.5 60)
     trim-zeros
     (draw-hist "Repeat Lengths, random hypothesis"))

Nucleotide repetition lengths for the random hypothesis

As expected by our analysis above, this shows the falling “power-law” distribution of randomly-occuring nucleotide repetitions. Compare this with the genome for the yeast S. Cerviciae:

(->> (genome-sequence yeast)
     get-lengths
     (make-hist 0.5 60.5 60)
     trim-zeros
     (draw-hist "Repeat Lengths, S. Cerviciae"))

Nucleotide repetition lengths for the yeast S. Cerviciae

And, for humans (here we did not wait for the result for the entire genome):

(->> (genome-sequence human)
     (take 10000000)
     get-lengths
     (make-hist 0.5 60.5 60)
     trim-zeros
     (draw-hist "Repeat Lengths, human genome"))

Nucleotide repetition lengths for humans

The actual genome data clearly deviate from our null hypothesis of randomness, as one might expect. However, these graphs raise more questions than they answer. In particular, note that there is a sort of bimodal characteristic or extended, secondary bump in the data for humans, with hints of an outlier feature on the tail which may or may not just be a statistical fluctuation.

One complicating factor we neglected is the role of “N-blocks”, which should be eliminated without introducing artificially longer lengths; e.g. ANNNNA with the Ns removed should be considered two 1-blocks rather than a two-block AA group. This simple modification to get-lengths is left as another exercise to the reader (the change does not affect the resulting distributions).

A real biologist could no doubt tell us a lot about these distributions. Meanwhile, the latest code has been pushed to GitHub.

Updating the Genome Decoder

Sat, 13 Jul 2013 00:00:00 +0000

In our last post we saw that so-called “N-blocks” (regions of the genome for which sequences are not available) were not getting decoded correctly by our decoder.

The solution is to look back into the so-called “file index” which specifies where the N-blocks are, and how long each block is. The modified decoder looks like this:

(defn genome-sequence
  "
  Read a specific sequence, or all sequences in a file concatenated
  together; return it as a lazy seq.
  "
  ([fname]
     (let [sh (sequence-headers fname)]
       (lazy-mapcat (partial genome-sequence fname) sh)))
  ([fname hdr]
     (let [ofs (:dna-offset hdr)
           dna-len (:dna-size hdr)
           byte-len (rounding-up-divide dna-len 4)
           starts-and-lengths (get-buffer-starts-and-lengths ofs 10000 byte-len)]
       (->> (for [[offset length] starts-and-lengths
                  b (read-with-offset fname offset length)]
              (byte-to-base-pairs b))
            (apply concat)
            (take dna-len)
            (map-indexed (fn [i x] (if (is-in-an-n-block i hdr)
                                    :N
                                    x)))))))

The primary modification is the map-indexed bit at the end, which looks up the position of the base pair in the index using the is-in-an-n-block function:

(defn is-in-an-n-block
  ([x hdr]
     (let [{:keys [n-block-starts n-block-sizes]} hdr]
       (is-in-an-n-block x n-block-starts n-block-sizes)))
  ([x n-block-starts n-block-lengths]
     (let [pairs (map (fn [a b] [a (+ a b)])
                      n-block-starts n-block-lengths)]
       (some (fn [[a b]] (< (dec a) x b)) pairs))))

I tested the new decoder against FASTA files of the first two chromosomes of the human genome using the same procedure we used for yeast in my validation post. The N-blocks (and all other sequences) were decoded correctly.

The astute reader will note the use of lazy-mapcat instead of =mapcat:

(defn lazy-mapcat
  "
  Fully lazy version of mapcat.  See:
  http://clojurian.blogspot.com/2012/11/beware-of-mapcat.html
  "
  [f coll]
  (lazy-seq
   (if (not-empty coll)
     (concat
      (f (first coll))
      (lazy-mapcat f (rest coll))))))

A version of genome-sequence which used mapcat instead of lazy-mapcat worked fine for individual chromosomes (file sections), but consistently ran out of memory when processing whole genome files. It took a bit of research and hair-pulling to figure out that there is a rough edge with mapcat operating on large lazy sequences.

Lazy sequences are a strength of Clojure in general – they allow one to process large, or even infinite, sequences of data without running out of memory, by consuming and emitting values only as needed, rather than all at once. Many of the core functions and macros in Clojure operate on sequences lazily (producing lazy sequences as output), including map, for, concat, and so on. mapcat is among these; however, mapcat is apparently not “maximally lazy” when concatenating other lazy seqs, causing excessive memory consumption as explained in this blog post. Using that post’s fully lazy (if slightly slower) version of mapcat fixed my memory leak as well. Though lazy seqs are awesome in many ways, one does have to be careful of gotchas such as this one.

So, in the process of handling this relatively large dataset, we have discovered two rough edges of Clojure, namely with mapcat and count. We shall see if other surprises await us. Meanwhile, the latest code is up on GitHub.

Back to Frequencies

Now that we are armed with a decoder which handles N-blocks correctly, let us return to our original problem: nucleotide frequencies. For yeast, we again have:

jenome.core> (->> yeast genome-sequence frequencies)
{:C 2320576, :A 3766349, :T 3753080, :G 2317100}

Same as last time. For humans,

jenome.core> (->> human genome-sequence frequencies)
{:N 239850802, :T 856055361, :A 854963149, :C 592966724, :G 593325228}

As expected, the GAC numbers are the same as what we had previously; only T and N have changed. The distributions look like so:

Updated nucleotide frequencies

We can see that Chargaff’s Rule obtains now: A/T ratios are equal, as are G/C ratios. (BTW, this rule is intuitively obvious if you consider that in the double-helix structure, As are paired with Ts and Gs are paired with Cs.) Interestingly, the relative abundance of GC pairs in the human genome is higher than it is in yeast.

Timing and Parallelizing

Many of these “queries” on our data take a long time to run (particularly for the human genome). So as not to tie up my REPL and preventing me from doing other (presumably shorter) experiments, I find it helpful to run such tasks inside a little macro (similar to Clojure’s time) which performs the computation in a thread and, when the task is finished, prints the time elapsed, the code that was run, and the result:

(defmacro tib
  "
  tib: Time in the Background
  Run body in background, printing body and showing result when it's done.
  "
  [expr]
  `(future (let [code# '~expr
                 start# (. System (nanoTime))]
             (println "Starting" code#)
             (let [result# ~expr
                   end# (. System (nanoTime))
                   dursec# (/ (double (- end# start#)) (* 1000 1000 1000.0))]
               (println (format "Code: %s\nTime: %.6f seconds\nResult: %s"
                                code#
                                dursec#
                                result#))
               result#))))

For example, our overflowing count runs as follows:

(tib (count (range (* 1000 1000 1000 3))))
Starting (count (range (* 1000 1000 1000 3)))
;; ...
Code: (count (range (* 1000 1000 1000 3)))
Time: 399.204845 seconds
Result: -1294967296

Another tool which has proved useful is:

(defmacro pseq
  "
  Apply threading of funcs in parallel through all sequences specified
  in the index of fname.
  "
  [fname & funcs]
  `(pmap #(->> (genome-sequence ~fname %)
               ~@funcs)
         (sequence-headers ~fname)))

This threads one or more funcs through all the sequences in the file, as mapped out by the file headers. The magic here is pmap, which is map parallelized onto separate threads across all available cores. pseq maxes out the CPU on my quad-core Macbook Pro and gives results significantly faster:

Code: (->> yeast genome-sequence frequencies)
Time: 33.090855 seconds
Result: {:C 2320576, :A 3766349, :T 3753080, :G 2317100}

Code: (apply (partial merge-with +) (pseq yeast frequencies))
Time: 16.056695 seconds
Result: {:C 2320576, :A 3766349, :T 3753080, :G 2317100}

With our updated decoder and these new tools, we can continue our poking and prodding of the genome in subsequent posts.

Getting Our Hands Dirty (with the Human Genome)

Wed, 10 Jul 2013 00:00:00 +0000

In the past two posts, we created and validated a decoder for genome data in 2bit format.

Let’s start actually looking at the human and yeast genomes per se. First, if you haven’t downloaded it yet, you’ll need a copy of the human genome data file:

cd /tmp
wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.2bit

(Depending on your local bandwidth conditions, this can take quite awhile.)

Again we define convenience vars for the two data files:

(def human "/tmp/hg19.2bit")
(def yeast (as-file (resource "sacCer3.2bit")))

The simplest thing we can do is count base pairs:

jenome.core> (time (->> yeast genome-sequence count))
"Elapsed time: 17577.042 msecs"
12157105
jenome.core> (time (->> human genome-sequence count))
"Elapsed time: 4513537.422 msecs"
-1157806032

WTF?! count has overflowed (if you convert -1157806032 to an unsigned int you get 3137161264, which makes more sense). This, and our running time of 75 minutes, is our first hint of pain associated with handling with such a large dataset.

It’s worth noting that number overflows in Clojure generally raise an ArithmeticException (or silently get promoted to BigIntegers, depending on the operators used) – I emailed the Clojure mailing list about this surprise and a ticket has been made.

The overflow is easily enough remedied with a non-overflowing count (inc' is one of the aforementioned promoting operators):

(defn count' [s]   ;; Regular count overflows to negative int!
  (loop [s s, n 0]
    (if (seq s)
      (recur (rest s)
             (inc' n))
      n)))

Which gives us the unsigned version of our previous answer, in 10% more time:

jenome.core> (time (->> human genome-sequence count'))
"Elapsed time: 4931630.301 msecs"
3137161264
jenome.core> (float (/ 3137161264 12157105))
258.05167

(Update 2/14/2014: Matthew Wampler-Doty came up with a more elegant count':

(defn count' [s] (reduce (fn [x _] (inc x)) 0 s))

which is about 25% faster than my slightly more naïve version.)

We now have our first bit of insight into human vs. yeast genomes: we humans have about 260x more base pairs than yeast do.

The next question is, What are the relative proportions of occurrence of each base pair? Easily answered by the frequencies function:

jenome.core> (->> yeast genome-sequence frequencies)
{:C 2320576, :A 3766349, :T 3753080, :G 2317100}
jenome.core> (->> human genome-sequence frequencies)
{:T 1095906163, :A 854963149, :C 592966724, :G 593325228}
jenome.core>

Those numbers have a lot of digits, so here are some bar charts:

Our first stab at nucleotide frequencies

Though we are playing ignorant data analysts rather than real biologists, this looks a bit fishy right out of the gate. Why so many Ts?

As a sanity check, perhaps we ought to look more closely at the actual data:

jenome.core> (->>
                human
                genome-sequence
                (map name)
                (take 100)
                (apply str))

;=> "TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT
;    TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT"

That doesn’t look good at all. How are the first 10000 base pairs distributed?

jenome.core> (->>
                human
                genome-sequence
                (take 10000)
                frequencies)
;=> {:T 10000}

How about the next 10,0000?

jenome.core> (->>
                human
                genome-sequence
                (drop 10000)
                (take 10000)
                frequencies)
;=> {:T 2012, :A 2065, :C 3028, :G 2895}

That looks a lot better. Playing around with this shows that exactly the first 10,000 base pairs are Ts.

The astute reader will recall that the file index in our decoder had slots for unknown (“N-block”) sequences. Though we didn’t explore them explicitly, /S. Cerviciae/’s genome had nothing in these slots. It’s time we went back and looked at our human’s file index:

  (->> human
       sequence-headers
       (map (juxt :name :dna-size :n-block-count))
       clojure.pprint/pprint)
;; =>
(["chr1" 249250621 39]
 ["chr2" 243199373 24]
 ["chr3" 198022430 9]
 ["chr4" 191154276 12]
 ["chr5" 180915260 7]
 ["chr6" 171115067 11]
 ["chr7" 159138663 17]
 ["chrX" 155270560 23]
 ["chr8" 146364022 9]
 ;; LOTS more entries...
)

Holy cow, they all have N-blocks. This is only the first section; this Gist shows the entire set of N-blocks for all the sequences. (It also shows that there are many more files in the index than the yeast genome had, and many more than one for each of the 24 human chromosomes – clearly there are many things to learn about this data.)

Moreover, if we look at the N-blocks for the first sequence:

  (->> human
       sequence-headers
       first
       ((juxt :n-block-starts :n-block-sizes))
       (apply interleave)
       (partition 2)
       (map vec)
       clojure.pprint/pprint)

;; Gives:

([0 10000]
 [177417 50000]
 [267719 50000]
 [471368 50000]
 ;; ... total of 39 entries.
)

As expected, our first 10,000 base pairs are actually unknown. The 2-bit file format apparently uses 0’s for the unknowns AND to represent Ts. The rest of the offsets and lengths are in this Gist. (Perhaps someone can explain to me why these are all multiples of 10000!)

All this means that our decoder needs a tweak to give us :N instead of :T when the position of the base pair is inside one of the N-blocks. This we will accomplish in the next post, before we proceed to the study of other properties of the genome.

Before we end, it’s worth noting that, had we been real biologists, we would surely have known about Chargaff’s Rule, which states that A and T ratios should be the same, as are G and C, as born out in our yeast distribution, above.

Though we haven’t gained mastery over too much of the genome yet, it is worth pointing out Clojure’s strengths in the above and previous posts. With comparatively little code we are able to investigate many of the features of these data sets, thanks to the power of the threading macro ->>, higher-order functions such as juxt and map, and sequence handling functions such as interleave, partition, and frequencies. We’ve also processed large data sets “lazily” without needing to fit large amounts of data in memory at once. And we have hardly scratched the surface of Clojure’s tools for parallel computation, though we may make more use of these in coming posts.

Validating the Genome Decoder

Sun, 07 Jul 2013 00:00:00 +0000

Today we’ll validate the genome decoder we described yesterday, once again with our friend the yeast Saccharomyces cerevisiae (you may want to enjoy a slice of freshly-baked bread and a stein of Pilsner with this post).

We are aided in this case by the availability of the SacCer3 genome in both 2bit and FASTA formats. We can get the FASTA version in the same place we got the 2bit file:

mkdir /tmp/sacCer3_fasta
cd /tmp/sacCer3_fasta
wget http://hgdownload-test.cse.ucsc.edu/goldenPath/sacCer3/bigZips/\
chromFa.tar.gz
tar xvzf chromFa.tar.gz

There is one FASTA file per sequence, starting something like this:

>chrI
CCACACCACACCCACACACCCACACACCACACCACACACCACACCACACC
CACACACACACATCCTAACACTACCCTAACACAGCCCTAATCTAACCCTG
GCCAACCTGTCTCTCAACTTACCCTCCATTACCCTGCCTCCACTCGTTAC
CCTGTCCCATTCAACCATACCACTCCGAACCACCATCCATCCCTCTACTT
ACTACCACTCACCCACCGTTACCCTCCAATTACCCATATCCAACCCACTG
CCACTTACCCTACCATTACCCTACCATCCACCATGACCTACTCACCATAC

Back in the REPL, we can now spit out our own copy in the same format (carrying over yeast and other functions and vars from the previous post).

First we need a new directory for the FASTA files we’ll generate:

(.mkdir (clojure.java.io/file "/tmp/decoded"))

Then we convert the keywords in our sequence to strings:

(defn genome-str
  "
  Convert e.g. [:A :G :T :C] to \"AGTC\"
  "
  [s]
  (->> s
       (map name)
       (apply str)))

A simple function will spit out the files, given a seq:

(defn write-seq
  "
  Write a (potentially very long) sequence of lines to a text file
  "
  [filename s]
  (with-open [wrt (clojure.java.io/writer filename)]
    (doseq [x s]
      (.write wrt (str x "\n")))))

And now for the actual converter:

(doseq [{:keys [name dna-offset dna-size]} (sequence-headers yeast)]
  (let [fname (str "/tmp/decoded/" name ".fa")]
    (write-seq fname
               (cons (str ">" name)
                     (->> (genome-sequence yeast dna-offset dna-size)
                          (partition-all 50)
                          (map genome-str))))))

Did it work?

cd /tmp/sacCer3_fasta
for f in *.fa; do diff $f /tmp/decoded/$f; done

No output – it succeeded! This builds more confidence that we didn’t screw anything up in the decoder. (There are also a few other hard-coded unit tests in test_core.clj.)

Comparing file sizes, the FASTA files are about 4 times larger than the original 2bit file. If you tar and compress both versions, the FASTA files are still about 30% larger. It is left as as an exercise to the reader (with some spare hard disk) to do the same comparison with the human genome. (Though larger, the FASTA files would clearly be simpler to work with, and these days 4 GB isn’t too terribly much data; nevertheless, we’ll continue to use the 2bit file and decoder for our explorations.)

Now we are ready to begin playing with the actual data – starting in the next post.

A Two Bit Decoder

Sat, 06 Jul 2013 00:00:00 +0000

The entire human genome is available as a single .2bit file here (click on “Full Data Set”, then download hg19.2bit). Unlike the stellar signal in His Master’s Voice, the 2bit format is reasonably clearly documented.

We want to write Clojure code to:

Provide base pairs in symbolic (rather than raw binary) form as lazy sequences – i.e., sequences which need not all fit in memory at once, but can be consumed and processed as needed;
Provide “random access” to this data selectively, e.g. by chromosome, rather than always reading through the entire file;
Provide access to metadata encoded in the file.

The functionality to do this is posted in the jenome project on GitHub. In this post, we’ll explore this code a little; in following posts, we’ll do some investigating of the actual genome per se.

Capability (2) is provided, in part, by implementing random-access reads from file fname of len bytes starting at offset:

(ns jenome.rafile
  (:import (java.io RandomAccessFile)))

(defn read-with-offset [fname offset len]
  (let [raf (RandomAccessFile. fname "r")
        bb (byte-array len)]
    (.seek raf offset)
    (.readFully raf bb)
    (.close raf)
    bb))

Armed with this, we can get the .2bit file header:

(defn file-header [fname]
  (let [[sig ver seqcnt resvd] (->> (read-with-offset fname 0 16)
                                    (partition 4)
                                    (map bytes-to-number))]
    (assert (= sig 0x1A412743))
    (assert (= ver 0))
    (assert (= resvd 0))
    seqcnt))

file-header basically just gives us the number of sequences (usually, chromosomes) in the file, doing some sanity checks along the way. bytes-to-number converts an arbitrary sequence of bytes to the appropriate unsigned integer. (For brevity’s sake, I won’t show every utility function in this blog post; the source code on GitHub is reasonably short.)

The next part of the file, as shown in Figure 1, is called the “file index,” and contains a list of sequences contained the rest of the file. It can be read as follows:

(defn file-index [fname seqcnt]
  (loop [i 0
         ofs 16
         ret []]
    (if (< i seqcnt)
      (let [[nlen] (read-with-offset fname ofs 1)
            name (apply str (map char (read-with-offset fname 
                                                        (+ ofs 1) nlen)))
            seq-offset (get32 fname (+ ofs 1 nlen))]
        (recur (inc i) (+ ofs nlen 5) (conj ret [nlen name seq-offset])))
      ret)))

This somewhat imperative code walks through the seqcnt sequence portions of the index, pulling out sequence names and lengths as we go.

It’s here that we introduce a new friend, the yeast Saccharomyces cerevisiae (SacCer3), used since antiquity for making bread and fermented beverages. Relatively small in comparison with the human genome, SacCer3 will be our “unit test” organism. Available online and checked into the resources folder in this repo, the file can be accessed as

(def yeast
   (as-file (resource "sacCer3.2bit")))

(I have imported resource and as-file from clojure.java.io; again, see the source code.)

Our index-reading code yields:

  (let [seqcnt (file-header yeast)]
    (file-index yeast seqcnt))
  ;=>
[[4 "chrI" 191]
 [5 "chrII" 57762]
 [6 "chrIII" 261074]
 [5 "chrIV" 340245]
 [5 "chrIX" 723245]
 [4 "chrV" 833233]
 [5 "chrVI" 977468]
 [6 "chrVII" 1045025]
 [7 "chrVIII" 1317776]
 [4 "chrX" 1458453]
 [5 "chrXI" 1644907]
 [6 "chrXII" 1811627]
 [7 "chrXIII" 2081188]
 [6 "chrXIV" 2312312]
 [5 "chrXV" 2508412]
 [6 "chrXVI" 2781251]
 [4 "chrM" 3018284]]

The apparent consistency of these values give us some initial confidence that we are reading the index correctly. Note, however, the curious fact that chrIX appears between IV and V.

With this encouraging start, we can now attack the sequences proper. These are laid out as shown in Figure 2, with block metadata prepended to the actual DNA sequences:

Sequence Record Layout

The “N blocks” are blocks of unknown sequences with specified offsets and lengths. Masked blocks are blocks which are known repetitions (indicated as lower case a, g, c and t in the text-based ‘FASTA’ file format). We are obviously most interested in dnaSize, the number of base pairs in the sequence, and the actual sequence values themselves.

Unpacking the above data format (except the base pairs per se) makes heavy use of get32, which just returns the unsigned 32-bit integer at the specified file location. This code doesn’t need to be super efficient, since the block headers themselves are quite small. The metadata for the entire file is returned as a sequence of maps.

(defn getblk [fname offset n]
  (let [ret (map #(get32 fname (+ offset (* 4 %))) (range n))
        offset (skip offset n)]
    [ret offset]))


(defn sequence-headers
  "
  Get sequence headers from .2bit file, as documented in
  http://genome.ucsc.edu/FAQ/FAQformat#format7. Returns a list of maps
  with details for each sequence.
  "
  [fname]
  (let [seqcnt (file-header fname)]
    (for [[nlen name ofs] (file-index fname seqcnt)]
      (let [[[dna-size]         ofs] (getblk fname ofs 1)
            [[n-block-count]    ofs] (getblk fname ofs 1)
            [n-block-starts     ofs] (getblk fname ofs n-block-count)
            [n-block-sizes      ofs] (getblk fname ofs n-block-count)
            [[mask-block-count] ofs] (getblk fname ofs 1)
            [mask-block-starts  ofs] (getblk fname ofs mask-block-count)
            [mask-block-sizes   ofs] (getblk fname ofs mask-block-count)
            [[reserved]         ofs] (getblk fname ofs 1)]
        (assert (zero? reserved))
        {:name name
         :nlen nlen
         :dna-size dna-size
         :n-block-count n-block-count
         :n-block-starts n-block-starts
         :n-block-sizes n-block-sizes
         :mask-block-starts mask-block-starts
         :mask-block-sizes mask-block-sizes
         :dna-offset ofs}))))

A few sanity checks are included in test_core.clj to make sure we’re decoding the metadata correctly. Requirement (3) is done!

The final step (Requirement (1)), is to actually get our base pairs (BPs). Since we have to assume the data set is very large (as is the case with the human genome), we cannot read the entire DNA sequence at once. The first part is to break the dna-size base pairs (remember we have 2 bits per BP, or 4 BP/byte), starting at dna-offset. First we obtain the “coordinates” of the sequences we want to read:

(defn get-buffer-starts-and-lengths 
  "
  Return buffer offsets (starting at ofs) required to cleanly read a
  total of m bytes no more than n at a time
  "
  [ofs n m]
  (loop [a ofs
         len n
         ret []]
    (if (>= a (+ m ofs))
      ret
      (recur (+ a n)
             n
             (conj ret [a (min len (- (+ m ofs) a))])))))

;; Example:
(get-buffer-starts-and-lengths 100 200 512)
;=> [[100 200] [300 200] [500 112]]

At long last, having obtained the locations and lengths we want to read from, we can get our sequences out:

(defn genome-sequence
  "
  Read a specific sequence, or all sequences in a file concatenated
  together; return it as a lazy seq.
  "
  ([fname]
     (let [sh (sequence-headers fname)]
       (mapcat #(genome-sequence fname %1 %2)
               (map :dna-offset sh)
               (map :dna-size sh))))
  ([fname ofs dna-len]
     (take dna-len
           (apply concat
                  (let [byte-len (rounding-up-divide dna-len 4)
                        starts-and-lengths (get-buffer-starts-and-lengths
                                              ofs 10000 byte-len)]
                    (for [[offset length] starts-and-lengths
                          :let [buf (read-with-offset fname offset length)]
                          b buf]
                      (byte-to-base-pairs b)))))))

;; Example:
 (->> yeast
      genome-sequence
      (take 30))
;=> (:C :C :A :C :A :C :C :A :C :A :C :C :C :A :C
;    :A :C :A :C :C :C :A :C :A :C :A :C :C :A :C)

This function allows us to choose the entire genome, or that for a given offset and number of base pairs (whether from the metadata for an entire chromosome, or some smaller region, thus satisfying (2)).

The next post will focus on verification of correctness of this code; subsequent posts will begin to explore the characteristics of this data for various genomes, human or otherwise.

Exploratory Genomics with Clojure

Fri, 05 Jul 2013 00:00:00 +0000

In Stanislaw Lem’s His Master’s Voice, humanity receives a “letter from the stars,” in the form of regularly repeating binary data encoded in a beam of extraterrestrial neutrinos. The protagonists know nothing a priori about the content of the letter. The plot unfolds as the message slowly, as a result of careful study, yields a few (not many) of the signal’s secrets.

What would you do if you received such a dataset? How would you study it? What real world datasets are similarly mysterious?

As most readers will know, every living being contains a “message” of sorts, encoded, not in base 2 (binary), but rather base 4: A, G, C, or T, arranged in paired, sequential molecular strands called DNA (in which every A is paired with a T, every C with a G). From 159,662 base pairs for the smallest known genome, to 3.3 billion BPs for humans, to 149 billion BPs for the largest genome, that of Paris japonica, each organism carries its own unique tome locked in its cells.

In the past ten years, the human genome has been sequenced and made widely available. Though downloadable in a variety of formats, the most compact seems to be the ‘2bit’ format (two bits per base pair): the genome in this format is just under a gigabyte, uncompressed. Plenty of genomes of simpler organisms are available for study as well.

I am not a genomicist or biologist, but am curious about the properties of the “data” stored within us all. In the next few blog posts I intend to investigate genomic data (human or otherwise) using Clojure, a modern Lisp developed for the JVM. Clojure’s strengths are simplicity, concurrency, expressiveness and speed – assets which will be helpful as we tackle a few simple analytical tasks aimed at understanding some of the properties of life’s very own “Big Data.”

Rosalind Problems in Clojure

Sun, 09 Jun 2013 00:00:00 +0000

This weekend I’ve been having a lot of fun working the Bioinformatics problems from Rosalind. Most people work them in Python, but so far they have been very amenable to Clojure except where BioPython libraries are used for access to online databases. The problems have been straightforward so far but I have enjoyed the elegance and brevity that Clojure lends the solutions.

In particular, I like this short translator from RNA sequences to amino acids:

;;; Translating RNA into Amino Acids

(defmacro deftable [tname & rest]
  `(def ~tname (apply hash-map '(~@rest))))

(deftable proteins
  UUU F      CUU L      AUU I      GUU V
  UUC F      CUC L      AUC I      GUC V
  UUA L      CUA L      AUA I      GUA V
  UUG L      CUG L      AUG M      GUG V
  UCU S      CCU P      ACU T      GCU A
  UCC S      CCC P      ACC T      GCC A
  UCA S      CCA P      ACA T      GCA A
  UCG S      CCG P      ACG T      GCG A
  UAU Y      CAU H      AAU N      GAU D
  UAC Y      CAC H      AAC N      GAC D
  UAA Stop   CAA Q      AAA K      GAA E
  UAG Stop   CAG Q      AAG K      GAG E
  UGU C      CGU R      AGU S      GGU G
  UGC C      CGC R      AGC S      GGC G
  UGA Stop   CGA R      AGA R      GGA G
  UGG W      CGG R      AGG R      GGG G)

(defn to-protein [s]
  (->> s
       (partition 3)
       (map (partial apply str))
       (map symbol)
       (map proteins)
       (take-while #(not= % 'Stop))
       (apply str)))

The body of the proteins table is literally cut-and-pasted from the problem page (click on “RNA codon table”). I think it’s a good example of using macros to provide a little bit of syntactic sugar to make the code just a little more readable and elegant, or to encode domain facts as directly as possible.

This and my other solutions so far are up on GitHub.

Introduction to Context Managers in Python

Sat, 20 Apr 2013 00:00:00 +0000

Context managers are a way of allocating and releasing some sort of resource exactly where you need it. The simplest example is file access:

with file("/tmp/foo", "w") as foo:
    print >> foo, “Hello!”

This is essentially equivalent to:

foo = file("/tmp/foo", "w")
try:
    print >> foo, “Hello!”
finally:
    foo.close()

Locks are another example. Given:

import threading
lock = threading.Lock()

then

with lock:
    my_list.append(item)

replaces the more verbose:

lock.acquire()
try:
    my_list.append(item)
finally:
    lock.release()

In each case a bit of boilerplate is eliminated, and the “context” of the file or the lock is acquired, used, and released cleanly.

Context managers are a common pattern in Lisp, where they are usually defined using macros. Examples include with-open and with-out-str in Clojure and with-open-file and with-output-to-string in Common Lisp.

Python, not having macros, must include context managers as part of the language. Since 2.5, it does so, providing an easy mechanism for rolling your own. Though the default, “low level” way to make a context manager is to make a class which implements the context management protocol, by implementing __enter__ and __exit__ methods, the simplest way is using the contextmanager decorator from the contextlib library, and invoking yield in your context manager function in between the setup and teardown steps.

Here is a context manager which you could use to time the use of threads vs. processes in Python:

import contextlib
import time

@contextlib.contextmanager
def time_print(task_name):
    t = time.time()
    try:
        yield
    finally:
        print task_name, “took", time.time() - t, “seconds.”

with time_print(“processes”):
    [doproc() for _ in range(500)]

# processes took 15.236166954 seconds.

with time_print(“threads”):
    [dothread() for _ in range(500)]

# threads took 0.11357998848 seconds.

Composition

Context managers can be composed very nicely. While you can certainly do the following,

with a(x, y) as A:
    with b(z) as B:
        # Do stuff

in Python 2.7 or above, the following also works:

with a(x, y) as A, b(z) as B:
    # Do stuff

with Python 2.6, using contextlib.nested does almost the same thing:

with contextlib.nested(a(x, y), b(z)) as (A, B):
    # Do the same stuff

the difference being that with the 2.7+ syntax, you can use the value yielded from the first context manager as the argument to the second (e.g., with a(x, y) as A, b(A) as C:...).

If multiple contexts occur together repeatedly, you can also roll them together into a new context manager:

import contextlib

@contextlib.contextmanager
def c(x, y, z):
    with a(x, y) as A, b(z) as B:
        yield (A, B)

with c(x, y, z) as C:  # C == (A, B)
    # Do that same old stuff

What does all this have to do with testing? I have found that for complex integration tests where there is a lot of setup and teardown, context managers provide a helpful pattern for making compact, simple code, by putting the “context” (state) needed for any given test close to where it is actually needed (and not everywhere else). Careful isolation of state helps eliminate bugs and makes code easier to reason about.

Processes vs. Threads for Integration Testing

Fri, 19 Apr 2013 00:00:00 +0000

Python has become notorious for being somewhat problematic when it comes to concurrency. The language supports threads, but performance is known to be dismal in some cases. In particular, as David Beazley discovered, when I/O-bound and CPU-bound threads run in the same program, the GIL performs very badly (worse than if you ran the tasks one after the other on a single thread/CPU). Alternative approaches which have become popular are to use asynchronous or event-driven programming techniques (Twisted, asyncore), or to make heavy use of the multiprocessing module.

These wrinkles aside, there are situations where threads in Python are still very useful. One of these is integration testing of distributed systems.

Imagine you have a distributed system with some 30 (or 3, or 300) software components running on some cluster. These programs might interact with a database, with a user via command line or a GUI (Web or otherwise), and with each other via a messaging or RPC protocol (XML-RPC, ZeroMQ, …).

While low-level (functional, unit) testing will perhaps be the bulk of your tests, integration tests are important to make sure all the programs talk to each other as they should. Like your other tests, you want to automate these. And they should run as fast as possible to optimize the feedback cycle during development.

A straightforward way to test these components is to run them all in separate programs (locally or distributed), each in their own process. However, you’re likely to get much better performance from your many short-running tests if you run the components as local threads in a single process. The components running in these threads would, of course, still talk to each other via RPC, ZeroMQ, etc., same as if they were processes. But for short tests the setup and teardown for threads is much faster. The most trivial example (assigning a value to a variabled) shows the difference dramatically:

# in ipython:

import threading, subprocess

doproc = lambda: subprocess.Popen(["python", "-c", "'a=1'"],
                                  stdout=subprocess.PIPE).communicate()

def dothread():
    def run():
        a = 1
    th = threading.Thread(target=run)
    th.start()
    th.join()

time junk = [doproc() for _ in range(500)]

# CPU times: user 0.18 s, sys: 1.81 s, total: 1.98 s
# Wall time: 14.30 s

time junk = [dothread() for _ in range(500)]

# CPU times: user 0.09 s, sys: 0.05 s, total: 0.14 s
# Wall time: 0.16 s

That’s a factor of about a hundred – something you will most definitely notice in your test suite.

Another advantage of this approach is that, when you write your code so that it can be run as threads, you can put as many or few of these threads in actual processes (programs) as you’d like. In other words, the coupling of processes to components is loosened. Your automated tests will force you to implement clean shutdown semantics for each thread (otherwise your test program will likely not terminate without manual interruption).

Finally, it’s much easier to interrogate the state of each component when it’s running as a thread, than it is to query a subprocess (via e.g. RPC). This greatly simplifies the assertions you have to make in your integration tests, since you don’t have to send a message of some sort via RPC or message queues – you can just query variables.

I found, while writing the automated tests in IceCube Live (the control and monitoring system of the IceCube Neutrino Detector), that making components that could be instantiated in threads (for testing) or in processes (for production) greatly sped up my test suite and simplified the actual tests quite a bit. I should note that, prior to release, there is still a final integration test done on a mirror test system which simulates actual data collection and makes sure IceCube Live can play along with other systems. The mirroring, however, is not exact, since the actual detector elements we deployed at South Pole are expensive and rely on a billion tons of crystal-clear ice to work as intended.

In the next post we will explore the use of context managers, which are helpful for organizing setup and teardown of complex tests cases involving multiple components.

Resources for Learning Clojure

Mon, 21 May 2012 00:00:00 +0000

I thought I’d take a moment to present one possible path, or set of alternate paths, for learning Clojure, a modern Lisp implemented on top of the Java Virtual Machine.

Update: Since this was written in 2012, a number of additional excellent books have come out. These include Carin Meier’s Living Clojure; Vandgrift and Miller, Clojure Applied; and Daniel Higginbotham’s, Clojure for the Brave and True. These are all great choices, and Clojure for the Brave and True is a strong competitor for the best introductory Clojure book.

In addition to a surprising number of very good resources out there for such a young language, there are resources specific to other Lisps which map reasonably well onto Clojure.

Clojure does have a learning curve, especially for one coming from imperative languages such as Java, Python, C++, or Ruby. These languages are, in some ways, more similar to each other than they are to functional languages such as Clojure (though they all have features which were pioneered originally in implementations of Lisp, one of the oldest computer languages).

Since the approach I took was somewhat haphazard and I have the benefit of some hindsight now, I present here a rough sequence which might make the road a little less bumpy for others. Though not an exhaustive list of resources by any means, the following diagram summarizes the discussion points that follows.

A Flow Chart for Learning Clojure

The approach I recommend goes something like this:

If you are going to pick just one of these resources to get started, I recommend the O’Reilly book Clojure Programming by C. Emerick et. al. I have both the PDF and print versions of this book, and it’s not only an excellent book on Clojure, it’s one of the best O’Reilly books I’ve seen in the dozen or so years I’ve been reading them. The other two books are also good (I have not checked out the 2nd edition of Programming Clojure yet)… Clojure is particularly blessed in having not just one but at least three excellent “starter books” on the language.

If you’re curious why one might want to learn a Lisp as your next (or last?) programming language, Paul Graham evangelizes very effectively about Lisp in some of his many essays. You might want to check those out before diving into Clojure per se.

Additionally, videos of Rich Hickey’s talks, particularly Simple Made Easy, though not so much about Lisp or Clojure per se, explain the challenges and limitations involved with stateful Object-Oriented programming and motivate the ideas behind functional programming in general and Clojure in particular. I very much recommend these videos, though reaction to them among programmers I’ve shown them to has varied wildly from enthusiasm to extreme irritation.

People with exposure to Common Lisp or Scheme may be familiar with Abelson and Sussman’s Structure and Interpretation of Computer Programs; Friedman’s The Little Schemer and its sequels The Seasoned Schemer and The Reasoned Schemer ; and P. Seibel’s Practical Common Lisp, which I’ve only peeked at but which has an excellent reputation. These books could provide a gateway into Clojure, or might merit a look at once you’ve got some Clojure basics under your belt. I find them to be interesting supplemental reading.

Before getting to the more advanced books, the Clojure Koans are a fun and pleasantly gradual way to start reinforcing your Clojure basics (I worked most of them on various airplane flights to the South Pole last year). Similar but somewhat more challenging are the 4clojure problems (I’m about 3/4 of the way through these).

I find these kinds of problems alternately soothing (when I get them right away) and maddening (when I actually have to work at them). They are like crossword puzzles for programmers, and they do really help reinforce the material from the books. Due to the way the problems are implemented, they are unfortunately limited in terms of how they cover Java interoperability, macros, and concurrency.

After you’ve read an introductory text, worked some problems and maybe started one or two personal projects with the language, you’re ready for The Joy of Clojure. This book is somewhat hard to describe; I started it too early, then came back after six months and it made a whole lot more sense. The best way to explain The Joy of Clojure is that it helps you to understand both some of the quirks of the language and what makes it truly great, and will help you evolve from writing clumsy translations of what you’ve learned in other languages, to truly compact, elegant, idiomatic Clojure code.

After, or perhaps during, Joy of Clojure, take a deep dive into macros, the secret sauce of Lisp. Macros allow one to implement new language features in order to make one’s code elegant and expressive, and to optimize the fit to one’s problem domain. Paul Graham’s On Lisp is mostly about macros; Hoyte’s Let over Lambda refers to On Lisp many times and builds on some of the foundations laid in that book.

To summarize (strongest suggestions in bold):

Graham essays
Hickey videos
Introductory book (Clojure Programming, Programming Clojure, or Practical Clojure)
Possibly detour into SICP, Practical Common Lisp and/or The * Schemer books
Clojure Koans
4Clojure Problems
Joy of Clojure
On Lisp
Let Over Lambda

There are many other Lisp books which I haven’t read yet, as well as a Clojure book or two I’ve left out. Also, lest I forget, the Clojure community has a reputation for being extremely welcoming and helpful. There is a Google Group / mailing list and an IRC chat as well.

The bottom line is that for such a young language (or young variant of an old language), we are fortunate to have such a rich selection of resources to pick and choose from. Enjoy!

Continuous Testing in Python, Clojure and Blub

Sat, 31 Mar 2012 00:00:00 +0000

A separate monitor is handy for showing tests results continuously while working. The paintbrushes are strictly optional.

What follows is a somewhat rambling introduction to continuous, test-driven development, focusing mainly on Python and influenced by Clojure tools and philosophy. At the end, a simple script is introduced to help facilitate continuous TDD in (almost) any language.

For the last four years I have increasingly followed a test-driven approach in my development. My approach continues to evolve and deepen even as some of the limits of TDD are becoming clearer to me.

Initially, I had a hard time getting my head around TDD. Writing tests AND production code seemed like twice as much work, and, even before I wrote tests, I typically ran the program under development, e.g. with print statements added, to test each change. But making changes to old code was always a fairly daunting proposition, since there was no way to validate all the assumptions I’d checked “by eye” just after I’d written the code.

TDD helps reduce risk by continuously verifying your assumptions about how the code should perform at any time. Using TDD for a fairly large project has saved my bacon any number of times.

The basic approach is that the test code and the production code evolve together more or less continuously, as one follows these rules:

Don’t write any production code without a failing unit test;
Write only enough production code needed to make the tests pass.

Once I started writing tests for all new production code, I found I could change that code and make it better without fear. That led to much better (and usually simpler) code. I realized I was spending much less time debugging; and when there were bugs, the tests helped find them much faster. As I have gained experience with this approach I have found that the reliability of, and my trust in, my code written with TDD is vastly superior than otherwise. The two-rule cycle also tends to foster simplicity, as one tends to eschew any modifications that don’t actually achieve the desired objectives (i.e. meet the goals of the software, as specified formally in the tests themselves). The process is also surprisingly agreeable!

It goes without saying that if you follow this approach you will be running your tests a lot. The natural next step is to automate this more. In the book Foundations of Agile Python Development, Jeff Younker explains how to make Eclipse run your unit tests every time you save a file in the project. The speed and convenience of this approach was enough to get me to switch from Emacs to Eclipse for awhile.

Most of my daily programming work is in Python, but I have been an avid hobbyist in Clojure for several months now. It wasn’t until I saw Bill Caputo’s preparatory talk for Clojure/West here in Chicago that I heard the term continuous testing and realized that this is what I was already doing; namely, the natural extension of TDD in which one runs tests continuously rather than “by hand.” Bill demoed the expectations module and the autoexpect plugin for Leiningen, which runs your tests after every save without incurring the overhead of starting a fresh JVM each time.

(One point Bill made in his talk was that if your tests are slow, i.e. if you introduce some new inefficiency, you really notice it. Ideally the tests should take a few seconds or less to complete.)

Back to Python-land. Not wanting to be always leashed to Eclipse, and inspired by the autoexpect plugin, I started looking for an alternative to using Eclipse’s auto-builders — something I could use with Emacs or any other editor. There are a lot of continuous build systems out there, but I wanted something simple which would just run on the command line on my laptop screen while I edited code on my larger external monitor. I found tdaemon on GitHub; this program walks a directory tree and runs tests whenever anything changes (as determined by keeping a dictionary/map of SHA values for all the files). This is most of what I want, but it restricts you to its own choices of test programs.

In a large project with many tests, some fast and some slow, I often need to specify a specific test program or arguments. For example, I have a wrapper for nosetests which will alternately run all my “fast” unit tests, check for PEP-8 compliance, run Django tests, etc. In some cases, such as debugging a system with multiple processes, I may need to do something complex at the shell prompt to set up and/or tear down enough infrastructure to perform an existing test in a new way.

One piece of Clojure philosophy (from Functional Programming, a.k.a. “FP”) that has been influencing my thinking of late is the notion of composability: the decoupling or disentanglement of the pieces of the systems one builds into small, general, composable pieces. This will make those pieces easier to reuse in new ways, and will also facilitate reasoning about their use and behaviors. (Unfortunately, the merits of the FP approach, which are many, have poisoned my enthusiasm for OO to the extent that I will typically use a function, or even a closure, before using an object, which would perhaps be more Pythonic in some cases).

So, in the current case under discussion (continuous testing), rather than making some kind of stateful object which knows about not only your current file system, but also what tests should be allowed, their underlying dependencies, etc., it would be better (or at least more “functional”) instead to simply provide a directory-watching function that checks a dictionary of file hashes, and compose that function with whatever test program suits your purposes at the moment.

The result of these thoughts is a small script called conttest which is a simplification of tdaemon that composes well with any test suite you can specify on the command line.

Some examples follow:

$ conttest nosetests  # Runs nosetests whenever the files on disk change

$ conttest nosetests test.path:harness  # Runs only tests in 'harness'
                                        # object in path/to/test — handy
                                        # for developing a single feature
                                        # or fixing a single bug.
# Check both PEP-8 style and unit tests:
$ conttest 'pep8 -r . ; nosetests'

It would work equally well with a different language (“blub language”) with a separate compilation step:

$ conttest 'make && ./run-tests'

Using this program, depending on my needs of the moment, I can continuously run a single unit test, all my “fast” unit tests, or, if I’m willing to deal with slower turnaround times, all my unit and integration tests.

The script is on GitHub for your continuous enjoyment of continuous testing. May you find it helpful.

(Ironically, this script does NOT work that well for JVM languages like Clojure since the startup time is lengthy (a couple of seconds on my MacBook Pro). Most of Clojure's testing frameworks have autotest capabilities built in, which work great.)

Programming Languages

Thu, 22 Dec 2011 00:00:00 +0000

Working on Clojure Koans while riding on a C-17, bound for Antarctica

Today I am inspired to ponder many languages at once and review which ones I use regularly, which ones I’m curious about, which ones I avoid, and what I’d like to use, if it were to exist.

Programming Languages I Use Regularly

Python

By far the language I use most for work. I like it for its clean philosophy, its expressiveness, its “batteries included” extensive set of libraries, and, first and foremost, for its readability.

C

Of all the languages I use regularly, C is the one I learned first. I maintain a large Linux kernel device driver I wrote for the IceCube project as well as an embedded system written for 5000+ sensors designed for the same.

C now feels like assembly language to me but I still appreciate its power and elegance.

Clojure

I have dabbled in Lisp since the 1980s but not seriously until recently. Somewhat seduced by Paul Graham’s essays on Lisp and encouraged by a bit of a Lisp renaissance, I have started reading up on Clojure and working through problems on 4clojure.com. While not without its warts, I like many things about Clojure, including the Lisp “code-as-data” philosophy, availability of macros (something I wish Python had) and its interoperability with Java classes. While I doubt I’ll be able to use this in my paying work any time soon, I have started playing with Clojure for personal projects. Having to deal with significantly concurrent systems in my work, I am intrigued by functional programming, as opposed to the usual object-oriented approaches where state is king and where tangled hierarchies of relatively meaningless relationships can obscure intent. Clojure takes an interesting approach, with its emphasis on immutability, software-transactional memory and other concurrency primitives.

Bash / Unix Tools

It amuses me slightly to include Bash here, but combining simple iteration with conditional statements and adding basic Unix concepts and tools such as pipelines, grep, awk, sed, wc, etc. is surprisingly powerful. Every small Bash trick or new tiny-Unix-tool I learn seems to eliminate the need for some number of actual programs, at least for quick-and-dirty work. The results tend to be obscure and hard to parse; if I can do something in a single line of bash, I will; otherwise I’ll resort to Python for most things.

Javascript / Coffeescript

Not my favorite language by any stretch, but you can’t avoid it if you’re working in the browser (I don’t consider closed-source Flash an option). The language has a lot of warts, but some good parts too. I can feel the Lisp bones deep underneath the surface of the language when I dive into JavaScript. CoffeeScript is sweet because it’s so much more readable and offers protection from common JavaScript gotchas, but has some flaws of its own.

Programming Languages I Have Used in the Past but Tend to Avoid

Perl

I fell in love with the power of Perl (“the duct-tape of the Internet”) back in the 1990s, but now dislike its strange, ad-hoc syntax and the relative inscrutability when compared to Python.

Java

I haven’t done a ton of Java development, but have done enough to be irritated by certain things about it: its extremely verbose syntax, strict typing, distance from the actual hardware, and lack of (at least until now) anonymous functions (“lambda”). Also the JVM startup time is irritating, a problem Clojure inherits from Java (though there are workarounds).

Java has become so ubiquitous, however, that it’s hard to avoid, and it does have a certain self-consistent habitability to it. I think current JVM languages such as Clojure and Scala will only strengthen the role of Java and the JVM in modern computing, unless Oracle massively screws things up.

C++

Another language I’ve played with a bit. A language that splits the difference between C and Java (I realize C++ came before Java); I would prefer to write in a “real” higher level language and glue C in where needed.

FORTRAN

I’m sorry to say that, coming from physics, I’ve written more FORTRAN code than I care to admit. I find it interesting, however, that while Lisp and FORTRAN are almost the same age, Lisp still holds interest where FORTRAN does not (except to pure number-crunchers, due to ancient and venerable numeric libraries).

Languages I’m Curious About But Haven’t Had Time to Look At Much

Exposure to purely functional programming and lazy evaluation in Clojure made me curious about Haskell.

I am curious about Erlang, which is supposed to have excellent concurrency features.

I saw some talks about Go at OSCON. Go looks like it has some really nice features compared to C (compilation speed, concurrency support, and improved readability), but it may be a bit low-level for my interests.

I have only tinkered with Objective-C, but that is the language of choice for serious Mac OS X or iOS development. Its syntax looks pretty odd, but perhaps that’s a small price to pay for running on all that pretty hardware.

Purely logical languages such as Prolog (equivalents of which can be easily implemented in Lisp) are of interest for their ability to process large amount of semantically-related content. I’m curious about expert systems, ontologies, the Semantic Web, and many other related areas of AI research.

The Language I Wish Existed

The perfect language would:

Be very readable, like Python (whitespace or other visual cues probably playing a significant role)
Support full Lisp-like macros (“homoiconicity”)
Have very broad library support (Python, Java, …)
Have built-in features in support of test-driven development (Python’s doctests and Clojure’s :test metadata seem like just the beginning of what might be possible)
Handle concurrency very well (Clojure, Erlang, … but not Python)
Run in the browser, or be implemented efficiently on top of JavaScript
Allow you to get very close to the machine if necessary, or at least the bytecodes of the virtual machine or interpreter (Python, C, C++, …)

Points 1-3 are the most important to me. Resolving the tension between points 1 and 2 is of particular interest.

I doubt such a language will come along any time soon. But I’m taking a class next month which, who knows? … might help someday.

Milvans and Container Malls

Fri, 02 Dec 2011 00:00:00 +0000

Dec. 2, 2011, 08:10 NZDT Tranz Coastal train from Christchurch to Picton

Observatory Hill in McMurdo

The new Cashel Mall in Christchurch

Another gorgeous summer day here as the countryside rolls by: grassy, green hills punctuated by trees passing singly and in groups, as well as cattle, horses, and the ubiquitous sheep, which appeal strongly for some reason today, signaling tranquility, domesticity… humans living in pastoral harmony with the land. Icebreaker, perhaps my favorite clothing brand, puts tags on their merino wool clothes, each with a unique code you can type into their Web site to find out which sheep farm in New Zealand your woollen item comes from. I wonder if I’ll pass the farm that made the wool in the shirt I’m wearing. It’d be nice they would sell the sheep as well from their Web site. Or maybe even whole farms….

I’m on my habitual route from Christchurch to Wellington to see friends Neil and Amelia for a few days before heading back to Chicago. The usual blue, slightly creaky train cars have been upgraded to sleek new tan-colored ones with power outlets under the seats, bigger windows and a quieter ride — overall, an improvement. Flying would take one-tenth the time but I’m spending enough time on airplanes, and at any rate the scenery is quite gorgeous, an antidote to both the polar plateau and the coming Chicago winter.

I had just over a full day in Christchurch. As with the Elms hotel on the way down, the Pavillions was a disappointment. Expensive WiFi and rooms, noisy, with somewhat cheesy decor. Paying for WiFi is particularly irksome since it I know how cheap it is to implement — it feels like paying for tap water, by the cup. However, beggars cannot be choosers and the hotel situation in Christchurch is dismal after the February quake.

The advantage of the Pavilions is that is it relatively close to downtown. Had a big breakfast yesterday at a nearby cafe, and an interesting conversation with a friendly fellow diner who is very into meditation and marathoning and was accompanying a group of kids coming out of some month-long South Island adventure. Then I trotted down Victoria Street into the downtown area. Much is still fenced off so I had to circumambulate a couple-block area in various stages of demolition. Eventually I found myself on Armagh street staring at a familiar sign, that of the Winsor hotel, the first place I ever stayed in New Zealand. The familiar building, however, is gone, replaced by a generic looking construction-in-progress. Half a block closer to downtown, the Devon, my favorite place to stay, is completely gone, recognizable only by the fence behind the original parking lot. It’s all parking lot now. One thing to be said about Christchurch is that, on average, you can see farther now.

I’ve laid down a lot of pre- and post-Antarctic memories in these places so it was a blow to see them gone… gone so completely that it was hard to even tell where they’d been. Certainly a vivid illustration of the impermanence of things. Walking back towards the hotel I was snatched up by IceCube colleagues driving by: Stephanie and Anthony from Canterbury University. We were joined by Sarah at a combination cafe and garden store, where I had a delicious salad (still making up for polar freshie deficit) and “chocolate ice” (a sort of shake made with chocolate and whipped cream). After lunch we drove downtown to the new Cashel Mall, an area that was hit so hard that no original buildings remain. Instead, a few dozen structures made of shipping containers occupy a roughly one block area, serving as a new shopping mall. This sounds dubious on paper but in execution it was, like many things New Zealand, fabulously gorgeous and chic — imagine FEMA as tutored by hip architects gunning for a multi-page spread in Dwell Magazine. Walked back alone past the Christchurch art gallery, the Arts Centre, and the Dux De Lux, all still closed. Then through the Botanic Garden, in full bloom and betraying no evidence of seismic calamity, and “home” for Thai dinner and an early bedtime to prepare for the early morning departure of the train.

Christchurch and McMurdo seemed to lift me out of my slightly numb South Pole funk. In McMurdo, I felt some of the thrill towards the gritty texture of the town I used to experience each time I came down. Much of that texture has been bypassed in recent trips because they were straight-throughs, landing at Pegasus only to change planes.

My roommate in McMurdo was Bill, a kindly videographer from Chicago who’s lived in Louisiana long enough to acquire the Southern drawl. Bill said out of the blue, “It really is the simple pleasures down here which make life worth living.” A couple of showers and salads and nine hours of sleep, concurrent with continual re-saturation in oxygen, and I was inclined to agree. By Wednesday morning, I had enough time and energy to walk to Hut Point and summit Ob Hill before lunch, taking lots of pictures of “milvans” (shipping containers), buildings, and rock and ice along the way. At Hut Point I sat for awhile with eleven seals sunning themselves nearby on the ice, some the size of small Chevies. Trails from the seals to nearby holes in the ice which looked far too small to accommodate the animals… and some blood stains near one of these were a reminder that this was not a sanitized game park but actual habitat, with life and death coexisting precariously as always… perhaps more so, at the base of Mount Erebus (whose volcanic plume could be seen in the clear, continual daylight), there on the edge of the highest, windiest, driest, coldest, iciest, emptiest, cleanest, darkest, quietest continent.

At lunch I met a longtime colleague Ryan, waiting for his LC-130 ride to Wais Divide. Ryan went to the Pole as an undergraduate on AMANDA, years before I did, and now he does mostly glaciology and climate studies at UC Berkeley, logging dust concentrations as a function of depth in deep holes across Antarctica. We talked about deploying to the Ice over and over again. His perspective interested me. “It’s a nice break from your routine, back up North. And you come back stronger each time.” I thought about it for a bit, and I think I agree. As one gets older it is easy to get into a comfortable routine — I have a really good life in the North: Eden, meditation, art, work, Chicago, Madison, family and friends. But the months do speed by, and deviations from routine seem more and more jarring. Traveling does break you out of that. Even simple things like remembering what needs to be done without relying on a computerized to-do list seem easier after a few weeks of having to keep track of gear, travel dates, and ever-changing circumstances.

I had time for a nap before reporting to the Movement Control Center for the flight to Christchurch. Eighteen of us packed our gear and selves into a Delta and headed out to the Ice Runway, one of the last flights to do so this season (flight operations transition to Pegasus airfield as of today, considerably farther away from town). Despite the quick trip to the plane, we wound up spending a good hour parked nearby, while the back of the Delta gradually turned into a greenhouse, cooking us in our ECW gear until we opened, against strict instructions not to, the back door of the vehicle to let in some cool air. Finally we made it on board and got a good look at the cause for delay — two empty liquid helium dewars, each the size of a small school bus, which no doubt were tricky to tie down correctly (I, for one, was happy for the extra attention to detail on the matter).

Eighteen passengers is a pretty light manifest for a C-17. After takeoff some people took several seats along the side walls and managed to sleep that way for most of the five hours to Christchurch. I was perhaps the only one to mosey up to the flight deck. Talked with one of the pilots for awhile, who said he’d been to 55 countries since he started flying in his squadron. Most commonly Iraq and Afghanistan, of course, but also frequently to Southeast Asia — they are apparently still finding remains of American soldiers in Vietnam (DNA techniques being the evident game-changer there), and aren’t above sending C-17s down to collect them, with appropriate ceremony, one assumes.

We’re pulling into Kaikoura now, so I’m going to tie this off for today and get a breath of fresh air. Already saw some seals in the water from the train. Liquid water, as far as the eye can see… what a strange and marvelous improvement, over an ocean of ice.

New train in Kaikoura

Marlborough Sounds

Oxygen

Tue, 29 Nov 2011 00:00:00 +0000

Nov. 29, 2011 17:30 NZDT Crary Lab, McMurdo Station

Made it back to McMurdo where liquid water runs in the streets, mountains can be seen in all directions, helicopters take off and land every twenty minutes or so, and the oxygen flows thick and warm into your lungs.

Our flight to McMurdo had only five passengers. After packing a lunch and waiting with bated breath for the announcements from Comms indicating the plane’s arrival, I headed out, with Carlos, Sven, Ralf, Gary, Martin and David coming out as well to say farewell (thanks guys!). We took a few group photos in various combinations of cameras and individuals, and then those of us who were leaving headed onto the plane. There were no instructions of any kind from the crew (indeed, it would be hard to hear anything anyways)… just strap in and off you go… and so, in a few moments, we were aloft over the Polar Plateau.

Leg room

I managed to sleep a bit after take-off (I’ve been up since 4 AM), slumped over and wrapped up in Big Red, with no shortage of leg room (the plane was nearly empty). Read a bit, ate my makeshift burrito-and-cookie lunch, and then took pictures out the windows for 45 minutes or so.

After ten trips here, one thing I do not tire of or take for granted is the view from the plane of the Trans-Antarctic mountains. As a Midwesterner I have always found mountains to be a bit alien and fascinating, but mountains in Antarctica are ever more so. I think I’ve probably taken the same photos of the same features over and over again, but the terrain is stunning. It feels like looking down from orbit onto another planet — a land you know you will never set foot upon (and, in many spots, nor will anybody), but you cannot help but wonder what it would be like to see it up close.

Arriving in McMurdo

We arrived in McMurdo to a buzz of activity. There was a mass casualty drill just before we arrived; they are also packing up the Ice Runway in preparation to move out to Pegasus airfield, much further away from town. We had to wait for a shuttle into town for half an hour or so. When the shuttle did arrive, both passengers and crew from South Pole piled on, as well as other airmen, and crew and passengers from a Twin Otter flight arriving from somewhere out in that aforementioned crazy terrain. I had to ask around to figure out where I would be staying the night — finally I found my room, got some linens, headed off to Crary Lab for Internet access and now am about to head off to the dining hall for dinner. Tonight we weigh in with our bags and gear and should get the flight schedule for the trip back to Christchurch tomorrow.

Starting tomorrow, hopefully, I am going to get spoiled with a week of summer before heading back to the beginning of Chicago winter. But first, dinner.

Ghost

Mon, 28 Nov 2011 00:00:00 +0000

Nov. 28, 2011 15:51 NZDT B2 Science, Amundsen-Scott South Pole Station

Walking home

Last full day here, insh’Allah. We had flights today, so whatever weather problems were plaguing McMurdo seem to have sorted themselves out. I’m ready to leave, despite the short and comfortable trip (relatively speaking). So far on this trip, unlike some previous trips, I have not had to:

Go to Medical for assistance with altitude/sinus/cold/flu issues (roughly 1/3 of trips)
Work outside for 16-20 hrs straight (AMANDA deployments, ’97, ’98, ’00)
Give an impromptu presentation or tour to Distinguished Visitors or 'tourists’ ('08 at least)
Implement software without which IceCube would not be able to take data ('05, ’06, ’07)
Prototype a brand-new Experiment Control system in situ ('08)
Sleep in Summer Camp ('97, ’98, ’00, ’05)
Try and sleep during daytime through fire alarm testing ('07 at least)
Wait due to flight cancellations in McMurdo or Christchurch (knock on wood)
“Boomerang” (have flight turn around, a relatively common occurrence which I have so far avoided) (” “)

This is going to be short because I have to go pack. No satellite connection to the real world anyways at the moment.

21:54 NZDT Greenhouse

Now it’s a few hours later. Having cleaned out my room and packed, eaten paella and bread pudding in the galley, dropped in on Pole Mart (the Station store), chatted for awhile in B2 and watched two episodes of Firefly in the video lounge, I’m headed now to bed but am first making a slow farewell circuit of the Station — possibly the last one “for awhile” (let’s leave it at that). After hitting the sauna for a few minutes (yes, there is a real cedar sauna in the Station), I’m sitting on a couch in our tiny greenhouse, to write and read a bit for a few minutes before bed.

Leaving is always bittersweet, but differently so now. As I was sitting in the sauna I realized that, having said my goodbyes so thoroughly in January, I feel a bit ghost-like here, like someone who needs to move on but hasn’t yet for some reason or another. It’s a feeling I think I need to take seriously, regardless of what happens in the future.

I think that’s all I have to say for tonight. We shall see what the trip North brings tomorrow.

Update Nov. 29, 08:24 NZDT B2: Plane from/to McMurdo is en route, so my next post will be from there or Christchurch….

Today's Flight Schedule

Turkey, Stuffing, Eclipse

Sun, 27 Nov 2011 00:00:00 +0000

Nov. 27, 2011 18:37 NZDT B2 Science, Amundsen-Scott South Pole Station

The iClipse

Interesting last couple of days. Friday (two days ago, for us) we had a partial solar eclipse, my second here. Whereas two years ago I traipsed outside with colleagues to take pictures, this time I stayed in the Galley where a viewing party formed. Aside from being much warmer and cozier, this also turned out to be a good move because, while nearly perfect for every other photographic task, my little Canon S95 was not up to direct shooting of the event; a big DSLR with a telephoto lens (and a mylar partially-silvered filter) is really needed to do justice to a direct exposure.

That's a nice eclipse you've got there, ma'am….

70% totality

Getting just a little bit closer to that eclipse

Fortunately, one scientist set up a pair of binoculars in reverse atop a tripod, projecting the image of the sun on a cardboard box on an adjacent table for convenient viewing and photographing. This was cool even before the eclipse started, since you could see sunspots on the projected image. The eclipse got to about 70% of totality, during which the snow outside turned the same liquid-silver-grey colors I saw here two years ago during the last eclipse. Quite beautiful and a little eerie. The viewing turned into quite a gathering, with maybe fifty or so people turning up. A few die-hards also went outside to view the mercurial snowscape up close and personal.

Carlos and Julie at Thanksgiving

Yesterday, finally, was “Thanksgiving,” or Thanksgiving dinner at any rate, two days after the official date — my first major holiday meal at the Pole. Nearly the entire station turned out for appetizers, followed by dinner, in three separate seatings (most of us IceCubers took the 2nd). The galley windows were blocked off by screens, and lit candles adorned the tables — probably the first open flames I’ve ever seen at the ever-so-fire-safety-conscious South Pole. Rather than displaying information about how cold it was outside or the day’s flight cancellations, the overhead monitors showed images of flickering fireplaces. It was downright cozy, I have to say, and we had a fabulous meal:

Turkey
- Smoked
- Roasted
- Fried
Mashed potatoes
Sweet potato medley (probably kumara from New Zealand – tasty)
Stuffing
Gravy
Cranberries
Fresh salad with REAL tomatoes and REAL cucumbers
Pie with whipped cream
- Pumpkin
- Walnut
- Apple
Red and white wine
Other great stuff which my food coma has forced me to forget

Dinner is served

After our seating, several of us stayed to help out with the next. I served wine and pie for an hour or so, then helped with the dishes. It really felt like a community effort, with so many people helping, from the kitchen staff who clearly had worked hard for days, to Katie the station manager making sure the place settings were correct, to several wine-n-pie servers. Fun, also, to do physical work with other people — while bustling to and fro, pouring wine, busing tables, and cleaning plates, I remembered taking pleasure in moving quickly, with purpose, with others, in my restaurant job days (so many years ago now).

Today was fairly quiet, with much of the station recovering from dinner and the follow-on parties (which I mostly skipped, creeping back to my room and watching Avatar on my laptop before conking out). But the friendly communal feeling still hangs in the air, a feeling I will miss when I head North in two days… something I am nevertheless very much looking forward to doing (weather and mechanical gods permitting).

Serving wine at the third seating

Wind Storm and Moon Dust

Fri, 25 Nov 2011 00:00:00 +0000

Nov. 25, 2011 15:32 NZDT B2 Science, Amundsen-Scott South Pole Station

The "South Pile" in windstorm

For those of you who don’t know, December and January are the centennial anniversaries of Amundsen and Scott’s arrival at the South Pole, respectively. I think a lot of people are writing online about the event, and at any rate, I’m going to miss the celebration here, so I won’t go too much into the preparation and planning that’s being done to prepare for a record number of 'tourists’; but it is interesting to contemplate the history a bit and to think about how comfortable it is here compared to what Amundsen and Scott went through.

Today we received an email 'status report’ of where Amundsen and Scott were about this time one hundred years ago. Amundsen recorded that the dogs were doing well and that he had pitched his tent in the most beautiful campsite in the world. Scott wrote, “quite the most trying march we’ve had…. If it were not for the surface and bad light, things would not be so bad. There are few sastrugi and little deep snow. For the most part men and ponies sink to a hard crust some 3 or 4 inches beneath the soft upper snow…. Our forage supply necessitates that we should plug on the 13 (geographical) miles daily under all conditions, so that we can only hope for better things. It is several days since we had a glimpse of land, which makes conditions especially gloomy. A tired animal makes a tired man, I find, and none of us are very bright now after the day’s march, though we have had ample sleep of late.”

Other interesting tidbits on Thanksgiving Day (in the States, anyways — it’s Friday here): there is a partial solar eclipse this evening at about 6:30. Unfortunately it is cloudy and windy here and you cannot even see the horizon. Yesterday is was even windier. We have not had flights for three days or so because of the weather here and in McMurdo.

Halo in windstorm

Last night I went to the greenhouse to read a bit before bed. I’ve learned that it’s OK to be in there when the greenhouse guy is working, so I didn’t clear out when he came in. He introduced himself as Jon and we talked for quite awhile. We both agreed that one of the best things about coming to Antarctica is meeting all sorts of really interesting people and learning about what they do, but Jon himself personifies that very phenomenon. He explained in fascinating detail about the greenhouse, its control systems and its yearly cycle of operation. In the winter, it produces enough veggies for one salad per person each week (!). But what I found most interesting is that he also studies greenhouse designs for possible lunar and Martian habitats. Did you know that there are recent indications that the dirt on Mars is extremely toxic, which could severely impact the possibility for creating long-term habitations there? That the International Space Station has (or has had) a small greenhouse that revolves around an axis to create simulated gravity for plants? Or that the lunar south pole contains lots of water ice and is possibly the coldest place in the solar system? He has also touched (studied) moon dust — I dare say a lot more people have been to the South Pole than have touched objects from the moon.

Today I’m feeling a bit under the weather, perhaps due to something in the food not agreeing with me. But I have less than four days to go, the winter-overs are doing well, and I’m eager to start heading back northwards. The weather will have to clear up first, however!

Ceremonial Pole from B2

Sven and David

Shower Instructions

Wed, 23 Nov 2011 00:00:00 +0000

Nov. 23, 2011 14:53 NZDT B2 Science, Amundsen-Scott South Pole Station

Forlorn Slippers

How to take a twenty second shower

First, work up a sweat in the gym or the sauna. Change out of all clothes except flip flops (you might want to go into the bathroom first). Turn water temperature to maximum heat without turning on water yet. Get towel, soap, washcloth and shampoo ready.

Apply water to washcloth for 1/2 second, then turn off shower immediately. Wet hair and dampen body. Apply shampoo to hair and lather. Rub end of soap on towel to get it soapy. Rub soapy towel across entire body and scrub with towel.

Step into shower and turn on water. The first five or six seconds of water will be COLD. Use this to rinse hair. As the water warms, rinse off the rest of your body. It only takes a few seconds to get the soap off (short hair is definitely an advantage here). If the water gets too hot either turn off the shower and get out or turn the heat down for the last five seconds.

Dry off and enjoy that clean feeling.

Busy last few days. My departure time has been moved forward by two days, so I’m pushing a little harder to get through all the training exercises I came up with for the WOs, as well as helping out with some software upgrades on our servers. My ragged sleep schedule is back, but I’m in pretty good spirits today. I’m also keeping busy in my free time, going to the gym, reading a novel (“Surface Detail”) left here by ex-WO Freija, listening to Neal Stephenson’s REAMDE, learning a new programming language (Clojure), and have started writing a book of my own….! (Don’t hold your collective breaths, it’s going to be awhile.)

Thanksgiving is on Saturday here. That’s Friday for most of you; I suppose the reason for the delay is to avoid having to give people an extra day off (as far as I can tell, non-scientists here get one weekend day off per week; scientists take off when they want, which is usually never). That may seem unreasonable but if people take time off there are no flights, no cargo, no meals, etc. — it costs a lot to keep people here so I guess it makes sense to use that time as productively as possible. Thanksgiving and Christmas are a big deal at the Pole (I’ve heard). There are three seatings, the second one of which I signed up for, as well as dish duty afterwards. I like washing dishes here — it’s an excuse to plunge your hands into warm water (compare with shower instructions, above) and move around after a day sitting at the computer.

Gary prepares to meet the plane

Off-roading it

Fresh Air and Bananas

Sun, 20 Nov 2011 00:00:00 +0000

Nov. 20, 2011 12:49 NZDT B2 Science, Amundsen-Scott South Pole Station

Peek-a-boo with the Ceremonial Pole

Sunday here, so a light work day. Station has been pretty quiet all day. Slept a solid 8 hours, had a banana and oatmeal for breakfast, then suited up and went out for a walk. It’s funny, yesterday morning I had a dream about bananas, and then the first bananas showed up on-station. I’m eating one now (hi Dad!).

If you don’t go outside you forget how amazing it is out there. You get to feeling like you are living on a cross between a cruise ship (comfy and isolated) and an aircraft carrier (not so comfy but still isolated). We are indeed floating on an ocean… a solid one. But the solidity of the water doesn’t always make it easier to get out and about, if you’re sleeping on station and your duties don’t take you outside.

Get out, though, and it’s another story. The air is cold, yes (100 degrees colder than inside the station) but it is fresher than any most other air you could conceivably breathe — the cleanest air in the world, they say. When you walk, the snow squeaks like styrofoam. It actually sounds hollow and had me wondering if it was at all possible to encounter crevasses at this part of the Polar Plateau. After several days of crystal-clear skies, there are a few clouds, doing their usual it’s-so-cold-and-flat-here-let’s-also-do-the-flat-thing thing. Nearby, the power plant belches a steam cloud the size and speed of about one Graf Zeppelin flying away every ten seconds.

Went out to the Geographic and Ceremonial Poles just to say hi and look around. I have been noticing that the horizon here is not strictly flat — there is a slight rise between the end of the skiway and the Clean Air building, which is easy to see if you look for it from the Galley, and still noticeable from the Pole markers. What passes for terrain is a barely visible inflection of an otherwise perfect horizon.

Went to the gym and did some stretching, sat, and just got back from brunch. In short, between some fresh fruit, a couple of good night’s sleep, and a stroll, I’m in better spirits today. I know that getting outside is part of it and so I moved all my ECW gear to the coat room near Destination Zulu so I don’t have to drag it out of my room every time I feel like a stroll. I don’t think I want to do a lot of running outside (my co-worker Carlos regularly does 10k or more out on the skiway—he just passed in front of the window in front of my desk mere seconds ago, covered in frost), but will make more time for walks, and continue my runs on the treadmill.

Traveller and the Human Chain

Thu, 17 Nov 2011 00:00:00 +0000

Nov. 17, 2011 11:39 NZDT B2 Science, Amundsen-Scott South Pole Station

Summer Camp from the A4 wing

Just helped form a human chain to bring the mail into the Station, brought to Destination Zulu by a snowmobile/sled combo and a front loader from Cargo, and swapped hand-to-hand up the stairs into the airlock to the Post Office. Mail is a big deal here, and I’ve enjoyed getting it in the past, so it’s fun to be a part of that, even if just a few minutes.

Though I’ve continued to acclimate and have started running in the Gym, I have not been sleeping well. Not sure why, because, though noisy, it’s not as loud as it’s been in the past. It’s still bearable. But I have to admit it feels as there is no getting away from this place sometimes. Getting through a few weeks is enough of a challenge that I find it hard to imagine a whole year (much less multiple years like some winter-overs here have done). I do remember, however, that it always seems to take a period of adjustment to get into a rhythm here. The irony of the short trips I usually take is that I’m generally more or less settled in by the time I leave.

At some point I owe returning readers to this blog an explanation. Though I’ve always looked at my trips to the Pole as potentially “the last time” (as any visit to an extremely strange/strangely extreme place can be), I was pretty sure last season WOULD be the last time, since we were done with IceCube construction. I kind of made a big deal out of it, and said my goodbyes to this place in this blog last January.

Clearly I was wrong, since here I am. The reason why is that we still have to run the experiment we built, at great cost and effort, for many years to come. Each season we have new winter-over scientists, who have to get trained in many different, rather technical and esoteric areas relating to the operation of the detector itself. This includes the maintenance of roughly 150 servers in our laboratory building, keeping an eye on network switches, disk and tape drives, UPSs, etc.; and running a dozen or so homebrew software subsystems, each written (mostly) by physicists, and each with its own characteristic quirks and features. Though we do some days training on a test system in Madison, Wisconsin, there is no substitute like training in situ, and so my task for this early-season visit is to help facilitate that training, to answer those questions I can answer, to provide exercises and training tasks which will help them prepare for the coming year. My hope was also to get a bit more caught up on various programming tasks (a hope so far not really realized).

All this raises the question, how many more times am I going to do this? Some of you probably figured out long before I did that I might have to actually “decline further missions” if I want to be certain that this trip (or any future one) is the last.

To contemplate further trips requires some notion of how such work aligns with one’s goals and aspirations.

In the 1980s, at the height of the Dungeons and Dragons-inspired craze, my friends and I would occasionally take a break from D&D to play a science-fiction variant called Traveller. The basic idea is the same as D&D — one person 'creates the world’ and the other players create characters who do various things in the world, with a sort of story unfolding spontaneously out of the interactions between the characters and the world, and each other. But rather than knights and magicians entering lairs of monsters, in Traveller, one learns esoteric skills and then travels from planet to planet to carry out various adventures. Worlds can vary wildly in climate, culture, and technological sophistication, from backwards forested planets to dazzling urban arcologies to frozen wastes….

The first few times I arrived at Pole, it felt like I was in a Traveller game. The military modes of transportation; the quirky, unwashed characters; the utter lack of scenery; the only-slightly-breatheable atmosphere; and the elusive sub-atomic quarry we were hunting… it was all just so damned sci-fi. Of course the old Station with its ’70s-era geodesic aluminum dome and the somewhat ramshackle orange buildings within, the slightly chaotic, dirty charm of the place, contributed to the atmosphere of duct-tape-and-gritty-humor. I loved that aspect of it… in a way, I guess I was living out the dream we had carved out gaming in my friend Fred’s basement fifteen years earlier, and in the books and films I consumed wholesale throughout my (er, extended?) adolescence.

Now, much of that feeling has been worn away with familiarity. I’ve been asking myself what’s left when you take away that sci-fi feeling. The South Pole is a job site. People make money here, make and lose friends, fall in and out of love, go to the bathroom, get inspired or depressed (or both), have accidents, get older. What arises to fill the absence of the fantasies that this place used to fulfill? Other than, of course, the physical and emotional discomfort — sleeplessness, chapped hands/lips/elbows/…, headaches, light-headedness, shortness of breath, sinuses made of wood, lack of fresh fruits and vegetables, two minute showers twice a week, lack of privacy, restricted entertainment options, lots of rules, nowhere to go.

I don’t have the answer now but I’m living it, whatever it is. It seems to be something about how to maintain cheerfulness under discomfort and duress. About trying to stay curious about other people when your own life doesn’t feel that easy. About the gap between when someone coughs in an adjacent berthing and when the wind rattles the wall behind your pillow. About enjoying small things like a good cookie or a shared laugh. About accomplishing what you set out to do, with as much contentment as you can manage. And, when you leave, about the satisfaction of seeing night time, and leaves, and (liquid) water, and loved ones again.

Regardless of whether or how I return here in the future, at the moment, getting to know some of these things better is the way forward: both the effort, and the reward.

Gary tries out a helmet

Departing Hercules

Shadow and Summer Camp

Station Exterior

Destination Alpha

Carlos after a 10k run at -40F/-40C

Reveille

Mon, 14 Nov 2011 00:00:00 +0000

Nov. 15, 2011 05:36 NZDT Berthing A4-209, Amundsen-Scott South Pole Station

Fire sled next to refueling pit

Well, that’ll wake you up nice and quick. We just had a fire alarm which went on for long enough to get everyone out of bed and the fire team half-dressed for action. False alarm, apparently.

I’ve had a checkered history with fire alarms here. A few years ago when they were testing out the system, I was on night shift. Of course, testing only occurred during the day. So for about four days I could be practically guaranteed to be pulled out of bed (or at least woken up by the shrieking of the alarm in my room) a couple of times a night. Sleep is a precious commodity at South Pole (altitude and noise conspire to make it challenging for me), and I was not a happy camper.

But testing or no, frequent fire alarms are a fact of life here. Though it may seem counter-intuitive with the cold, the practically zero percent humidity and frequent high winds make fire an ever-present hazard. Couple that with the remoteness of the Station and, well, you take it seriously. If a fire starts, help will not arrive from McMurdo on time, except to collect the injured.

And even that assistance can only come in summer. As I understand it, every winter-over is either on the fire team or the trauma team. There is no “Let’s sit around and let’s see how it plays out” team.

We had a real incident here a few days ago with a glycol spill in the power plant. Fortunately everyone was OK but we had to reduce power consumption for several hours afterwards while they cleaned up and brought everything back online. A similar episode happened a few years ago which resulted in injuries and medical evacuations.

So I’m OK with false alarms.

Now to try to get back to sleep. Morning, all! :-)

(Postscript: beat the odds and managed to sleep a few more hours. Sometimes you’re so tired that you can just sleep no matter what.)

Drifts

Sun, 13 Nov 2011 00:00:00 +0000

Feeling a bit run down this afternoon. Got up and decided it was my first “workout” day, working out consisting of twenty minutes of walking since this is still just my fourth day here. Had a nice walk and a bit of a stretch, and then a SHOWER, my first since McMurdo. Feeling pretty peppy, I had decided today would mostly be a day off, to rejuvenate and prepare for the week ahead. I was headed back to my room to practice, and ran into WO Carlos who informed me that a server had crashed in the IceCube Lab (ICL) and that the detector was therefore running short one string. So much for a day off, but this was another training opportunity, training being primarily what I came down for. Sven and Carlos and I suited up and Sven got a snowmobile with passenger sled ready (I was happy not to have to walk the half-mile distance there and back, having perhaps already pushed it a bit by walking in the gym).

I’ve been thinking more about what I wrote the other day, about being tuned into differences between this trip and previous ones. In fact, I have mostly been struck by the sameness. The rhythm of the plane rides here, the same faces year after year, the process of acclimatizing to altitude, the same nooks and crannies (unmarked bathrooms, locations of semi-private phones for calling home) — much has simply not changed at all (at least in the past several years, since the new station was built). It feels as though this place is on another plane altogether, not part of Earth or the world at all, a kind of parking ground for (temporarily?) lost souls. I would liken it to the Tibetan notion of bardo, but bardo is an in-between state between incarnations, and this place doesn’t feel in-between at all to me, but somehow out of the normal flow of things. Limbo, perhaps.

A shift in perspective came to me as I was riding the sled out to the ICL. The winds are still quite high, keeping the wind chills below -60F, and making snow drifts and sastrugi everywhere. As we crossed the skiway I looked out across the vast space and thought about the drifting snow, the constant shifting and drifting, and realized that the change I was curious about, in myself and in others, is more subtle, a building and shifting over time which is hard to notice even on the time scale of months or years. Yesterday in the galley a colleague of many years, Robert, looked at me and said, “You’re getting grey, man.” True enough. The wearing of time sometimes is noticed not in sudden shifts but in gradual drifting, accumulation, wear and tear and the greying of hair. Doesn’t sound cheerful, I guess, but it was actually a pleasant notion to contemplate as we zipped across the wide white waste.

After which, of course, we quickly fixed what needed fixing and headed back for brunch.

Bon Voyage

Fri, 11 Nov 2011 10:51:00 +0000

Nov. 12, 2011 11:46 NZDT B2 Science, Amundsen-Scott South Pole Station

Jens and Freija departing South Pole after 12 months

Just said farewell to IceCube winter-overs Freija and Jens on the “flight deck” (outdoor waiting area near the plane… wind chill -65 F). Bittersweet partings for all concerned, I’d say. Their plane is taxiing as I write this. Carlos and Sven take over now as winter-overs for the coming year.

This was the first time I’ve been outside since I arrived two days ago. It was cold but great to get out. I’m feeling even better today and may try a walk in the gym tomorrow.

Heading out to meet the plane

Farewell

A Nicer Guy?

Fri, 11 Nov 2011 06:08:00 +0000

17:01 NZDT B2 Science, Amundsen-Scott South Pole Station

En route to South Pole

Been here just over a day now and am settling in. Once I get here it’s easy to settle into a work routine, which makes it harder to do much writing. But I’m still acclimating to the altitude, trying to rest and stay hydrated, and the schedule isn’t too intense so far.

Did a debrief today of our outgoing winter-over (WO) scientists, who are leaving tomorrow, and did some training of our new WOs as well. Our old WOs are in good spirits but also happy to leave, and I don’t blame them. In addition to having been here for most of a full year (with just a brief R&R break before station closing), it sounded like a pretty challenging winter, with medical emergencies you can Google about (and some you can’t). I’m glad I came when I did because it has been fascinating to hear their stories and their perspective.

McMurdo Evening

Building 155

My last post left off somewhere in the air between New Zealand and McMurdo. We arrived safely at the seasonal Ice Runway and were quickly put onto Ivan the Terra Bus. I couldn’t believe how close we were to town — the last several seasons I’ve landed at Pegasus which is a good hour (at minimum) from town. But Observation Hill and McMurdo proper were clearly visible and seemed but a stone’s throw from the Ice Runway, which I think they only operate early in the season when it’s cold enough to land there. The trip into town took a mere 20 minutes or so, after which we were briefed on life in town — a briefing which I’ve attended ten times now and portions of which I think I can recite from memory, despite the fact that much of it doesn’t really apply to short-timer Polies (I have never done laundry in McMurdo, used the gym, or even walked very far out of town, something which one cannot do without attending an outdoor safety lecture).

Street texture

From the briefing I went to Crary Lab, got my laptop screened, and checked email, and then went to settle into my room… but not before grabbing someone else’s Big Red parka by mistake (they all look alike, and his name is similar to mine, and …). Before long I noticed my gloves, which were in my parka pockets, were missing, so I figured I’d left them somewhere, and chased around trying to find them for an hour or so (losing and re-finding USAP issue clothing seems to be my karmic fate each and every trip). My hat, I noticed, was also gone — that tipped me off to the fact that I had the wrong jacket, so I went back to Crary lab, gave back the jacket I “stole,” got mine (gloves and hat included), and went to a very crowded galley for chow. After dinner we weighed-in and I managed to get to bed early, in a room with only one other fellow in it (McMurdo facilities manager and unicycle aficionado).

Flight manifest

Our flight schedule was not nailed down by the time I went to bed, so I was up at 5 AM and went off to check the flight manifest. By 8 we were at the Movement Control Center along with passengers headed to two other places on the continent. Our flight to the Pole consisted of just five regular passengers, plus the Guard crew, and a woman, already on the plane, who I assumed was a Distinguished Visitor or reporter of some sort. The plane was jammed full of cargo and some medical equipment and stretchers I assumed were bound for South Pole Medical. Later I learned that our flight was actually a medevac flight and that they were taking someone back to McMurdo in a hurry, which may have explained the rather hasty briefing (barely audible over the roar of the props) that we got on the plane.

Baseler bound for Byrd Camp

Though there were minor differences when compared to previous trips, it was mostly quite familiar, and I found myself going through the motions, not really being surprised by anything. At one point I sort of woke up and said to myself, Dude, you are on a military aircraft bound for the South Pole. Pay attention.

That more or less worked, and though I did play with some programming stuff on my laptop for a lot of the flight, I also enjoyed the strange familiarity of it, the familiar strangeness. Somehow being bundled up in ECW gear and earplugs in a roaring C-130, with all its green guts clearly exposed for the viewing, just feels like the canonical Antarctic experience to me.

I managed to get up to the flight deck around when we passed over the Trans-Antarctic Mountains. The vast Beardmore Glacier stretched below, through black ridges of rock, a creamy, motionless river riddled with great crevasses which looked like tiny scratches at that distance. Though our window views were limited throughout the flight (due to the bulk of the cargo), that view out the front windows of the aircraft was stunning. The crew made sure I could take pictures and video from various angles, moving aside and motioning me forward. After I got my shots in, I talked to one of them and asked him about the geography we were passing through; he said, “I don’t know, I’m just a doctor.” (Presumably he was assisting with the medevac.)

Video from Flight Deck

At some point between McMurdo and arriving at Pole, maybe on the airporter bus to the Ice Runway, I thought about how much of a collaborative effort all of this is, this whole Antarctic program. The shuttle driver; the passenger manager who gave us our lunches, earplugs and a safety briefing; the pilots and crew; the airplane mechanics; the satellite techs; the waste management folks… there are so many people working together here, professionally and (for the most part) harmoniously… so many people doing their jobs, to keep us and each other alive and safe and to enable scientists to do their jobs. It is quite humbling, actually, to be on the receiving end of that. That of course is the reality of the “Real World” too — it’s just that so many of those people Up North are actually invisible… the water treatment engineers, air traffic controllers, homicide detectives, construction workers, priests… people you don’t normally see but who keep you alive and safe. Here, things are turned inside out, the machinery is exposed for all to see, you see each other more — everybody pitches in, whether it’s “house mouse” duties (cleaning bathrooms, etc.), working in the dish pit, helping to load “freshies” into the galley, … you eat dinner with the people here who load your cargo and handle fuels from the Hercs, with the cooks and the managers and the dishwashers. People who could and would save your life if needed. Somehow it’s just more clear who you rely on here… and they rely on you too. It’s a good experience to have.

(It is also interesting to go somewhere where your comfort is not anybody’s primary concern, and where nobody is trying to sell you anything, but perhaps that’s a topic for another post.)

Briefing prior to boarding LC-130 to Pole

Arriving at the Pole was a rush, as always. It was about 40 below, maybe -66F with the wind chill. Freija, Sven, and Carlos met us at the skiway. Strangely enough, of all the places I’ve passed through in the past few days, this place feels most like home. And, as usual, I know a bunch of people here and recognize many more faces. But it’s less crowded than I’m used to, and I’m a bit more relaxed too, both of which are nice.

I sat down at lunch with a communications engineer named James who I met last season. James likes to discuss particle physics and quantum mechanics. I sat down with him and a GA named Pete, a young fellow with a robust blond beard who fights wildfires back in the real world.

James, sitting across from me, asked, “So, have you learned anything in the past year?”

“You mean has IceCube made any discoveries, or have I personally?”

“You personally.”

I thought about it. “Yes, I think I’ve learned a lot, actually.”

“Such as?” asked James.

“Well, I think I’ve really learned a lot about programming.”

“Okay… I guess that’s good,” he replied.

“Also, I think I’m a nicer guy.”

At my side, Pete nodded sagely, and said, “That’s good. That’s good.”

It is.

The Quiet Earth

Mon, 07 Nov 2011 18:21:00 +0000

2011/11/8 05:01 AM NZDT Christchurch

Spring in Christchurch

Woke up a few minutes ago and started thinking about a blog post. I would have happily slept more but my body tells me that’s not going to happen. It’s quiet here, with the only sound the faint snoring noises coming from the next room over.

I’m at a hotel in an area of Christchurch called Papanui whose distinguishing feature seems to be a shopping district with a mall and a number of smaller stores and eateries. The shops include several Asian restaurants, a comic store, and a small model train store. It’s not bad area, but it feels like it could be anywhere in New Zealand. I’m a bit sad not to stay at the Devon B&B, not to hear the trolleys rattle by every so often, not to wander down to the Arts Center or stroll down to the shops near Cathedral Square.

After my arrival yesterday, I tried to get out see a bit of what could be seen. There is a rail line nearby with a path alongside it leading to Hagley Park, which is close to the center of town near where I used to stay. I went for a run down this path, perhaps a few miles, until I reached the far side of the park. But I was out of water and energy, so I didn’t go further.

Perhaps I’ll make it farther today or some other time. But, truth be told, I’m not sure I have the stomach to see it. When we arrived at the airport yesterday, we were met by a young woman who briefed us on the conditions here. She made sure we kept our emergency information cards on us at all times. She told me that, though she was from Christchurch, she had yet to get the nerve up to go downtown and look around. The cordon around the entire downtown area was apparently just opened up a week or two ago — she said that the area is mostly a write-off, with most buildings needing to be torn down and rebuilt. Most of the Christchurch I have come to know over the years is apparently gone. The newer parts around here seem mostly fine (aside from a few fenced-off buildings), but the main reception area of our hotel is closed off, with a rather lovely, elegantly-designed shipping container structure serving as temporary office.

Ice People checking in at temporary reception container

The young woman described what it was like when the big earthquake hit on February 22. She said that, out by the airport, it was eerily silent just after the big shakes stopped. I was reminded of the 1980’s New Zealand film, The Quiet Earth, in which most of humanity is suddenly spirited away. She said they could tell by the number and intensity of the sirens which gradually filled that silence that it was very bad for the downtown area.

Fenced-off buildings in Papanui

My trip to Christchurch was pleasant. I asked for, and got, an exit row window seat from Chicago to LAX. It even looked as though our row’s middle seat would be empty (even more room!), but at the last minute a large, Middle Eastern man got on and sat there. As we taxied he pulled out a small book in Arabic, which I guessed was the Koran until I noticed the picture of Jesus on the cover. We conversed a bit and he told me he was a Copt, Copts being Egyptian Christians who, according to Wikipedia, were the dominant religion there before the Muslim conquest in the seventh century. He told me of recent atrocities against Christians in Egypt by the majority Muslims, and said that despite the Arab Spring things were not good now in Egypt for Christians; in fact, he had won the Green Card lottery in the States and went back to Egypt to settle some affairs before moving to LA permanently. Nice guy.

At LAX, I managed to recharge my laptop and iPhone batteries at a charging station and get my boarding pass for the Qantas flight to Auckland. I asked if there was any choice of seats, and the guy at the gate said, “no, but I got you an aisle seat.” Fair enough, I thought, and went back to the charging station to do some work. When they started calling my flight, I looked at my boarding pass and did a double-take: they had upgraded me to Business Class! I was pretty excited about this — the difference between Business and Economy for me is 1-2 hours of sleep versus 5-7 hours, due to the extra space and the fact that Business seats recline almost fully. I marched right on the plane and sat down in the first row of seats and was happily chatting on the phone and sipping orange juice in minutes. The flight was very smooth; I really enjoyed the conversations with the guy next to me, a lawyer from New York, and managed to both catch some sleep and get some work done (there was a regular electrical power outlet under my seat, another major bonus).

Electrical power and leg room are interesting to me; possibly less so to potential readers. What does one write during one’s tenth trip to the South Pole? I have written a journal of sorts during most or all of my previous trips, usually some kind of blog. You can go back and read a lot of that writing on this site. What can I say that hasn’t already been said?

One answer that comes to mind is to work with difference — with what has changed for me, or in external circumstances. Christchurch is certainly different; we shall see how the Ice is different, if at all. And I am different… older, certainly. Wiser? Perhaps. I plan to explore some of these external and internal changes in future posts.

If you are interested in reading shorter updates (say, < 140 characters), I’ll be posting those to my Twitter feed. And, as I did in January, I’m posting the pictures to Flickr.

Hotel fire alarm just went off. False alarm, but so much for the Quiet Earth. Time to hunt down some breakfast. We get our cold-weather gear at 1 PM.

Ten

Tue, 01 Nov 2011 21:03:00 +0000

T minus four days and I’m getting ready for yet another trip to the Pole — this one is number ten. I guess I’m not sure what I’m going to write about on this trip, yet, but I will be posting updates as time allows. Just spent a few hours getting the blog software up to speed (there are lots of improvements I want to make if I have time). It’s interesting coming back to a project after nearly a year and seeing what I’ve learned since then.

Perhaps other aspects of the trip will put changes in other areas into similar relief.

The Wheel

Wed, 19 Oct 2011 00:00:00 +0000

Details of drawing in progress

The wheel turns, and returns — during a very busy fall season work-wise I have been unable to get much painting done but have lately been drawing in the evenings with ink and gouache — such a fun and flexible pairing of media. Drawing seems to be what I come back to, again and again.

Will be headed to the Pole again in just a few weeks, so check back for some more blog posts!

Plein Air

Sat, 27 Aug 2011 00:00:00 +0000

Today’s painting — Town Line Road at OO, oil on panel, 8“x10”

I’ve had a week off and have been painting outside, at my mom’s house out in the country near Chippewa Falls, Wisconsin. The investment which has made this possible is called a pochade box which is essentially a small portable easel which fits on a tripod. The one I got is called the Prochade by Artwork Essentials. So far I’m pretty happy with it, and slightly amazed and bemused at how fun it is to use.

I’ve had a lot of resistance to painting outside over the years. The few times I’ve done it have been enjoyable, but something about the bugs, the cold, the heat, the sun, the rain, the lightning, the tornadoes, the spectators, earthquakes, muggers, cosmic rays, etc. has kept me from doing much of it. And my passion has always been surrealism, fantasy & sci fi, nudes, painting from imagination, the darker architecture of private spaces. Many of my favorite artists also work this way, and I don’t think I met anyone in art school who painted from life outdoors (or who admitted to doing so) … painting outside seemed almost corny, a Sunday painter’s habit.

Lately, however, I’ve been interested in getting out there and working from nature. Possibly some of that interest is thanks to blogs like James Gurney’s. Gurney interests me because he is an artist of the fantastic (see his Sci Fi book covers or the Dinotopia series), but is also an avid plein-air painter, which very much informs his fantastic work. (He even goes so far as to build sculptures or 'maquettes’ of his characters so as to be able to see and paint the real, natural light on them.)

I’m also interested in making some slower, visual contact with the real world, and perhaps ever so slightly less interested in aesthetic navel-gazing (though I’m still very interested in painting imaginary worlds).

Usually I paint sitting down at my easel in the studio, with a computer, music, food, etc. all close at hand. I never liked painting standing up in the studio; but standing up alone outside for a few hours is surprisingly easy and pleasant. Also it is really nice to be away from computers for awhile (an iPod can be nice while painting outside, though being in the actual space with the crickets and the wind is part of the experience too).

The French term for painting outdoors is “en plein air” — which could be re-translated as “in the full air”. There is something pleasant and “airy” about just choosing a subject and painting it alla prima — no preliminary sketches, no layers of glazes, photo transfers, reference materials, SketchUp studies or traumatic mid-stream adjustments of content — just walk up to the easel and make a painting! I finished three paintings this week… my rate in the studio is sometimes less than three per year. Plus the paintings themselves are more lightweight — things I could just destroy or give away as gifts rather than agonize for months about whether they’re done, where to show them, how much to charge for them, etc.

I love photography and take lots of photographs in addition to the studio paintings. Painting with the pochade box combines some of the things I love about both art forms — you scout your terrain and frame your image (à la photography) and then bring all your gestural and color sensibilities to the material. You just do it, like clicking the shutter, except it takes a couple of hours rather than a couple of milliseconds. In the end your 'shot’ is juicy, oily, full of bugs, and, maybe, ready to hang on a friend’s wall for awhile. Nice and easy!

Next, maybe we’ll see how it goes painting around Chicago.

Pochade box in action

ISO50

Thu, 28 Jul 2011 20:10:00 +0000

Thanks to Jon at ISO50 (Scott Hansen’s beautiful Web site/blog) for mentioning / highlighting my Antarctic photography!

SketchUp and MakeHuman

Mon, 27 Jun 2011 04:09:00 +0000

Mockups for paintings

Makehuman figures imported into SketchUp

I’ve been playing around with the Golden Open acrylics a lot lately and preparing a bunch of small and not-so-small supports for new paintings. After trips to MOMA and the Chicago Art Institute’s Architecture and Design room, I’m fired up to try more content using SketchUp as a planning tool for dreamy, sci-fi spacious otherscapes.

While playing around with this I stumbled across a new program called Makehuman. It pretty much does what it suggests — makes a 3D model of a figure. There are for-pay software packages (Poser) which do this, but Makehuman is open source. You can turn knobs to change gender, body type, facial parameters, hair, etc… strangely addictive. Some blog posts online suggested how you could import Makehuman figures into SketchUp, but I had to hack around a bit before I got a recipe which worked:

Make model in Makehuman
Export it as Wavefront .obj
Import that into the latest Blender release
Export THAT model as a 3D Studio Max (.3ds) file
Import the .3ds file into SketchUp

You can’t really pose the figures yet in Makehuman, though that feature is apparently in the queue. This stuff is evolving fast and I think some really cool art could be made using the tools that are being made now.

In Defense of Hobbies

Sun, 29 May 2011 00:00:00 +0000

I’ve been thinking about hobbies lately. Though “hobby” isn’t a word I hear much these days, when I was a kid I thought about my hobbies all the time. In fact, I can subdivide my youth into the various hobbies I was interested in:

Model trains
Model airplanes
Rock collecting
Shell collecting and study of fish and ocean life
Model rockets
Electronics
Science fiction
Skateboarding
Gaming (role playing, military, board games, …)

The first two were big ones. Trains were a first love, and my grandfather had a beautiful old HO-gauge steam train set with which I was obsessed. During one family visit I was allowed to sleep with the engine and tender. They smelled faintly of multi-purpose electric motor oil and of the tablets which were placed in its smokestack which, when heated during normal operation, simulated smoke. I can also remember the smell of the HO-gauge green and white Burlington Northern diesel engine I had, and the excitement and frustrations involved with setting up my first layout in our basement.

Sleeping with Trains

My favorite store at the mall was called Hobby Horse [the etymology of hobby is late Middle English hobyn or hoby, from nicknames for the given name Robin, which also, along with Dobbin (aka Robert), denoted a small horse]. Hobby Horse had a glass case with N-scale train engines and cars in crystal boxes which you could shuffle or rotate like a heavenly train-jukebox by pressing green “forward” and “backwards” buttons.

When I was a little older I used to wander off on my own to spend my allowance at the Midvale Hobby Shop. This trip used to involve a several-mile trek on foot out the train tracks on the west side of Madison, Wisconsin (this was back in the days when parents used to let their kids wander around unattended as opposed to driving them everywhere in minivans). After buying gum at the nearby grocery store, I would wander the densely-packed aisles of the hobby shop, lusting after the model airplanes and occasionally buying the cheaper ones that were within my budget. These would get assembled by me and my friends, painted and decal-ed somewhat haphazardly, and then have bullet holes burned in the fuselage with magnifying glasses and sunlight (like plenty of boys of that era, we were obsessed with the Second World War and its aircraft in particular).

Long before it became its own focus of interest, drawing was a meta-hobby — all my childhood hobbies involved drawing. Drawing could be done just about anywhere, required very little hardware (a ball point pen on IBM punch cards worked just fine, thank you very much), and afforded the free ranging of the imagination over any aspect of one’s obsession, whether trains, sharks, or Curtiss P-40 “Flying Tiger” aircraft.

In High School, drawing metastasized into Art, including ceramics, sculpture, woodworking, painting, and graphic design. No longer hobby-like, since it was school, and therefore not leisure time (“hobby” is defined as an activity done regularly for pleasure during one’s leisure time), art was a social nucleus and a way to understand and define who I was — not a mere hobby, but a core of identity.

After childhood, I hardly thought about hobbies at all. A decade-long obsession with physics started (physics is a bit unforgiving to have hobbyists, though there are some); I also went to art school and spent some time growing an art career, and the fastest way to piss off an art student or fledgling “serious” artist is to suggest that he or she is doing it as a hobby.

Now it is another decade or two on and I hardly ever hear the word “hobby” anymore. This may be in part because people have less leisure time to begin with (particularly in the wake of the economic downturn and shrinking incomes in general). It may also be due to the enormous increase in variety and sophistication of entertainment, from video games to movies and television, to social media and other online distractions, any of which could I suppose could be called hobbies but which lack the sort of active creativity I associate with the word.

Last night over dinner with my mom and stepdad and friends of theirs, we were discussing hobbies. Ron’s brother simulates baseball games. He and his pals gather virtually over the Internet (and, from time to time, in the real world), accumulate deep wells of player-related statistics, trade actual real-world players back and forth on virtual teams, and slug out simulated National League games using intricate rule systems. Ron’s brother apparently thinks (and talks) a great deal about this activity despite, or perhaps because of, the fact that it provides absolutely no illusion that, if only one worked hard enough, one could eke out some sort of living as a “professional.”

The boundary between hobby and profession seems to be whether the activity is paying the bills (this distinction is especially important to the IRS, which essentially uses the profitability of one’s activity to determine whether expenses in support of that activity are tax deductible). Yet many of my friends in the visual or performing arts, myself probably included, will never support themselves and their families solely on their artistic activities (even art and dance professors still have to teach, and teaching is a profession in its own right). Nevertheless, they would not consider themselves hobbyists — a word that we associate with dilettantism or lack of seriousness.

I think it’s time to rehabilitate the word “hobby.” Unlike professions, hobbies can wax and wane without shame. To have a hobby is to be an amateur, meaning one who loves something. To have a craft, whether it’s needlework, growing orchids, or fixing your car. My dad’s hobby is making deep Web sites for family genealogy and media. I know guys who collect, fix and modify 1970s and ’80s era personal computers. Another friend restores prairies.

At the moment, I am, among other things, a professional programmer. I wonder what it would be like to be a hobbyist programmer. One who programs solely for the love of programming, who has the luxury of taking time to make things as beautiful and as clean as possible, to polish and craft code like a jeweler or a fine cabinetmaker. Instead of walking for hours on the railroad tracks to the hobby store (the store is long gone and the tracks have been replaced by a bike trail), I imagine a similar preparatory activity, walking outdoors among the trees, calming my head and refreshing my body, and then sitting down to engage with … my hobby.

Closure

Tue, 25 Jan 2011 05:22:00 +0000

Once again on the train bound for Picton, to catch the Interislander ferry to Wellington. I think this will be my last post for this trip, so I may try to sum up a few things. First and foremost I want to thank those of you who read along with me; knowing that there were at least a couple of regular readers made the writing more enjoyable, and helped keep me to the daily discipline of posting something, which in turn helped establish a sort of rhythm for the trip.

I have called this my last trip to the South Pole and have tried to treat it that way. Helping to hunt for neutrinos in Antarctica has perhaps been my life’s biggest adventure, and it certainly is bittersweet to say goodbye to it these past few days, thinking of each step of my northbound journey as the last repetition of a pattern established over years. Of course it is a fact that as long as life continues one doesn’t know where one will wind up — just as one never knows if the time one spends with a friend will be the last time or not (and, in fact, every time is unique, never to be repeated, whether you notice or not), I can’t say for certain whether or not I will ever go back to the Pole. There will be other projects on the Ice and I could see myself involved with those. Also, IceCube will hopefully run another 15-20 years and they will be sending people every year, and while I’m not sure I’ll be needed on the Ice I certainly wouldn’t rule out helping if asked. But with the completion of IceCube’s construction and the diminished likelihood of further trips, it seems appropriate to let the place go in my mind. I feel that this is a time in my life to start to let go of certain things, to let things fall away and see what takes their place.

I remembered a sentence from Rudolf Steiner, … that when something ends, we must think that something begins. His advice is salutary, but the execution difficult, for we only know what we have lost, not what we will gain. We have a very precise image — an image at times shameless — of what we have lost, but we are ignorant of what may follow or replace it. — Jorge Luis Borges, Seven Nights

I wrote previously of success or Fruition, and tried to give a sense of the excitement of putting a capstone on a body of work, related not just to this two-week trip but to a decade or even half a lifetime of work. Now I want to think for a moment about more mundane challenges of the trip.

Towards the end of last year I got onto a sort of minimalism kick, reading books and blogs on living simply and thinking about what sorts of things I could give away or live without. The idea of paring down, eliminating distractions and noise, multitasking less, and focusing more on what matters, appeals lately. Of course traveling always helps to encourage a certain minimalism. Going to the actual Ends of the Earth, living out of two bags in a room no wider than my wingspan and not a lot longer, living with four minutes of shower time a week, not having to (or being able to) cook for myself, having no Internet for most of the day … this level of enforced minimalism is something I have enjoyed in the past, but it seemed this time to throw into relief the three choices available at any given time. These were: to distract myself with work (for there was always more work to do, always someone to help or a question to answer or a test to run, even before considering the ever-present backlog of pending software fixes and upgrades); to distract myself with play (movies, books, socializing, blogging, photography, games, …) or, often less appealing, to just stop and be where I was: at the Pole; short of sleep; too hot or too cold; irritated by all the noise; worried about this or that; chapped or bleeding in various places because of the dryness; bad hair; missing fresh fruit and veggies; and generally being out of sorts at the Bottom of the World.

“This year I would like to set one goal, which is to be as fully awake and alert to the entire experience as I can.” Though there vivid and joyful moments, many of which involved stopping and just looking out at the lovely white waste, or moments of fellowship with colleagues, a shared laugh or the ordinary pleasure of lining up for food in the Galley, being awake often meant taking stock of just how uncomfortable I was and how I was searching for the next distraction. Having “awake” as a goal meant bumping against that over and over again, and confronting my resistance to whatever was going on. Perhaps by occasionally noticing, I wore a small nub off of that great ball of resistance.

My Antarctica has been:

The first acid bite of wind on my face after getting out of the plane after the flight to the South Pole
The smell of SimpleGreen™ used to mop the floors and clean the waterless urinals
Workouts in various gyms in the old and new stations
The hisses and rumbles of a hundred invisible machines inside and outside — heaters, generators, snow tractors, skidoos, Hercs
The metallic crunch of half-ton doors closing, and steps on metal stairs
Stash tea: licorice spice
The sound of the flags flapping from the platform above Destination Alpha
Having noisy neighbors, and (probably) being one
Carrying a radio instead of a cell phone
Music: David Bowie, “Outside” (1998); Soundtrack to “Pi” (2000); Kris’s 7-CD mix (2009); the fellow playing guitar down by the gym (2011).

“Everybody has his own Antarctica.” —Thomas Pynchon, V

I say goodbye to mine.

Thanks again for reading. I plan to keep posting pictures for awhile since I haven’t gone through all 2500 of them and because I’m still taking a lot here in New Zealand.

Takeoff

Sun, 23 Jan 2011 22:09:00 +0000

Yesterday’s post was a bit of a tome and I am using today to do some shopping. Heading up to Wellington tomorrow by train + ferry. So I will leave you with a short post today consisting mainly of a link to the video I took taking off in the C-130 from South Pole:

Mummification

Sun, 23 Jan 2011 11:18:00 +0000

A full day in Christchurch, a Pole month’s worth of showers, sushi, photos of graffiti, walking around, going to the Christchurch art gallery, good conversations, and a nap have all helped get me back into the Real World™. I would like to try and remember the trail of events that led here from the last long blog post, because if I wait another day they will be gone down whatever tunnel of emptiness my leaky memory feeds into. They are gone already, of course, but I perhaps I can share bits and pieces with you nonetheless.

My last day at Pole was good. I disengaged with work long enough to wander the Station quite a bit, shooting videos of the place and the people and savoring and digesting the smells and the sights and the low, thrumming vibrations of the place. One nice thing about having the camera which does stereo video is that it captures the sounds almost adequately (I have long dreamed of bringing enough equipment to the Ice to do really good audio recordings of very banal things like wind and machinery and radio traffic and the ebb and flow of conversations in the Galley — the Canon S-95 let me do that with the video well enough as an approximation without having to haul a lot of gear all over the place).

Friday morning I got up as usual, skipped my gym workout, meditated, had breakfast in the Galley, did the 0800 and 0830h meetings, had a good talk w/ our cadre of Polies about software development within IceCube, then did the final packing and cleaning up of my room. There was a lot of discussion about whether we would actually go that day — there had been a lot of delays due to weather, people were delayed Southbound into McMurdo and the forecast for Pole hadn’t been great the previous day, but it cleared up to be fairly nice (partly sunny and -18 F or so, which was the coldest it ever got for me this trip — quite a bit warmer than previous trips, though I came midsummer rather than the usual end-of-season). Also the northbound Polies from a few days previous were all stuck in McMurdo, including a bunch of “DV“s (distinguished visitors) who I’m sure had to get back home for Various Important Things. It wasn’t at all clear whether we would get stuck in McMurdo with them or whether the logjam would break with our arrival.

The day you leave, the cadence and excitement builds, starting, if the weather and mechanical gods allow, with an all-call announcement around 9AM of your plane taking off from McMurdo (“Attention South Pole… we have our first off-deck of the day, skier 93 arriving South Pole at 12:10 PM… this is our inbound/outbound passenger flight…”). Comms tells everyone your flight “has reached Pole-3” (about 25 minutes away) and then “the flight is 10 minutes out… the crossing beacon is on… please stay clear of the skiway…” and then “aircraft is on deck… crossing beacon is off” and finally, “passengers please proceed to the flight deck.”

We were four 'Cubers and one radio guy, and a few people I didn’t know. It was nice to have several of those who were staying come to see us to the plane — a sort of karmic reward for having greeted several of them, I suppose. We all made our way out, took the obligatory group photos, and killed time while Cargo unloaded and loaded the plane and the fuels person sucked AN-8 fuel out of the Herc to add one more increment to the winter’s fuel supply. Some over-eager passengers headed to the plane a few seconds early and had to wait until the Air Guard gave the all-clear; I was the last person to the plane, waved a final goodbye to the remaining IceCubers out on the snow, and then piled into the Herc after everyone else. As soon as they closed up the hatches the flight crew blasted the heat and it went from -18F to probably 60F in about 5 minutes, and people shed various hats and gloves (the parkas, though warm, are excellent to nap in, since you can pull your hood over your head to shut out light and some sound and sort of self-mummify yourself to sleep). We all juddered in our seats as the plane literally skated down the skiway, gradually gaining speed… it takes a good bit of time for the C-130s to actually get airborne and I cheered the plane on, twisting around in my seat to film my goodbye takeoff video from the window behind me. Then we were in the air and past the last skiway flags and over the great white void that is the polar ice cap.

The cabin pressurized almost immediately and I happily sucked down the extra oxygen. I spent the flight writing a bit and photographing the Transantarctic Mountains. Though I thought I recognized some of the places, it is scenery one doesn’t tire of. Then I dozed a bit and before I knew it we were descending down to Pegasus field. When we got out it was nearly 60 degrees F warmer than when we left — above freezing, not quite slushy but definitely a different character of snow. We snapped a few pix and then got a brief van ride to a much larger Delta transport vehicle, the least comfortable transportation I know of (the passengers rattle around in a compartment completely separate from the driver, so she tosses in a radio after loading them in so they can call her in case of emergency).

It was ironic to have to take the hour-plus journey into McMurdo, knowing that the plan was to have us back out there for a night flight in just a few hours. I took it as a sign that they didn’t actually expect the C-17 to arrive from Christchurch, and that we would be stuck in Mac Town for several days. But, whatever, I was at low altitude, with warm air, in a place with actual landforms. We jostled and jolted into town and immediately bag-dragged for Christchurch. Afterwards it was nearly dinner time so we headed towards “building 155” which is the main hive of activity in town… Sebastian and Andrew in T-shirts, all of us skipping down the wooden steps and crossing actual an actual stream of water, then headed down the road and kicking at rocks and watching dust blow hither and yon.

When Shackleton’s crew, having survived a desperate winter and the crushing of their vessel in the sea ice, reached Elephant Island, the men were delirious with joy to tread upon something other than ice. “They were laughing uproariously, picking up stones and letting handfuls of pebbles trickle through their fingers like misers gloating over hoarded gold,” he wrote. Though McMurdo is quite an ugly place by most peoples’ standards, to see actual dirt, rocks, and running water in McMurdo is quite a pleasure after the Pole, a pleasure I wasn’t immune to this trip despite my relatively short stay at the Pole.

At dinner we ran into the other 'Cubers who were stuck in town waiting to go North, and I saw several other people I recognized from other Pole seasons. The consensus was that there was no way we were going to get out that day; the next day was Sunday (no flights) and the rumor was that no flights had gone on Monday either for several weeks. So I basically was preparing to have to completely rethink my New Zealand plans. Afterwards we headed to Discovery Hut and saw the lone Adelie penguin which has apparently been hanging around for several days or more. Colleague Mike came out with us in just a thin shirt and in fact all of us were a bit underdressed; the wind started picking up quite quickly and we hurried back and then headed to the Coffee House (which is really more like a bar) and killed time waiting for news of the flight from Christchurch. Again the consensus was that there was no way, we were all going to be stuck there for days.

Finally, two bottles of wine later, around 11 PM someone came in and gave us the news: the C-17 had taken off from Christchurch and was en route; we were to report at 1 AM for transport to the airfield. At this point I began to simultaneously get optimistic that we would go, and very tired (I typically went to bed around 10 PM at Pole). From then on things gradually became more and more dreamlike, as we headed back to Pegasus (this time in Ivan the Terra Bus, a step up from the Deltas but still a long ride in the middle of the “night”… though we did see five emperor penguins!!!), rattled around the “terminal” (a square orange trailer on skis with chairs for maybe 80 people) while the C-17 was unloaded, then, finally, about 45 minutes standing around the snow watching while they filled the C-17 with our bags, a bunch of retrograde waste cargo, and, oddly, the propellor of an airplane. We left the Ice around 4 AM as I sunk into a slightly delirious cycle of half-sleep, dreaming for a few minutes, blearily waking up and fidgeting and then sleeping a bit more. As in all long distance flights, it took forever and was over before I knew it.

When we arrived it was raining. People even complained.

I was happy to be back in New Zealand, to have a full week of rest and creativity and the pleasures of summer, and to go home after that. Immigration and customs took only a few minutes (“Did you leave for the Ice from Christchurch? OK, thanks very much.”). I have no idea how they de-palletized our bags and got them to the terminal so quickly. Before I knew it I was shedding 40 pounds of gear (it is always SUCH a relief to give it back) and heading to my hotel. By 10 AM I was at the Grange B&B waiting for my room to be readied (I had expected to have to go nap by the river since check-in time is usually late afternoon, so this was a big relief as well). While waiting I talked to another guy from the C-17 — he had spent two months in the Dry Valleys, a helicopter ride away from McMurdo, with NO SHOWERS, sleeping in constant daylight, drilling ice cores. He spoke of 2000-year-old mummified seals and penguins (the air is so cold and dry that normal decay is essentially stopped).

Since arriving I have basically sleeping, showering, eating and walking around. And speaking of sleep, it is time to do that. I was going to add a bunch of photos and videos to make this a very multimedia post but it is too late… perhaps tomorrow will be an all-media day. Good night to all.

McMurdo: Actual dirt (and water!)

New Zealand, Summer, the Real World

Eleventh Hour

Sat, 22 Jan 2011 10:44:00 +0000

Just a few more minutes to get this post in for January 22. This will be very brief since I need to catch up on sleep. The short summary is that we arrived safely in Christchurch this morning after a very eventful 24 hrs. It is great to be back in the real world but our night flight was quite exhausting and it will be a day or so before I feel rested. I will try to write much more in length tomorrow and post a lot of photos (and maybe even a video or three).

Diamond

Thu, 20 Jan 2011 18:57:00 +0000

Morning on my last day (hopefully) and, like a kid on the night before Christmas, I’m excited and nervous. But in some way it feels like my last Christmas. I have been a bit sad off and on. Not a bad sad, but I’ve spent so much time absorbing the peculiarities of this place that it is a bit hard to say goodbye to them, and to it. Last night I spent quite awhile in Comms, the place with the best view on the station, looking out over the skiway and the Dark Sector. While listening to the radio traffic from McMurdo flight control, the attendant static pops and hisses and beeps, and the occasional words from the woman on shift, I looked out across the skiway at IceCube a kilometer away. From Comms one can see the extent of the entire hexagonal IceCube array, 1 km on a side — each of the 86 strings marked with a large red or orange flag, with the ICL building in the middle; and, surrounding this enormity, the much vaster enormity of the wide, white, expanse of the ice shelf we are on, flat as glass and encrusted with powdered diamond, textured with a million tracks of wind, vehicles, and footprints. There are so many beautiful vistas in the world, but that particular view is entangled with my life in myriad and deep ways, from the words on this page to the code I wrote which is running as you read this in 5500 computers deep in the ice, from the lines on my face from the years I have spent working on this, to the relationships (ranging from passing greetings to deeply challenging to strong affection) with so many people in the project… to the artwork I have made, to the artwork I want to make… all these ties and tangles feel, to some extent, a bit more exposed in these final moments.

The weather forecast last night predicted low visibility, which often means canceled flights. But today the forecast looks better and there are four flights scheduled. I expect this chapter does in fact get to close today. With any luck I won’t get stuck in McMurdo. Time for breakfast, two meetings, then clean my room and sail on outta here.

Late season rush of activity begins

Antarctic footwear: flip-flops for shower, boots for normal walking around, Baffin boots for major work outside

View in B2 with emergency sleds

Baby, It's Cold Outside

Wed, 19 Jan 2011 18:12:00 +0000

Holy frozen Batman, I’m leaving tomorrow! I still have a lot I want to do! I was just getting into the swing of things! But I am also excited to leave. A colleague always says, “the best days at the South Pole are the day you arrive and the day you leave.” After all these years, that feels as true as ever.

I know my time is short today because I have to:

Pack (bags out by the station door known as “Destination Zulu” by 7 pm)
Sort my garbage (the US Antarctic program has an extensive recycling and waste management program, and I have two weeks of trash in the can in my room)
Finish editing documentation for our Winter-Overs
Help the new arrivals get set up for their work
Possibly, hopefully, visit the antenna fields with Marco, the second of the four marathon finishers here. His site is a few km from the station and will present a view of the South Pole I’ve never seen before.

That will be a very full day even assuming nothing else comes up. But 24 hours from now I will be listening intently to the flight announcements. I just got email from Northbound colleagues in McMurdo saying there are flight delays today due to fog (dang) but that they saw penguins (cool!) — the ever-present uncertainties of travel in Antarctica start to kick in for me now.

But I’m going home soon!

I really can’t stay (but baby it’s cold outside) I’ve got to go away (but baby it’s cold outside) This evening has been (been hoping that you’d drop in) So very nice … So really i’d better scurry (beautiful please don’t hurry) but maybe just a half a drink more (put some records on while i pour)

Ooooo Baby it’s cold… out…. side!

– Lyrics and melody by Frank Loesser … shout-out to E

Cargo berms

Warming my hands in the GOES antenna shack (a tiny plywood room inside the antenna dome; photo by M. Dierckxsens)

Fruition

Tue, 18 Jan 2011 23:25:00 +0000

Back in the Galley again, the smell of dozens of lunch burgers grilling and the very same licorice spice Stash tea I always drink here setting the olfactory tones for this post.

Almost exactly twenty years ago, when I was working at CERN in Geneva, Switzerland, I was preparing to return to Madison to take some art classes and I emailed some physics colleagues there to see if there was any student work to be had. Colleagues Bob Morse and Francis Halzen replied that there was a new project under way, which was to build a neutrino detector in the Ice in Antarctica. Bob had done an experiment at the Pole to detect cosmic rays, and Francis was a particle theorist who specialized increasingly in astro-particle physics, and neutrinos in particular. Computer simulations were needed to see how effective light sensors, placed deep in the ice, might be at detecting neutrinos.

What started as an art-school day-job turned into a doctorate in physics as the Antarctic Muon and Neutrino Detector Array (AMANDA) became a reality and the computer simulations I and other students wrote became important tools for the understanding of the data from the first instruments deployed by the growing collaboration in the polar ice cap. But in the very earliest years, it was understood even by using back-of-the-envelope calculations that a much larger detector would be needed… a Cubic kilometer of Ice, in fact … an Ice Cube. AMANDA paved the way by detecting high-energy neutrinos, by setting limits on astrophysical sources, and, perhaps most crucially, by providing technological prototypes for the hot water drilling which would be needed to deploy the sensors, and for the design of the sensors themselves which evolved from simple photomultiplier tubes at the ends of long coaxial cables to sophisticated computers in their own right which would provide the maximum signal quality by digitizing the light down in the ice and sending them to the surface with no loss of signal quality.

The difficulties faced by AMANDA were daunting. The drilling in particular was an immense engineering challenge, requiring a kind of specialized expertise found in only a handful of individuals (if that many) worldwide. One man in particular, the late Bruce Koci, brought a kind of preternatural understanding, intuition and tenacity to the task which allowed the 2-km holes for the AMANDA instrument to be drilled successfully, though not without delays, logistical and engineering nightmares, burst hoses, and endless worry Collaboration-wide. After the holes were drilled, deployments of the “strings” of sensors themselves were an ordeal in their own right, sometimes involving 'stuck’ strings, free-falling cables, injuries and, in the best of times, 20-hour deployments out-of-doors in a single shift (one of my most vivid memories of these deployments was the enormous amounts of food one could eat during the breaks, and the utter contentment of exhaustion when the task was done).

Handling the data in AMANDA times was a challenge in and of itself. The data volume and mathematical complexity of the analysis strained the computing resources available at the time; crunching the numbers to find the first neutrino candidates required time on Cray supercomputers and a scrum of dedicated grad students and postdocs working at the limits of their programming abilities.

Since it was understood that AMANDA would eventually give way to a larger instrument, work began well before the final stages of AMANDA to design the IceCube detector and to procure funding. Several competing prototype designs for a next-generation sensor were created and tested in later AMANDA deployments, including a design created while I was at LBNL in Berkeley. The LBNL “Digital Optical Module” (DOM) took some of the signal processing computations down into the ice itself in order to get the maximum signal quality. Though more sophisticated (and expensive) than the usual AMANDA modules, the interface to the surface required only the usual copper coaxial cables, with far better quality of data. A test AMANDA “string” (#18) deployed in 2000 worked well enough (barely!) to show that the DOM technology was workable and could be used to detect particles in the ice. The early LBNL prototype would be extensively improved for IceCube.

IceCube itself received a first real jolt of funding in 2002 and work began to quickly implement the vision of a kilometer-cubed detector. I was put to work on the communications interface which would connect the deployed DOMs down in the ice to a network of off-the-shelf computers running on the surface, a design which would make IceCube an experiment with relatively little custom-built electronics from a high-energy physics standpoint (the DOMs, cables, and a computer interface card plugged into a commodity chassis are the only hardware we had to design ourselves… the rest of the experiment looks like a data center you’d find at any medium-sized Internet company).

The first three IceCube holes were planned for early 2005, but the first version of the new drilling equipment (an enormous engineering task in and of itself) proved exceptionally challenging, and after an injury in the drill camp and many other difficulties, the first IceCube string was deployed in early January. After freeze-in, the DOMs were powered on for the first time by National Science Foundation Representative Jerry Marty and Station Manager BK Grant, with all of us watching with bated breath to see if the DOMs communicated. They did.

The following seasons were all very challenging. Drilling improved substantially with experience, with 8, 12, 19, 19, 20 and 7 strings deployed in the following years, respectively. Software was written, and rewritten, to read out the data generated by 5500 DOMs and to progressively reduce the huge volume of noise to a size small enough to be sent over the satellites to the North. Tasks such as writing the embedded software for the DOMs themselves, to the system which sifts through the detected light to “trigger” on signals from real particles traveling through the Ice, to the filtering of these particles for neutrinos, to the overall control and monitoring of the system, to the simulation of the entire process to be able to make firm quantitative statements about what we are seeing — all this has taken a tremendous amount of effort, collaboration, acquisition of knowledge and skill, political struggles, failures and triumphs.

It has seemed that since we started this project, there has been barely enough time to catch one’s breath. Now we are at the end of the construction of this utterly strange device. I have put nearly a decade of my life into this project alone. If you count AMANDA, I have spent almost half my life in this field.

Last night we took the first runs with the completed IceCube detector — all 86 strings. Though hopefully the experiment will run for decades, in some sense last night’s runs were for me the fruition of years of effort, since I have been more a builder than a user of this instrument. I was going to write more about how it feels to hit this milestone (one which I was never at all certain we would reach), and less about the history… but how I feel seems somehow not that important right now. We built it. Many of us. It is good. May it run long and see deep into the fabric of things. May it teach all of us about many new and beautiful things. May we all benefit from working together and take the knowledge gained, from the trivial to the profound, out into the world to do good things.

Time to greet the dozen or so of new IceCube arrivals from the flight landing now — the last big batch of 'Cubers for the season… maybe the last big batch ever.

IceCube on-Ice lead Greg Sullivan starts the first 86-string run using IceCube Live

Celebrating the run

Friendly Radiation

Tue, 18 Jan 2011 00:00:00 +0000

Tuesday, Jan. 18 2011 1:14 a.m. UTC

Running out of time on this satellite pass for a blog post. Took tons of photos today… this morning we had a tour of the satellite antennas at the very end of Summer Camp. The weather was fantastic, maybe -15F with no wind and high clouds which shed a magic crystalline light on everything. Satellite technician James and his winter-over colleague Glen showed us around, with some of the best views of the Station I’ve ever had.

The satellite stations and the Hercules flights are our ONLY regular link to the outside world. So aside from the inherent technical interest (what geek doesn’t like large satellite dishes and racks of hyper-specialized electronics), there is also an emotional impact to seeing the intricate fragility of our communications uplinks. Every bit of data I’ve sent off station (or sent from various offices in Chicago, Madison or elsewhere TO the Pole) has passed through these humble structures and the networks that connect them to the rest of the Station.

One of our tour guides spent three years in Tikrit, Iraq. Now he is wintering at the Pole — from +130 F to -100 F. I expect his situation to be safer here.

Satellite Uplinks

John and James approach the SPTR Dome

John Vader

A Place That Wants You Dead

Mon, 17 Jan 2011 00:00:00 +0000

Sunday, Jan. 16 2011 11:32 p.m. UTC

Marathon finish line at the Geographic Pole

It’s a little before the lunch rush here in the galley and I’m using a bit of the quiet time here to write today’s post. From where I sit I can see nearly half the entire horizon — as close to a completely flat line as you can get, apart from being on an ocean-going boat, I suppose. There is a meditation technique in which one raises one eyes to whatever horizon, real or imaginary, is in front of one, and one simply watches thoughts arise and fall away. That horizon is nowhere more perfect than here.

The minimalism of the view means that any human construction (antenna, caterpillar tractor, futuristic building, or any of the many thousands of flag markers), no matter how distant, stands out in relief. That visual fact underscores a notion I have about this place. Except for featureless plains of snow, a wide-open sky, wind, and extreme cold, one’s experience of this place is almost exclusively that of humans and human-made things. No trees, no terrain, no insects or animals, no smells, nothing — except what we have brought here. In this way, this alien location is perhaps the most human place I will ever visit. Music in the galley, jokes over email and posted on the corridor walls, footsteps in the hallway outside your room while you’re trying to sleep, the fuel smells mixing with food cooking in the galley, one’s own body odors and those of one’s colleagues, and, most of all, one’s own mundane or esoterically technical work — these are the parameters of life here. The strange irony of this place is that it is all fellowship and craft, in one of the harshest spots on Earth.

There is a New York Times article today about Antarctic tourism commemorating Scott’s and Amundsen’s expeditions here:

Despite the potential circus atmosphere, some veterans insist that Antarctica is not for novices.

“It’s a place that wants you dead,” said Robert Swan, an environmentalist who walked Scott’s route to the South Pole in 1985. “Scott found that out 100 years ago.”

True enough — particularly for tourists who try to walk, ski, or drive here (yes, people do drive to the South Pole from time to time). But for us, ensconced relatively comfortably in this space station / aircraft carrier / submarine, it is certainly less hazardous than it was for Amundsen and Scott, and nothing more than human. When we visit the stars, we will take everything about ourselves with us… and that may provide the biggest challenge, and the biggest joy.

Yesterday after dinner I took a turn in the dish pit washing dishes. The galley serves 200 people for dinner and so there is a real after-dinner rush for the dishwashers; they rely partly on volunteers on Sunday (and Ricky, one of the dishwashers, had just finished winning the marathon). It was a pleasure to work fast, with my body and hands, to feel warm water and the spray of errant droplets on my face. Computer work is exceptionally hard to do with full-body awareness, but when washing dishes that awareness is an easy pleasure. Fleetwood Mac off of someone’s iPod kept the beat. The gratitude of the galley staff was palpable. It was a high point of this trip, simply to turn dirty dishes into clean ones.

The lunch crowd is building. Time to eat.

Marathon

Sun, 16 Jan 2011 00:00:00 +0000

Sunday, Jan. 16 2011 1:11 a.m. UTC

I just got back from running in/photographing my first South Pole marathon: temperature -18F, windchill -41F. Every year there is a Christmas Day race here called the Race Around the World. The winner gets to go to McMurdo and run in a marathon there (and usually wins, since they are trained at high altitude). But flights yesterday got cancelled because all the Hercules C-130s had to go out to field camps. Since “our” runner couldn’t get to McMurdo, South Pole is having its own marathon right now… outside. Starting gun was an hour ago and they are still out running. I ran the first mile or so (lungs still hurt!) and then cut across the course to photograph the leaders and, well, everyone else — there were only ten or so of us.

It was so fun! Probably not the safest or sanest thing I’ve ever done, but I’m very glad I did it. It was made even more exciting by the fact that we raced the clock to get the first 84-string IceCube run going just before the marathon. I literally had to run to my room to get dressed and then out to the Geographic Pole marker to make it in time for the start. The Station doctor gave a short exhortation to stay safe before we started… hopefully the people still out there are being careful! The station turned out to provide water stops, sleeping bags, toilets, aid stations, cheering and photography.

There is a lot more I want to write about but I need to use the rest of the satellite pass for other things, so hello to everyone and good luck to the runners!

Yours truly after 1/26th of the maraton

Deep Fried Macaroni and Cheese Balls

Sat, 15 Jan 2011 00:00:00 +0000

Friday, Jan. 14 2011 11:22 p.m. UTC

Today is showtime, with several special calibration runs happening, so I leave you with this video taken yesterday of the flight taking the IceCube drillers back to McMurdo Station (best in HD).

Today’s cool song in the galley: Curtis Mayfield, “Pusher man.”

Today’s lunch highlight: deep fried macaroni and cheese balls.

Sebastian grapples with some analysis code.

Would you buy a used, cubic-kilometer, high energy neutrino detector from this man? (Photo: Mark Diercksens)

Retrograde

Thu, 13 Jan 2011 00:00:00 +0000

Thursday, Jan. 13 2011 12:59 a.m. UTC

Retrograde cargo as seen from the roof of the ICL

I have been here a week and my trip is half over! The trip is actually going quite quickly. One big mercy compared to previous years is that I have slept relatively well, 7 hours of sleep a night which is better than usual for me here.

Went out to the IceCube Lab this morning to show new arrival Mark D. around. From the roof of the ICL one can see orange flags, spread out over 250 acres, indicating the position of each of the 86 IceCube strings. Until you survey the scene that way it is difficult to get an adequate sense of the scale and strangeness of this whole endeavor.

Tomorrow we may already be able to attempt runs with the complete 86-string IceCube detector, which will be a pretty exciting milestone. Other than that there is not too much to report, so I will keep today’s entry brief and catch up with you again tomorrow.

Mark D. and South Pole Telescope/Dark Sector Lab. Note position of IceCube Strings as marked by orange flags.

The author posing as an alien from Planet Earth

Three

Wed, 12 Jan 2011 00:00:00 +0000

Wednesday, Jan. 12 2011 1:12 a.m. UTC

Geographic South Pole

Today will probably be another short post since it’s getting late in the satellite pass. I’ve been outside three times already today — first for a morning jog out to the IceCube laboratory (ICL), then to take the obligatory “hero shots” out at the Pole markers (there is a geographic pole marker, which moves every year as the ice, with the Station and everything else, gradually glides towards some distant oblivion in the Southern Ocean; and the Ceremonial Pole a few dozen meters away which is ringed with the flags of signatories of the Antarctic Treaty). Finally, I went out to meet Mark D., a new collaborator from Belgium who I hadn’t met before. On his plane were half a dozen “DVs” — Distinguished Visitors, which in this case included reporters from the National Science Foundation and National Geographic, which may stroll by here any moment.

IceCube Laboratory

It was interesting going out to the ICL and seeing all of the strings cabled to computers there. It looks quite well organized and impressive to me, very 21st Century, and I thought of how far we’ve come since the AMANDA days in the 1990s when we ran everything off of a Macintosh desktop computer. We have perhaps several thousand times as much computing power on the Ice now, and though there are plenty of loose ends and improvements to be made, I think the overall design is correspondingly more advanced, robust and complete.

Yesterday we collected some test data to pave the way for the first runs of the full 86-string detector, which required some trial and error but which were ultimately successful. Progress is being made, though schedules might be a bit tight if we are going to get the first full runs in before I leave in little over a week (!).

South Pole Architecture (Jamesways)

Transitions

Tue, 11 Jan 2011 00:00:00 +0000

Tuesday, Jan. 11 2011 12:44 a.m. UTC

Scoop and Polie

Things are starting to get more hectic here for me, with various special runs, feature requests, new arrivals (including some journalists arriving tomorrow), not to mention “House Mouse” duties (station cleaning duties which we all share in addition to our normal “jobs,” whatever they may be). I am trying to stick to a schedule, which so far seems to be working: wake at 6:30 or so, go for a walk OUTSIDE, stretch and light weights in the gym, sitting meditation, followed by a quick breakfast and the 0800 IceCube meeting, after which I start on whatever items take the highest priority of the day (I am not getting to the medium or low priority items very often).

Ben and Hutch say goodbye before the passenger flight departs.

The morning walks are quite nice, a sort of continuation of what I do some mornings in Hyde Park, though I should wake up a bit earlier here so they aren’t rushed. It is very quiet outside that time of day (I suppose that might be strange to hear given the remote location, but there are times when the flurry of C-130 flights, snowmobiles, caterpillar tractors, pedestrians, adventurers on skis, etc. make the place almost seem crowded), and there is a lot to see. If you keep moving and don’t stop TOO often for pictures, it can be warm enough that you keep your parka unzipped (it’s still quite warm here, just below -10F or so). Since my own work here doesn’t require me to work outside much, it’s good to have a habit of going outside — otherwise, it can start to feel like living in a submarine or an aircraft carrier.

Spoolhenge

The Future

Mon, 10 Jan 2011 00:00:00 +0000

Monday, Jan. 10 2011 12:05 a.m. UTC

Destination Alpha

I have a few minutes before today’s passenger flight arrives. Several 'Cubers are leaving on this flight and are very excited to go (some have been here since November). One colleague says the last thing he does on the last day is strip the linens off his bed — and only when the flight is 20 minutes out and there is no danger of it going back to McMurdo without landing and picking up passengers. Some new people are arriving too and I will go out to meet them.

Work has been going reasonably well from my standpoint. Until the last seven strings are handed over, I am basically supporting the WOs [winter-overs] and others and trying to catch up with my backlog of programming tasks. Once the final set of strings is tested, then it will be my job (with one other fellow and the winter-overs) to bring those new strings in with the other 79 and to test the completed detector for the first time. Jim H. gave us a bottle of champagne to celebrate the first 86-string IceCube test run — this will be a real milestone and I’m excited to get there, having worked on this project for many years.

Last night some colleagues gave the Sunday Science Lecture in the Galley and described a new neutrino telescope in the very early stages of construction here at the Pole. It will use radio frequencies rather than light to detect the very highest energy neutrinos. Since the target neutrino energies are so large (and the particles themselves are correspondingly so rare), the radio detector will be 80 square kilometers in size — which makes IceCube (at 1 km squared) look like a potato chip. The technologies differ in many ways from how IceCube was built and is operated; hey are exploring the use of solar and wind power to energize the stations, since it will be very difficult to supply power over such a wide area. It was neat to get a glimpse of the future.

Talking to the drillers and other IceCube folks I can sense a slightly bittersweet feeling in the air. The task of drilling and deploying all of IceCube, which seemed extremely daunting in 2005, is now done successfully, with a remarkable safety record, on schedule and on budget. It is a big victory, but we will all be doing other things sooner or later, and as with all accomplishments I think that along with the sense of victory there is a small sense of loss in completion of the task. This enterprise which consumed so much of the attention of so many of us for nearly a decade is entering a very different phase. The completed experiment will continue to run for the next 10-20 years and will continue to measure some of the most esoteric and mysterious fluxes of energy detectable by current methods, in the hopes of discovering new sources of neutrinos, and possibly stranger kinds of matter as well (supersymmetric particles, monopoles, …). Meanwhile, many of us who are not as concerned with the actual analysis per se will be moving on to other pastures. (As someone active in the actual operation of the detector, I currently still have a lot to do, but this, too, will eventually change, as things become more streamlined.)

Sunday

Sun, 09 Jan 2011 00:00:00 +0000

Sunday, Jan. 9 2011 1:50 a.m. UTC

Fire Sleds

Today most people on station have the day off; some 'Cubers are working, though — I’m taking it a bit easy to try to finally overcome the twin poisons of altitude sickness and paucity of sleep caused by the usual jet lag, lack of nighttime, noise on station, exhaustion, et cetera.

Yesterday a large group of us had snowmobile training, focused largely on the rules (helmets for all drivers & passengers always; 10 mph speed limit), how to inspect the machine before driving it, and how to start two different kinds of snowmobiles (pull-starters only — batteries really don’t work well down here, and you have to leave the machine idling when you get where you’re going, or you might not get it started again).

Hmm, something has come up which needs my attention, so today’s post will be short (I have to post it now before we lose satellite connectivity). Talk to you tomorrow.

Radio Waves

Sat, 08 Jan 2011 00:00:00 +0000

Friday, Jan. 7 2011 6:12 p.m. UTC

Radio measurement

Woke up before 0500h to do some work on the detector and get caught up on emails & such. Today’s post is going to be brief since it is almost time for breakfast and the morning meeting.

Last night was the first time I’ve worked (or even gone) outside since arriving the day before. Sebastian and I did some radio noise measurements out in Summer Camp (a collection of Korean-War-era Jamesways, wood-frame-and-canvas buildings, caterpillars rattling to and fro around us). It was fun to be out there but yikes, COLD since I’m not used to it (about -40C/-40F right now with wind chill). Eyes tearing, struggling with 4 different head coverings, multiple glove layers, taking pictures, working with equipment — it’s a matter of reacquiring simple motor skills with new parameters before you can do anything efficiently out there on the snow. I just spoke to our drill manager, who runs 4 miles outside here regularly… I’ll be quite lucky if I can get that far on the treadmill in the gym by the time I leave.

Yesterday afternoon I headed through the lounge to the bathrooms. There was a thing floating in the air, like a soft white spider or some kind of pale insect. I watched it for a second with two other guys, and realized it was a down feather, probably from some pillow somewhere. We all chuckled, and one of the guys said, “I wish it was alive — I’d capture it and put it in my room.” Sort of sums it up. I saw a few baby plants in the greenhouse, but other than that, it’s just us humans down here.

More pictures from the trip out to Pole

To the plane

Ivan the Terra Bus

From the air

Settling In

Thu, 06 Jan 2011 17:59:00 +0000

Been here about nineteen hours and am settling in, though I have yet to do any real work. The current barometer reads 10,369 feet and, while I really felt the altitude last night, today I feel almost normal. My dorm room is noisy as usual, and not light-tight (recall that it is always daylight outside), but somehow I managed to sleep 6 hours or so, without earplugs.

The first couple of days you are supposed to take it real easy and I know from experience that I need to do that. Headaches, dizziness and shortness of breath are common (I always seem to get the headaches, though “vitamin I,” aka Advil, has been my friend so far); less common are more serious forms of pulmonary or cerebral edema which we won’t get into here but which we’re all taught to be on the lookout for symptoms of so they can put us on oxygen if need be.

But, happily, so far so good. The first hour or so is exhilarating — to get off the plane and see the wild stark alienness of this place (but so familiar, too!) is such a joy and a thrill — to be met by colleagues is also very nice (thanks Jim H. and everyone else who came out to meet us) — then to be handed my packet inside the station and sent off to settle into my room (I am in the station again, which does make life easier — some people have to stay out in “Summer Camp,” a few hundred meters away, where the toilets are in separate buildings. Done that, don’t need to do it again).

Many familiar faces here — people I have seen for years but whose names I don’t know, plus of course my colleagues from IceCube. Winter-over Freija exclaimed with relief, “we’re so glad you’re here!” which was gratifying. It’s great to see people in Madison when I get up there a few days a month, but here it’s a real treat to see familiar friends and faces.

After a few hours, the altitude really starts to hit. After dinner in the galley, I holed up in my tiny room with a copy of “Iron Man II” and went to bed.

Given the recent days of travel, I have a long backlog of things to do, and some preparations to make for the work at hand which I will try to detail in a future post. Now I need to do my morning routine and get ready for the morning meetings.

Another video from the C17 flight to McMurdo

Revolution Number Nine

Thu, 06 Jan 2011 00:46:00 +0000

The Emperor and I

If you are reading this post, then we have made it to the South Pole.

At the moment we are airborne on Flight P-106. There are 17 of us today plus a palette or so of cargo, with plenty of room on the C-130 to spread out. I, for one, am standing in the middle of the plane, with the laptop perched atop some bags. A few others have found spots on the floor or are checking out the flight deck one by one.

Yesterday’s blog post was written in Crary Lab, still one of my favorite places on the Ice. A building in multiple wings or “Stages” which reminds me of the architecture from “Space, 1999” (now you know how old I am if you didn’t already), the lab is Ground Zero for much of the science carried out in Antarctica, with dozens of different rooms, each serving various support roles for biology, physics, astronomy, meteorology, geology, vulcanology and probably even sociology. Stuffed penguins, seal skulls, fish in formaldehyde jars, dried sea sponges, alternately shiny and encrusted instruments from days of Antarctic yore, and similar artifacts are on display at various locations, as are posters from dozens of science projects and even a few lovely oil paintings of landscapes which can be reached by helicopter from McMurdo.

The top floor of Crary has enormous picture windows looking out over the nearby mountains — Black Island, White Island, Mount Discovery. This is a great place to sit and work if you get stuck in McMurdo for a few days.

One of my favorite labs in Crary is the aquarium, where there is a “touch tank” containing sea creatures such as anemones, small fish, sea stars, something which looks like an ocean cockroach larger than a sparrow, and extremely alien-looking orange sea spiders, things which are all legs, very creepy and fascinating.

After exploring Crary and touching the sea monsters, we headed off to the cafeteria for a dinner of butter chicken, dal, and various vegan salads — I am trying to eat healthy this trip, and I think so far it has paid off in terms of overall energy and fitness despite the truly whacked sleep schedule. Following dinner was Bag Drag, which is an extremely aptly-named procedure (it involves schlepping all your Bags and is a Drag) which is more or less the equivalent of what you do when you check in your bags at any commercial airline, except that they weigh you with your Extremely Cold Weather (ECW) gear as well.

After Bag Drag, Sebastian and I hiked up Observation Hill to see what could be seen. The town below was stretched out in all its glory like an Alaskan stevedore’s acid-trip version of a malignant model train layout, a matrix of pale oranges, greens and dusty browns spattering the basin between Arrival Heights and Ob hill, the distant sounds of construction equipment competing with the wind and our conversation as we slowly climbed, shooting lots of pictures on the way. We talked about the early Arctic and Antarctic explorers and the race to be “the first,” and how those missions (and, in smaller ways, the current expeditions across the Ice, which strive to be the first person or group of category X or nation Y to traverse such-and-such piece of Antarctica) are similar to the field of High Energy Physics, where the first person (or, almost always, group) to discover a new particle or breakthrough result, gets the “prize” — the prize being all the credit, or in rare cases the actual Nobel Prize. The drive to be the first seems to be a key driving factor in both science and exploration, but seems to me actually somewhat antithetical to the true collaborative nature of each (collaborative in that the teams involve several or many people, but also in that multiple teams have to try in order to reach the extremely challenging goal, since some teams will fail). IceCube is a collaboration of a few hundred people, and, having experienced both collaboration and competition in spades, I have found, perhaps as a result of getting older, collaboration to be a much more satisfying mode.

We also considered the difficult question, what is the intrinsic value of large science projects in Antarctica (or elsewhere)? IceCube cost upwards of $270 million US to build (not including future operations), which could have been used for other science projects, or, say, for several schools on the South Side of Chicago — or for a small fraction of a new fighter jet. Comparing schools with neutrino detectors is difficult, not just because they are so different from each other, but because the things they attempt to do are, as they say on the Master Card ads, truly priceless. What would you pay to give your child a safe and good education and prepare them for a good life? What would you pay to allow humanity to understand the intricate and gorgeous workings of the universe? What would you pay for peace (for I do think that cooperative, international ventures such as science in Antarctica are conducive to world peace)? What would you pay to own a piece of artwork you fall in love with… or to create it? For your own spiritual awakening or that of others? In today’s world, these things all take money, and it boggles my mind somewhat to contemplate such costs… but as I approach the work site for the project I have so long been a part of, supported by taxpayer money and the hard work of thousands of people, it feels appropriate, somehow, to contemplate the matter, even if there are no easy answers.

Returned back to our dorm in Building 155 to sleep early prior to our transport for our flight at 0645h. Around 0200 our roommate came in an promptly began snoring. I have never heard snoring so loud — it was like listening to the vivisection of a dying bull. Real nice guy, though (awake). Up and at 'em at 0530 with breakfast and then packing up to catch the van for the hour-long ride to Williams Field.

I’m excited to see the Pole, to unpack and have a place to call home for a few weeks, and to take a 20 second shower.

Update: just arrived safe and sound and it is good to be back! Will post more news and pictures when I can.

Red Eye to McMurdo

Wed, 05 Jan 2011 03:14:00 +0000

As we began our descent to Pegasus Airfield, it finally came to me, bleary-eyed after a brief but intense bout with REM sleep, why we had left Christchurch at the unholy hour of 11 PM rather than something more civilized. I had chalked it up to the whimsical vagaries of military flight schedules (picturing some tough hombre of a McMurdo Ops flight sergeant displaying active and deliberate lack of consideration for everyone’s sleep schedule). The clue, though, was that the previous day’s flight also left at the same time — I realized that they were probably putting us down when the runway would be hardest, most frozen. The reason for the 10-day delay for the previous groups going in was a soft ice runway. In other words, by scheduling us as a red-eye, they were simply keeping us safe, and indeed, we landed just fine — though I could see the “softness” (gaping holes, actually) in the ice just off to the side as we approached the dirt road to Scott Base.

C-17 Flight Deck En Route to McMurdo Station, Antarctica

The flight had its moments, despite the increasing aches, pains and mental fogginess caused by the sleeplessness and discomfort of multiple transoceanic night flights. I got up to the flight deck and talked to one of the airmen for a bit. He was clearly excited to be flying this particular mission; postings to Antarctica are apparently highly sought after and this was the 7th continent he had flown a C17 to — repeated deployments to Iraq and Afghanistan being the norm.

McMurdo seems (in what little I have seen of it — I slept for five hours after breakfast) exactly like I left it in 2009. After dinner and “bag drag” (carrying ourselves and our gear up the hill to the Movement Control Center for weighing and palletization of the cargo), I hope to walk around and take some more pictures.

In short, we are here, safe, even slightly rested, and ready to fly to the South Pole tomorrow.

More pictures

That's the Way

Tue, 04 Jan 2011 00:00:00 +0000

Thursday, Jan. 13 2011 11:37 p.m. UTC

ARO, antenna, geographic pole marker, and pedestrian

Satellite is up in a few minutes so I’d better prepare today’s post. Been jonesing for a bath the last couple of days. Those 1-2 minute showers every couple of days are getting old.

We have a massive exodus of IceCube drillers leaving in a few minutes (all but one, 21 in all) on today’s passenger flight. Hopefully they won’t get stuck in McMurdo — the C-17 from Christchurch has apparently boomeranged two days in a row due to weather in Mac Town. Or rather two nights in a row, since they are still doing night flights to McMurdo. I feel bad for those people who have had to get up in the middle of the night several days in a row. Unfortunately I too will probably have a night (or early morning) flight to Christchurch in a week, unless the weather gets a lot colder.

I had planned originally to spend some posts explaining what IceCube was and what exactly we’re doing. I guess I haven’t really done that yet. I apologize to any readers who haven’t already heard me babble at length about neutrinos in person; this may seem to you like an even more random and strange endeavor than it already is. A detailed description of our tasks here right now would probably be too technical anyways. But a simple explanation that might suffice for now is that we are testing the new equipment installed this year, both deep in the ice (the final seven “strings” of light sensors, deployed about a mile and a half below the surface), and the new computers in the IceCube Lab on the surface, and, most crucially, getting everything to talk to each other and perform the initial checks that everything is working OK. After we leave, two winter-over scientists will remain behind to run the experiment throughout the Austral Winter, supported by a scrum of experts in the North who will reach down over the satellite, perform periodic calibrations and software upgrades, and generally help if/when things break from time to time.

I had expected to use some of my time here to beat down my backlog of features to add and bugs to fix in my primary project (“IceCube Live,” a server used to control the experiment and a Web site used to monitor it). I have done some of that, but in reality most of my time has been spent helping other people here… which, I suppose, is really how it should be.

Evening game of nine-ball

Everybody's at lunch

Favorite tunes/bands heard in the galley here:

Kruder and Dorfmeister’s K&D Sessions
Tom Waits, intro song from “Down by Law”
Led Zeppelin, “That’s the Way”
Sufjian Stephens, “Come On Feel the Illinois(e)” (I miss Chicago! Not to mention hot baths…….)

Faults in Ice and Rock

Mon, 03 Jan 2011 09:35:00 +0000

My stay here in Christchurch may be a short one. Arrived yesterday afternoon to the heat of high summer and found out directly that our clothing date would be this evening — and that we were scheduled to fly immediately after receiving our cold weather gear. That gives just over a day here, which is shorter than I might have liked, but which might result in arriving at the Pole and getting settled in sooner, which would also be nice.

The flight schedule has been screwy, apparently because of cracks in the ice runway at Pegasus airfield. Some folks have been here in Christchurch for 10 days waiting for a flight (lengthy delays or even cancelled trips are an ever-present danger when traveling to Antarctica). Another advantage to reaching McMurdo tonight would be to avoid such a fate. I once stayed here for a week while waiting for a part to arrive for a C-130; it was a nice vacation, but my stay at the Pole will be short enough that to lose 10 days would be a bit of a setback.

Speaking of cracks, I felt a small earthquake yesterday just after arriving (we also had one in Chicago last week, followed by a Thunderstorm on New Year’s Eve!). Evidence of the big earthquake which hit in September is everywhere, mostly in the form of scaffolding, missing buildings, and friendly signs saying DO NOT APPROACH OR ENTER THIS BUILDING.

It is nice to smell New Zealand again. Pictures and words can convey a bit of the feeling of being here, but I realized yesterday after leaving the International Terminal in Auckland how much of my experience here is tied up in smells. Part of that may have been reinforced by the repeated shock of going from the Pole, where smells of burning fuel, body odors, stale beer, steel, and various synthetic things dominate, to the opulent, moist smells of the late summer air in Christchurch. Even my hotel, the Devon, has its own distinctive and instantly recognizable smells, whether from the flowers outside, the cleaning substances they use, or some combination of the two.

I am a bit jet lagged, but slept well. Sleeping the whole night, with the windows open and the breeze blowing in, with no screens and only the occasional electric tram going by, was lovely. Tonight what little sleep I get may be on a C-17, bundled up in my red parka with a bunch of similarly-attired humans and God knows what sort of cargo. Or, the schedule may slip and I may sleep in a real bed again… we shall see.

Bardo

Mon, 03 Jan 2011 02:39:00 +0000

I didn’t get a post in yesterday, because it was only January 2 for a few hours, just before I crossed over the International Date Line. Not to mention the fact that there was no WiFi on the flight (though apparently some planes have that now).

My second flight didn’t take off until nearly midnight, local time. I dread the flight to Auckland more than any other stage of the trip; it feels like the bardo state from Tibetan Buddhism, in which one has died and is waiting to incarnate into one’s next form — a state of unsettled in-between-ness, which somehow passes both quicker than expected and with excruciating slowness (a little like some meditation sessions, come to think of it). I did manage to sleep a bit (breathe deeply, follow the out-breath, notice thoughts and sounds, …. zzzzz), waking at least half a dozen times, and then watched “The Social Network,” which I enjoyed but which I dearly hope will resemble the coming trip not in the slightest.

One sees “Ice people” in increasing amounts as one gets closer to The Ice (slang for Antarctica). I met an IceCube colleague from Madison in Chicago; noticed a few others at LAX and several more in Auckland. You can tell them by their clothes (take all the geeks or professors you know, and all the rock climbers you know, and imagine how the intersection of those two sets would dress) and by the little yellow penguin tags hanging off of their luggage or the red, white and blue US Antarctic program patches on their jackets. Often we don’t talk to one another, but there are all sorts of interesting people with interesting stories. For example, on the flight to Auckland I managed to sit next to the Program Psychologist on his way to evaluate the psyches of people staying over the winter at South Pole and at McMurdo, but didn’t know that (or anything about him) for the 13 hours we sat together until we landed in Auckland and finally indulged in some small talk.

In short, the trip so far has been relatively painless — by far the worst of it was the Mexican food (and service) I got at LAX, not realizing that they would have dinner on the plane despite the late departure. Now I’m waiting for my flight to Christchurch and, soon, soon! a shower and a real bed.

Chasing the Sun

Sun, 02 Jan 2011 03:32:00 +0000

Happy New Year to everyone from the skies somewhere over the western USA, en route to LAX. This is my ninth and probably final trip to the Geographic South Pole, and if you’re reading this you probably know me and know I’ve been preparing for this trip off and on for a number of months (and, in some ways, for decades — more on that later). Now I am finally under way, and I’m relieved to be launched and finally headed towards whatever this journey will bring.

I have done some writing and some photography on every trip, and published writing or photos in some form during most of them — in some cases just emails to friends and family; more recently, in blogs. There are a few reasons for doing this, not the least of which is to let loved ones know where I am and that I’m safe. I also find that the writing process helps me to capture, absorb and explore the experience of going somewhere few people get to go. But it is also an opportunity to give some of that experience back to all the people who have contributed (at least in the form of tax dollars) to support me and my colleagues in the US Antarctic Program.

This Year’s Blog

My intention is to post photos and text here every day for the next 28 days or so. My entries may therefore be briefer than they have been in past years. I will try to cover some of the the basics of the trip (why go; logistics; etc.). I also may indulge in personal musings on the whole process and about the overall experience over the past fourteen years. Finally, I want to invite you to contact me and let me know if there is anything you would like me to explain, describe, etc.

Special Goals

In previous years I have tried to enumerate certain personal goals or projects in advance (in addition to objectives for work, obviously). These goals have been things like workouts, regular mindfulness meditation practice, art projects, places to see, books to read, etc. I have supplies along for all those things, but for this year I would like to set one goal, which is to be as awake as possible to the entire experience as I can. This may mean fewer activities than in past years, (or shorter blog posts, except for this one!); but this is probably my last trip down here and I’d rather enjoy the things that come along of their own accord than to try to chase around to see or do everything I missed on previous trips (or re-do things I’ve already done).

Why Is This The Last Trip?

Many people have asked why this is the last trip (and have pointed out that I’ve said that every trip would be the last). This year truly is different in that this is the last construction season for the IceCube project — we have deployed the final of 86 strings just a few weeks ago, and when these seven latest strings are brought online, the detector (already operational in limited forms since 2005) will be complete, so there will be very little need to send people like myself in future years (of course who knows about future projects, etc. — but really, folks, this probably is the last time for me).

How to Get Updates

Some people have asked how they can get notified when the blog is updated. I would prefer not to spam people’s inboxes with notifications, since I don’t know who really wants to get a daily update and who would rather not. So, there are a few ways which should all work well:

Just visit http://johnj.com/ and look at the Recently Updated links;
Subscribe to the RSS feed for this site (use Google Reader or other blog reading software);
Follow me as “eigenhombre” on Twitter
If we are Facebook friends, follow my status updates;
If you want just the photos, they will be posted on Flickr, possibly a day or so before a subset of the pictures make it to the blog. There is an RSS feed for just the photos, as well.
If you really want an email notification, there are free services on the Web that will notify you when an RSS feed is updated, so please use one of those.

If you think of someone who would be interested in the blog and who I am not in contact with on Facebook or Twitter, please do forward them the link to the blog.

Many thanks, and a fruitful, peaceful and prosperous 2011 to all.

Downhole

Mon, 13 Dec 2010 12:07:00 +0000

As thoughts turn in earnest to shopping and packing to depart January 1, here is a South Pole video from old times to get one in the mood for the coming trip. This was from my second trip, 13 years ago, taken by a camera deployed in one of the AMANDA strings, designed by our Stockholm collaborators.

Coming Out of Hibernation

Mon, 13 Sep 2010 16:04:00 +0000

I have been moving things around a bit here in preparation for my ninth and probably last trip. (Yes, I know, I always say it is going to be my last trip. But this season marks the end of the construction of IceCube, so far fewer people will be going to Pole after this austral summer.) I am tentatively scheduled to leave Chicago for the South Pole on Jan. 1, 2011. After taking last season off, I’m excited to go back for one last time.

Managing the Most Remote Data Center in the World

Fri, 30 Jul 2010 00:00:00 +0000

This video is from an interview I did with James Turner at OSCON in July of 2010.

From Slashdot:

“Imagine that your data center was in the most geographically remote location in the world. Now imagine that you can only get to it 4 months of the year. Just for fun, add in some of the most extreme weather conditions in the world. That’s the challenge that faces John Jacobsen, one of the people responsible for making sure that the data from the IceCube Neutrino Observatory makes it all the way from the South Pole to researchers across the world. In an interview recorded at OSCON, Jacobsen talks about the problems that he has to face (video), which includes (surprisingly) keeping the data center cool. If you’re ever griped because you had to haul yourself across town in the middle of the night to fix a server crash, this interview should put things in perspective.”

Ruby Plugins for Sketchup

Sat, 03 Apr 2010 00:00:00 +0000

Here are some Ruby plugins I developed recently during my experimentation with SketchUp:

Auto-reload Plugin

The scan_plugins plugin monitors any given directory and looks for any changed .rb files. When your code changes, it is reloaded into SketchUp automatically. This means you can work in your own plugin directory, and won’t need to restart SketchUp to test changes to your scripts.

I found the plugin-writing process somewhat frustrating because, in order to install a new plugin, you typically have to quit SketchUp and restart it. This is murder if you test continuously as you develop (as we all do… right?). scan_plugins avoids this problem.

Caveat: because each file will be executed as soon as it is changed, the Ruby files in the specified directory should consist ONLY of SketchUp plugins! Put your Ruby-based disk cleanup programs, WEP-cracking utilities, etc., elsewhere.

I have found this script indispensable while developing other plugins, by greatly reducing the turnaround time for testing.

Crowd Plugin

The crowd plugin places a selected component at random throughout an area on the x-y (red/green) plane. This was the first plugin I wrote, just to try out the process.

Crowd Plugin result

Current Project

I have been working sporadically on a set of mesh manipulation tools for subdividing, smoothing, joining, and sculpting polygonal meshes. Meanwhile, Subdivide and Smooth [video no longer available] and Tools on Surface are great plugins to get you started with advanced mesh manipulations.

The Cruel Stranger

Tue, 10 Nov 2009 00:00:00 +0000

Came across an interesting post today by author R. Scott Bakker, writing on Tor.com about his novel, “Neuropath.” He has some interesting things to say about the sometimes alien and unpalatable conclusions provided by science, an institution which he likens to a “cruel stranger”:

So here’s the question: What other bitter pills does science hold in store for us? The cruel stranger isn’t finished, you can bet the family farm on that simply because nothing is final in science. So what other stomach churning surprises does it hold in store for us? And what happens if it begins telling us things that are out and out indigestible?

What if science, the greatest institutional instrument of discovery in history, starts telling us there’s no such thing as choices, or stranger still, selves? What if the portrait of humanity that science ultimately paints strikes us as immediately and obviously inhuman?

I have been long intrigued by the ability of science to dish up conclusions in which are difficult to accept. The obvious example is evolution, but evolution is something I personally grew up with and actually find comforting (our literal kinship with the rest of the animal kingdom). However, some of science’s conclusions about the physical world are hard to swallow. Einstein, for one, could never accept the randomness inherent in the quantum theory (“God does not play dice with the universe”). I, for one, am pained by notions such as the inaccessibility of distant stars or the notion of the end of the universe.

The latter idea is particularly difficult and intriguing for me. As a ~~non-theist with strong~~ Buddhist ~~leanings~~, what I want to believe (so long as I have beliefs!) is that the universe will continue to manifest forever in a sort of infinite, awesome, endless dance of change. However, while the cosmological picture has evolved tremendously even in my lifetime, and will probably change more, the dominant theory (as I understand it) currently has the universe continually expanding at ever increasing speeds under the influence of dark energy, as the stars gradually burn out, grow cold and dark. Ultimately, very little can happen in such an asymptotic universe… certainly not any form of life as we can currently imagine it.

On the human scale, one struggles to accept one’s own mortality, but it is a comfort to think of the world continuing on. Of course, the eventual fate of the Earth is likely evaporation as the sun expands. But the cosmos will always be there, with new beings, new civilizations, or at least, the vast beauty of astronomical phenomena. Or will it?

Forget mortality, forget the utter insignificance of humanity in the crushing vastness of the cosmos. If the universe itself dies (becomes static), then change itself will come to an end. Where is the comfort in that? Fortunately, cosmology as a science is truly in its infancy (for example the vast majority of the mass-energy in the universe, in the form of dark energy and dark matter, are still almost completely unknown to us), so the picture will probably continue to evolve. But what if science tells us that the universe really has a finite (if very very very long) life span? The Cruel Stranger will have struck yet again… and, certainly, not for the last time.

Photoshop on a Dime

Mon, 12 Oct 2009 00:00:00 +0000

As a side benefit of transitioning back from European time since returning home last week, I have been waking up early most mornings and getting in a bit of painting. Some days there is nothing like gradually improving old paintings, scumbling, glazing and blending with some quiet music going and the sun slowly rising outside.

As satisfying as that can be, I don’t want to only rehash the same unfinished paintings endlessly (some may be beyond help). I keep coming back to the problem of composing new pictures — what to put in a piece, and where to put it. When I draw (or just close my eyes), many images or ideas arise… not to mention the thousands of photos I take or the images I “clip” from books and the Web. But it can be hard to knit this slurry of ideas and source material into something which works, which has, in other words, a unique and compelling aesthetic logic. Painting is such a slow process (at least the way I do it) that it can be painful to make the required compositional choices once a piece is “launched.”

I have posted before about making studies and preliminary drawings, but I keep tweaking my methods, looking for processes which will help generate the most compelling images (at least to me) and will add more momentum and fun to art making.

One thing which I’ve tried before and which I returned to this morning is the use of transparent overlays to make preliminary studies. I first encountered this method of drawing in slides of Eric Fischl’s work. A few years back I tried a variant of the process for composing pictures, using transparent sheets known as Kimodesk. While trying to add some content to a sketchbook drawing this morning, I tried it again, and it seemed powerful and slightly magical, as if I had a physical version of Photoshop at my fingertips:

One 'composes’ multiple figures, buildings, etc. quickly, simply by moving sheets around on top of each other;
You can draw on both sides of the sheet for a greater expressive range;
The surface takes graphite like soft, black butter; gouache works too if it’s very 'dry’ (no washes… probably oil would be fine);
Unlike a drawing in a sketchbook, where you have to erase whenever you move or change something, there is very little penalty for making adjustments;
It is easy to duplicate a picture element or series of marks;
You can trace existing drawings from sketchbooks and incorporate found photos, etc., or just run the Kimodesk directly through a laser printer, making a nice, rich black;
You can flip your drawing and see the image in reverse at any time (a good compositional check, essentially the same looking at your painting in the mirror¹).

Kimodesk looks really good and can be used directly in finished works (as Jack Damer, who introduced me to the material, has done), but probably glassine or other transparent paper would work nearly as well for the preliminary drawings I’m interested in at the moment. Unfortunately I don’t believe anybody is making or selling the stuff at the moment. If anybody knows otherwise, please let me know.

(Why not just use Photoshop? I like Photoshop, but the lack of physicality is offputting at times, and you have to get the images into the computer and back out again. Plus, I already spend 99.9% of my time in front of a computer.)

The eye gets used to flaws in a work-in-progress. To counter that problem, in addition to the mirror trick, some people turn their work upside down, or use a convex lens to make it look farther away. Salvador Dalí used to have his wife Gala place his paintings at surprise locations around the house where he could bump into them unexpectedly and be struck by things he hadn’t noticed about them… lacking Gala, most of the rest of us use other tricks.

Man on Wire

Fri, 09 Oct 2009 00:00:00 +0000

Still from James March's "Man on Wire"

Growing up in the Midwest speaking English like everyone else around me, I always thought my native tongue was somewhat ordinary, even ugly. In high school and beyond, I loved studying French for the glimpses it afforded into other ways of thinking, ways which I may have presumed were more culturally sophisticated. French was the language of Baudelaire, of wine and cuisine, of Louis XIV and Napoléon; of Monet, Delacroix, Ingres.

I think it took me many years of reading English prose and making small forays into other languages (something I still enjoy) to truly appreciate how marvelous a language English really is. For one thing, at roughly 1 million words, it far outstrips most other languages in terms of vocabulary (though perhaps half of those are technical terms). The sheer lexicographic heft of English provides enormous expressive power — when several similar words exist for the same thing, one can choose the one with exactly the right associations or poetic impact.

Yet, as with any language, English has words whose equivalents in other languages are more beautiful. “Beautiful,” for example, is not a beautiful word. It sounds a bit prim, or prissy. “Beau,” on the other hand, is such a simple word, barely an exhale, or a sigh. It is far superior to its English cousin, in my opinion.

“Beau” is a word you hear a lot towards the end of Man on Wire, a film by James March which portrays the efforts of Frenchman Philippe Petit, in 1974, to cross between the Twin Towers of the World Trade Center on a tightrope. While the film itself is very well made, what is particularly captivating is the story — how Petit and his friends combined vision and inspiration, careful planning, stealth, stamina and, above all, courage, to stage an act which transformed not just themselves, but also the public’s view of the Towers.

I was somewhat incredulous to hear, years after the fact (I was only eight years old at the time of the stunt) that someone had tightroped between the two buildings. Now, of course, the imagery in the film is amped and torqued in gut-wrenching ways by the ghastly absence of the Towers themselves and the collective memory of their destruction, burned on the retinas and the psyches on so many of us who remember seeing them fall on live television. Yet the tears shed by the participants in the film, interviewed decades after the fact, do not seem to be about the fallen towers themselves, but about the purity and intensity of the moments before, during, and after Petit’s walk, and the beauty created, however briefly, on that day — August 7, 1974.

Seeing the film (made just last year) transforms the memory of the Towers from one of trauma to something more like transcendence. One of the film’s ironies is that it invites comparison between the clandestine, overseas preparations of the youthful Petit and his friends to what one imagines the nineteen hijackers’ final months and days were like (it would make an interesting film to try to capture their point of view honestly, but I doubt the world is ready for such a film). The contrast, very vivid, and never stated in any form in the film itself, is an obvious one: the sheer horror of Sept. 11, versus the joy and amazement generated by the simple feat of walking between the two buildings, more than 400 meters above the streets below. Yet perhaps it shows the power of art that such comparisons fade when one is shown Petit’s feat on its own terms. In the space left behind by the towers, one imagines the space between them — and the courage required to face imprisonment by a foreign police force or, equally likely, death at the hands of high winds and gravity. To achieve the impossible — not for the sake of destruction, but for beauty. Comme c’est beau.

South Pole Videos

Fri, 01 May 2009 00:00:00 +0000

For your enjoyment: YouTube videos from the last two trips to Antarctica.

Posing Rigs

Sun, 05 Apr 2009 00:00:00 +0000

Been following along with Justin Chin about how to create posing rigs with SketchUp. Whereas he is using it to pose detailed 3D models (with an eye to making a comic and/or film using SketchUp), my interest relates more to orienting figures in space for paintings — placing the masses, in perspective. So far it seems promising, though I have yet to carry it through to a real painting.

Posing rig example

Though the tools are contemporary, the approach dates at least to the 16th Century:

Drawing by Luca Cambiaso

Metric

Mon, 02 Mar 2009 00:00:00 +0000

Being a geek, I’ve always liked the word 'metric.’ A metric is a number or other quantitative object (physicists like 'metric tensors’) which measures something.

I have three paintings drying right now (i.e., too wet to work into without screwing up areas). A thought occurred to me as I was thinking about what to work on next, that the number of drying paintings could be a metric of how artistically busy you happen to be at the moment. Call that number of drying paintings the 'D-number.’ If I start working on another, it would make a 'D-number’ of four. A moderately busy period, by my standards.

I wonder what my highest D-number has been. Seven? Surely not ten.

I like working on lots of pieces at once. When you get bored or stuck, just pick up another. It’s especially nice if your paintings take a long time to finish, because you might be in the wrong mood for one particular painting, but in the right mood for another. If you glaze a lot, you have to wait for layers to dry (I don’t use resins because of the fumes), which artificially inflates your D-number.

Other painters just pick up a panel and don’t touch anything else until it’s done. Their D-numbers are always equal to one (that is, if they paint in oils — acrylic painting makes your D-number usually zero, due to the quick drying times).

Then there is the U-number — the number of paintings which are unfinished, kicking around the studio, that you hope to get around to sometime. Mine’s probably about 15 or 20. A recent goal of mine is to make U=0, though probably that’s a fool’s hope.

Finally, there is the I-number, which counts the paintings you plan to do next, or sometime soon, or in a future lifetime. As Van Gogh quoted to Emile Bernard, “the most beautiful pictures are those one dreams about when smoking pipes in bed, but which one will never paint.” I like to keep my I-number as high as possible — it means I’m feeling especially inspired.

Cuba

Sun, 22 Feb 2009 00:00:00 +0000

Cuba Street Fair, Wellington

At the end of a few days here in Wellington, I am flying home today.

This is one of my favorite cities in the world. It has been fabulous to see Neil and Amelia again, to go shopping (the US dollar is finally stronger after several years in the gutter), to go to art galleries and to walk up and down Cuba Street.

Yesterday we braved the crowds at the Cuba Street Carnival, then came home and made dinner for some of N&A’s friends, including one of my favorite New Zealand painters.

Friday, I met Amelia’s mum and dad and uncle. Uncle lives in US and goes to McMurdo every year to lead expeditions on Mt. Erebus. Dad is an amazing architect. They have a full acre quite close to Wellington, with a garden full to bursting with strawberries and corn, and chickens aka 'chooks’ (your Kiwi word of the day). Mom, sure wish you could have seen their garden.

Cuba St. Store Display

Cuba St. Carnival Garlic Seller

Cuba St. Carnival Vendors

Cuba St. Doorway

Olive Cafe

Wellington Alcove

Minty Chook

Neil and Amelia

More pictures

Every year at the end of my South Pole trip, primed by a bit of rest following weeks of unceasing mental effort, and stimulated by the intoxicating beauty of this country, I get inspired to put more effort into my art. This year has been no exception and I’m excited to get back into the studio.

This will be my last post of the year on this blog, but I will continue blogging on my art blog as time and energy allow. Thanks to everyone who followed along.

Peace!

Wickets

Mon, 16 Feb 2009 00:00:00 +0000

Monday morning and Jan, Timo and I are on the Northbound train from Christchurch to Picton. Outside, greens and browns of grasses, dry scrub, and trees blur together with grey skies, rain and the dun coats of sheep. It always seems to rain during my train trips here. It’s hard to believe that just Friday morning we were at the South Pole.

Thursday night, having Bag Dragged and then sweated in the sauna ('Epic’ was how colleague Stephanie described that last, crowded, lengthy steam-bath — with multiple trips outside to Destination Zulu to photograph the steam rising off our bodies, trips which were surprisingly pleasant despite the 260 degree Fahrenheit temperature transitions involved), the younger crowd stayed up partying all night, while comparatively 'old’ folks like myself turned in for a few hours of sleep. Friday 'dawned’ (not a word which applies literally at South Pole for more than one long week per year) abruptly, with the announcement of the early off-deck of our plane from McMurdo… an announcement which I missed somehow, but caught on the online 'scroll.’ Given that the flight was going to arrive, and therefore leave, early, I spent a few minutes tracking everyone down so that nobody would miss the flight.

Anna at Bag Drag

Waiting for the flight

The last few summer flights from Pole are somewhat bittersweet for all concerned. The winter folks are glad to have the Station to themselves at last, but of course it marks the beginning of eight and a half months of isolation. The Summer crew eagerly await oxygen, moisture, 'freshies,’ night, family, vacation and sleep. But many of us will never return to the Pole.

Erik documents the departure

Boarding the flight North

Winter-overs Camille and Erik came out to say goodbye to the last ten or so of us 'Cubers to leave. The flight came and we huddled around in the -50F air like a bunch of penguins until we were signaled to approach the plane. Awash in the deafening roar of the props near the nose of the aircraft, I turned and waved to Erik and saw his wave in response. Then it was climb the stairway, stuff myself and my bags into the nearest available seat, and buckle in. There actually weren’t enough seats for the available passengers, so Stephanie, who got on last, was pulled aside and got to experience take-off from South Pole with the flight crew in the cockpit. After takeoff the pilots pressurized the cabin to more atmospheres than I’d had since Jan. 20.

Timo takes one last picture of the South Pole

LC-130 Flight Deck

Stephanie and Anna

We arrived at the Pegasus ice runway to overcast weather, with Castle Rock and Ob. Hill barely visible, Mt. Erebus not at all. The Herc disgorged us and our bags and immediately took to the air again to land at Williams field, a mere handful of miles away. We had our choice of rattling around on the ice outside (the weather was certainly warmer than Pole, with Real Snow™ but rather windy) or the 'pax terminal,’ a grungy orange box on skis, big enough to hold fifty or so people. After taking the obligatory photos of the various firefighting vehicles nearby, we packed inside and settled in for the two hour wait for the C-17 from Christchurch.

We debated whether we could feel the difference in the altitude. Stephanie was the only one who said she felt nothing at all. I could, as usual: I find the sensation of returning to sea level to be a vertiginous cross between a caffeine dose, a feeling of having come in from the rain and changed clothes, and, paradoxically, being so relaxed I feel like taking a nap (though that last part could have more to do with shifting from night shift to days in less than 48 hours).

To kill time we set up my laptop and fired up The Incredibles (with subtitles, because of the noise of other people talking), the audience eventually expanding to include a couple of non-Cubers. The C-17 arrived just after Helen and the kids were shot down in their aircraft and 'paddled’ to shore. The plane’s arrival meant we all felt we had to suit up and watch the unloading and loading of cargo despite the fact that it clearly would be a full hour before we were waved onto the plane.

Waiting to board the C-17

At some point each of us started to remember that it doesn’t have to be -50F to feel extremely cold (it was probably about 20F but much more humid than at Pole). But soon enough we were packing into the C-17. Just before I entered the plane, I turned around and took what may have been my last look at Antarctica: a panorama of whites and subtle, pale grays… a few distant mountains shrouded by low-hanging clouds, outcroppings of rock, and a handful of human artifacts out on the airfield. Not much to look at, but special somehow in some way I have never quite been able to articulate.

One’s first look at Antarctica is thrilling: here you are in a place most people will never see except in pictures or on film or TV. It is an otherworldly place whose few native inhabitants, while familiar through media and popular culture, are quite alien up close. (Penguins, for example, are somehow much odder 'in the flesh’ than one would expect). The people, too, tend to be a breed apart somehow. Many of them young, most of them white and lower to middle class, they come from all over the world but often hail from the US Midwest or mountain states; are often better educated than their on-Ice job descriptions would suggest; and have robust senses of humor and very tight social ties with one another. 'Repeat offenders’ such as myself tend to be both specialists who have niched themselves so tightly into their work that they are hard to replace, and people who cherish the sense of community that exists on the Ice. In addition to the colleagues you know and don’t get to see except on the Ice, it is somehow reassuring to see all those other familiar faces season after season, even when you don’t know their names. One feels like one is part of something, though it is hard to say what that something is. Surely, we are there to do or to support science… but there are geopolitical angles as well, a fact which adds a slight tinge of artificiality to the whole enterprise. And then there is the fact that one reason we are there is … because it is there. Antarctica is a mountain to be climbed, a planet to visit, a river to cross.

In November, an expeditioner named Todd Carmichael set off from a coastal location called Hercules Inlet and broke the world’s speed record for unassisted traverse to the Pole. His ski bindings broke soon after beginning his trip, so he proceeded the rest of the way on foot. By the time he reached the Pole he was so disoriented that he talked to his sled and had trouble finding the station even when it was right in front of him. A colleague interviewed him on video while he was recovering in the medical clinic at the Pole. At the other end of what had turned out to be a near-death experience, Carmichael talked about endurance not just as a means to reach a goal but as a value in itself. “Hard things are good for you too.” That value resonates for me, and I suspect it does as well for other travellers to the Ice.

Sometimes, after sweating through my daily session in the station’s gym, I’d step directly outside into the cold air on the upper level deck outside the Station. With nothing on but running shoes, shorts and a t-shirt, someone who is warmed up can last up to a minute or two outside without even getting particularly cold. It’s enough time to reestablish the sense of where you are: the incredible brightness; the view of the experimental buildings in the Dark Sector; the crossing beacon for the skiway; a thousand half-buried flags indicating buried cables, buildings, fuel lines; the view, off to one’s left, of Summer Camp and the cargo berms. The cold so sharp it makes your lungs ache to take a deep breath. A momentary flash of the South Pole, of the diamond-crisp beauty of the place, the infinitely flat wisps of clouds on the rim of the sky at the very edge of the world. No longer novel, Antarctica, but personal, and deadly, and dear.

Boarding the last flight

The C-17 flight back to New Zealand seemed short thanks to my having slept, sitting up, through most of it… waking often, sore, desperately wishing for a bed, and then falling unconscious again. By the time I woke completely we were readying for our descent into Christchurch. People stripped out of their gear and emerged as individuals, with long hair, piercings, tattoos, shorts, sandals. You have to go through customs again before they’ll let you back into New Zealand. The customs officials ask, “Did you fly to the Ice from Christchurch?” Not being Todd Carmichael, the answer, of course, is always yes. Night time, humidity, wait for a shuttle, then back at the Devon, were the beckoning of the bed trumps even that first, long shower.

The weekend was spent shopping for gifts from home, delighting in real food (bananas, beer, fish, salads, Banoffee Pie from the Dux again!) and, Sunday, watching a four hour cricket match at a park on the outskirts of town. Stephanie and Suruj, both colleagues from Canterbury University in Christchurch, were there to teach the rules and the finer points of strategy and culture (trivia question: Who played the first international cricket game? Answer: the United States vs. Canada). I would always rather watch a sporting match in person than on TV, and would rather play than watch. Suruj presciently obliged by bringing a cricket bat and ball: at half time we took to the field with about a thousand underage Kiwis and he taught us how to bowl and to bat.

Now it is Tuesday and I am perched in the top floor of a house looking out over the Queen Charlotte Sound, about twenty minutes by boat from Picton. My gracious hosts at the Fern Ridge Homestay are Nicki and Neil. I spent the morning walking a four-hour portion of the Queen Charlotte Track, similar to what I did last year. Nicki took me by boat to Torea Bay and picked me up at Mistletoe Bay. As I walked along the Track, cicadas as large as small hummingbirds thudded against my hat, their collective cries throughout the bush loud enough to drown out a C-130. I also passed a six hundred year old rimu tree, a specimen of the kind that used to cover much of New Zealand, a tree similar in size, age and kind to the redwoods of California. On our way back, after picking me up, Nicki stopped the boat in a small harbor along the way to feed breadcrumbs to the herring and mackerel, which swarmed so thick to the food that I could feel their slimy, wriggling bodies as I swept my hand through the water.

Queen Charlotte Sound

View from Queen Charlotte Track

Fern Ridge still life

I have another night here and another day on the Sounds; then a night in Picton, after which I take the ferry to Wellington to see Neil and Amelia. The it’s back to winter in Chicago, which will hopefully be almost over. I am looking forward to seeing Eden, family, friends, to being in my studio, and to being in one place again for awhile.

Night view from Fern Ridge Homestay

More pictures, from the Pole

More pictures, Post Pole

Safe

Sun, 15 Feb 2009 00:00:00 +0000

Just a quick one-liner to let everyone know I’m safely in New Zealand enjoying oxygen- and moisture-rich air. Will post photos and more details tomorrow or as time allows.

Broken Glasses

Thu, 12 Feb 2009 00:00:00 +0000

South Pole Telescope

Less than 30 hours to go before we leave, and time is whipping by so fast my ears are starting to ring. As always, the last few days are the big crunch time due to the fact that we have to wait for our holes to freeze completely before turning on new strings.

Just a few hours ago we ran the first successful test with all 59 deployed strings. It was very touch and go for awhile since several things didn’t work properly at first, but now it looks good and it’s a big relief – the climax of the third act.

As our test was running, we traipsed out in -50F (-75F windchill) to the Dark Sector for a tour of the South Pole Telescope. IceCube and SPT are the 'big two’ at South Pole and it was very cool to see the other one. Like IceCube, SPT is a simple device by the standards of modern physics (compared with detectors at CERN’s Large Hadron Collider, for example) but the SPT folks are doing very exciting, fundamental physics, using galactic clusters to probe the role of Dark Energy in the evolution of the Universe. (Not only is the science exciting, but their device makes a cool sound while it’s running, unlike the IceCube counting house which just sounds like any old data center.)

Back in the station and running our final tests now. There is enough time to finish our tests, sleep a bit, pack and get out of here — barely. A great season, though. If it’s the last time I come here I will feel good about it, even though I haven’t gotten out of doors much, at least in comparison with my earlier seasons.

The biggest technical glitch of the summer seems to have been the breaking of my glasses this morning out at SPT. I have a spare pair, but they are too geeky even for me. But, if looking like an uber-geek is the sacrifice to be paid for getting 19 new IceCube strings commissioned and running, I guess I’m willing to pay it.

Home Sweet Home

Heading out to SPT

Tilo

SPT and the Dark Sector Lab

Martin A. Pomerantz Observatory

Timo and Tilo during the tour

Doin’ the SPT Shuffle

The Roof Opens

Winter-over Erik Descends from the roof of the DSL

Night shift on top of the Dark Sector Lab

Night shift at the South Pole.

More pictures

End of the Second Act

Fri, 06 Feb 2009 00:00:00 +0000

Station as seen through the window of IceCube's Ford "Gran Neutrino" truck

Almost exactly one week to go, for me. It is sort of the end of the second act, when you think you know what is going to happen, but surprises can still occur, and the final conflict has yet to sort itself out.

Things are going smoothly enough that we (the night crew) were able to watch “The Big Lebowski” in the B1 lounge after midnight. Those who got up early were able to see the Super Bowl (at breakfast, I caught the amazing play just before half-time, which was enough football for me, and got to watch all the Springsteen-style Americana with my German, Dutch and New Zealand colleagues). I cannot actually remember the last time I saw a Super Bowl in the US… but they are entertaining to see here, because of all the Armed Forces Network commercials in between plays, tempting people to reinlist. Hmmm…. Army, Navy or Marines? So many choices….

Actually, it really is like being on a very roomy and safe aircraft carrier in some ways. I haven’t been outside except to poke my head out on the upper deck for a couple of days. When planes land here, comms broadcasts the announcement that “the aircraft is on-deck.” And then there are the two-minute 'Navy’ showers, which I am actually feeling used to. Though I could get used to a hot bath pretty quickly. Fortunately nobody is shooting at us, the pay is better (for some of us, anyways), we don’t have to salute, and, for us summer folks, the tour of duty is mercifully short.

Though at times uncomfortable and stressful, this trip has been quite vivid in a way, with almost a sort of real-time nostalgic feel to it. I feel peaceful about the likelihood that it’s the last time I’ll be here, and I find myself noticing the texture of the station floor, the views out the windows, the galley food, the unwashed heads of hair, the grumbling in my stomach and the smiles in the hallways of people I’ve seen year after year, whose names I have never learned.

Pigs and Fish

Wed, 04 Feb 2009 00:00:00 +0000

There is a short list of adventures one can have in the US Antarctic program. You can experience landing in the cockpit of a C-130 (done it). You can fly over the crater of Mt. Erebus (done it). You can go to 'Happy Camper’ school and learn how to sleep in your own snow- or ice-shelter (never done it). Here at Pole you can ski out to the 'Love Shack’ a few miles from the Station (never done it). You can do an outdoor hot tub in -30F and feel your hair freeze solid (done it). One thing I’d never done until tonight was go down into the utility tunnels underneath the station. I’ll let the pictures tell most of the story, including the temperature (-60F). It was really cool! (And cold.) Afterwards we had a sauna, followed by the Midrats Season Finale with Filet Mignon, seafood skewers, stuffed portabella mushrooms, sushi, chocalate chip cookie chocolate sandwiches, and upside down peach cherry pie.

Life isn’t all bad at the South Pole.

Suiting up to hit the tunnels

Tunnel crew

I wish I owned this sign

A half-mile long tunnel 60 feet below the snow surface

Dead end

Stephanie &co.

Timo Griesel

Deep Enough

Jan Luneman

Pig shrine (yes, it’s a real pig’s head)

-60 F

17-year-old sturgeon with caviar: gift of the Russian base (I seem to have lost the hi-res version of this picture :-( )

Yesterday I headed out to the IceCube Laboratory (ICL) with Thorsten from LBNL to debug a nasty little low-level problem. We made some progress and I took some more pictures in and around the ICL:

Back of 'DOM Hubs’ where strings are plugged into the computers, for which I had the privilege of writing a device driver

Winterized 'Seasonal Equipment Site’ (Drill Camp)

Cable conduits into IceCube Laboratory

Our neighbor: the South Pole Telescope (whose principle investigator lives near me in Hyde Park)

Last Arrivals

Sat, 31 Jan 2009 07:00:00 +0000

Timo and Jan arrived from Mainz today — the last arrivals for IceCube this season. They are working on a system within IceCube to detect stars which explode in or near our galaxy. Our nightshift roster is now complete.

Night shift serenade

The flight arrives

OK, you can stop taxiing now (IceCube Laboratory in the background)

Passengers from the Plane

Lily White

Sat, 31 Jan 2009 00:00:00 +0000

Followup to yesterday’s post: then there are the times when you are quietly pedaling the stationary bike in the gym while your iPod plays your favorite audiobook, and a couple of lily-white guys (did I mention it’s mostly lily-white guys here?) come in and crank up some of the more vulgar and un-musical hip-hop on the gym stereo.

This is a place of compromises.

In a Dry and Waterless Place

Fri, 30 Jan 2009 00:00:00 +0000

I’ve had some of my most intense music experiences in this place, often while working out. (Anyone else had similar experiences at high altitude?)

Just got back from my daily workout in the darkened aerie above the gymnasium where they were playing the U2 concert film “Live in Paris: The Joshua Tree.” Songs from The Unforgettable Fire taking me straight back to freshman year, 1985: friendships and first loves… my incense-filled room across the street from the hospital where I was born; learning to program on a 256k IBM PCjr… taking my first steps on the long, strange trajectory that lead me to this work, in this place.

Music has the power to reach across time and space, to cement and trigger memory and to connect people, all in ways that film, literature and visual art cannot. I wonder if Bono & co. could have imagined during their concert in 1988 that their music would fuel the workouts of a few South Polies twenty years later. That’s the way it works, I guess… most of the time we swim in a sea of ignorance about the effects we have on each other and the world.

Time for a 20 second shower before today’s first meetings.

And who are you that reads this? In what day / year / millenium? And in what strange place?

Immortality

Thu, 29 Jan 2009 07:00:00 +0000

Despite the morning’s wind and low visibility, colleague Joanna just arrived from Berkeley hand-carrying some cables and a special surprise from Jerry at LBNL, carefully packaged in anti-static plastic and bubble wrap.

It’s the printed circuit board from one of our Digital Optical Modules, one which failed visual inspection and is therefore scrap for engineering purposes. I.e., a nice, flat, attractive paperweight. I’m touched, particularly because it has my name etched on it along with those of several of my current and former LBNL colleagues, and because I have just been told that all the the DOMs deployed in IceCube are thus enscribed.

Though it’s not quite a burial fit for a king (not to mention the fact that I’m still alive and kicking), our names entombed inside the polar ice cap will endure for many millennia, far beyond the current age of the Pyramids of the pharaohs. When some future civilization visits the barren ruins of earth and goes prospecting for technological artifacts in the polar ice, they will find the sigils which make up our names and wonder at their significance. Or, in a hundred thousand years, when the flowing river of ice finally reaches the ocean, they may rust free of their mutual tethers and drift the ocean currents… five thousand messages in spherical glass bottles to be found by whatever forms of life might remain to roam the seas or shores of future Earth.

Thanks Jerry and Joanna.

Routine

Thu, 29 Jan 2009 00:00:00 +0000

It’s 10:05 AM and getting close to bed time. The weather today is strange – quite warm (almost minus 10F) but with 21 knot winds, making the wind chill about 50 below. The snow is blowing high and visibility is poor at times. But it’s quite beautiful.

Life is settling down into a routine, of sorts. Wake up as late as possible, usually before dinner. Clean up room, go to gym, meditate, have breakfast. Start work when satellite rises around 11:30 PM. Work until mid-morning, and then wind down until I have to do it again.

Most of the other late-season 'Cubers are on night shift. Our 'days’ consist of testing new pieces of the detector and new bits of code, troubleshooting, adding features, all somewhat endlessly. We have regular consultations with colleagues up North via phone meetings and instant messaging; and a daily 8 AM face-to-face meeting here just to check in and make sure everyone has what they need to get things done. There are a lot of things happening at once and it turns out I have to orchestrate and schedule quite a bit. But, so far it’s going ok. We have tested an additional 9 strings combined with the 40 we ran throughout 2008. If we can get the remaining 10 more strings added in and calibrated, that will be a good start to the winter season. In addition, the experiment control software I wrote throughout 2008 is starting to come together and be fully operational here.

IceCube Laboratory

One strange feature about being here is that current and past summer seasons blend together into a sort of day-lit constancy, where the real world up North seems not exactly unreal, but somehow parallel and separate in both space and time. The sameness of the surroundings, the work, and, to some extent, the people (many of the denizens here are 'repeat offenders’ like me) all contribute to this. Lots of people have a summer home… mine seems to be a January slot in a late-season population of 250 or so people dreaming of Christchurch and points beyond. Part of the strange timelessness of this place is also the absence of people who have been here before, never to return, and whose spirits somehow knock about the place still. What part of me will be left behind when I get on that northbound plane for the last time?

Last night I dreamt a great rainstorm fell here. It was delightful at first, but quickly turned terrifying because everything started to melt and submerge. I knew that when the cold came back everything would be frozen in place. I have no idea what that means, but images of liquid water are powerful in a place where showers are tightly rationed and the daily per-person water consumption is flashed up on the video scroll in the galley.

Tourists

Wed, 28 Jan 2009 00:00:00 +0000

In the next few days, the last few 'Cubers are arriving on Station to help us with end-of-season commissioning of the new strings. Many more IceCube people are leaving, however… most of them drillers. It will be nice to be here when things are a bit quieter. The winter-overs are already talking about how great it will be when we summer folk are gone.

Some people pay a fair chunk of change to fly here for an hour or two, look around, and leave. Tourists, we call them. Others fly in for McMurdo ('sleigh rides’) to see the place, or to work for a day. Still others ski here from various points remote, and camp a few hundred yards from the Station (I’ve already given a few impromptu tours to various frost-bitten groups). Those of us here for the Summer are called 'tourists’ by the winter crew.

Of course, everyone is a tourist in Antarctica. Despite what repeat winter-overs may sometimes feel, there are no true Antarcticans. We are as much visitors here as we would be to Mars. Take away the airplanes, and we all perish.

I have posted a video of the last string deployment.

Meanwhile, a group of five or so of us are up all night, every night, fussing with software to read out the newly-deployed strings as they freeze in. We still have two weeks but it will go very quickly, and we tourists will redeploy to various points North.

Passing Notes

Thu, 22 Jan 2009 00:00:00 +0000

My room in the A4 wing of the Elevated Station

Been here about 58 hours so far and the reason I am counting in hours instead of days is that my sense of days is pretty shot. I have a screwy sleep routine: 10 hours awake, 3 hours asleep. Lather, rinse repeat. I am trying to be up at night (we keep New Zealand time here) for the satellite hours but usually that entails a long nap in the middle, which curtails my sleep during the day. I’ll figure it out.

Despite the goofy schedule and the altitude, last night I felt well enough to walk with Dave out to drill camp and watch the nineteenth and final IceCube string deployment. Being outside reminded me of how beautiful it is here and how hard that is to see until you are actually out in it, seeing the changing sky, the sastrugi, the myriad structures scattered about the snow, structures which do not qualify as architecture but are solely and completely engineering.

Ryan takes a well depth measurement during deployment.

Seeing the last deployment was a real pleasure, partly because I didn’t have to work it — I was a tourist, a real treat after having worked on perhaps a dozen or so deployments myself. It was also sweet because the drilling and deployment went so well this year… and also bittersweet, perhaps, because it was the last deployment I’m likely to see. I took photos and some video which I will post as soon as I can.

Heading back from the Dark Sector

Around midnight we decided to head back. A bunch of the drillers were driving back in the “Ford Gran Neutrino” (the modified van which IceCube uses to shuttle back and forth to the Dark Sector), and Dave and I squeezed in. After we crossed the skiway, the van stopped and the drillers started discussing the expedition which was camped near the station. A small handful of jacked-up and modified Toyota trucks have driven here from a Russian base in support of a ski race across Antarctica. It was decided to pay the campers a visit, so we drove the van over there and piled out, admired their trucks, and started talking with a member of the expedition.

Apparently about 300 people have skied, walked, or driven to South Pole this Summer. These people live in a sort of parallel universe which is disconnected with that of the station. They are allowed a cup of coffee inside and not much else; and few people walk over to their makeshift camps near the Pole marker. So it was a treat to talk to this fellow who has had a pretty different experience in Antarctica. After a few minutes of conversation he pointed to Sven and said, “Hey, you’re Sven!” Then he handed him a note written on a scrap of paper from a case of beer. It was given to him by a member of yet another expedition which he encountered during his trip, to give to Sven at the South Pole! We all laughed. Even though it is nearly as empty as Mars (i.e. nearly zero people per square kilometer), Antarctica is a small place!

Sven greets an expeditioner

After delicious burgers at Midrats, we started work. There is ever so much to do and I am starting to get acclimated enough to be able to focus on it. The task at hand is to connect the nineteen new IceCube strings to the rest of the detector, and to calibrate them to prepare for the new physics run starting April 1. Also to carry out a variety of software upgrades and make sure everything works correctly. Also, to report progress to the north and to coordinate all the activities to minimize conflicts.

Watching the Obama Inauguration

This evening, after another short sleep, we watched highlights of the Obama inauguration, hand-carried on a DVD on today’s flight. (The CBS news clips carried over the Armed Forces network carried the word “Live!” at lower right, which was certainly ironic in this setting, days after the fact.) It was pretty amazing to see our Hyde Park neighbor take the oath for the Oval Office.

A solemn moment

Now for a first foray to the gym, and to start into work again.

Translation

Tue, 20 Jan 2009 00:00:00 +0000

10:07 PM New Zealand Daylight Time and I am well-socketed into my room in the station at the South Pole, watching videos, napping when I can, drinking as much water as I can stomach and waiting for my body’s chemistry to adjust to a physiological altitude of… let me check… 10,455 feet (the local air density here depends on the weather, and is displayed continuously on the monitors and the local Web site).

To kill time while I acclimate, I just watched “Lost in Translation” which, with its parables of jet lag and alienation, seem somehow appropriate even for this very different destination. “Translation” in physics means moving from one place to another. I have definitely undergone some serious translation.

My stay in McMurdo was very short for a southbound trip — less than twelve hours. Which suited me fine. We arrived after 10PM, fairly exhausted from the long and crowded flight, had the usual (relatively extraneous) briefing, and then immediately bag-dragged for the Pole flight. The weather was pleasant and sunny, with a small stream of water flowing downhill past the Movement Control Center (MCC) where we weighed in.

After a brief midrats (midnight meal) we turned in for a few hours of sleep in the dorms. Then we trudged back up the hill at 0645h to the MCC for transportation to Williams Field our flight South. The air had chilled quite a bit during the 'night’ and the stream had frozen.

Willi Field Air Control Structures

Main Street at Willi Field

Today’s flight was quite a bit easier than yesterday’s. Two of the Air Guard loadmasters hung out and chatted with us in the airfield cafe while we waited for our plane to be readied, and then rode out in the van with us. There was no briefing, just 'get in and go.’ There were only 7 passengers this time, so there was room to spread out after take-off (I even got to lie down for awhile).

Our ride

The scenery was lovely, as in previous years, but another veteran and I compared notes and we were certain that today’s flight took a different route up to the Polar Plateau than in the past. (I think it’s fun to be able to know the route well enough to be able to tell the difference with no map and no human landmarks.) Low clouds obscured some of the terrain which made it both harder to see anything and more beautiful, as sky and ice merged into one milky mixture punctuated by craggy, chocolate-colored rock.

Watching the Scenery

Before we knew it the plane began its landing approach and we suited up, goggles, balaclavas, neck gaiters, boots, glove liners, mittens, fleece, and Big Reds (the ubiquitous parkas). Always a student of economy of means, I enjoyed watching the senior loadmaster giving the order to his subordinate across the plane to put on her gloves, by wagging his leg at her and raising one (gloved) hand.

And then we were down, out, breathing those first breaths of cold, rarefied air (an experience I will miss, assuming this is my last trip) and looking out at the minimal snowscape punctuated by the hyper-functional structures of South Pole Station.

Our flight

Arrival at South Pole

Yours truly arriving – Photo by Mark Krasberg

Mark and Dave came out in the -21F cold to meet us, which is always such a nice thing — to come to the end of the Earth, and see friends and colleagues, familiar faces, to have someone carry your carry-on while you suck your first breaths of high-altitude air, to make your way to your room and start the slow process of unpacking and settling in. I am here until February 13… long enough that I have to think of it as home-away-from-home. After seven times already it should be pretty easy.

I will post more drawings and photos soon, and will back-populate some of the posts with pictures, so check back for updates.

Soon the satellite will be up!

More pictures from the flight to the Pole

Gouache, and a new system for conquering the world

Sun, 30 Nov 2008 00:00:00 +0000

Here’s my new system for conquering the world. It will solve all my problems, at least the artistic ones.

Ingredients:

sketchbook
pencils of various hardness/thickness/sharpness
white gouache paint, water and brushes
Uni-ball micro pen
Mitsubishi 0.28 micro pen
2-3 Kuretake brush pens filled with black ink, and diluted ink
Kneadable erasers
Helix battery-powered electric eraser
Presto correction pen
My standard sketchbook (I go through several of these a year).

The system is simple. With this extremely portable set of tools I can capture and eliminate ideas freely and quickly and iterate until I have a drawing that is worth making a painting of (or at least not want to rip out of the sketchbook and toss into the fireplace).

In the past, what was missing was the 'iteration’ part, especially while working in ink. Although I sometimes lay in material in pencil before going to ink, I find that I 'think better’ in ink much of the time… also, pencil drawings tend to get muddy if erased too much.

The gouache solves that by allowing you to quickly obliterate a part of the drawing. And, unlike acrylic or white-out types of liquid, it is easy to erase and takes graphite and brushed ink nicely.

I guess I’ve steered clear of gouache and watercolor because I like the rich, layered textures of oil and acrylic. But I know that comic book inkers use gouache frequently, and I’ve had fun playing with it today.

Yesterday, at the Chicago History Museum, I saw an aerial rendering of the 1933 World’s Fair, likely done completely in gouache. Perhaps 72“x30”, it was my favorite thing out of few thousand artifacts there — utterly photographic from far away, delightfully graphic when viewed close up.

Recent drawing: gouache, ink and graphite.

Settling In

Tue, 30 Jan 2007 00:00:00 +0000

Been here a bit more than 24 hours, and I'm settling in. Mostly I've been preoccupied with debugging a particularly nasty system error in our device driver; hopefully I'll have that wrapped up soon and can join Kael, Keith and Dave in higher-level deployment and testing of DAQ (the data acquisition software).

Once I get acclimated to the altitude I'll be much happier. It's been hitting a bit harder than last year, though I took the diamox they encouraged us to take in McMurdo. Unfortunately the diamox makes my hands and feet and face feel like they're buzzing, which is irritating. And I'm generally out of breath, and have to pee constantly (apparently the body adjusts its pH to the new altitude this way). Out in the Dark Sector (the radio-quiet area where our experiment is) the facilities consist of a "solar" (i.e., painted black) toilet; the solar toilet is essentially an otherwise unheated outhouse with a styrofoam toilet seat built over a half-buried waste barrel. Ah, the life of luxury.

All is pretty good, though, and it's fun to see familiar, sunburnt faces with unwashed hair (remember, 2 showers per week, 2 minutes each, folks!), and the same old tracked vehicles and space-age buildings and the endless horizon and the great big open sky with long simple cloud shapes a seemingly infinite distance away. I'm settling in and with any luck we can start to get stuff done quickly.

Batteries and Sheep

Wed, 24 Jan 2007 00:00:00 +0000

John and Dave on the Anaconda MTB Path

With a day to kill yesterday, Dave, Kael and I put our heads together with Sandy, our proprietress at the Devon, and decided to rent bicycles for a trip to the ocean. Sandy said that there were a bunch of WWII gun batteries worth seeing near the entrance to the harbor where Lyttleton lies, south of Christchurch. That sounded pretty good to us. Within 20 minutes the bike rental guy was there, providing us with mountain bikes, helmets, locks, and a suggested route and we were off, pedaling furiously first through Christchurch traffic, then up a fairly wicked mountain grade, past residential zones at first and then just trees, rocks, wet clouds and, when the clouds parted, amazing views.

The first photo set for this trip will tell the story better than any writing I can come up with - suffice it to say I was pretty astonished that such rugged beauty lay within a vigorous pedal of Christchurch. And, that we were all thoroughly wasted after more than eight hours of biking up and down the mountains south of CHC (sometimes on paths as challenging as a BMX bike course). At one point, as we juddered down an especially challenging path, I jokingly asked Kael if this wasn't one of those team-building exercises to get us ready for the Pole.

Of course, no blog about New Zealand would be complete without a picture of a sheep:

A Sheep

The gun emplacements were also amazing. For reference, they are at Godley Head south and east of Christchurch. We had to get the bikes back by 1830h so we could only see half of what was there. I've seen WWII defensive emplacements in southern France, on Corsica, and in California, and these were much better preserved and much more extensive than any I'd ever seen. The coolest part was a set of underground rooms and tunnels, completely dark (Kael's flashlight and my camera focus-flash kept us oriented). Here's a teaser, but do see the complete set:

Kael descending into a bunker

Today we get our clothing, if I can actually get my battered carcass going…. Breakfast may help.

All for McNaught

Tue, 23 Jan 2007 00:00:00 +0000

Since its swing through perihelion a few days ago, we should be able to see Comet McNaught from here. Last night, a decent meal of sushi (punctuated by green tea ice cream to die for), we went looking for the comet in Hagley Park, a large open area ringed by trees not far from the Arts Centre and our B&Bs. As the hot winds from the north buffeted us, we saw Venus, a lovely sunset, and amazing, hurricane-like clouds of the sort I suppose you only get to see in the middle of the Pacific Ocean – but no comet. We're going to try again tonight. No other plans yet.

It's strange to have summer, an 80 F January thaw… but nice. I'm already starting to get used to it, which is a very bad idea.

Descent

Mon, 22 Jan 2007 00:00:00 +0000

Dawn over the Pacific

Dawn over the Pacific, and 30 minutes left in the Big Flight from LA to Auckland. The first couple of hours were the worst. We had to wait for luggage at the gate for almost an hour and I was already sore and restless before we took off. But I managed to distract myself with "Gravity's Rainbow," songs on the iPod, a decent dinner, a glass of Australian chardonnay, and, most helpfully, Gregory's suggested Tylenol PM which put me out while trying to focus on the first few minutes of "Miami Vice."

So the upshot is that I managed to sack out for at least 1/3 of the flight….

… now I'm safe in Christchurch having landed perhaps 30 minutes ago. Staying at my favorite hotel, the Devon, and not flying until Thursday. So it's time to relax and enjoy summer for a few days (it was rainy in Auckland, but it's hot and sunny here).

Giacometti

Mon, 22 Jan 2007 00:00:00 +0000

Just got back from a tasty lunch at the Dux De Lux followed by a visit to the Christchurch Art Gallery, an attractive little museum a block from the Devon.

They had a lovely show of Giacometti prints, drawings and sculptures. Giacometti and I share a birthday (Oct. 10) and he died the year I was born (1966). His sculptures are haunting and deceptively simple-looking - tall and thin figures standing rigidly erect, features all but obliterated in a field of surface vibration. But I found his drawings and prints even more attractive. Very simple subject matter again: a head, a chair, a small group of figures. But rendered loosely as spare lattices of visceral response to…. something, seen in the few models he employed over and over again (his mother, his brother, his wife, his mistress). Made me want to make etchings again, and lithographs.

Also a wonderful set of small rooms filled with drawings and paintings by the spookily gifted 19th c. New Zealand/Dutch artist Petrus Van Der Velden - in particular, a series of sketchbooks and studies for a painting called the Dutch Funeral. Gorgeous drawings à la Rembrandt or certain 19th c. Russian artists like Ilya Repin, but moodier, darker, quieter, seemingly obsessed with beauty in death and suffering. Some of the sketches were tiny, no more than a couple of inches on a side, brown pages taken from battered sketchbooks… haunting, inspiring, and very humbling.

Tomorrow we have the day off; Wednesday we get our cold weather gear, and Thursday we fly to McMurdo.

t=0

Mon, 22 Jan 2007 00:00:00 +0000

Under way at last.

My flight to LA leaves in 40 minutes - the first of five flights, a migration of sorts. Flying south for the winter. Well, summer…

This is trip number six; as such, what I write about won't be a complete blow-by-blow as I have done in earlier years ~~(see the links to previous years on the sidebar)~~, but you will get to see the trip digested through the eyes of an "old hand." To get in the mood, let's start with a few excerpts from my first trip, ten years ago…

December, 1996.

FLYING TO NEW ZEALAND

8:47 AM New Zealand time, Aukland airport.

New Zealand is cool!!!!

Everyone drives on the wrong side of the road. It's green and wet.

A cyclone is expected to hit tonight. I totally love this place already. Gotta go, we're taking off.

[…]

I think I have found the one place in the world more beautiful than Wisconsin. Way better than Hawaii, California, Switzerland… we'll have to see about France.

On the coast near the Aukland airport, tiny green trees stick out of the shallow water like little puffballs. Everything is green and lush, a shock after Wisconsin. Apparently it frosts only a few times a year, and those are the bad winters. Lots of tiny roads and houses scattered about on irregularly-shaped land. Impending cyclones with 150 km/hr winds provide added excitement. I believe our aircraft took off just ahead of the storm.

Before takeoff, I thought we had a problem–smoke or steam was drifting into the cabin above our heads through long slits. No one else seemed to mind, so this is apparently normal. Maybe they want to keep people from drying out. I sure could use a hot bath, come to think of it.

Horse racing and cricket figure predominantly in sports news in the New Zealand Herald.

On the airplane telly last night (gotta start talking like the locals) there was a blurb on some sport that looked like it might be rugby, but I couldn't tell. It was total mayhem–people running and kicking a football-like thing, catching it, tossing it around. Pure action but much more three-dimensional than soccer or football; it looked more like the pure hand-to-hand combat that all sports must have arisen from. [It was Australian Rules Football.]

Time for the second airplane breakfast of the day. I'm having fun!

4:45 PM, Jan 1, 1997, Hotel California, McMurdo Base, Antarctica Temperature, about 30 degrees F.

[…]

FLYING TO MCMURDO

"Few people have ever seen this," said Neck [our pilot]. I felt amazed that I had ever doubted that I would want to make this trip.

What began as a faint line of mountains gradually became a spectacular view. As we approached, I studied the approaching topography on the navigation chart, noticing the special symbol for crashed aircraft, and finding where one lay on the map not all that far from where we were flying. But the view soon overwhelmed my interest in the chart. I took many pictures which I hope will mitigate my inability to describe the mountains and long, winding glaciers. I [tried to see] the aerial view with both scientific and artistic eyes. Looking at rivers of ice and the clouds that flowed around them, I could see that ice and clouds/air have a lot in common in appearance and underlying form. Both flow through available channels, though on different timescales, and reflect the light hitting them with so much purity that one is not distracted by color, and can appreciate the near-still dance of pure form.

Especially beautiful were the contrasts between the raw, jagged faces of exposed rock and the breathlessly smooth contours of white, flowing ice, occasionally ripped in striations caused by sudden shifts of inclination in flow.

[…]

We passed close by town before landing on the skiway on the ice sheet beyond. From the air (and from the ground too), the town itself looks more like a collection of industrial freight boxes on black dirt and rock than a real town. It is, however, an actual town, with a peak summer population of 1200, a post office, a bank, a chapel, bars, apartments and dorms, laboratories, "roads" (stretches of dirt that people drive on, as opposed to stretches of dirt people don't drive on), a hospital, and a power plant. As far as I know, the only thing McMurdo lacks is a cemetery. None of this is really apparent to the new observer from the air, cockpit view notwithstanding.

We passed the town and the rocky point it is on, and descended to the runway just behind it. All this while the crew was going through their landing checklist, acting very efficiently and professionally. I always like to see people do something they're good at. And it was of course a tremendous thrill to see the runway rise up to greet the aircraft with a soft thud. Suddenly we were sliding down the ice, slowing down with reverse thrust, watching odd, orange tracked vehicles and equipment slide past. We went to the end of the skiway, turned 180 degrees, and parked. Not long after, I thanked the crew, and the other "beakers" [scientists] and I were led out on the ice towards a van with big tires and high suspension (like most McMurdo vehicles). It was quite warm, warmer than the Wisconsin I had left just a few days ago. I looked around. There was a great expanse of ice, and then mountains across the bay; nearby, the black hills hiding McMurdo from us jutted into the white plain. A single seagull or skua stood not far away. I was very excited. I had landed on The Ice.

[…]

AT THE POLE

I dozed and listened to music for a few hours, unpacked a few things, and pondered my situation while taking several trips in the snow out to the toilet and shower shack. It was there I met the guy who told me about the piss cans [makeshift chamber pots]. We also talked about other things about the Ice in general. I told him my impression of McMurdo was better because there was scenery there. He replied, "There's scenery here, too. All you have to do is close your eyes." Words to live by.

There really is no scenery here. It's like being out in the middle of a frozen lake, except there is no end to it! The dome, various buildings, odd tracked vehicles known as "sprytes" as well as other trucks and construction equipment, the runway, and the ceremonial and true geographic poles, are it. They are within about a mile of each other. This is a small speck on an immense plain of ice. To think that Scott and his men made it here on foot boggles the mind. (Amundsen, who beat Scott by a few weeks, used sled dogs both to move men and gear, and to feed the men as well towards the end of the expedition. Amundsen made it back alive. Scott didn't. The station is named after both of them.)

Fast-forward to 2007.

Airborne now en route to LAX, listening to "Dub Side of the Moon," a thoroughly postmodern Reggae-Floyd mash-up (thanks, Todd) on the new red Nano courtesy of Santa Jobs and the Apple-cheeked elves of Cupertino. Fell asleep already before takeoff, and realized as I awoke to the trill of the engines that I could happily stay asleep until magically waking up in my small room in the A wing of the station, go for a quick jog on the treadmill, take my 2 minute shower, and start working. Sad, eh? I'm ready to be there before the trip has even gotten under way. I may think I know what to expect, but I know enough to know that there can be surprises¹ good and bad when traveling to the Ice. If nothing worse than previous years occurs, I'll be perfectly content. Once I get to Christchurch I'll be in good shape. The concatenated series of three flights (4+13+2 hrs) is the hardest part of the trip.

For newcomers to the blog, the flight sequence is ORD->LAX->CHC->MCM->NPX: Chicago to LA to Auckland, NZ to Christchurch, NZ; get extreme cold weather (ECW) gear, fly to McMurdo station on the edge of Ross Island, then to South Pole. Acclimate to the altitude, then accomplish miracles.

A suitcase, a small backpack and a manly-purse are my only luggage items. Packing light is a challenge and a pleasure. Most of the way I won't have to schlepp the big bag, but a long schlepp it is indeed, and I know what not to take now. Colleague Bob Morse told me, "all you really need is a toothbrush and some underwear." He's not far off. A laptop and a towel are the other necessities, though some people manage to do without the laptop (how?) and one colleague I know forgot his towel and had to suffer through until a fellow South-bound traveler could liberate one from a hotel in Christchurch for him. Some things I do take I wouldn't have thought of on the first trip: spray bottle for extra humidity, clothesline for same (most clothes hang dry in <8 hrs in the driest, highest desert in the world). An extra bag to store summer clothes and other stuff I won't need in Christchurch, and for gifts on the way home. Similarly, I give back perhaps 1/3 of the ECW gear in Christchurch, since I know I won't wear it.

As the engines gunned for takeoff from Chicago I thought of the software tests on "new DAQ" we have been running for the past few weeks. Three or more of us in separate offices, separate cities, tethered together by online chat and daily conference calls, putting the software out on the "runway" (a bunch of networked computers), giving it lots of gas, and watching with bated breath to see if it took off. Cheering when it did, sighing when it crashed and burned, sending us back to the proverbial (digital) drawing board. It's not exactly the way you build jet engines – thankfully, I thought, as we soared into the skies above O'Hare.

Or is it?

A few small personal and work goals for the trip:

Help to achieve and maintain hitherto-unseen stability and elegance in the data acquisition (DAQ) software;
Stick to a routine of sleep, meditative stretching, and running that keeps me healthy and positive. Keep up my running conditioning for Shamrock Shuffle and '07 Chicago Marathon;
Maintain good-natured and constructive dialogue with colleagues. Steer as clear as possible of any of the ugly politics that tend to crop up during summers at Pole;
Finish or at least take a good whack at Pynchon's "Gravity's Rainbow," a brown-paged paperback copy of which I "borrowed" from the AMANDA lab in MAPO in the Dark Sector at Pole in 1998. I'll be leaving the book (sans overdue book fines) behind in the library in the new station. I like Pynchon but have had trouble finishing anything except "The Crying of Lot 49." But I've beaten Stephenson's amazing Baroque Cycle (almost 3000 pages) twice now, so I may be ready for the "'Rainbow," and if I can finish that maybe I'll try "V" again or his latest brain-bashing brick of a novel;
Keep moving my drawing forward in bits and pieces. It has been a real source of pleasure of late to return to drawing. My focus has been on figurative drawing from life and from imagination, steered by a measure of storytelling or narrative intent. Originally I was going to make this a blog of drawings, and I may post some drawings if they "work," but I'm going to leave it open-ended for the moment. But I have four sketchbooks and plenty of implements to play with;
Get completely caught up on episodes of Battlestar Galactica (my new favorite guilty pleasure);

Of course, the first goal (doing the actual work) will probably take up 99% of my time and energy.

Other differences between this trip and last year:

I'm supposedly lodging in the A wing rather than the B wing of the station - which may mean a larger, or at least a quieter, room.
The new gym (and, in fact, the remaining half of the newly-constructed station) is available.
I have no management responsibilities, since Kael (IceCube DAQ software lead) is coming at the same time - whew!
This trip is shorter by a week - also a good thing. Anything shorter than three weeks is perhaps not worth the long slog to the Ice. Anything longer starts to feel too long, though I've done six weeks and, of course, some people go for a year or more….

About this blog: apologies in advance if things are not explained enough or background details (such as what we're building) go missing - if people have specific questions, they're more than welcome to write – I'll try to answer them here. Check out the old blogs too.

Later!

Surprises I or other colleagues have experienced en-route: wandering herds of penguins; tours of Scott's Hutt; unexpected cabin depressurizations; being invited to sit in the cockpit during landings; flights "boomeranging" (having to turn back due to weather or mechanical issues); scenic flights over active volcanoes or through canyons; and, of course, sitting around Christchurch or McMurdo for hours to dozens of days, waiting for the weather to clear or for a needed replacement aircraft part to arrive from somewhere in the US military's global infrastructure.

The Big (Really really big...) Picture

Mon, 22 Jan 2007 00:00:00 +0000

In case I don't get to it any time soon, Keith has done a decent enough job explaining the big picture of Icecube on his blog.

How to subscribe to blog updates

Mon, 11 Dec 2006 00:00:00 +0000

This year I will be posting directly to the blog, rather than sending out e-mails to everyone like I've done in previous years.

The easiest way to get notified of updates is via a news aggregator. A news aggregator is similar to your Web browser, but with the added feature that it lets you know when new posts have appeared. This saves you the chore of having to check the Websites you are interested in "by hand." In other words, it's sort of like an e-mail inbox for the Web.

It's a big advantage for the blogger too, because he or she doesn't have to collect e-mail addresses from everyone or send out daily e-mails to hundreds of people (this is especially nice when posting from a remote site like the South Pole). Anyone can subscribe or unsubscribe at any time without the blogger having to do anything.

In order to set up your news aggregator, first you'll have to install some software. ~~I recommend FeedReader for Windows, and NetNewsWire for Mac (free version available here)~~ [Feedly is what I use these days].

Once you have your news aggregator installed you "subscribe" to a blog by giving your news aggregator software the URL for the blog. This will be something called an "Atom feed" or an "RSS feed." Use ~~this link~~ as the feed for this blog (right-click or, on the Mac, option-click on the link and say "Copy link location" to capture the URL). The exact method of "subscribing" depends on the software, but should be fairly obvious once you have the software installed – on NetNewsWire Lite, I use "File->New Subscription" and paste the URL in.

Many of you already look at blogs - most newspapers have RSS feeds too. It's a really nice way to get fun stuff off the Web (cuts down on the advertising, too!).

Happy reading!

p.s. to colleagues, friends and family - If you have a blog you'd like me to link to, please drop me a line.

Example news aggregator:

example news aggregator

What The Blog is For

Sun, 10 Dec 2006 00:00:00 +0000

On my sixth trip to the South Pole, I will try to publish at least one drawing or photo each day, possibly including some text.

Auckland; Leaving Already?

Tue, 11 Jan 2005 00:00:00 +0000

Europe in Miniature

09:00h

Made it back to the Midwest of the South Pacific, where the most threatening thing is the drug dog that will point you out to Customs if you have had any fruit (or drugs) in your backpack at any time in the last 10,000 years.

How is it that a 12 hour transpacific flight can seem to take forever, and yet when it's over it's hard to say what exactly happened during those 12 hours? The memory of too-small seats and futile attempts to sleep sitting up is imprinted more on my body, which feels like the 747 actually rolled over it, than in my mind. The in-flight film "Collateral" made more of a mental impression (Tom Cruise blowing away various people on the LA streets I just went jogging on "yesterday"). But the silent view from the back of the darkened plane of hundreds of personal LCD screens embedded in the back of every seat, all tuned to different movies, TV shows, games, etc. was lovely… something straight out of a contemporary art gallery. In the 21st Century, apparently, everyone gets their own in-flight media smorgasbord.

At any rate, though I missed my flight to Christchurch, I'm on the 10 AM flight, just an hour or so away. All I hope is that I don't have to show up at the CDC to get my Ice clothing TODAY. I'm hoping for a hot bath and a nap at the Devon.

Everything is exactly as I remembered it so far.

Leaving Already?

12:43h

Arrived in Christchurch just a short while ago and I have to get my Extreme Cold Weather Gear in 15 minutes…. may leave for McMurdo at 6 AM tomorrow; no rest for the weary….

Halfway Around the World; Dragging the Soul Behind

Mon, 10 Jan 2005 00:00:00 +0000

Halfway Around the World

09:14h

I'm still stuck in LA but feel as if I'm already halfway around the world. Went out for a jog, just around the block, smelling forgotten, friendly smells, but somehow got stuck in a run down neighborhood and had to jump two fences just to get back to my hotel, which was within line-of-sight the whole time. As everyone knows, this place is not set up for pedestrians. All the trees here third-world-nation different and people drive gleamingly new cars but live in shabby apartments… I love California but even after living here for five years it feels more like a tourist destination than Home.

I'm on the same flight to Auckland tonight so I still have 8.5 hours to kill. Time for a nap.

Dragging the Soul Behind

12:13h

I'm listening to an audio book on my iPod: William Gibson's "Pattern Recognition." The main character is in London with major jet lag. I'm not officially too jet lagged yet but am tired from being airport-bound until 3 AM this morning. Gibson's protagonist describes jet lag as your soul being reeled in from whatever far away place you came from, drifting like a kite through the troposphere, slower than the trans-oceanic jet you came in on. I'm trying to reel in my soul just as quick as can be, 'cause soon he'll have a long distance to make up.

When you're tired but in your normal routine, you've still got the glue of that routine keeping you together and productive. When you're in a new place, with time to kill, but alert and rested, you can happily go exploring, or, if you're a mobile tech warrior, productively tick items off your lists of things to do. But one of the paradoxes of killing time while on travel is that you're both uprooted and exhausted, so you live moment to moment, in a kind of blurry fog, not sure what to do next, and too tired to care very much.

Time to repack, head to LAX and take another stab at Aotearoa, the Land of the Long White Cloud (New Zealand).

Launched

Sun, 09 Jan 2005 00:00:00 +0000

GPS: N40d00.000' W88d35.625' (but, bad signal in O'Hare)

13:01h

Well, I'm underway, waiting for my flight to LAX. Managed to get my form stamped at customs, and found a seat near my gate with a power outlet.

Itinerary: Chicago -> Los Angeles -> Auckland, NZ -> Christchurch, NZ -> McMurdo Station -> South Pole Station. Five plane rides, something like 30 hours in the air.

This is my fourth trip but it's been a five year wait and enough time has gone by that I am not 100% sure what to expect. Will I get a few days to unwind in Christchurch? Will I get to stay at my favorite place, the Devon Private Hotel? Is Gloria still the inkeeper, will I get to eat her tasty if fattening breakfasts? Will I get to go jogging along the tiny river Avon?

Is the Clothing Distribution Center (CDC) still the same? Is all the gear they give you the same? Will I have room to stretch out on my flight to the ice? Will I get to see the view from the cockpit?

I'm hoping to have a few days to walk around in McMurdo and take pictures. McMurdo and the South Pole are very different, but both are dead ringers for sets for the game Traveller, a game I played when I was a rather younger man, a sci-fi role-playing game where you travel from one world to another and have adventures in dusty, desolate and desperate places.

             McMurdo vs Pole
             -------    ----

    Dirt, rock & Ice    Ice

           Sea Level    10,000 ft

Skuas, orcas, seals,    People
 penguins and people

Pop.: a few thousand    Pop.: a few hundred (summer)

       24/7 internet    ~8hrs/day internet

  Get money from ATM    Can cash checks

"Chapel of the snows"   Meditate on makeshift zafu

    Shared bunkrooms    Private, closet sized room in
                        wood+canvas "Jamesway" building

      Summer weather    Summer weather like a
         like a nice    really, really bad winter
       winter day in    day in Minneapolis
             Chicago

     Short stroll to    Half-mile walk across skiway
  Crary Lab to check    to Dark Sector to work on
              e-mail    experiment

 Jog up "Observation    Run on treadmill, lift weights,
  Hill" to catch the    bake in sauna.
                view

   4 cafeteria-style    4 home-cooked, delicious
  meals/day in huge,    meals/day in cozy
multi-room meal hall    galley

   2 minute showers,    2 minute showers,
         twice/week.    twice/week.  :-(

Stymied, Just Out of the Gate

19:14h

Thanks to a broken de-icer (actually a faulty computer circuit board) we are still at our gate in Chicago, so I may get stuck overnight in LA.

Why am I doing this?

When I describe trips to the South Pole, people tend to split into two camps: those who think it's very cool and would love to go, and those who think it's utterly crazy and wouldn't dream of doing such a thing.

There are some good reasons (and maybe a few not-so-good reasons) for doing a high energy neutrino astrophysics neutrino experiment at the South Pole. Rather than go into the why's and how's of the project, I will refer you for the moment to the IceCube Web Site.

But as to me personally, there are several reasons why I wanted to make this fourth trip. Professionally, I have worked on IceCube and it's predecessor AMANDA for over 13 years. This is the first season when IceCube will put sensors ("DOMs") in the ice. It is a crucial season for this reason and I want to do whatever I can to help ensure the successful deployment and operation of the instrument. My work on IceCube in the last two years has been to write the software for communicating with the DOMs in the ice, and I hope to help debug any problems that may arise in the communications. I also wrote some of the software that runs inside the DOMs themselves, so it will be good to be "on hand" when we shake out the DOMs deployed in their proper environment (a few kilometers below the surface of the ice). Also to help with the actual deployments if need be, helping with other parts of the software, or troublshooting anything that comes up.

Personally, I get to indulge my thirst for barren, strange places (a taste acquired, no doubt in part, on my previous trips to the Pole…. I love the idea of doing systems software for other projects in similarly remote places – the ocean floor, the desert, down in a mine, or in deep space). And after painting seriously for several years in the Bay Area I have been pretty caught up in photography in the last year or two, and I hope to take advantage of the trip to capture some of the otherworldly imagery which abounds in Antarctica. On previous trips I fantasized about taking pictures with a medium or large format camera to show the kind of vivid texture and detail that make the otherwise very minimal ice-scapes so compelling when seen in person. So I have brought my Dad's old Horseman 6x7 camera and a '50s era Russian folding camera along to try to capture some of this detail, along with a Canon 20D digital camera for the flexibility to shoot absolutely any number of pictures of absolutely anything that catches my eye. I have no idea if any of this will be at all successful, but it's worth a try.

Finally, it's another chance to re-connect with colleagues (since I telecommute from Chicago most of the time I feel like "the ghost in the machine" too often) and hopefully get back into shape after the long holiday season (in past trips South, exercise has kept me sane and happy).

Best wishes to all, Happy Birthday to Pete, and thanks again to Gregory for the GPS receiver.

Camping Out in LA

19:19h

Ugh. Finally airborne after about 7 hours on the plane parked at the gate. They finally switched us to another plane. Fortunately I get to rest in a hotel at LAX before doing it all again tomorrow night. Blech. Next time I'm taking a cruise ship to the ice.

Anyways, I hear folks are very backed up in Christchurch as it is because of bad weather on the ice, so maybe it's just as well if I get delayed a day or so going into New Zealand.

Getting Ready (t minus 2 days)

Fri, 07 Jan 2005 00:00:00 +0000

Last latched GPS: N41d59.099' W87d39.934'

T minus 2 days. Biggest snow storm in 2 years getting me warmed up for the Pole. I've gone out each morning for the past few days on "trial runs" with Dad's old Horseman camera and tripod. Yesterday, by the time I got to Lake Michigan, the shutter cock had frozen (not a good sign) and I only got half a roll. But the film grain looks very fine (APX-100) and today the shutter worked ok.

On the beach everything was pure white except the lake itself (still a stormy dark grey), two men out exercising their dogs, and a snowbreak fence winding along the beach. I got some good photos of that and finished up my roll. After that I decided to get up close to the water and noticed many shapes bobbing up and down in the water. At first I thought it was some sort of clot of weeds, but then my head flashed grimly to the image of tsunami victims washing up.

What it actually turned out to be was quite beautiful and mysterious – thousands of basketball-sized, weed-covered snowballs (iceballs?) washed up on the water's edge. One of the dogs attacked them one after another and pulverised them for fun. It was an amazing thing to see and got me in the mood for the trip. After a few weeks at the Pole, even a few weeds on snowballs are going to seem pretty darn organic.

Meanwhile, so much to do…

Mon, 01 Jan 0001 00:00:00 +0000

Artworks

A sample of recent work created since 2018. See also: Instagram feed.

Noli Appropinquare, 2020-2025
Oil on linen
40"x28"

Evening Coastal Landscape, 2022
Oil on wood
4"x5"
SOLD

Red Light Skull, 2022
Oil on copper
4"x6"
SOLD

War Permutations, 2022
Relief print on paper
7.5"x9"

Skull, 2022
Woodcut on paper
11"x16"

Steelworkers' Park Blast Furnace Bell, 2021
Oil on Panel
5"x7"

Skull Study, 2021
Oil on Panel
5"x7"
SOLD

Flower Study - Peonies, 2020
Oil on Paper
10"x12"
SOLD

Tempera Skull Study, 2021
Egg tempera on paper
Approx. 6"x10"

Small Skull Study, 2020
Oil on wood
Approx. 4"x5"
SOLD

Crows and Civilization, 2019
Oil on linen
40"x28"

Portrait of John, 2018
11.5"x7.5"

NUPOC Training Prosthesis, 2018
Oil on Canvas

Figure study, 2018
Silverpoint on paper

Mon, 01 Jan 0001 00:00:00 +0000

Crows and Civilization
View image only

Oil on Canvas

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Graphite drawing in progress
View image only

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Portrait of John
View image only

Oil on Panel

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Noli Appropinquare
View image only

Later state of painting.

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Noli Appropinquare
View image only

Early state of oil painting.

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Noli Appropinquare
View image only

Oil on canvas, recent detail

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

NUPOC Training Prosthesis
View image only

Oil on Canvas

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Skull Study, 2021
View image only

Oil on Panel

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

War Permutations
View image only

Relief print, series of 362880 unique arrangements.

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Paintings

"Crows and Civilization", 2019

"Noli Appropinquare", 2020-2022

Mon, 01 Jan 0001 00:00:00 +0000

A., Seated
View image only

Graphite

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Composition Study
View image only

Gouache on paper

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Composition in Black and White
View image only

Gouache on Paper

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Seated Figure
View image only

Silverpoint

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Foreshortened Figure
View image only

Oil on Panel

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Monochrome Figure Study
View image only

Gouache on paper

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Sketch of N.
View image only

Graphite

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Noli Appropinquare
View image only

Study for work in progress

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

Quick-Sketch Figures
View image only

Graphite

Next
Previous

Mon, 01 Jan 0001 00:00:00 +0000

2026 - What I'm Doing Now¹

As of the beginning of the year I have been continue to assist with the IceCube Upgrade (with a very exciting Pole season underway); continuing a regular drawing practice, learning to play Mom's old concert flute, and caregiving duties.

I have also been reengaging with Clojure programming, being reminded how fun Lisps and a good REPL can be. The burgeoning and amazing AI tooling landscape has not changed that – if anything, Clojure is even more fun than ever.

2025 Projects

I showed fourteen pieces in the August group show, "Sites of Arrival", at the Palette and Chisel Academy of Fine Arts in Chicago. Participated in a few other shows as well.
I continued work for IceCube: embedded systems programming, test visualization, and communications tests in preparation for the 2024-2025 Antarctic field season ("IceCube Upgrade");
I did regular "quick sketch" figure drawing at the Palette and Chisel;
I spent more time than usual on family care, reading, and private writing.

2024 Projects

IceCube work
Playing with compilers and toy languages; learning OCaml;
Drawing and painting practice
Taking care of family.
Making A Tiny E-Paper Status Display for the Raspberry Pi Zero
Wrangling Half a Thousand Dreams

2023 Projects

Worked on my painting and drawing practice, including a series of self portraits. Worked as an independent software developer for the IceCube Neutrino Observatory.

2022 - Sabbatical

Prior to September, 2022, I took a five month sabbatical from paid work. Here are some of the things I worked on:

Improved l1, my Lisp dialect written in Go, and wrote the beginnings of a roguelike-inspired mini-game in that language.
Made a tiny 1-d roguelike in Python.
Improved my Scheme implementation in Python.
Tinkered with Common Lisp projects, including this library of functions inspired by Clojure, a text-based histogramming library, and a bookkeeping program for War Permutations.
Did the first 50+ Common Lisp problems on Exercism to train myself better in this old, strange, fast language.
Paintings and relief prints. Example: War Permutations print. See also works in progress here, or my Instagram feed.
Practiced Python programming (again!). I did a lot of Python development up until about 2014, after which I switched to mostly Clojure for day jobs. But I still like Python and it's fun to get back to it for awhile.
Learned about Quantum Computing by reading books and playing with Qiskit.
Worked on the Web site you're reading.
Journaling and working on improving my handwriting.

2022 - Prior to Sabbatical

Implemented a Lisp in Go, with detours into Crafting Interpreters and Structure and Interpretation of Computer Programs.
Learned the Go programming language. I've written a few modestly useful things and am really enjoying the experience after years of focusing primarily on Lisps and functional programming. Go emphasizes a different kind of simplicity than Clojure does, and it seems to occupy a sweet spot on the speed vs. power spectrum. I'm regularly impressed by how fast Go programs are, including the program used to build this site.
Learned some Mandarin (reading, mostly). I got my vocabulary up to about four hundred words. It's a beautiful language, and a slog to learn, but thanks to technology it is definitely possible!
Until late April, 2022 I was a staff software engineer at OppFi, where I wrote software (mostly in Clojure), mentored and taught other engineers, and tried to improve process, code quality, and architecture wherever I could. I spent nearly four years at OppFi, learned a lot, helped a lot of people, and am happy to move onto other things.

Prior Decades

2010-2020

Lived in Chicago. Finished last few South Pole trips. Stopped contracting for IceCube and worked in industry. Invested more heavily in art practice.

2000-2009

Lived in San Francisco Bay Area, then left LBNL at moved to Chicago, working for IceCube. Ran a few marathons and half marathons, went to the Pole several more times.

1990-1999

Graduated, worked at CERN for a year, then spent a few years in art school, then grad school in art and physics at UW-Madison. Finished Ph.D. 1996, then moved to Bay Area. First two trips to the South Pole.

For more about Now Pages, see this link.

John Jacobsen

"Sites of Arrival" Group Show

Sites of Arrival

Location and Hours

Works in the Show

Artist's Statement

Biography

For news and updates

Lisp Testing Fun

Fun

To The Metal... Compiling Your Own Language(s)

The Problem In General

Small Is Beautiful

An Approach

Enter Babashka

Building A Compiling Calculator

Lisp

Alternatives and Future Directions

Conclusion

Wrangling Half a Thousand Dreams

(In-)consistency

Dream Content

Making the Ebook

Common Words

Parting Thoughts

Programming Languages and Art Media

eBook of Dreams

Making A Tiny E-Paper Status Display for the Raspberry Pi Zero

Supply List

Dependencies

Virtualenv-based setup

Running the Demo

Rolling Our Own

Prototyping a Better Display

Temperature

Memory

WiFi

Disk Space

CPU Load and Uptime

What else is there?

Building the Mockup

Deployment

Running "In Production"

Running It Periodically

Future Directions

In Praise of Small Shell Scripts

2024 Blogging Intentions

Emacs, Continued

Writing prose

Git operations

"Literate Devops"

Future Work

Summary

Practices for Software Projects

Table Of Contents

Checklist

Automated Tests

Test-Driven Development

Probing With Tests

Tradeoffs

Speed

Clean Tests

Test Frameworks

Test Coverage

Style Checks

Automated Builds

Repeatability

READMEs

Versions and Releases

Issue Tracking

Other Helpful Practices

Using a Common Build Tool

Getting “Invisible Feedback”

Auto-Generating Documentation

Summary

Acknowledgments

Adding Tail Call Optimization to A Lisp Written in Go

Motivation

The Optimization

The Approach

How Do I Use `Curses` (or equivalent)?