Wrangling Half a Thousand Dreams

As of this morning, there are 470 dated entries. Each entry, typically written first thing in the morning, captures at least one dream from the previous night (unless you wake up in between, it's hard to know when one dream ends, and another begins). Though the numbers average to about one entry per month during my adult life so far, my actual rate of dream journaling has been highly variable, and has tended to track my rate of daytime journaling. Here are the number of dreams per year, as of today:

1985:   68 ....................................................................
1986:   59 ...........................................................
1987:   10 ..........
1988:    7 .......
1989:    9 .........
1990:   37 .....................................
1991:    7 .......
1992:   23 .......................
1993:    4 ....
1994:    1 .
1996:    2 ..
1998:    2 ..
2000:   10 ..........
2001:    2 ..
2002:   34 ..................................
2003:    3 ...
2004:    3 ...
2005:    4 ....
2006:    1 .
2011:    4 ....
2017:    2 ..
2019:   68 ....................................................................
2021:    2 ..
2022:   22 ......................
2023:   29 .............................
2024:   57 .........................................................
TOTAL: 470

I will say more about how the statistics were generated in a moment. Each dot indicates a dated entry of one or more dreams from the previous night. Some years I averaged an entry or two a week; other times I went for years with no entries. That the data would be this variable piqued my curiosity.

Naïvely, I might expect to have written more during times of crisis or big change in my life, and that is approximately true. For example, there are peaks during my first year in college (1985-1986), the year I lived abroad in Switzerland (1990), and the year I had various personal crises and moved away from the Bay Area (2002). 2019 was a year of some tumult as well. But my interest in dreaming has waxed and waned over time, and interest, while hard to track, certainly drives the rate as well.

It is also well-known that remembering and writing down dreams one morning makes it easier to do so the next morning, so I'd expect the journals to clump somewhat in time. However, given the sheer number of entries, it surprises me a little to have gone literally years with no entries at all. There are very few entries in the interval from 2003 through 2018, and a few multi-year gaps. For any "missed" year, if you assume 3-6 dreams per night (a very rough, possibly bogus average I found online), over a thousand dreams completely slipped through the net. (By the same measure, my total average dream recall since reaching adulthood has been less than one percent.)

The handwritten journals have some nice features compared to digital files. A few of them have small drawings or thumbnail sketches of a particular moment or image from a dream. They also show the variability in my handwriting over the years (which has never been great), and their proximity to "normal" diary entries help show what else was going on at the time. (I typically would write the diary entries forward from the front page of the journal, and dream journal entries back-to-front from the last page, starting a new journal book when the two met in the middle.)

Obviously, I'll continue to hold onto the physical journals. But the combined digital journal has two major advantages: it is much easier to read, and all the dreams are in one place. These two together make reading the journal as a whole much easier. This, in turn, made repeated images and common themes more apparent.

It is a hallmark of dreams that they seem to defy characterization, lurching from image to image in surprising ways, but I find some dream topics cropping up consistently over the years, whereas others seem to have come and gone. Common topics over the years have included:

• Moving through big, open landscapes, especially at night;
• Making art;
• Fighting, with people or animals;
• Family and friends;
• Teachers, mostly physics and art teachers I’ve studied with;
• Physics, and technology in general;
• Nuclear war/power/devices;
• Storms;
• Sex;
• Trains;
• Travel, especially Paris, Greece, New Zealand, and the South Pole.

Some of these (sex, family) are probably common for most people; others (nuclear devices, trains, Antarctica) maybe a little less so.

I also tried to spot changes over time. Common topics from earlier years, that are now less common, included: flying or falling, flooding or shallow water, war / soldiers / cops / authority figures, and being in school. More common topics these days include being in art workshops; traveling through disused, dark interior spaces; and programming or other computer work.

I have always resisted thinking about dreams as something demanding interpretation, but the fact is they do come from somewhere, and are probably informed by the events in our lives. Reading through all the repeated variations, I get the sense of my mind working through things, though in exactly what way is obviously a mystery.

My point in thinking about evolution of dream content is not so much to drive specific insights about my life, but rather to take a step back and experience a broader sense of dreaming as a part of life… to feel what it is like to be "one who dreams."

I won't weigh this post down with too much technical detail, but present a few details here in case anyone finds it helpful; others can obviously skip this section.

When I started organizing things this year, the largest dream journal I had was a single Emacs Org Mode file dreams.org. I used this as the nucleus of the project, copying or typing all the content from the other sources, sorted by increasing date. Here are the first few lines:

#+TITLE: Dreams
#+DATE: <2017-08-29 Tue>
#+OPTIONS: toc:nil num:nil

* 1985

** <1985-02-04 Mon>

I'm at Fred's and the cab is 45 minutes late.

Leaping over an empty oil drum on a highway.

** <1985-02-05 Tue>

[...]

The plain text format is readable on its own, and editable in any editor, though obviously Emacs has some advantages for this format. Without any extra work, having the content in Org Mode immediately allowed me to view the entire document as plain text, as HTML in the browser, or as a PDF, thanks to Org Mode's export functionality.

During the long process of typing older dreams into this document (doing only a few per day to keep the project from taking over my life), I started looking for ways to turn the dream document into something I could view on my phone, Kindle, iPad, Kobo reader, etc. I found I preferred reading old dream content on an e-ink display such as the Kindle or Kobo rather than on a backlit screen, so an e-book format such as EPUB seemed a natural next step.

I described my initial approach in an earlier blog post. The basic idea was to tweak the Org Mode formatting slightly to make it equivalent to Markdown and then use Calibre's ebook-convert program to turn that output into an EPUB file. However, that program ran more slowly than I liked, and I found myself wishing I could customize the output (especially the table of contents). I also was curious what exactly went into all those e-books I've been reading over the years – what, exactly, did they look like, under the hood?

So, I decided to skip the middleman and write my own Org Mode to EPUB converter. The result of that work is up on GitHub. While some of the details are out of scope for this article, I will say two things about the approach I took. First, I was very happy with my choice of Babashka for the implementation. Babashka is fast and expressive, and I did not need any external libraries, relying only on the very standard zip program as the only external dependency. My gratitude to Michiel Borkent for Babashka continues undiminished.

(Ironically, while looking up Babashka tricks in Daniel Higgenbotham's excellent Babashka Babooka, I discovered he also uses a dream journal as his example Babashka project.)

The other approach worth mentioning is the strategy I used to produce a working EPUB. This turned out to be more difficult than I thought, as EPUB is a rich and complex format with a number of historical variants. I spent a good amount of time with Google, ChatGPT, and Wikipedia to try to get a simple example to load in both Calibre and Apple Books; when this did not succeed, I switched gears and picked a random e-book from Standard Ebooks out of my collection, gradually stripping out all the actual text content and extraneous data out (zipping, importing and visually checking the files after each iteration) until I had the barest minimum set of files, with two "Lorem ipsum…" style chapters and a table of contents, which worked in both Calibre and Books.

Then I automated the production of that minimum example using Babashka, and posted the results. This provided something I could build on to make my final dream e-book implementation, which has the following features:

1. Cover art;
2. Introduction and Collophon sections;
3. Organization by month and year, with table of contents entries for each;

4. Correct conversion of the following formatting elements from Org Mode into the EPUB HTML:

   1. Italic, boldface, strikethrough, underline, code
   2. Sections and subsections (used to organize the dates as well as subsections for each day, as needed);
   3. Correctly-formatted beginning and ending quotes and apostrophes;
   4. French guillemets (a few of the dreams are written down in French or have French dialogue).

The script uses an intermediate Markdown representation for the content before converting to HTML, since the transformations are simple and I have the idea I might eventually support Markdown in addition to, or instead of, Org Mode. The formatting requirements are minimal enough to be easily supported by a sequence of regular expressions to convert from one format to another.

The script's output is the EPUB file itself, along with the statistics display I posted above.

A special mode of the program (invoked with the -w command-line option) prints out the 300 most common words in the dreams:

above across actually again against almost along am a... another
apartment are area around arrive art ask asks away beautiful bed been
before behind being below between big bike bit black blue boat body
both building buildings bunch can't car cars city class close clothes
comes coming corner country couple dad dark d... different does
doesn't dog doing don't door down drawing dream drive driving each
e... either else end everyone family feel feeling few fight fighting
figure finally find fish floor flying food f... french friend friends
front game gets getting girl glass goes going gone gray great green
ground group guy guys had hair hand hanging hard has having he's head
hear help here high hill home house huge i'll i'm i've inside is it's
j... j... keep kind lake large last later least leave left light
little long looking looks lot lots love made madison makes making man
many maybe meet men mom more mostly move moving much myself name near
need next night off office old older open others outside painting
paper paris part party past path perhaps person p... p... phone
physics piece place plane playing point possibly probably pull put
ready real realize really red remember restaurant ride right road room
rooms run running same says school seems series several she's short
should show showing side similar sit sitting slightly small somehow
someone something somewhat somewhere sort space spot stairs standing
start starts still stop strange street stuff suddenly sure swimming
table taken takes taking talk talking tell there's thing things
thinking though three through together too top towards town train
trees truck try trying turn under very wait wake walk walking wall
walls wants was water we're where while white whole why window

(In this list, the most common 100 English words have been subtracted out to help distinguish the content from any other text.)

The words that stand out for me, aside from first names (which I elide here using ...), are nouns which indicate settings, actions or activities, and commonly occurring objects from the dreams. I suppose this list, or something like it, could be considered a sort of "dreamer's fingerprint." It would be interesting to compare with someone else's output (another English speaker, perhaps). If you're a programmer and have a dream journal of your own, here is a link to the relevant code that you could adapt to generate the list. (Or, of course, you could convert your journal to Org Mode and use the software I wrote "out of the box.")

Digging into this material, both the combined journal itself, and automating the text conversion, has been engaging enough to consume most of my limited free time over the past month or so. I was surprised at how fun it was to read through the collected dreams, though I suspect the material would be much less entertaining for anyone who didn't know me well. The writing itself tends to be more compact and faster moving than my other writing. In this regard it sets an example I want to emulate in the rest of my writing.

The work also provoked contemplations about memory, a topic I tend to think more about as I get older. One mysterious aspect of dreams is that, for the most part, we forget them in the morning. They disappear completely unless we make a record of them, giving them that kick of energy they need to turn from virtual particles into real ones. In re-reading the dream entries, I was struck by the fact that, not only did I have no memory of most of the dreams I'd collected, but for the most part I had no memory at all of writing them down, either. It was strange to confront a whole tranche of my life of which I had basically no memory, having only the "evidence at hand" to work with. In this regard, I felt at times a bit like a scientist studying someone else's dreams.

The collection of dreams is packed with myriad images, whether strange, disturbing, or funny, or all three… each of them somehow connected with my life. As an artist who sometimes struggles to figure out what to draw or paint, having a trove of material to work with feels exciting and a little daunting. Regardless of what any of it means, the material is more "mine" than anything inspired by other artists, or films, books, etc. It's up to me to see what, if anything, I can actually do with it.