Sign in

From Software:

Python Concurrency Workshop

Monday, May 18 2009 UTC

Thursday and Friday, fellow IceCube-r Dave Glowacki and I took David Beazley’s workshop on concurrency in Python here in Chicago.

Dr. Beazley is a former math/physics geek and University of Chicago professor, musician, and consultant, currently teaching on Python full time. He is the author of the Python Essential Reference (soon to be out in a new edition).

I felt quite positive on the whole about the class. Most of the software I’ve worked on recently for IceCube has been parallelized in various ways (relying heavily on RPC and/or threads), and I’ve also wrestled with some frightening concurrency issues while writing device drivers, but it was pretty mind-blowing to see how many multiprocessing tools (and hazards!) there are in Python:

  • Threads. We covered locks, reentrant locks, R/W locks, context managers, decorators, metaclasses, semaphores, queues, and condition variables. Also race conditions, deadlocks etc…. unfortunately Python performs poorly when it comes to CPU-bound threads on multiple cores, due to somewhat broken contention semantics relating to the Global Interpreter lock. We covered this in some detail, even studying the interpreter source code itself.
  • Traditional Unix tools for IPC: pipes, named pipes/FIFOs, sockets, memory mapping.
  • Object serialization strategies and getting data into/out of low-level data structures.
  • The ‘multiprocessing’ library, new to Python 2.6. This is a very cool and flexible module which provides nearly the same semantics as the threading library, but which has add-ons for using and managing pools of worker processes on the same physical machine… an approach which becomes more and more relevant as
    the limitations of physics push CPU design into more and more cores.
  • Standard concurrency algorithms and patterns: pipelines, worker pools, map-reduce, ...
  • ‘Asynchronous’ I/O—both the true asynchronous I/O which is based on signals, and the more common sort which is based on poll() or select(). Using ‘select’ to create event-driven programs.
  • Generators and coroutines.

Though I was pretty fried by the end due to the massive info-dump and the simulated jet lag (the class schedule was early for me), I found the very last part to be the most interesting. I’d used generators before, but I had only seen coroutines in extremely gnarly C-language contexts and in Lisp.

Coroutines were introduced into Python in version 2.5. With coroutines and generators, you can make really wild, elegant and powerful patterns in modular programs which pass data between sub-sections, simulating a sort of concurrency without any of the usual race condition hazards one has when using threads. David even showed an elegant, mind-blowing task scheduler implemented with generators in pure Python. It’s not completely obvious to me why you’d want to do this, but it’s really cool that you can!

David’s presentation skills were enviably clear, efficient and easygoing. One of the best things about the class was that we would talk for awhile, and then Dave would toss us a problem, and we’d hack for awhile on that. Then we’d go back to talking, followed by more hacking, and so on. This really allowed the concepts to cement much more quickly than they would have if Dave just talked the whole time.

I have always loved seeing languages (human and computer) pushed to the limit, showing their expressive power in new ways. For example, one of the things I’ve always loved about Lisp was the implicit emphasis on metaprogramming (for example, Lisp macros allow one to create arbitrary new language constructs, not just objects or functions). Before this class, and our study of metaclasses and decorators, I didn’t realize Python had as many metaprogramming capabilities as it does (and I hope it gets more). Metaclasses in particular seemingly don’t (and probably shouldn’t) get used for many ordinary programming tasks, but it’s good to know they are there, because occasionally one does find situations where a whole mess of repeated code can be eliminated with the judicious use of metaprogramming.

It was a real pleasure to discover new capabilities of what has become my language-of-choice for most personal and work-related tasks. Though I had my doubts about the cost of the course (I do wonder if the course cost could be trimmed slightly with a cheaper venue and communal lunch in a nearby restaurant), the experience on the whole was energizing and informative… so much so that I spent the weekend hacking Python code for my various Web sites.

I look forward to more locally-grown and -taught advanced Python classes from David Beazley.