I thought I would do a brain-dump about something in the computing world that has interested me for a long time: the Lisp programming language, or family of languages. Lisp, in it’s various forms, is known as a language with a passionate but small following, which has never reached the level of industry adoption of C, Java, Python or many other languages on the TIOBE. But being one of the oldest high-level languages means that it has provided inspiration and ideas for many of these later languages; for example, it was the first language with garbage collection.
I came across Lisp early in my programming journey, when I first realized that computing technology was an area I wanted to explore, but without having much direction other then my own curiosity and what came up in web searches. In one of these searches I discovered the writing of a guy named Paul Graham, whose perspectives on software and startups placed what I was learning in a bigger picture. At the time, I wasn’t aware of the whole culture of Silicon Valley startups, having grown up on the opposite shore, nor that there was this language called Lisp, with its funny parentheses-heavy syntax. Graham wrote glowingly of it, and so following this lead I learned about Common Lisp and Scheme, macros and recursion. I read SICP and PAIP. And I can agree with what others have said about Lisp, that it is worth knowing for the perspective it gives you about how programs can be written, even if you don’t end up writing much Lisp code.
Lisp is unique among programming languages for a number of reasons. There is its unique history, as the brain-child of John McCarthy, the father of Artificial Intelligence and christener of that field. There is its singular syntax, composed of expressions delimited by parentheses, giving Lisp code a visual presentation unique among programming languages, instantly identifiable to anyone trying to pick it out of a source code lineup. There is the meta-programming capability that this simple syntax affords, allowing the programmer to easily write code that writes code. There is also its infamous inability to gain traction in the technology industry despite its perceived benefits. Here I’d like to spend a few words discussing some of the interesting things I’ve discovered while exploring this language.
Beating the Averages
Lisp is a bit like an underground artist, with a small but dedicated following. One of the most high-profile individuals among that following is the aforementioned software entrepreneur and author Paul Graham. I don’t mean to put the guy on a pedestal, I’m sure there is plenty to criticize about him, but here I am just going to focus on my reaction to his writing at the time I first encountered it.
Graham has written a number of essays (he does not stoop to being a mere blogger, rather he styles himself as an essayist who publishes on the web), of which one of the most well known is Beating the Averages, published April, 2001, in which he recounts the experience of building his successful early web startup, Viaweb, which allowed its users to create web-based stores, and was eventually acquired by Yahoo. For me, reading it at a time when this was all new territory, it told a compelling story about scrappy startups, about the unique challenges and opportunities of the early web, and about Lisp as a “secret weapon” that gave them an edge over the competition.
It is flattering to programmers to think that it is their choice of the superior programming language that makes all the difference. In today’s web programming landscape, I think it unlikely that you are going to reproduce Viaweb’s results by following them in using Lisp. But it is certainly possible that at the time that Beating the Averages was written, before the proliferation of front-end frameworks and libraries for anything you can imagine doing in a web application, picking Lisp as their implementation language really could have been a deciding factor in their success.
The title has the obvious meaning of beating the statistical averages, but also beating the average developers who were stuck hacking away at C and Perl (the web languages of the time) while Graham and his colleagues were blazing ahead in their Lisp-powered machine. There is a certain elitism associated with Lisp that is part of its mystique as well as part of the reason for its low adoption. Graham says as much when he discusses his idea of the Blub paradox, the idea that the average programmer is happy with their average programming language, while looking at more powerful languages (such as, obviously, Lisp) as too weird to bother with.
Lisp History
Lisp did not take over, in the way that some thought it would, but at one time it seemed like it was in position to be the lingua-franca of computing. There was a time when a creature called the Lisp Machine roamed the Earth. These were systems where everything, from operating system to network stack to file system, was written in Lisp, and even the processor architecture it ran on was optimized for running Lisp code. These were produced by some now defunct companies, like Symbolics and Lisp Machines, Inc., but also some familiar names like Xerox and Texas Instruments. Check out the unlambda website if you’re interested in learning more about this branch in the evolutionary tree of computing, or even getting one running as a virtual machine.
The Lisp Machine operated with a different paradigm than the one adopted by the current crop of operating systems; instead of a collection of individual binaries running in their own sequestered memory spaces, all applications in a Lisp Machine were loaded into the same memory space and formed the “world”, the entire state of the runtime system which was always available for your inspection and modification. Each function and variable defined in the world was available to the user, and adding new capabilities to a Lisp Machine was more like extending the whole system rather than creating a standalone application. Graham refers to this property in Beating the Averages when he talks about being able to fix a bug while still on the phone with the user who reported it.
Today we have a kind of Lisp Machine-lite in the form of Emacs. While associated with the world of Unix and Linux, it stands out from the Unix toolbox approach of composing programs that do one thing well by being much more than a tool, more than a text editor, being more like a platform for text-based applications, useful for everything from editing source code to playing text adventure games. This is because Emacs comes from the same culture that produced the Lisp Machines, that of universities like MIT and Stanford where people were working on Computer Science and Artificial Intelligence using Lisp. It is a sort of survival of the Lisp Machine way of doing things that has found its way into the modern world while all of its cousins have disappeared, a kind of software Coelcanth.
Parens, Parens Everywhere
People get hung up on the plethora of parentheses in Lisp code, but I think this is because of expectations set up by the languages they are used to working with that don’t apply to Lisp. In your average language, with its many keywords, operators and syntactic characters, you are constantly making micro-context switches, from infix to postfix to prefix, from definition to assignment to condition. With Lisp, you have a uniform syntax based on the application of the first item to the rest, all enclosed in parens. Sometimes that is a function application, sometimes a macro or special form, but the syntax for all of these cases is the same. Because parens in Lisp perform this simple duty of delimiting applications, you have to pay much less attention to them then you would in other languages, and can almost block them out mentally, relying on them only when the structure is not evident from indentation. If you are irritated by them, you can change the syntax highlighting in your code editor so that they are barely visible.
People sometimes say that everything in Lisp is a list, something that is not really true, but it is true that Lisp makes it easy to work with lists. This makes solving problems with recursion very natural, because of the fact that recursion requires a base case and a recursive case, and a list is naturally decomposed into “head” and “rest” parts, or “car” and “cdr” in Lisp parlance; the first item in the list which can be processed, and the rest of the list which can be recursed over. Then there is the fact that the syntax for representing lists is the same as that for representing code, which is the basis for Lisp macros, something Lisp aficionados, Paul Graham included, believe is one of its most powerful features. There is a certain elegance to how these various unique aspects of the language work together that is, I think, part of what makes it a favorite language for those who appreciate it.
The Lisp Book That Could Have Been
One indication of the popularity of a technology is the number of current books published on it. One of the big publishers of technology-related books for many years has been O’Reilly media. I’ve read many, many books by O’Reilly in my career; being self-taught, technical books that were not primarily targeted to students but to professional developers, were a primary source of educational material. While they do have a few titles covering Clojure, a modern re-imagining of Lisp which has managed to gain a level of traction, in all of their vast catalog, they have never published a book on the classic Lisps, on Scheme or Common Lisp.
I suppose this is not surprising given their focus on providing knowledge to the working technologist, who with very few exceptions would not have been working with Lisp. I discovered, however, that at one point, a Common Lisp book was planned. It was going to be titled “Lisp Outside the Box”, and was in the process of being written by Nick Levine. Unfortunately, he was not able to complete it for health reasons. The written chapters are available on the archived website, if you are interested.
O’Reilly’s books until recently followed a design standard that I think was very effective, as it provided strong brand recognition with an immediately recognizable look, but was also able to individualize each title by pairing it with a unique woodblock print of an animal, while also providing a higher level of topic grouping using a single distinct color for different categories of technology. For example, books on system administration featured navy blue, books on web technologies were teal, and so on.
One interesting note about this book that never made it to print: if you look at the cover on the site linked above, you’ll see the familiar O’Reilly cover design. Take a look at the covers in the O’Reilly animal menagerie. You don’t see that exact shade of purple on any of them. Was this just a mockup that didn’t reflect what the final printed version would look like? Or maybe, possibly they were going to start a new category for books on Lisp, signified by that color? A piece of Lisp lore, lost to history.