SICP: The only computer science book worth reading twice? (2010)

It’s interesting, SICP and other many other “classic” texts talk about designing programs, but these days I think the much more important skill is designing systems.

I don’t know if distributed systems is consider part of “Computer Science” but it is a much more common problem that I see needs to be solved.

I try to write systems in the simplest way possible and then use observability tools to figure out where the design is deficient and then maybe I will pull out a data structure or some other “computer sciency” thing to solve that problem. It turns out that big O notation and runtime complexity doesn’t matter the majority of the time and you can solve most problems with arrays and fast CPUs. And even when you have runtime problems you should profile the program to find the hot spots.

What computer science doesn’t teach you is how memory caching works in CPUs. Your fancy graph algorithm may have good runtime complexity but it completely hoses the CPU cache and you may have been able to go faster with an array with good cache usage.

The much more common problems I have is how to deal with fault tolerance, correctness in distributed locks and queues, and system scalability.

Maybe I am just biased because I have a computer/electrical engineering background.

You're in luck. Part 5 of the book is about building a virtual machine to run lisp simalated at the register lelvel: https://mitp-content-server.mit.edu/books/content/sectbyfn/b...

Writing a network between n such machines is left as an exercise to the reader.

If you knew how to design programs you could run it all on a single box and wouldn’t have to design “systems.”

I’m being slightly facetious, but only slightly. If you really think everything is solvable with arrays, you are not going to scale well and of course you’re going to need to throw a lot more hardware at the problem.

I find books like SICP interesting and not very useful. I love reading them because I like this stuff, but I don’t get to apply their teachings in real world software. It’s a problem because naturally I want to spend my time reading these kind of books, but if I do that I would be jobless. I need to divide my time between reading pearls like SICP and boring Kafka/Postgres/Golang/K8s/AWS documentation.

Have you seen

"Software Design for Flexibility: How to Avoid Programming Yourself into a Corner" by Chris Hanson and Gerald Jay Sussman

It's from 2021.

I hadn't, that looks excellent.

True. However I find that most junior and even experienced programmers struggle with tactical level coding. I'm really suffering with this right now because the small component I'm tasked with making a small change to is annoyingly stateful and deals with 2 abstractions at once. (It processes files and uses the file system and database to store it's state). I'm shocked how badly it has been thought out. I've spent days trying to avoid doing what has gone before, bits bolted on that make it even more difficult to understand. It really seems that pull request culture has just led to any old crap being approved because no one has the band width to think deeply about the actual code. Bring back in person code reviews !

I don’t find them useful in the sense of directly applying practical techniques in my day job, but I consider them somewhat necessary background reading to get into the right state of mind. You can very quickly tell when someone never acquired any academic knowledge in this area (or never played with functional languages or similar pastimes) - you can’t explain to those people why modifying global variables all over the place in a large program is a bad idea and other things like that. They just nod along skeptically and then somehow keep stumbling into the same kind of mess over and over.

Well CS and software dev in trenches moved a bit.

There are still jobs where people write frameworks, database engines or version control tools. Those jobs require heavy CS and algorithms, data structures day to day. But there are less of those jobs nowadays as no one is implementing db engine for their app they just use Postgres.

Other jobs that is vast majority is dealing with implementing business logic. Using database with understanding how it works in details is of course going to produce better outcomes. Yet one still can produce great amount of working software without knowing how indexes are stored on disk.

Also a lot of CS graduates fell into a trap where they think their job is to write a framework - where in reality they should just use frameworks and implement business logic- while using CS background to fully understand frameworks already existing.

> while using CS background to fully understand frameworks already existing.

Most frameworks today are so complicated that you typically cannot understand them fully, and even understanding them somewhat partially is more than a full-time job.

> these days I think the much more important skill is designing systems

That's true, but that doesn't mean that there is no value in having an understanding of how established technology works under the hood.

> What computer science doesn’t teach you is how memory caching works in CPUs.

That is also a very good point. There is a lot of daylight between the lambda calculus and real systems.

> Yet one still can produce great amount of working software without knowing how indexes are stored on disk.

I agree... up to a point. Most software will likely be replaced/obsolete before it even reaches a scale where indexes even matter (at all) given how fast the underlying hardware is at this point.

... but I don't think this is particularly relevant wrt. the "to CS or not CS" question. If a CS grad has been paying any attention they usually have a decent idea of what kinds of problems are intractable vs. problems that are tractable (but maybe expensive to compute) vs. easy. Also just general exposure to different ways to approach solving a problem (logic programming, pure functional, etc.) can be very valuable. There's just much that one couldn't come up with on their own if one weren't exposed to the ideas from the vast expanse of ideas that are known in CS. (And even a master's doesn't come close to scratching the surface of it all.)

The right book for the right problem. SICP isn't meant to teach you how to tackle fault-tolerance in a complex distributed system. Here is a textbook that talks about distributed systems (van Steen and Tannenbaum):

https://www.amazon.ca/Distributed-Systems-Maarten-van-Steen/...

There are still innumerable people writing standalone programs, single-purpose embedded systems, independent components and libraries, etc

The industry has expanded to include a lot of large-scale distributed cloud projects (often where we might have expected mainframes and cobol before), with many of today's largest employers doing most of their work there, but none of that other stuff really went away. It's still being done every day.

You need a book for what you're doing, and not every book is going to be that. Apparently, SICP is not it. I possess and have read many books, and only some small number of them are applicable to the projects I'm working on at any time.

They don't compete with each other, they complement each other.

I think the classic CLR text is great but, yeah, caches through quite a monkey wrench into naive big O analysis.

Still, I think the time investment to learn algos and data structures isn't too much of a burden.

That's true, though new technologies -- web browsers, mobile devices -- have necessitated some framework writing.

So yes, some people working at Microsoft, Apple and Google wrote those frameworks. But that's like a drop in the bucket.

+Meta ;)

IMO it's not excellent. It's not like SICP, it's obtuse for no reason, I find it a hard slog. Flexibility is good but it seems to try to make every bit of your program flexibile and pluggable and you just need to do something eventually.

My opinion, I'd welcome others on the book; there was a small splash when it came out but not much discussion since.

The first reason why I really loved SICP is that it is based on Scheme, a language with powerful primitives. I came from a self-taught world of PL/1, Algol, C, then later C++, Java etc. None of them had closures, hygienic macros, anonymous functions, functional programming, call/cc, and of course, "amb", the non-deterministic choice operator. At an even more basic level, SICP taught me that a lot of non-trivial code can be written with just sequences and maps, with good enough efficiency!

Because SICP's starting point was so high, they could describe many concepts easily from the ground up, from object oriented programming, backtracking, constraint programming and non-determinism.

This taught me a number of techniques to apply in real-life, because I could readily identify the missing building blocks in the language or system I was given to work with. For example, I was able to build a lightweight threads system in Java quite readily because I knew that the missing piece was a continuations feature in Java.

See https://github.com/kilim/kilim

I haven't read the book, but my experience is that the way to make things flexible is to make them simple as possible.

When I've used (or built) something that was built in the style like you're talking about, it's almost always wrong, and the extra complexity and stuff now makes it harder to do right. It's not surprising: unknown future requirements are unknown. Over building is trying to predict the future.

It's like someone building a shed and pouring a foundation that can work for a skyscraper. Except it turns out what we needed was a house that has a different footprint. Or maybe the skyscraper is twice the height and has a stop for the newly-built underneath. Now we have to break apart the foundation before we can even begin work on new stuff; it would have been less work if the original just used a foundation for a shed.

Is there another book you'd recommend - more recent than SICP - for how to avoid programming yourself into a corner?

Last week, after reading Methodology is bullshit here, this was my first thought!

https://x.com/henri__OK/status/1854813243916882365

Scouring SICP cannot imbue the student with mechanical sympathy any more than poring over analysis of Coltrane makes me a saxophonist.

Nevertheless. It must be done. Theory and practice.

I don't know about books, but I think the best approach is functional programming in a dynamic language. That could be because I'm currently an Elixir fanboy, but I think Lisps, especially Scheme or Clojure, or a functional-restricted approach in JavaScript could do it as well. I agree with parent comment that it's better to keep things as simple as possible and make the changes when necessary vs. building in all the flexibility in the beginning.

You can also get a free PDF version of that textbook here https://www.distributed-systems.net/index.php/books/ds4/ (you only need to provide an email)

I wish someone told me this back when I was trying to get a programming job as a self taught programmer. I would do things like try to build a simple React clone thinking it would help me overcome imposter syndrome to fully understand things from the base up, but it was pretty futile because no one really has time to wrap their head around something that big unless they are paid full time to do it.

Nicely said. The way I think about it: if we can’t write legible and adaptable functions, then we have no chance at making viable systems. All the same engineering skills are at play, just on a different scale.

> but these days I think the much more important skill is designing systems.

It is hard to design systems if you don't have the perspective of implementing them. Yes, you move up the value chain to designing things, no, but no, you don't get to skip gaining experience lower down the value chain.

> What computer science doesn’t teach you is how memory caching works in CPUs.

That was literally my first quarter in my CS undergrad 30 years ago, the old Hennessy and Patterson book, which I believe is still used today. Are things so different now?

> The much more common problems I have is how to deal with fault tolerance, correctness in distributed locks and queues, and system scalability.

All of that was covered in my CS undergrad, I wasn't even in a fancy computer engineering/EE background.

You kind of defeat your own argument. You say it's important to learn "academic knowledge", but then acknowledge the organization will not value your knowledge.

I do agree with you though.

Unfortunately, while I really want to love Software Design for Flexibility, it's clear that Hanson and Sussman haven't really solved (or even come close to solving) the problem they have identified in the book.

The introduction to that book is brilliant at identifying just how much room software has to grow (you can find similar talks from various Strange Loop sessions Sussman has done), and is really quite inspirational for anyone seriously thinking about the future of computing.

But the rest of the book fails to provide a coherent answer to the questions that are brought up in the intro. It shows off some neat functional programming tricks, but repeatedly fails to deliver on solving the (admittedly ambitious) challenges it provides for itself.

I'm still glad I have a copy, and have re-read the first half multiple times now, but sadly it's not the book it wants to be. To be fair though, that is because we haven't come close to understanding computation enough to solve those problems.

It's a very ambitious book that falls short of it's own ambitious.

One of the problems I've seen is that when new learners and self-taught individuals ask for advice, a lot of software engineers give recommendations based on what they wish their job was or how they would like to imagine themselves.

Well in my experience good organizations do recognize that the better design means lower costs in the long run, and people who don’t get that tend to not get promoted. Communicating this effectively up and down the chain is a whole different art in itself though.

>What computer science doesn’t teach you is how memory caching works in CPUs. Your fancy graph algorithm may have good runtime complexity but it completely hoses the CPU cache and you may have been able to go faster with an array with good cache usage.

Traditionally, the field of databases is largely about solving algorithm problems in the scenario where you have much more data that can fit in memory. Data exists on disk as "pages", you have a fixed number of "page slots" in RAM. Moving pages from disk to RAM or RAM to disk is slow, so you want to do as little of that as you can. This makes trivial problems interesting -- e.g. there's no notion of a 'join' in classic computer science because it's too trivial to bother naming.

We're used to thinking of the study of algorithms as a sort of pure essence, but one could argue that algorithmic efficiency is only meaningful in a particular data and hardware context. That's part of what keeps our jobs interesting, I guess -- otherwise algorithm expertise wouldn't be as useful, since you could just apply libraries/cookbook solutions everywhere.

I think CS 30 years ago was closer to computer engineering today.

At my uni 10 years ago the CS program didn’t touch anything related to hardware, hell the CS program didn’t even need to take multivariable calculus. In my computer engineering program we covered solid state physics, electromagnetism, digital electronics design, digital signals processing, CPU architecture, compiler design, OS design, algorithms, software engineering, distributed systems design.

The computer engineering program took you from solid state physics and transistor design to PAXOS.

The CS program was much more focused on logic proofs and more formalism and they never touched anything hardware adjacent.

I realize this is different between programs, but from what I read and hear many CS programs these days start at Java and never go down abstraction levels.

I do agree with you that learning the fundamentals is important, but I would argue that a SICP type course is not fundamental — physics is fundamental. And once you learn how we use physics to build CPUs you learn that fancy algorithms and complex solutions are not necessary most of the time given how fast computers are today. If you can get your CPU pipelined properly with high cache hits, branch prediction hits, prefetch hits, and SIMD you can easily brute force many problems.

And for those 10% of problems which cannot be brute forced, 90% of those problems can be solved with profiling and memoization, and for the 10% of those problems you cannot solve with memoization you can solve 90% of them with b-trees.

A top tier CS program is going to make you learn computer architecture along side automata and proofs. MIT went the extra mile with the SICP, it was honestly a hole I didn’t have access to in my top tier program, but I only realized this because I studied PL in grad school. You should go through it if you haven’t, I think it would have made my ugrad experience better and I definitely benefited from a great well rounded curriculum already (UW CSE is still no slouch, but it isn’t MIT!).

If you are into physics and mechanics, then you have to check the SICM (SICP’s less famous cousin) out as well. Again, MIT went the extra mile with that as well.

My argument is that 90% of problems can be solved with arrays, 5% of problems can be solved with memoization, 3% of problems can be solved with b-trees, and 2% of problems with other data structures.

It is good to know that solutions to the 2% exists, but what we should be focusing on is writing the simplest code possible which solves the problem and then only optimize afterwards using a profiler. God forbid you have to work on some codebase written by someone who believes they are the second coming of haskell with crazy recursion and backtracing, monads, red black trees, and a DSL on top of the whole thing.

You are right that many problems can be solved with a single box, but my argument is that you do not need fancy algorithms to solve problems on a single box. We should strive to use single boxes whenever possible to reduce complexity.

Computation is designed by humans to serve humans, we should make it as easy as possible for humans to understand. I’m probably going to start a flamewar here, but this is why simple solutions like UNIX and golang have prevailed in the past. Simple code is easy to understand and therefore it is easy to modify and reason about. Some people think simple means that you decompose programs into the smallest possible functional parts, but simple to me is a 500 line main function.

Yes, I have the distributed system book from van Steen :)

I have a saying that program's complexity is always exactly equal to the human-intelligible complexity + 1.

If not, the developer would add one more feature. It is due to the entirely human-made aspect of this discipline.

i did build a react clone in a long weekend. but I built the first 90% that takes 10% of the time and not the last 10% that takes 90% of time.

> What computer science doesn’t teach you is how memory caching works in CPUs.

Yes it can, and there are tons of papers about data structures to use in various scenarios to handle not just L1, L2, L3, but also NUMA. Sure, this isn’t in SICP, but claiming CS as a field completely ignores how memory works is incorrect.

Since you emphasize designing systems over just programs, do you have any go-to resources or references ?

> big O notation and runtime complexity doesn’t matter the majority of the time and you can solve most problems with arrays

I have the exact opposite experience.

Software comes out best if you always ensure to use an approach with sensible runtime complexity, and only make trade-offs towards cache-friendly-worse-O implementations where you benchmarked thoroughly.

Most cases where I encounter mega slow programs are because somebody put in something quadratic instead of using a simple, standard O(n logn) solution.

Check out https://www.tumblr.com/accidentallyquadratic for many examples.

You don't have to understand them fully to make use of them. One has to understand design patterns and underlying reasoning, understand context of the framework like for example if it is JS framework that it runs in browser to understand which parts are "because we are running in the browser" vs "that is just why framework implemented it" and if it is Typescript then how it blends into that mix.

Then for any details or unexpected behavior knowing where to look in documentation.

Computers today are slower than they have ever been. And tomorrow's computers are going to be even slower.

In many applications, the amount of data grows at least as quickly as computer performance. If the time complexity of an algorithm is superlinear, today's computer needs more time to run it with today's data than yesterday's computer did with yesterday's data. Algorithms that used to be practical get more and more expensive to run, until they eventually become impractical.

The more data you have, the more you have to think about algorithms. Brute-forcing can be expensive in terms of compute costs and wall-clock time, while low-level optimizations can take a lot of developer time.

We (at a public research university in the US) designed a rudimentary CPU, wrote mips assembly, and understood computer architecture for our CS degree. I graduated 6 years ago

Edit: we also did formal methods and proofs as part of core curriculum

> I don’t know if distributed systems is consider part of “Computer Science"

It surely was part of my Informatics Engineering degree, with Tanenbaum book being one of the required reads.

Concur. 15 years ago, as sophomores we built a functional virtual cpu from the transistor level up. As I recall that was about 4 weeks, for the rest of the course, we extended the CPU with cache, extended opcode support, bus expansion, implemented a lexer/parser and ended writing our own assembly subset implementation and using that to write programs.

As for the OP's contention that computer science doesn't teach you to look for higher level things like cache thrashing, I wholeheartedly dissent with that supposition.

I recall at least 3 courses where substantial coursework was devoted to hacking, stack smashing, reverse compiling, profiling/introspection, kernel modification, so much beyond 'dynamic typing is an OO antipattern' stuff that gets IMO erroneously conflated with CS degrees.

Maybe these shit schools exist, but in a top 20 program you will definitely learn cache pitfalls.

This is a real problem that people with experience put on learners. If you asked them how they learned it, they'd tell you an entirely different story about starting small, practicing often, trying many ideas, having a strong motivation and some occasional guidance. But they tell others to follow a rigorously defined path which creates the opposite mindset that a learner needs.

A more humble book, with a more grounded approach might be:

_A Philosophy of Software Design_ by John Ousterhout (the guy behind Tcl)

https://www.goodreads.com/book/show/39996759-a-philosophy-of...

Written as the textbook for a software engineering course, it developed out of that course being taught multiple times _and_ all the code reviews which that entailed.

Previous discussions/mentions here which had a notable number of comments:

https://news.ycombinator.com/item?id=41017367

https://news.ycombinator.com/item?id=34733120

https://news.ycombinator.com/item?id=17779953

https://news.ycombinator.com/item?id=8055868

I re-wrote my current project in the course of reading it (I would read a chapter, then read through the code and where appropriate apply the relevant principle) and once I finish the current re-write (from OpenSCAD to Python) will be repeating that process to see if what I was supposed to have learned stuck/survived the re-write.

> but these days I think the much more important skill is designing systems.

Isn’t t it what SICP is all about?!

I think abstraction layers exist at the system design level as well. Many of the things SICP teaches apply there as well.

Do you have any distributed systems design books to recommend?

if we're talking about relative human-experienced performance, the slowest computers i ever owned were in the early-mid 00s. they sped up as multicore and ssd's entered the picture and plateaued about ten years ago ime.

> the old Hennessy and Patterson book, which I believe is still used today.

Do you mean "Computer Organization and Design - The Hardware / Software Interface" or "Computer Architecture: A Quantitative Approach"? Thanks.

> I think the much more important skill is designing systems

Systems engineering is a seperate discipline in engineering fields, same applies to CS I would think, specifically if you look at what most developers do as computer engineering it would only make sense that there is forward progression within the discipline that builds on the foundations tought in most CS courses.

Regarding CS not teaching caching, well any course writer in a CS program that doesn't at least touch on that should feel like they failed their students, it is really something fundamental that should be tought pretty soon.

I feel like profiling should be something that is tought somewhere in a CS program, even a half semester course can dramatically improve peoples understanding of what the machine is actually doing.

On that note, even if you never intend to use the language, watch some C++ conference talks, almost every year at every conference thwre is a talk discussing performance.

> What computer science doesn’t teach you is how memory caching works in CPUs. Your fancy graph algorithm may have good runtime complexity but it completely hoses the CPU cache and you may have been able to go faster with an array with good cache usage.

Computer architecture and organization should teach this, no?

The most striking performance observation of my experience was that Apple took a system, OpenStep 4.2 which ran okay on a 33 MHz 68040 (and acceptably on my 25MHz Cube) and made it run only a little bit better on a 400MHz G3 as the Mac OS X Public Beta.

The difference of course was anti-aliasing, and much greater bit depth, and running multiple programming environments/toolkits (Carbon and Java).

The CS degree at my school was pretty much just Java. Mostly UIs with java. And applets lol.

The only kids who learned anything else learned C++ so they could get jobs with DOD contractors

Distributed Systems by Van Steen and Tanenbaum is very good and freely bailable:

https://www.distributed-systems.net/

The latter, I’m pretty sure.

I think that is what the book does. Step by step introducing new requirements, by telling you about a situation, where the previous code would not be flexible enough. I am sure it doesn't state, that one should apply all of its ideas all the time, regardless of the project at hand.

My CS degree covered hardware architecture, disk design, OS design, and also system design, alongside algorithms, data-structures, various programming language and paradigm, etc.