←back to thread

1298 points jgrahamc | 1 comments | | HN request time: 0.209s | source
Show context
jgrahamc ◴[] No.22883548[source]
I posted this hours ago and then stepped away. The story captures so much about the Lee I knew so well. I'll add one piece of praise for Lee's early architecture of Cloudflare.

Everything was controlled by a single Postgres database that made very heavy use of stored procedures, that called other procedures, that called others. It was one giant program inside the database. It took me a while to comprehend what he'd done but it was really great. The database ran everything and all those functions made sure that audit logs were kept, that the calls were allowed for the user ID being passed in, and some of these procedures made external calls to APIs including getting things like SSL certificates.

It was a magnificent monolith inside a database.

I worked on the periphery of the database (it was truly Lee's domain) and he'd tell me what output to expect or API to create and I'd code to his spec. and we'd just hook it up.

If any single artefact represents what he did at Cloudflare, it's that database. And he used to code it on a laptop we called "The Beast" because it was so crazily heavy and overloaded with memory etc. that he'd carry around a mini, test Cloudflare wherever he went.

replies(2): >>22883751 #>>22883786 #
woah ◴[] No.22883786[source]
I’m sure he did a great job, but I’ve seen those things be a huge nightmare for anyone other than the irreplaceable genius code wizard who created them.
replies(2): >>22884527 #>>22884915 #
dang ◴[] No.22884527[source]
I guess you're using the word genius ironically, but I've also seen such things be radical in the sense that they're built in a way that no one else would have done, yet it ends up working perfectly for the business and being far more successful than anyone would have predicted. There's a certain type of mind that is immune to fashionable or normal technical choices, that often goes along with remarkable productivity. I suppose the word 'genius' might be a good fit for that.

The interesting thing is that such cases can also actually be the huge nightmare you write about, in the sense that there are tradeoffs in the original design that originally made sense but become harder to deal with later, but also in the sense that their nonstandardness can be hard for more typical engineers to work with. (This is often referred to as the 'hiring problem', though that can be misleading.) The trouble with the typicalness of more typical engineers is that they're steeped in a technical culture which they regard as the proper and obvious way to do things, but which really is mostly just the set of conventions they're used to. Those two things are easy to conflate when all your peers agree on what's proper and obvious (for example that you should never build a complex system entirely in Postgres stored procedures, or whatever the weird thing is). Those conventional assumptions are at least part of what makes such systems a 'nightmare', because if you're working on a system that feels 'wrong', then you fight it rather than blend with it, and that makes it harder to maintain. Everything that's objectively wrong with it will loom larger in your experience than it would if you didn't have the feeling that it should never have been built that way in the first place. Worse, many things that aren't objectively wrong with it, in the sense that they aren't intrinsic to the system or the domain, end up blowing up into major issues because people find they can't work with this system in the way that they're used to, and it's too hard to abandon ways that you're used to—it always feels like the system is at fault. Meanwhile, everything it got right will fade into the background as you assume that any system that had been built 'properly' (i.e. the way you'd have done it) would have gotten all those things right too, at the same cost or better.

It's hard for the later engineer who experiences the 'nightmare' to fully appreciate that if it weren't for that weird, obviously 'wrong' system, they wouldn't have their job in the first place (nightmare and all), because the business that succeeded was built on the weird system.

I don't know anything about this case other than what John posted above, but I've seen something like it in other situations, and in fact I would say that PG and YC have this in common with the case of Lee and Cloudflare as John describes it—as remote from one another as Postgres procedures and custom Lisps must be in every other respect.

What I wish we could teach younger engineers is: when you run across a system like this, pause. Then, pause. Then pause again. The creativity at the core of it can be remarkable. Often you see things that one person did which would have required a large team in other hands, and therefore would never have happened otherwise. Maybe that's one reason why these systems tend to be found at the start of companies: more conventional approaches wouldn't have worked at a stage when hiring a team wasn't an option yet; and on the other hand, the teams you do hire at the later stages tend to have more conventional engineers whose minds (or rather, whose conditioning) can't grok why you'd ever have built something so weirdly.

Usually the later engineers bond with each other around how much they dislike the 'legacy' system, and lobby to replace it, and usually get their way in the end. That's a cycle-of-life thing, but it's also sad when we prefer our conventions and assumptions over seeing, and honoring, the creativity in what came before. Sometimes masterpieces get paved over because of that. Of course, not every nightmare system is an unappreciated masterpiece. But some are, and the bias in our industry is extremely strong toward perceiving them as nightmares. I've seen people take chainsaws to masterpieces without realizing that that's what they were doing, and it makes me sad to think of it. But it makes me happy to read what John wrote to honor Lee, because that's how I'd want someone to write and feel about something I'd built, if I built something like that.

replies(7): >>22884638 #>>22884764 #>>22885451 #>>22885678 #>>22887562 #>>22891436 #>>22892099 #
jgrahamc ◴[] No.22884638[source]
And those conventions change over time. So you might write something using Java EJBs at some point and then the world moves on and engineers no longer know how those things work. What looked like a great architecture is now confusing for most engineers.
replies(1): >>22884663 #
1. dang ◴[] No.22884663[source]
That's true, and weird unorthodox architectures can actually end up being more stable in the long run, especially if their design was an intrinsic fit for the domain. A weird architecture that is a good fit for its domain often has simple ways of meeting key requirements that would have been more costly to develop with more conventional techniques. I'm just hallucinating this from your comment upthread, but a requirement like "making sure audit logs are kept" is the sort of thing I mean. Any architecture can do that, but some make it much easier. When a business starts out with a weird architecture in its early days and ends up succeeding, it's possible that that weirdness gave some competitive advantage.