Fly.io: The reclaimer of Heroku's magic

At one point (a very long time ago now) it was declared that Dogwood was the future and as a result Go would be the language of choice at Heroku and Erlang would be no more.

Trouble is that Erlang ran all the important Cedar code (it might still today) and the Erlang engineers didn't particularly like the news that Erlang code was essentially deprecated so they left and nobody knew how to maintain the stack. This definitely wasn't the only problem we had but it was a big one.

What do fellow Herokai think? Was Dogwood a fool's errand? Or did we just not get enough staff to build it properly?

Sorry I couldn't find a reference, what is dogwood?

The next phase after Cedar. It was mostly a "Go 2" style mythical benchmark that they compromised on for private spaces but was never really able to meet.

Have seen similar transitions in different companies, and while I can't say anything about Heroku, it is often not the target technology or architecture that matters.

For instance I've seen PHP -> nodejs transition while moving to micrservices, and while the ideas made sense on paper:

- It didn't come from the engineers at large. Most weren't phased by the prospect and the main engine for the change was the top architects and the engineering management.

- target architecture and language were very popular, and easy to hire. Incidentally salaries would be cheaper for same level of experience engineers.

Predictidbly a ton of the existing engineers left and new blood came in, and it was mostly according to plan from what I gathered. Some of the "old folks" stayed at a premium once they were the only ones left with enough knowledge, but they got relagated in side "experts" roles and the product as a whole saw a big change in philosophy (I think mentally what was seen as the "core" also became "legacy" in everyone's mind as engineers moved away)

I think that it’s a very complicated answer to this question, .

This is pretty common in corporates too. Shiny new project gets all the attention, funding and management focus. The poor guys left to deal with the day to day crap get screwed over then leave. If they leave too soon its an emergency, if they stay too long they get laid off.

It’s mentioned in the article.

Heroku named all of their stacks after an alliterated color + tree convention:

- Argent Aspen

- Badious Bamboo

- Celadon Cedar

I can’t remember what to color name for the Dogwood stack was meant to be, we mostly only ever referred to them by the tree name and dropped the color. Suffice to say that Dogwood was meant to be the 4th major evolution of what Heroku was.

I worked at one of the beneficiaries of that exodus with some ex-Heroku Erlang folk. Can't speak to if Dogwood was a fools errand or not but dumping Erlang in favor of Go considering the quality of Erlang talent they had on staff was definitely a colossal fuckup.

Oh for sure. The last company I worked at did a similar thing and decreed that all Ruby projects would be maintenance-only and all new projects were to be in Golang, despite there being a single golang dev in the entire company (who did not even work in the backend dev team but rather in a "platform" team). Within five months all the Ruby devs had left, so despite not a single Golang project being released yet the company now had nobody who knew how the current system works. Several large outages ensued.

Second paragraph in the article

I just don't understand all the fetishism for go. At its core, its a brutally pragmatist language optimized for engineers who barely understand closures while running reasonably fast.

need a garbage collected native compiled language?/

- ocaml fits the bill and provides better type checking

need a fast as possible system?

- rust is faster and has much better abstraction for type checking. unless you're writing a throwaway or a very short lambda function, rust is almost always a better choice here as handling errors and maintaining the code is going to be more important overtime and go is just now getting its generics story straight

need a networking application?

- elixir (and erlang) do this so much better. it has 30+ years of high reliability networking built in and its about as fast as go. additionally, fault tolerance and error handling is so much better. I have real parallelism out of the box and async primitives that make goroutines look like a joke.

additionally, all 3 (ocaml, rust and elixir) give you proper tools for handling error branches. go downgrades you back to c style which works but means your code is going to evolve into a god damn mess as there's no way to separate your error path from your happy path

Literally the only place I see go making sense are small scripts that need to be written asap and wont' need much long term maintenance. for everything else, go seems woefully inadequate.

Rust probably didn't exist when this decision was made.

Out of the box from day one, Go is great at writing HTTP services including proxies, which is a large part of what Heroku needed. Ocaml is harder to use and not a popular choice for such things. Go has easy to follow docs and tonnes of useful contemporary libraries. Go is especially easy to pick up for anyone with older C experience.

I've found Go excellent for the long term, when you come back to something that hasn't been touched for years have passed, it compiles and run quickly and easily. I wouldn't have thought it was any good until I actually used it for something.

Also, concurrency in Go is braindead easy, there are multiple choices of "worker pool" libraries and queue/messaging choices. You don't even have to know about channels to do work across cores.

EDIT: Having said that, if you already had Erlang and an experienced team, you wouldn't ditch that for Go. Why do companies do this? Is there some famous historical case where keeping and growing a highly experienced team has backfired?

> Having said that, if you already had Erlang and an experienced team, you wouldn't ditch that for Go. Why do companies do this?

You don't get promoted until you do the new shiny thing. This seems to be especially prominent in Google and the likes. Big companies do not give out bonuses/promotions for those who quietly sit and work with existing, proven stack, maintaining it and fixing bugs. Shiny new things get noticed.

It might also be caused by staffing issues? Erlang/Elixir to this day are pretty niche stacks. It's much easier to find Go developers(or developers willing to switch to Go), than Erlang.

Yeah.. I'm seeing the same thing where I work. Moving from ruby/rails to elixir. No outages, but dev speed is at a crawl. All the legacy stuff just gets ugly patches.

There was zero need to switch because even though elixir/BEAM allows for a pile of processes, postgres does not.

If I had to guess, the switch was made because someone saw something shiny.

Lets re-write it they said. It will be easier to maintain they said. It will be more use friendly they said. We're Netscape and there's nothing like us they said.