The Grug Brained Developer (2022)

I keep trying to explain this to tiny dev teams (1-2 people) that will cheerfully take a trivial web app with maybe five forms and split it up into “microservices” that share a database, an API Management layer, a queue for batch jobs to process “huge” volumes (megabytes) of data, an email notification system, an observablity platform (bespoke!) and then… and then… turn the trivial web forms into a SPA app because “that’s easier”.

Now I understand that “architecture” and “patterns” is a jobs program for useless developers. It’s this, or they’d be on the streets holding a sign saying “will write JavaScript for a sandwich”.

> Now I understand that “architecture” and “patterns” is a jobs program for useless developers.

Yet, developers are always using patterns and are thinking about architecture.

Here you are doing so too, a pattern, "form submission" and an architecture, "request-response".

>I keep trying to explain this to tiny dev teams

I'm curious what role you have where you're doing this repeatedly

It's all they've seen. They don't get why they're doing it, because they're junior devs masquerading as architects. There's so many 'senior' or 'architect' level devs in our industry who are utterly useless.

One app I got brought in late on the architect had done some complicated mediator pattern for saving data with a micro service architecture. They'd also semi-implemented DDD.

It was a ten page form. Literally that was what it was supposed to replace. An existing paper, 10 page, form. One of those "domains" was a list of the 1,000 schools in the country. That needed to be updated once a year.

A government spent millions on this thing.

I could have done it on my todd in 3 months. It just needed to use simple forms, with some simple client side logic for hiding sections, and save the data with an ORM.

The funniest bit was when I said that it couldn't handle the load because the architecture had obvious bottlenecks. The load was known and fairly trivial (100k form submissions in one month).

The architect claimed that it wasn't possible as the architecture was all checked and approved by one of the big 5.

So I brought the test server down during the call by making 10 requests at once.

The customer is a government department formed by the merger of a bunch of only vaguely related agencies. They have “inherited” dozens of developers from these mergers, maybe over a hundred if you count the random foreign outsourcers. As you can imagine there’s no consistency or organisational structure because it wasn’t built up as a cohesive team from the beginning.

The agencies are similarly uncoordinated and will pick up their metaphorical credit card and just throw it at random small dev teams, internally, external, or a mix.

Those people will happily take the credit! The money just… disappears. It’s like a magic trick, or one of those street urchins that rips you off when you’re on holiday in some backwards part of the world like Paris.

I get brought in as “the cloud consultant” for a week or two at the end to deploy the latest ball of mud with live wires sticking out of it to production.

This invariably becomes an argument because the ball of mud the street urchins have sold to the customer is not fit for… anything… certainly not for handling PII or money, but they spent the budget and the status reports were all green ticks for years.

Fundamentally, the issue is that they're "going into the cloud" with platform as a service, IaC, and everything, but at some level they don't fully grok what that means and the type of oversight required to make that work at a reasonable cost.

"But the nice sales person from Microsoft assured me the cloud is cheaper!"

> So I brought the test server down during the call by making 10 requests at once.

Back in the very early 2000s I got sent to "tune IIS performance" at a 100-developer ISV working on a huge government project.

They showed me that pressing the form submit button on just two PCs at once had "bad performance".

No, not it didn't. One was fast[1], the other took 60 seconds almost exactly. "That's a timeout on a lock or something similar", I told them.

They then showed me their 16-socket database server that must have cost them millions and with a straight face asked me if I thought that they needed to upgrade it to get more capacity. Upgrade to what!? That was the biggest machine I have ever seen! I've never in the quarter century since then seen anything that size with my own two eyes. I don't believe bigger Wintel boxes have ever been made.

I then asked their database developers how they're doing transactions and whether they're using stored procedures or not.

One "senior" database developer asked me what a stored procedure is.

The other "senior" database developer asked me what a transaction is.

"Oh boy..."

[1] Well no, not really, it took about a second, which was long enough for a human button press to to "overlap" the two transactions in time. That was a whole other horror story of ODBC connection pooling left off and one-second sleeps in loops to "fix" concurrency issues.

Omg this is something I have experienced too many times, and constantly warring with the other side of the coin: people who never want to make any change unless it is blessed by a consultant from Microsoft/VMWare/SAP and then it becomes the only possible course of action, and they get the CIO to sign off on some idiocy that will never work and say "CIO has decreed Project Falcon MUST SUCCEED" when CIO can't even tie his shoelaces. Giant enterprise integration will happen!

In fact we're going through one of these SAP HANA migrations at present and it's very broken, because the prime contractor has delivered a big ball of mud with lots of internal microservices.

Is this DCS in NSW? If so that would explain so much about my own work interactions with them.

The only useful definition of a "service" I've ever heard is that it's a database. Doesn't matter what the jobs and network calls are. One job with two DBs is two services, one DB shared by two jobs is one service. We once had 10 teams sharing one DB, and for all intents and purposes, that was one huge service (a disaster too).

> It's all they've seen. They don't get why they're doing it, because they're junior devs masquerading as architects. There's so many 'senior' or 'architect' level devs in our industry who are utterly useless.

This is the real, actual conversation to be had about "AI taking jobs."

I've seen similar things a lot in the private sector.

There's just loads of people just flailing around doing stuff without really having any expertise other than some vague proxy of years of experience.

It's really not even exactly their fault (people have lives that don't revolve around messing about with software systems design, sure, and there's no good exposure to anything outside of these messes in their workplaces).

But, outside of major software firms (think banks, and other non-"tech" F500s; speaking from experience here) there's loads of people that are "Enterprise Architects" or something that basically spend 5 hours a day in meetings and write 11 lines of C# or something a quarter and then just adopt ideas they heard from someone else a few years back.

Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.

No… but you’re close with your guess!

These new superdepartments all have the same issues to the point that I sometimes confuse them.

> They'd also semi-implemented DDD.

One of my pet-peeves. "We're doing DDD, hexagonal architecture, CQRS". So, when was the last time your dev team had a conversation with your domain experts? You have access to domain experts don't you? What does your ubiquitous language look like?

So no, some "senior" read a blog post (and usually just diagonally) and ran with it and now monkey see monkey does is in full effect.

And you get the same shit with everything. How many "manager" read one of the books about the method they tell you they're implementing (or any book about management) ? How many TDD shop where QA and dev are still separate silos? How many CI/CD with no test suite at all? Kanban with no physical board, no agreed upon WIP limits, no queue replenishing system but we use the Kanban board in JIRA.

>Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.

Most other fields are similar, frankly.

On the other side of the coin, once some part of government contacted us about a project they wanted done. I don't even remember what it was, but it was something very simple that we estimated (with huge margins) to be 3 months of work end-to-end. What we heard back was that they need it to take at least two years to make. I suspect some shady incentives are in play and that exceedingly inefficient solutions are a plus for someone up the chain.

Yeah, exactly, and you see it all over the place. It's not even cargo-culting, it's more half-arsed than that.

"We're all-in on using Kanban here"

"Ah, great. What's your current WIP limit?"

"Um, what's a whip limit?"

As a consultant, I don't actually mind finding myself in the midst of that sort of situation - at the very least, it means I'm going to be able to have a positive impact on the team just by putting in a bit of thought and consistent effort.

A word and its meaning are usually antonyms.

Yes! My viewpoint also. I've been very alone in that view over the last few decades. Nice to know there's someone else out there.

This is in no way unique to software. Virtually all fields I have insight in are the same and managment is one of the worst.

Great point. I've only worked at a couple places that architected the system in this manner, where the data layer defines the service boundary. It really helps keep the management of separate services sane IMO, vs. different "services" that all share the same database.

> some backwards part of the world like Paris.

This alone earns my upvote.

At least it's widely considered bad practice for two services to share a database. But that's different from orienting your entire view of services around databases. The case of one job with two DBs matters too, mostly because there's no strong consistency between them.

A more precise view is that there are boundaries inside of which certain operations are atomic. To make this more precise: the difference between a “dedicated schema” and a “database” is that the latter is the boundary for transactions and disaster recovery rollbacks.

If you mix in two services into a single database - no matter how good the logical and security isolation is — they will roll back their transactions together if the DBA presses the restore button.

Similarly they have the option (but not the obligation!) to participate in truly atomic transactions instead of distributed transactions. If this is externally observable then this tight coupling means they can no longer be treated as separate apps.

Many architects will just draw directed arrows on diagrams, not realising that any time two icons point at the same icon it often joins them into a single system where none of the parts are functional without all of the others.

The atomicity is the main reason, but it's also about API boundaries. Sometimes someone makes a "service" that's more like a database exposing a full suite of transactions and CRUD operations, then other teams call themselves "stateless" interacting with it. But really they're sharing a DB, and thus building one giant service, because the API they're using is so broad that it creates an inseparable coupling.