So many gems in here but this one about microservices is my favorite:
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
replies(8):
Now I understand that “architecture” and “patterns” is a jobs program for useless developers. It’s this, or they’d be on the streets holding a sign saying “will write JavaScript for a sandwich”.
Yet, developers are always using patterns and are thinking about architecture.
Here you are doing so too, a pattern, "form submission" and an architecture, "request-response".
One app I got brought in late on the architect had done some complicated mediator pattern for saving data with a micro service architecture. They'd also semi-implemented DDD.
It was a ten page form. Literally that was what it was supposed to replace. An existing paper, 10 page, form. One of those "domains" was a list of the 1,000 schools in the country. That needed to be updated once a year.
A government spent millions on this thing.
I could have done it on my todd in 3 months. It just needed to use simple forms, with some simple client side logic for hiding sections, and save the data with an ORM.
The funniest bit was when I said that it couldn't handle the load because the architecture had obvious bottlenecks. The load was known and fairly trivial (100k form submissions in one month).
The architect claimed that it wasn't possible as the architecture was all checked and approved by one of the big 5.
So I brought the test server down during the call by making 10 requests at once.
The agencies are similarly uncoordinated and will pick up their metaphorical credit card and just throw it at random small dev teams, internally, external, or a mix.
Those people will happily take the credit! The money just… disappears. It’s like a magic trick, or one of those street urchins that rips you off when you’re on holiday in some backwards part of the world like Paris.
I get brought in as “the cloud consultant” for a week or two at the end to deploy the latest ball of mud with live wires sticking out of it to production.
This invariably becomes an argument because the ball of mud the street urchins have sold to the customer is not fit for… anything… certainly not for handling PII or money, but they spent the budget and the status reports were all green ticks for years.
Fundamentally, the issue is that they're "going into the cloud" with platform as a service, IaC, and everything, but at some level they don't fully grok what that means and the type of oversight required to make that work at a reasonable cost.
"But the nice sales person from Microsoft assured me the cloud is cheaper!"
Back in the very early 2000s I got sent to "tune IIS performance" at a 100-developer ISV working on a huge government project.
They showed me that pressing the form submit button on just two PCs at once had "bad performance".
No, not it didn't. One was fast[1], the other took 60 seconds almost exactly. "That's a timeout on a lock or something similar", I told them.
They then showed me their 16-socket database server that must have cost them millions and with a straight face asked me if I thought that they needed to upgrade it to get more capacity. Upgrade to what!? That was the biggest machine I have ever seen! I've never in the quarter century since then seen anything that size with my own two eyes. I don't believe bigger Wintel boxes have ever been made.
I then asked their database developers how they're doing transactions and whether they're using stored procedures or not.
One "senior" database developer asked me what a stored procedure is.
The other "senior" database developer asked me what a transaction is.
"Oh boy..."
[1] Well no, not really, it took about a second, which was long enough for a human button press to to "overlap" the two transactions in time. That was a whole other horror story of ODBC connection pooling left off and one-second sleeps in loops to "fix" concurrency issues.
In fact we're going through one of these SAP HANA migrations at present and it's very broken, because the prime contractor has delivered a big ball of mud with lots of internal microservices.
My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.
I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud.
I blame a few things for this:
1. Developers almost universally lack discipline.
2. Most programming languages are not designed to sufficiently account for #1.
It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story.
And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.
- Cannot be run without an orchestrator like K8S, which is a bear to install and maintain, which helps sell managed cloud.
- Uses more network bandwidth, which they bill for, and CPU, which they bill for.
- Makes it hard to share and maintain complex or large state within the application, encouraging the use of more managed database and event queue services as a substitute, which they bill for. (Example: a monolith can use a queue or a channel, while for microservices you’re going to want Kafka or some other beast.)
- Can’t be run locally easily, meaning you need dev environments in cloud, which means more cloud costs. You might even need multiple dev and test environments. That’s even more cloud cost.
- Tends to become dependent on the peculiarities of a given cloud host, such as how they do networking, increasing cloud lock in.
Anyone else remember how cloud was pitched as saving money on IT? That was hilarious. Knew it was BS way back in the 2000s and that it would eventually end up making everything cost more.
Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck.
By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem.
Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus.
The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
This is the real, actual conversation to be had about "AI taking jobs."
I've seen similar things a lot in the private sector.
There's just loads of people just flailing around doing stuff without really having any expertise other than some vague proxy of years of experience.
It's really not even exactly their fault (people have lives that don't revolve around messing about with software systems design, sure, and there's no good exposure to anything outside of these messes in their workplaces).
But, outside of major software firms (think banks, and other non-"tech" F500s; speaking from experience here) there's loads of people that are "Enterprise Architects" or something that basically spend 5 hours a day in meetings and write 11 lines of C# or something a quarter and then just adopt ideas they heard from someone else a few years back.
Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.
Don't forget various pipelines, IaC, pipelines for deploying IaC, test/dev/staging/whatever environments, organization permissions strategies etc etc...
When I worked at a large, uh, cloud company as a consultant, solutions were often tailored towards "best practices"--this meant, in reality, large complex serverless/containerized things with all sorts of integrations for monitoring, logging, NoSQL, queues etc, often for dinky little things that an RPI running RoR or NodeJS could serve without breaking a sweat.
With rare exceptions, we'd never be able to say, deploy a simple go server on a VM with server-side rendered templates behind a load balancer with some auto-scaling and a managed database. Far too pedestrian.
Sure, it's "best practices" for "high-availability" but was almost always overkill and a nightmare to troubleshoot.
But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like
@GET
@Path("/{id}")
public List<Data> fetchData(@PathParam("id") int id) { ... }It’s hard to hire people to do anything else. People don’t know how to admin machines so forget bare metal even though it can be thousands of times cheaper for some work loads (especially bandwidth).
You’re not exaggerating with a raspberry pi. Not at all.
The "entire generation of developers" paradigm is all over in different domains too--web programmers that seem to honestly think web development is only React/Angular and seem to have no idea that you can just write JS, python programmers that have no idea a large portion of the "performant codebases" are piles of native dependencies etc
The scaling problems we face could probably be solved by other solutions, but the company is primed and ready to chuck functionality into new microservices. That’s what all our infrastructure is set up to do, and it’s what inevitably happens every time
You could probably get most of this without the literal network hop, but I haven't seen a serious attempt.
Or maybe I’m missing what you mean - can you explain with an example an API boundary you can’t define by interfaces in Go, Java, C# etc? Or by Protocols in Python?
These new superdepartments all have the same issues to the point that I sometimes confuse them.
One of my pet-peeves. "We're doing DDD, hexagonal architecture, CQRS". So, when was the last time your dev team had a conversation with your domain experts? You have access to domain experts don't you? What does your ubiquitous language look like?
So no, some "senior" read a blog post (and usually just diagonally) and ran with it and now monkey see monkey does is in full effect.
And you get the same shit with everything. How many "manager" read one of the books about the method they tell you they're implementing (or any book about management) ? How many TDD shop where QA and dev are still separate silos? How many CI/CD with no test suite at all? Kanban with no physical board, no agreed upon WIP limits, no queue replenishing system but we use the Kanban board in JIRA.
> I think it's best to break things down
Factoring system = break things down.
Most other fields are similar, frankly.
I must scream but I'm in a vacuum. Everyone is fine with this.
(Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)
That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
"We're all-in on using Kanban here"
"Ah, great. What's your current WIP limit?"
"Um, what's a whip limit?"
As a consultant, I don't actually mind finding myself in the midst of that sort of situation - at the very least, it means I'm going to be able to have a positive impact on the team just by putting in a bit of thought and consistent effort.
But the guy 'is known to get the job done' apparently.
And to be fair, if folks didn’t get paid for reinventing basic Unix utilities with extra steps, the economy would probably collapse.
from submodule import pandas
microservices putting a network call in on the factoring is a feature in this case, not a bug. it’s a physical blocker stopping devs doing stuff like that. it’s the one thing i don’t agree with grug on.
HOWEVER — it’s only a useful club if you use it well. and most of the time it’s used because of expectations of shiny rocks, putting statements about microservices in the company website, big brain dev making more big brain resume.
Then you deploy the server program, and then you deploy the client program, to another machine, or machines, where they continue to talk to one another over http, maybe over the public Internet or maybe not.
Deploying can mean any one of umpteen possible things. In general, you (use automations that) copy your programs over to dedicated machines that then run your programs.
You can do this in Python as well, but it does involve a bit of care; I like the pattern of a module named “internal” that has the bulk of the modules code in it, and a small public api.py or similar that explicitly exposes the public bits, like an informal version of the compiler-enforced pattern for this in Go
* Information hiding by exposing a closed interface via the API
* Isolated, scalable, fault tolerant service
* Iterable, understandable and super agile
— Melvin E. Conway, How Do Committees Invent?
This alone earns my upvote.
So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks):
cp file1.csv combined.csv && tail -n+2 file2.csv >> combined.csvSame goes for users. There are about the same number of computer literate users as there were back then. There’s just a new class of iPad/iPhone user who is only a casual user of computers and the net and barely knows what a file is.
Of course the best is mono repo and monolith :3
If you mix in two services into a single database - no matter how good the logical and security isolation is — they will roll back their transactions together if the DBA presses the restore button.
Similarly they have the option (but not the obligation!) to participate in truly atomic transactions instead of distributed transactions. If this is externally observable then this tight coupling means they can no longer be treated as separate apps.
Many architects will just draw directed arrows on diagrams, not realising that any time two icons point at the same icon it often joins them into a single system where none of the parts are functional without all of the others.
If you prefer working on mono repo it s fine though I think micro services are more popular one now.
If so: mono repo is not an alternative to micro services. You can have both. You can have micro services in a mono repo. One single repository with many micro services. That is possible.
What is an alternative to micro services is “monolith”.
Curious how I’m misunderstanding this because I looked at it for a while and I really don’t see it.