Free software scares normal people

The usual situation is that the business department hires someone with a modicum of talent or interest in tech, who then uses Access to build an application that automates or helps with some aspect of the department's work. They then leave (in a couple of cases these people were just interns) and the IT department is then called in to fix everything when it inevitably goes wrong. We're faced with a bunch of beginner spaghetti code [0], utterly terrible schema, no documentation, no spec, no structure, and tasked with fixing it urgently. This monster is now business-critical because in the three months it's been running the rest of the department has forgotten how to do the process the old way, and that process is time-critical.

Spinning up a proper project to replace this application isn't feasible in the short term, because there are processes around creating software in the organisation, for very good reasons learned painfully from old mistakes, and there just isn't time to go through that. We have to fix what we can and get it working immediately. And, of course, these fixes cause havoc with the project planning of all our other projects because they're unpredictable, urgent, and high priority. This delays all the other projects and helps to give IT a reputation as taking too long and not delivering on our promised schedules.

So yeah, what appears to be the best solution from a non-IT perspective is a long, long way from the best solution from an IT perspective.

[0] and other messes; in one case the code refused to work unless a field in the application had the author's name in it, for no other reason than vanity, and they'd obfuscated the code that checked for that. Took me a couple of hours to work out wtf they'd done and pull it all out.

Part of the problem is that the novices that create these applications don't consider all the edge cases and gnarly non-golden-path situations, but the experienced devs do. So the novice slaps together something that does 95% of the job with 5% of the effort, but when it goes wrong the department calls in IT to fix it, and that means doing the rest of the 95% of the effort. The result is that IT is seen as being slow and bureaucratic, when in fact they're just doing the fecking job properly.

most of these teams only wants a straightforward spec, shut themselves off from distractions, just to emerge weeks or months later with something that completely misses the business case. and yet, they will find ways to point fingers at the product owner, project manager, or client for the disaster.

This reminds me of the "just walk confidently to their office and ask for a job to get one!" advice. This sounded bullshit to me until I got to stay with some parts of a previous company, where the hiring process wasn't that far really.

That's also the kind of companies where contracts and vendor choices will be negociated on golf courses and the CEO's buddies could as well be running the company it would be the same.

I feel for you.

The huge majority of devs want to understand the business and develop high quality software for it.

In one business I worked for, the devs knew more about the actual working of the business than most of the non-IT staff. One of the devs I worked with was routinely pulled into high-level strategy meetings because of his encyclopaedic knowledge of the details of the business.

If you want a developer to write good code quickly, put them in an isolated silo and don't disturb them.

If you want a developer to engage with the business units more, be prepared for their productivity to drop sharply.

As with all things in tech, it's a trade-off.

The single most valuable tool is user testing. However it really takes quite a few rounds of actually creating a design and seeing how wrong you saw the other person’s capabilities, to grok how powerful user testing is in revealing your own biases.

And it’s not hard at all at core. The most important lesson really is a bit of humility. Actually shutting up and observing what real users do when not intervened.

Shameless plug, my intro to user testing: https://savolai.net/ux/the-why-and-the-how-usability-testing...

You need a structure if you have org of 100+ employees. If it is smaller than that I don’t believe you get dev department.

IT should not be focusing on the theoretical, platonic Business Process. It never exists in practice anyway. They should focus on streamlining actual workflow of actual people. I.e. the opposite advice to the usual. Instead of understanding what users want and doing it, just do what they tell you they want. The problem with standard advice is that the thing you seek to understand is emergent, no one has a good definition, and will change three times before you finish your design doc.

To help company get rid of YOLOed hacks in Excel and such made by interns, IT should YOLO better hacks. Rapid delivery and responsiveness, but much more robust and reliable because of actual developer expertise behind it.

I.e. done right, it should be not just possible but completely natural for a random team lead in the mail room to call IT and ask, "hey, we need a yellow highlighter in the sheet for packages that Steve from ACME Shipping needs to pick on extra evening run, can you add it?", and the answer should be "sure!" and they should have the new feature within an hour.

Yes, YOLO development straight on prod is acceptable. It's what everyone else is doing all the time, in every aspect of the business. It's time for developers to stop insisting they're special and normal rules don't apply to them.

If you streamline a shitty process, you will have diarrhea...

Unfortunately, most processes suck and need improvement. It isn't actually IT's job to improve processes. But almost always, IT is the only department that is able to change those processes nowadays since they are usually tied to some combination of lore, traditions, spreadsheets and misused third-party software.

If you just streamline what is there, you are cementing those broken processes.

It's because of that condescending, know-it-all attitude that people actively avoid getting IT involved in anything, and prefer half-assed Excel hacks. And they're absolutely right.

Work with them and not over them, and you may get an opportunity to improve the process in ways that are actually useful. Those improvements aren't apparent until you're knee-deep in mud yourself, working hand by hand with the people you're trying to help.

I assume those processes weren't applied when deciding to use this application, why? Was there a loophole because it was done by an intern?

Also, if you have ever worked with anyone trying to get specifications worked out, you’ll see that most people (including devs) rely on intuition rather than checklists and will always forget to tell you something that is critical.

The thing is that cost of changes in the business can be a simple memo. But for software that usually means redesign.

The main reason you want a computer is cheap emulation (cad, daw,…) or fast (and reliable) automation. Both requires great deal of specifications to get right.

The loophole is that if you have Office or similar you have a variety of development environment, IT/compliance/finance aren't caring what files you produce with the applications you have, and no one else is paying attention initially either, but would have a say (and a procedure for you to follow) if you wanted to bring in or create a new application. The usual process is bypassed.

This is more commonly associated with Excel, but it applies to Access too (less so than it used to, but there are still plenty people out there who rely on it daily).

Once the demo/prototype/PoC is there it is a lot easier to “fix up” that than spin up a project in anything else, or get something else in that is already available, for the same reasons as why it was done in Excel/Access in the first place plus the added momentum: the job is already at least part way done, using something else would be a more complete restart so you need to justify that time as well as any other costs and risks.

[Note: other office suites exist and have spreadsheets & simple DBs with similar capabilities, or at least a useful subset of them, of course, but MS Office's Excel & Access are, for better or worse, fairly ubiquitous]

That's an illusion. The reality is, it's all hacky solutions on top of hacky solutions. Even in manufacturing: the spec may be fixed, and the factory line produces identical products by the million - but the spec was developed through an ad-hoc process, and the factory line itself is a pile of hacks that needs continued tuning to operate. And there is no perfectly specced out procedure for retooling a factory line to support the newest spec that came out of design department - retooling is, in itself, a small R&D project.

> Also, if you have ever worked with anyone trying to get specifications worked out, you’ll see that most people (including devs) rely on intuition rather than checklists and will always forget to tell you something that is critical.

This is the dirty truth about the universe - human organizations are piles of hacks, always in flux; and so is life itself. The sameness and harmony we see in nature is an illusion brought on by scale (in two ways - at large scale, because we live too short to see changes happening; at small scale, because the chaos at biomolecular level averages out to something simpler at the scale we can perceive).

Order and structure are not the default. It takes hard work to create and maintain them, so it's better be worth the cost. The prevalence of Excel-based hacks in corporate is a proof positive that, for internal software, it usually isn't worth it, despite what the IT department thinks.

> The thing is that cost of changes in the business can be a simple memo. But for software that usually means redesign.

Which is why you shouldn't be building cathedrals that need expensive rework every other week because of some random memo. Instead, go down to where people work; see them tweaking their shovels, take those shovels and make the tweak they want the proper way.

I disagree: it's a business prioritisation issue (not necessarily a problem). Ultimately, a lot of the processes are there because the wider business (rightly) wants IT to work on the highest impact issues. A random process that 3 people suffer from probably isn't the highest impact for the business as a whole.

Also, because it's not high impact, it makes sense that an intern is co-opted to make life easier (also as a learning experience), however it also causes the issues OP highlighted.

The problem is solvable, I think, but it's not easily solvable!

Love the assumption "when it inevitably goes wrong." In real life, many of these applications work perfectly for years and assist employees tremendously. The program doesnt fail, but the business changes - new products, locations, marketing, payment types, inventory systems, tons of potential things.

And yes, after the original author is gone, nobody is left to update the program. Of course, a lot of programmers or IT folks probably could update it, but ew, why learn and write Access when we can create a new React app with microservices-based backend including Postgres in the cloud and spin up a Kubernetes cluster to run it.

Then think again of those managers getting paid manager salaries who couldn't figure this out themselves - or worse, the ones who want to shut it all down because he didn't "follow the procedure" (the procedure of not doing anything useful???)

Which is a huge reason that learning a RAD (rapid application development - emphasis on rapid) tool is a pretty useful skill.

A bit of tangent: I think the idea of coddling users is what’s leading to the complexity of all those system. We’re building cathedrals when we need tents. Instead of having small, sharp software tools that can be adjusted easily, we’re developing behemoths that’s supposed to handle everything under the sun (systemd, a lot of modern package managers, languages that is tied to that one IDE,…)

http://lpd2.com/

I'm not really arguing for mega-tools with locked-down workflows. But there's usually some happy-ish medium between chaos and rigid monoliths.

My best example was a conversation I had with one of the scientists at my job when she mentioned that she had people spending hours every day generating reports from data our instruments produced. I pointed out that with the code we had it would be simple to generate the reports automatically.

Her response that she had asked repeatedly for a developer to be assigned to the task, but she kept being pushed away because it was low priority.

I couldn't just change the codebase on my own (it was for a medical device), but it was easy enough to spend a lazy afternoon writing a tool to consume the output logs from the device and generate the reports that she needed. That's it: about 4 hours of work and produced something this person had asked for a year prior, and that people were already spending hours each day doing!

The people in charge of vetting requests never even bothered to ask a developer to estimate the task. They just heard that there was a work around, so it immediately became "low priority."

I'm not "prioritizing" anything. The scenario we're discussing is when an intern or low-level employee is able to successfully automate, enhance, or simplify a manual, inefficient business process that management has not seen fit to improve - so the worker does it themselves.

Access and similar platforms aren't "rapid" because of shortcuts, they are rapid because they are visual-based, drag-and-drop, object-oriented and often make a component's properties and methods customizable also via a visual interface. It's a different way of programming, yes, accessible to the masses (which is likely the reason you have so much disdain), but not "shortcuts".

Kinda the whole goal behind (and "benefit" of) microservices, right? Totally independent dev teams, all uncoupled from each other, no need to look inside at the code, language-independent - just pass data according to an API and dont look behind the curtain.

We should not be thinking about architecture at the business process level. This is just repeating the mistake that needs to be avoided here. This is, and will ever be, a pile of ad-hoc hacks. They're not meant to add up to a well-designed system in a bottom-up fashion, because there is no system to design. The structure we naturally seek, is constantly in flux.

The right architectural/design decisions to make here is to make it possible to assemble quick hacks out of robust parts that fulfill additional needs the people on the ground may not consider - logs/audit trail/telemetry, consistency for ergonomic reasons, error handling, efficient compute usage, tracking provenance, information access restrictions dictated by legal obligations, etc.

The most important change needed is in the mindset. Internal dev needs to stop thinking of itself as the most important part of the company, or as a well-defined team that should own products. To be useful, the opposite is needed - such devs need to basically become ChatGPT that works: always be there to rapidly respond to requests to tweak some software by people on the ground, and then to retweak is as needed. They need to do this work rapidly, without judgement, and never assume they know better.

Only then people will stop weaving ad-hoc Excel sheets into business-critical processes.

Top ends of organizations need and want a lot of dumb, self-defeating things - this is very much the same thing that's described in "Seeing Like a State". Doing this blindly is, of course, a prerogative of the executives, but internal IT is actually low enough in the food chain that it could focus more on helping the org by improving the bottom layers, instead of embracing and encouraging the top to create more rigid structures.

EDIT: to refer back to the example I gave upthread:

"Steve from ACME Shipping" is not meant to be a special case of an "external vendor assigned to auxiliary shipping itinerary"; the system needs not to be designed to express the concepts of "external vendor" and "auxiliary shipping itinerary" and "shipping itinerary assignment for shippable resources". Steve is just Steve, the whole "extra evening run" thing is probably just a one-off emergency measure for Q3 that will disappear come Q4 (or persist, and then some executives will start talking talk about deeper changes to mailing process). Right now, all the mailing room needs is for someone to add a boolean flag:

  // Steve from ACME Mailing
  bool boundToACMEEVeningRun;

At some point in our industry, "service providers" started thinking of themselves as kings, instead of what they were supposed to be - servants.

Software development delivers products, internal products and solutions that should be leveraged by business to improve rate of growth.

If you have software development department chucked into IT and make them be supporters that run in the background you are wasting potential or wasting money on their salaries.

If you want supporters make it IT only and pay for SaaS solutions that everyone is using.