A Software Development Methodology for Disciplined LLM Collaboration

There's some irony; far from handling the details, LLMs are forcing programmers to adopt hyper-detailed, disciplined practices. They've finally cajoled software developers into writing documentation! Worth noting we've always had the capacity to implement these practices to improve HUMAN collaboration, but rarely bothered.

We’ve ultimately decided to treat the models with more respect, nurturing, and collaborative support than we ever did our follow human keyboard smashers. Writing all the documentation, detailed guidance, allowing them multiple attempts, to help the LLMs be successful. But Brenda, the early in career new grad? “please read this poorly written, 5 year-old, incomplete wiki, and don’t ask me questions”

I’ve been thinking about this for months, and still don’t know what to make of it.

Lol, Empathy and communication skills are important to develop after all.

The halo effect around LLMs is something crazy.

What i keep seeing missing for AI-labor replacement discussions is that technology may seem to replace human labor, but it doesn't really replace human accountability.

Organizations many times seem capable to diffuse blame for mistakes within their human beaurocracy but as beaurocracy is reduced with AI, individuals become more exposed.

This alone - in my view - is sufficient counterpressure to fully replace humans in organizations.

Shorter reply: if my AI setup fails I'm the one to blame. If I do a bad job at helping coworkers perform better is the blame fully mine?

I would also be motivated to write better documentation if I had a junior dev sitting right next to me, utterly incapable of doing any work unless I document how; but also instantly acting on documentation I produce, and giving me rapid feedback on which parts of the documentation are sending the wrong message.

I wonder if this is what will kill LLMs in the software development domain.

It turns out that writing and maintaining documentation is just that universally hated.

My experience writing in a professional setting is that people mostly don't read what I write, and the more effort I put into being thorough the less likely that it will be read.

Respect (or lack thereof) goes both ways: both the writer and reader. I have frequently felt disrespected by producing documentation, planning/etc that isn't read. In the end I mostly rely on oral transmission of knowledge because then at least I can read the room and know if I'm providing some value to people, and ultimately we're both trapped in the room together and have to invest the same amount of time.

The LLM isn't always smart, but it's always attentive. It rewards that effort in a way that people frequently don't. (Arguably this is a company culture issue, but it's also a widespread issue.)

That is an interesting observation. You're correct, the LLM inheritly reads and digests ever token you offer it.

Great framing of the problem. I do think it's a culture issue with "Agile" practices in particular - By design, there is no time budgeted for reading, writing, reflection, or discussion. Sprint, sprint, sprint.

In organizations that value innovation, people will spend time reading and writing. It's a positive feedback loop, almost a litmus test of quality of the work culture.

Agreed, and I would argue the super quick turnaround time an interactive discussion during a planning phase make this much more enjoyable.

I also enjoy discussing solutions with people in real time too. But writing documentation in a vacuum without any feedback or even knowing if someone will read the spec?? Soul draining stuff.

In fact, the best of both worlds would be having a discussion with someone else (real person) while an AI agent listens, takes notes, and provides feedback / insights using different models. Vetting your ideas etc.