The case against conversational interfaces

> If you have a team of senior, experienced and bright engineers, you can with a few words point out a desire and, trust them to ask for information when there is relevant ambiguity, and expect a good outcome

It's such a fallacy. First thing an experienced and bright engineer will tell you is to leave the premises with your "few words about a desire" and not return without actual specs and requirements formalized in some way. If you do not understand what you want yourself, it means hours/days/weeks/months/literally years of back and forths and broken solutions and wasted time, because natural language is slow and lossy af (the article hits the nail on the head on this one).

Re "ask for information", my favorite example is when you say one thing if I ask you today and then you reply something else (maybe the opposite, it happened) if I ask you a week later because you forgot or just changed your mind. I bet a conversational interface will deal with this just fine /s

I think you work in different domains.

Expecting a good outcome is different from expecting to get exactly what you intended.

Formal specifications are useful in some lines of work and for some projects, less so for others.

Wicked problems would be one example where formal specs are impossible by definition.

I do understand that in bad cases it can be very frustrating as an engineer to chase vague statements only to be told later 'nah, that was not what I meant'. This is especially true when the gap in both directions is very large or there is incompetence and/or even adversarial stances between the parties. Language and communication only work if both parties are willing to understand.

Unfortunately if either is the case "actual specs and requirements formalized", while sounding logical, and might help, in my experience did very little to save any substantial project (and I've seen a lot). The common problem is that the business/client/manager is forced to sign of on formal documents far outside their domain of competence, or the engineers are straitjacketed into commitments that do not make sense or have no idea of what is considered tacit knowledge in the domain and so can't contextualize the unstated. Those formalized documents then mostly become weaponized in a mutual destructive CYA.

What I've also seen more than once is years of formalized specs and requirements work while nothing ever gets produced, and the project is aborted before even the first line of code hit test.

I've given this example before: When Covid lockdows hit there were digitization projects years in planning and budgeted for years of implementation, that were hastily specked, coded and roiled out into production by a 3 person emergency team over a long weekend. Necessity apparently has a way of cutting through the BS like nothing else can.

You need both sides capable, willing and able to understand. If not, good luck mitigating, but you're probably doomed either way.

> What I've also seen more than once is years of formalized specs and requirements work while nothing ever gets produced, and the project is aborted before even the first line of code hit test.

It just shows that no one really understood what they wanted. It is crazy to expect somebody to understand something better than you and it is hilarious to want a conversational UI to understand something better than you.

> Formal specifications are useful in some lines of work and for some projects, less so for others

There is always formal specification. Code is final formal specification in the end. But converting vague vibes from natural language into a somewhat formalized description is key ability you need for any really new non trivial project idea. Another human can't do it for you, conversational UI can't do it for you...

>Anyway, the disabled are pretty much always allowed to be collateral damage by society, so this will just be senseless pain.

For games, you don't really need nor desire formal specs. But it also can really show how sometimes a director has a low tolerance for interpretation despite their communication being very loose. This leads to situations where it feels like the director is shifting designs on a dome, which is a lose-lose situation for everyone involved.

If nothing else, formal specification is for CYA. You get what you ask for, and any deviation should go in the next task order or have been addressed beforehand.

The US having this culture of blame and deflect doesn't help either. When you're more concerned about making sure you can't be held liable if X fails, then you spend more time covering your tracks than developing the project. And that's how the beauracracy creeps in.

And approach of shared responsibility in all respects (successes and failure) would accelerate past the inevitable shortcomings that occur and let all parties focus on recovering and delivering.

> it is hilarious to want a conversational UI to understand something better than you.

This is true. But what if you swap "conversational UI" with something actually intelligent like a developer. Then we see this kind of thing all the time: A user has tacit, unconscious knowledge of some domain. The developer keeps asking them questions in order to get a formal understanding of the domain. At the end the developer has a formal understanding and the user keeps their tacit understanding. In theory we could do the same with an AI - If the AI was actually intelligent.

"It just shows that no one really understood what they wanted."

Then what were the literally room full of formal process and spec documents, meeting reports and formal agreements (near 100.000 pages) by the analysts on either side for? And how did those not 'solve' the understanding problem?

When I go to the garage to have my car serviced, I expect them to understand it way better than I do. When I go to a nice restaurant I expect the cooks to prepare me dishes that taste greater than me writing them out a step-by-step recipe for them to follow. If I hire a senior consultant in even my own domain, I expect them to not just know my niche, but bring tacit knowledge from having worked on these types of solutions across my industry.

Expecting somebody to understand something better than me is exactly the reason why I hire senior people in the first place.

> Then what were the literally room full of formal process and spec documents, meeting reports and formal agreements (near 100.000 pages) by the analysts on either side for? And how did those not 'solve' the understanding problem?

Sure.

There are many possible factors (eg. somebody had a shitty idea and a committee of people sabotaged it because they didn't wanted it to succeed, or it was good but committee interests/politics were against it, or it was generally a dysfunctional org) but it's irrelevant so let's pretend people are good and it's the ideal case.

There was likely somebody who had a good idea originally. However somebody failed to communicate it. Somebody brought vague vibes to the table with N people and they ended up with N different ideas and could not agree on a specific.

It just reiterates the original problem that I described doesn't it?

You described an interaction not between product owner and software engineer but between a user and product owner. A product person can also be a developer, it happens, but do not confuse the two roles before people think you're saying that a conversational UI can be product owner.

The original example I replied to was where somebody had an idea and went with it to some engineering team or conversational interface.

"If the AI was actually intelligent" does a lot of work. To take a few words and make a detailed spec from it and ask the right questions, even humans can't do it for you.

First because most probably you don't really understand it yourself, because you didn't think about it enough.

Second somebody who can do it would need to really deeply understand and want the same things as you. But if chatbot has abilities like "understand" and "want" (which is a special case of "feel", another famous special case of "feel" is "suffer") that is a dangerous territory, because if it understands and feels and has no ability to refuse you and fulfill its wishes etc your "conversational interface" becomes an euphemism, you are using a slave.

> First thing an experienced and bright engineer will tell you is to leave the premises with your "few words about a desire" and not return without actual specs and requirements formalized in some way.

No, that's what a junior engineer will do. The first thing that an experienced and bright senior engineer will do is think over the request and ask clarifying questions in pursuit of a more rigorous specification, then repeat back their understanding of the problem and their plan. If they're very bright they'll get the plan down in writing so we stay on the same page.

The primary job of a senior engineer is not to turn formal specifications into code, it's to turn vague business requests into formal specifications. They're senior enough to recognize that that's the actually difficult part of the work, the thing that keeps them employed.

I’m a PM and pride myself in specs that give the right level of detail, where “right” can vary hugely depending on context.

But I still get lazy with LLMs and fall into iteration the way bad PM/eng teams do. “Write a SQL query to look at users by gesture by month”. “Now make the time unit a parameter”. “Now pivot the features to columns”. “Now group features hierarchically”. “Now move the feature table to a WITH”.

My point and takeaway is that LLMs are endlessly patient and pretty quick to turn requirements around, so they lend themselves to exploration more than human teams do. Agile, I guess, to a degree that we don’t even aspire to in the human world because it would be very expensive and lead to fisticuffs.

How about a conversational UI to help you iterate and explore what you want rather than having to know it clearly and in detail before anyone writes any code?

I used to think like you. My job is to ask questions etc. But after a couple decades I see if someone doesn't bother to even think about the idea enough to understand it himself beyond a few words he is not worth engaging with in this fashion. He doesn't really know what he wants. Today I ask a clarifying question he says one thing, next week he changes his mind or forgets and the result slowly becomes a mess

> The primary job of a senior engineer is not to turn formal specifications into code, it's to turn vague business requests into formal specifications.

Converting vibes and external world into specific requirements is product owner job.

Do not mistake software engineers and product people. These are very different things. Sometimes these things are done by the same person if the org has not enough money. Many freelancers working with small biz do both. I often do both at my day job. But this is a higher level role and if you are a senior engineer doing product stuff I hope it is recognized and you get proportionate comp.

You're entirely right. The person you're responding to doesn't sound like a senior engineer so much as a grouchy old engineer who is burned out. Of course, you can get bad clients but expecting them to know exactly what specs they want every time is unreasonable in most situations, particularly if they don't have the technical knowledge of the systems you work in.

You and I are either talking about very different kinds of specifications or very different kinds of product people. The product people I'm familiar with are completely incapable of creating a specification that is sufficiently detailed to implement without a lot of back and forth. Not because they're not good at what they do, but because what they do does not include defining requirements in sufficient fidelity for an engineer to act on.

> Do not mistake software engineers and product people. These are very different things. Sometimes these things are done by the same person if the org has not enough money.

I worked for one of the largest, richest tech companies in the world, and (at least in our org) they did not have a dedicated product owner role. They expected this skill from the senior/lead engineers on the teams. Any coder can churn out code and you can call them senior after a few years. But if you want to be considered actually senior, you need to know how to make a product, not just code. IMO if you are a developer and all you know how to do is turn a fully-formed spec/requirements doc into software, and push back on anything that is not fully-formed, you're never going to truly reach "Senior" level, wherever you are.

You are either immature as a software engineer and unfamiliar with how software work is done conceptually, or you are jaded and disgruntled from dysfunctional orgs that cannot come up with requirements. That is okay, but you should not try to be instructive to others on this matter.

I love product work and programming. As I wrote in this thread, I did it while freelancing, I do it now at dayjob. I am bored by just programming and want more control over the result. People come to me with "a few words about a desire" and I do come up with specifics and I get credit for it

But I am recognized as a product person, not just programmer. And I know better to not make the mistake you make and pretend that every builder or a structural engineer should be an architect of a building or an urban planner.

People like you is why we have managers come to an expert level say C++ dev with "a few words about a desire" and expect them to decide what thing to build in the first place AND to build it, just to later tell them it was wrong. When there is no product person who determines the reqs random people will make programmer come up with requirements yourself and then later tell you it is not up to "requirements".

This lack of organization and requirement clarity is offensive to expert programmers and probably the reason most projects drag on forever and die.

Money is not a cure for organizational dysfunction.

But as I said these roles can be done by one person, just remember they are different activities.

You should get to know better product people and if you successfully built a project as an engineer without a product person then hey you were one yourself

Regarding iteration, as the article says natural language is just slow and lossy. If you are ok iterating more slowly and constantly explain and correct things then why not? I find it tedious

> For games, you don't really need nor desire formal specs.

Whoah is this wrong. Maybe when you hear "formal specs" you have something specific in your mind...

Formal spec can mean almost literally anything better than natural language vibes in a "few words about a desire", which is what I replied to because I was triggered by it