Andrew Ng: Building Faster with AI [video]

Not sure why this has drawn silence and attacks - whence the animus to Ng? His high-level assessments seem accurate, he's a reasonable champion of AI, and he speaks credibly based on advising many companies. What am I missing? (He does fall on the side of open models (as input factors): is that the threat?)

He argues that landscape is changing (at least quarterly), and that services are (best) replaceable (often week-to-week) because models change, but that orchestration is harder to replace, and that there are relatively few orchestration platforms.

So: what platforms are available? Are there other HN posts that assess the current state of AI orchestration?

(What's the AI-orchestration acronym? not PAAS but AIOPAAS? AOP? (since aspect-oriented programming is history))

> So: what platforms are available?

I couldn't tell you, but what I can contribute to that discussion is that orchestration of AI in its current form would focus on one of two approaches: consistent output despite the non-deterministic state of LLMs, or consistent inputs that leans into the non-deterministic state of LLMs. The problem with the former (output) is that you cannot guarantee the output of an AI on a consistent basis, so a lot of the "orchestration" of outputs is largely just brute-forcing tokens until you get an answer within that acceptable range; think the glut of recent "Show HN" stuff where folks built a slop-app by having agents bang rocks together until the code worked.

On the input side of things, orchestration is less about AI itself and more about ensuring your data and tooling is consistently and predictably accessible to the AI such that the output is similarly predictable or consistent. If you ask an AI what 2+2 is a hundred different ways, you increase the likelihood of hallucinations; on the other hand, ensuring the agent/bot gets the same prompt with the same data formats and same desired outputs every single time makes it more likely that it'll stay on task and not make shit up.

My engagement with AI has been more of the input-side, since that's scalable with existing tooling and skillsets in the marketplace instead of the output side, which requires niche expertise in deep learning, machine learning, model training and fine-tuning, etc. In other words, one set of skills is cheaper and more plentiful while also having impacts throughout the organization (because everyone benefits from consistent processes and clean datasets), while the other is incredibly expensive and hard to come by with minimal impacts elsewhere unless a profound revolution is achieved.

One thing to note is that Dr. Ng gives the game away at the Q&A portion fairly early on: "In the future, the people who are the most powerful are the people who can make computers do exactly what you want it to do." In that context, the current AI slop is antithetical to what he's pitching. Sure, AI can improve speed on execution, prototyping, and rote processes, but the real power remains in the hands of those who can build with precision instead of brute-force. As we continue to hit barriers in the physical capabilities of modern hardware and wrestle with the effects of climate change and/or poor energy policies, efficiency and precision will gradually become more important than speed - at least that's my thinking.

> The problem with the former (output) is that you cannot guarantee the output of an AI on a consistent basis

Do you mean you cannot guarantee the result based on a task request with a random query? Or something else? I was under the impression that LLMs are very deterministic if you provide a fixed seed for the samplers, fixed model weights, and fixed context. In cloud providers you can't guarantee this because of how they implement this (batching unrelated requests together and doing math). Now you can't guarantee the quality of the result from that and changing the seed or context can result in drastically different quality. But maybe you really mean non-deterministic but I'm curious where this non-determinism would come from.

> I was under the impression that LLMs are very deterministic if you provide a fixed seed for the samplers, fixed model weights, and fixed context.

That's all input-side, though. On the output side, you can essentially give an LLM anxiety by asking the exact same question in different ways, and the machine doesn't understand anymore that you're asking the exact same question.

For instance, take one of these fancy "reasoning" models and ask it variations on 2+2. Try two plus two, 2 plus two, deux plus 2, TwO pLuS 2, etc, and observe its "reasoning" outputs to see the knots it ties itself up in trying to understand why you keep asking the same calculation over and over again. Running an older DeepSeek model locally, the "reasoning" portion continued growing in time and tokens as it struggled to provide context that didn't exist to a simple problem that older/pre-AI models wouldn't bat an eye at and spit out "4".

Trying to wrangle consistent, reproducible outputs from LLMs without guaranteeing consistent inputs is a fool's errand.