Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

My first impression is that this should enable approximately what Apple is doing with their AI strategy (local on-device first, then filling back to a first party API, and finally something like ChatGPT), but for web users. Having it native in the browser could be really positive for a lot of use cases depending on whether the local version can do things like RAG using locally stored data, and generate structured information like JSON.

I don't think this is a terrible idea. LLM-powered apps are here to stay, so browsers making them better is a good thing. Using a local model so queries aren't flying around to random third parties is better for privacy and security. If Google can make this work well it could be really interesting.

Apple might have an advantage given they’ll have custom hardware to drive it, and the ability to combine data from outside the browser with data inside it. But it’s an interesting idea.

Apple may have a bit of a lead in getting it actually deployed end-to-end but given the number of times I've heard "AI accelerator" in reference to mobile processors I'm pretty sure that silicon with 'NPUs' are probably all over the place already, and if they're not, they certainly will be, for better or worse. I've got a laptop with a Ryzen 7040, which apparently has XDNA processors in it. I haven't a damn clue how to use them, but there is apparently a driver for it in Linux[1]. It's hard to think of a mobile chipset launch from any vendor that hasn't talked about AI performance in some regards, even the Rockchip ARM processors seem to have "AI engines".

This is one of those places where Apple's vertical integration has a clear benefit, but even as a bit of a skeptic regarding "AI" technology, it does seem there's a good chance that accelerated ML inference is going to be one of the next battlegrounds for processor mobile performance and capability, if it hasn't started already.

[1]: https://github.com/amd/xdna-driver

For sure many devices will have them, but the trick will be to build this local web model in a way that leverages all of the local chips. Apple’s advantage is in not having to worry about all that. It has a simpler problem and better access to real local data.

Give my personal local data to a model running in the browser? Just feels a bit more risky.

> in a way that leverages all of the local chips

Which, in a way, is similar to building a browser leveraging all of the local GPUs to do render and HW-accelerated video decoding.

Is Safari on Apple Silicon better than Chrome on random Windows laptop for playing YouTube in the last 5 years? Hardly.

The video plays fine on both, yes, but the Windows laptop, generally speaking, gets hotter, and runs out of battery sooner.