Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

So it's loading an instruct model for inference? That seems a fair bit less useful than a base model, at least for more advanced use cases.

What about running LoRAs, adjusting temperature, configuring prompt templates, etc? It seems pretty early to build something like this into the browser. The technology is still changing so rapidly, it might look completely different in 5 years.

I'm a huge fan of local AI, and of empowering web browsers as a platform, but I'm feeling pretty stumped by this one. Is this a good inclusion at this time? Or is the Chrome team following the Google-wide directive to integrate AI _everywhere_, and we're getting a weird JS API as a result?

At the very least, I hope to see the model decoupled from the interface. In the same way that font-family loads locally installed fonts, it should be pluggable for other local models.

The base model can be found on HF (https://huggingface.co/wave-on-discord/gemini-nano) and run in any web browser using MediaPipe on WebGPU: https://x.com/niu_tech/status/1807073666888266157

As for temperature and topK, you can set them in the AITextSessionOptions object as an argument to `window.ai.createTextSession(options)` (source: https://source.chromium.org/chromium/chromium/src/+/main:thi...)

You should also be able to set it by adding the switches: `chrome --args --enable-features=OptimizationGuideOnDeviceModel:on_device_model_temperature/0.5/on_device_model_topk/8` (source: https://issues.chromium.org/issues/339471377#comment12)

The default temperature is 0.8 and default topK is 3 (source: https://source.chromium.org/chromium/chromium/src/+/main:com...)

As for LoRA, Google will provide a Fine-Tuning (LoRA) API in Chrome: https://developer.chrome.com/docs/ai/built-in#browser_archit...

Appreciate the info and links.