I discovered that VSCode has a very nice solution so I pulled the core VSCode libraries and injected them into a Chrome extension using the dependency injection, ipc / rpc, eventing to bridge the gap between all of these isolated JS contexts and expose a single, strongly‐typed messaging API, my IPC/RPC shim sits on top of each of the native environments and communication mechanisms.
Yesterday, Microsoft released the source code for the Copilot chat. Apparently, since the basis of my Chrome extension is the same core libraries I can drop the VSCode chat UI into the side panel without much friction. Although, I might continue to use Microsoft's FluentUI chat currently implemented in the extension.
Because Copilot chat has a lot of code that runs in node in Electron, now I'm working in porting all the agent capabilities for browser automation from the Copilot chat including the code for intent, prompt creation, tools, disambiguation, chunking, embedding, ect. I'm 4 to 6 weeks away from having feature parity of Playwright for automation from a Chrome extension side panel that can do most of the inference using huggingface transformer.js locally. Nonetheless, heuristics exposed as tools such that if the intent is playing a video, all that is required is a tool that collects all the video tags and related elements with metadata. No need to use $10 in tokens to figure out which video element to play.
Yeah, I think I'm 4 to 6 weeks away from having a Copilot chat in a browser doing agent automation.
If you want to see where I'm at today, https://github.com/adam-s/doomberg-terminal.