One thing that makes me nervous: Home Assistant has an extremely weak security model. There is recent support for admin users, and that’s about it. I’m sort of okay with the users on an installation having effectively unrestricted access to all entities and actions. I’m much less okay with an LLM having this sort of access.
An actually good product in this space IMO needs to be able to define specific sets of actions and allow agents to perform only the permitted actions.