I used to implement a system like this. Unfortunately, its not accessible. You at least need to keep a ref to the button in order to add the aria-haspopup attribute and return focus back to the button when the modal closes. This is why Radix requires you to place a trigger component. You're also not handling escape, not to mention integrating with any popover / dropdown library that will also listen to the escape button.
This pattern to invoke a modal is pretty useful though. I'm experimenting with a hook that returns an anchor component in order to implement the accessibility functionality. Not sure if that's a good idea yet:
const [ModalAnchor, showModal] = useModal()
function handleClick() {
showModal(<MyModal />)
}
return <ModalAnchor>
<button onClick={handleClick}>Open</button>
<ModalAnchor>