This is a very tempting and commonly used strategy in Rust to bypass the borrow checker. I've used it to implement tries/DFAs with great success (though I can't find the code anymore)
replies(3):
It's a problem in practice. Of the three times I've ever had to use a debugger on Rust code, two came from code someone had written to do their own index allocation. They'd created a race condition that Rust would ordinary prevent.
But I agree, it does give up some of the benefits of using native references.
I had one bug in a renderer where I'd see object shadows moving across a 3D scene, but not the object casting the shadow. Didn't crash. That was hard to find. A dead index was still in use, and it pointed to something valid enough to be drawn.