This is a very tempting and commonly used strategy in Rust to bypass the borrow checker. I've used it to implement tries/DFAs with great success (though I can't find the code anymore)
Often you are bypassing it. E.g. if you rebalance a tree then references into that tree may now be invalid, so the borrow checker will prevent you from doing that while you have live references. But you may also invalidate indices into the tree, and the borrow checker can't help you with that.