I have done this in various teams, often where I'm the manager or team lead.
You end up being an "external" person. They'll make plans without you. The primary communication channel is always in person. As a colleague put it, you end up the drunk person in the group who's going, "huh??? what are we doing??" while the rest of the team feels like they have to drag you along. You may well be the most productive person on the team, but people tend to resent accomodating you.
I do believe the entire team remote is better than the whole team onsite, but it's also a lot more work. It's like adopting a framework; it may be better or worse in the long run but it's complex and requires some investment, and you can't have one person running on a different framework.
We've also had things like drama and internal politics - these are always much better handled in person. As in we've had corrupt assistant managers just completely screw up the senior devs, but because that assistant manager met with us in person, we'd favor them instead of the people actually doing the work.