It’s especially hard when those beliefs have lead you to justify treating other people horribly. If you’ve gone around accusing people of grooming children for abuse because they thought gay or trans people deserved basic human rights, if you’ve said supporting immigration means supporting rapists and murderers, etc. it’s much harder to come back from that than if it was more traditional policy differences like whether we should have a particular tax rate. I think that’s intentional in some cases, just as with cults where pushing extreme claims and breaking outside ties makes it harder to leave.