Antifa is just short for Anti-Fascist. Their only goal is stopping fascism. If you see them as a threat, you're saying that you are a fascist.
Also, it's not an organisation, but a mindset that all people who oppose fascism share. And most of those oppose fascism in peaceful ways that don't hurt anyone.