Tell that to all the black new yorkers who had their guns and/or weed stolen by the government during the stop and frisk era.
Having the ability to stop people isn't worth much if you don't have the unaccountable bureaucracy that can unilaterally doll out whatever punishment it wants.
The usefulness of these sort of fishing exercises regardless of what kind of violation is being gone after is wholly predicated on the upstream system denying these people their rights. If enforcement bureaucracies had to give everyone subject to "serious punishment" (whatever we agree that is, incarceration likely fits it) real rights, follow real protocols involving warrants, evidince, etc, etc. and potentially drag them in front of a jury then you wouldn't have any of this.
And it's not just limited to ICE. Pretty much every other law (and I don't just mean criminal, civil too) enforcement excess evaporates overnight if people get full protection afforded by their rights after law enforcement interacts with them.
Now, you could argue that we should instead impose the constraints on law enforcement contact and then all these unaccountable systems that violate people's rights will be fine because the enforcers won't be shoving innocent people into them, but looking at the last ~50yr of executive branch (not just feds, states very much too) history, that's basically what we've done and it clearly hasn't worked.
This isn't a brown people problem. This is a bigger fundamental problem with how we've architected law (once again, not just criminal) enforcement in this country. The brown people are just the newsworthy implementation at this particular minute.