I don't get this interpretation of "free". Would you say that one should be "free" to kill someone else?
Nobody wants absolute freedom. We all want some set of rules (e.g. "You should not be allowed to burn my house for fun"). Of course, we may want rules that benefit us personally ("Taxes should be paid to me personally, not to the country"), but that obviously doesn't work (if taxes are paid exclusively to me, they can't be paid exclusively to you).
So as a group, we agree on a set of rules that benefits society the most. We want to "maximize the global utility", if I can say it like this.
If "not having unicorns" is better for the society at large than "having unicorns", then it works. And your short-sighted, convenient understanding of "freedom" doesn't change that.