A major con of the enterprisey ilom systems (such as the idrac) is their atrocious security track record. You are basically giving up your "the network is untrusted, I can survive its compromise" badge if you plug in one of those.
Well those ports should never face the internet anyway. Most servers will have a dedicated (physical) port you use for IPMI or whatever -- vlan that and only allow access from your VPN. If you're extra secure you can full on disable the switchport until you need it.