the ideal is that college should be very expensive for rich people and cheap, free, or at least more affordable, for less wealthy people.Dunno where you got this "ideal".
the days of outrageous student debt are thankfully fading away
..."fading away", to the tune of (at last glance ) one and three quarters of a trillion dollars in outstanding student loan debt.
it would make more sense to do this redistribution through taxes if possible
The ability of US higher ed to raise tuition prices will always overwhelm the ability of US taxpayers to meet those prices. The phrase "utility monster" comes to mind.
but many US institutions are private so that doesn’t really work.
Private, in the sense that nobody who answers to someone who must win an election is directly in charge of running them, but, who operate as charities for the purpose of donations, pay no taxes on either capital gains or real estate, and are permitted to act as government contractors skimming up to 85% of grant money they're tasked with administrating.
so the colleges basically have their own privately-run means testing programs, and like all such programs there are flaws and loopholes.
The flaw being that...the school is allowed to have total knowledge of a customer's ability to pay before it chooses to do business with them. Imagine if you had to give three years of your tax returns to the person you were trying to buy a house from.