> someone who is against racial diversity but isn't remotely racist
One way I see this working is a person who believes in individualistic meritocracy with libertarian leanings. In this post I'll try to lay out the point of view of hypothetical person (so not everything I say here represents my own personal beliefs or something I personally agree with).
First, the meritocracy part: Such a person would say that all workplace decisions should be made on a totally race-blind basis: "I don't care what race the people at my startup are, what matters is if they can code (if that's what their job title entails)." If it so happens that our society has relatively few people of X race who can code at the required skill level, then as an inevitable consequence on average startups will employ few people of race X, because there simply aren't enough skilled programmers of race X to go around.
Second, the libertarian part: Sure, it's indisputable that there are a ton of social and economic issues that people of race X encounter at home, in their communities, in school, that end up causing fewer young adults of race X to be coders. But it's certainly not this startup's job to try to fix the upbringing of employees that has resulted in their inadequacy to supply the needed labor. This startup's job is more along the lines of, if a person can't do the work the company needs them to do, they shouldn't be working here.
It's not even the government's job to fix this. When it tries, it only succeeds at wasting resources, turning the people it's trying to help into permanent dependents of the taxpayers, and poisoning the reputation of the actual high achievers of race X because everyone who sees them now assumes "Oh, he/she can't possibly actually be able to do his/her job, the only reason he/she's in that position is there's a quota of minorities to fill so the company doesn't get called out / boycotted / sued for insufficient diversity..."
Third, the individualistic part: Do we really want a society based on the group identities of different races? That seems like a recipe for perpetuating our race problems, not fixing them. If you enshrine "racial diversity" into any kind of official or quasi-official policy, then by definition the policy is treating under-represented races favorably and over-represented races unfavorably.