There's more infrastructure under urban roads. Crews come in to fix some utility, shred a section of a lane, patch it poorly with dissimilar materials, and leave.
Though given his definition of quality I expect he is actually ignoring all the real rural roads and only talking about major roads which while they get less traffic than urban roads are maintained to similar standards.
Like the other replies have indicated, I'm not so sure this is the case? I live in very rural northwest Iowa, and while there are certainly plenty of gravel roads around here, I'm only driving on them if I'm intentionally trying to go "off the beaten path." You'll take a gravel road if you live on a farm, or you're trying to get to somewhere secluded such as a lake, campground or maybe a county park; but (imo) it's rare for the average person to drive down a gravel road just going from Point A to Point B on their daily commute.
(Actual answer: I know a bunch of people who live in houses in the middle of seemingly-nowhere in rural Ohio, and almost none of them farm anything at all. They just seem to like the space and the quiet and the desolate isolation.
The only farmer who I know is my parents' neighbor, who has a house few miles away from their place.)
(Some adjustments may have to be made, but that's only another also-irrelevant expense.)
Not in California; we have an entire bureau, the Coastal Commission, that exists to prevent that very thing.