That's a problem in San Francisco proper too. If I think about my trips to Japan, in the population centers, at
best, a car trip will take about the same time as a public transit trip, but the majority of the time transit will be faster, sometime significantly so.
But inside SF, even during rush hour, it'll still nearly always be faster for me to drive (or get an Uber). The reason is because there's precious little transit infra that doesn't share the road to some extent, and even when there are dedicated bus lanes or off-street train tracks, there's still traffic lights, and the buses and trains are slow and make enough stops that any gains are lost. Then on top of that, transfers take time, and if you're even slightly off on your timing, you might have to wait for up to 15 minutes for the next bus at your transfer point.
I agree with your assessment of inter-city trips as well; SFO airport to my house in SF is also so frustrating, because I live a few blocks from a Caltrain station, but having to go from SFO->BART->Caltrain->home... that transfer in the middle is a killer. My home is only a few blocks farther to the freeway than to the train station, so even in rush-hour traffic it's still only a 20 minute drive, while BART+Caltrain will take 30-45 minutes, and that's during a time of day the Caltrain trains run at their most frequent.
I've lived in SF for 15 years, and I think I've only taken BART/Caltrain to or from SFO a handful of times. I can't even remember the last time; it's been at least 10 years (probably before Uber/Lyft was a thing). Nowadays I always take a Lyft, and while I cringe at the price ($30-$50, depending on time of day), it's so worth it when a) I'm worried about not making my flight if transit is slower than I expect, or b) I'm getting back home and just want to be home.
And yeah, I get that I'm privileged enough to be able to afford to take a car. Many people aren't; they have to pay with their time, which just really sucks. We never get that time back.