I hate to keep doing this to you, but I am yet again an existence proof that you're wrong about this. I've tried and played many sports in my life, on again, off again. Baseball and basketball mostly as a child, a bit of tennis. Lettered in high school in cross country, track (where I did hurdles rather than middle distance), basketball, and volleyball when I changed my mind from track. I did intramural soccer and dodgeball in college. Picked up running again in the Army and got into various outdoorsy stuff. Kayaking. Open-water swimming. Multi-day hikes. Alpine mountaineering. Rock climbing. I had terrible injuries through my late 30s and did next to nothing. In my 40s, picked up lifting, eventually got back into running, have recently started to learn how to surf and skateboard.
I can assure you that, in spite of not really mastering or finishing any of these things and being kind of flaky about it, it has at least lead to very good fitness.
In the same vein, I see no reason you can't simply practice and get in the habit of learning and being curious even if you never master a specific craft.