This in particular bothers me because I end up having this discussion with family members all the time who are convinced that "chemicals" are bad for you, and they only eat food without "chemicals".
If you can suggest a pithy replacement terminology for "what we really mean" here, I'm sure we'll all adopt it whole-heartedly. Until then, people are going to use the same (easily deciphered) shorthand. Tilting at this particular windmill doesn't actually improve anything or protect anybody.
Also, ‘things are bad for other reasons’ isn’t an argument for trying to eliminate this particular reason.
It's one. Volatile organic compound. VOC.
It doesn't take too long to tally up the O's, actually. But since this is such a math-oriented crowd, you can be sure someone will come along to do the counting for you.
You're conflating the colloquial usage of the word chemical with the naturalistic fallacy. These are two different things, however they interrelated because of our collective failure to embrace the precautionary principle when it comes to novel synthetic compounds. The result is that newer compounds tend to be correlated with less safety testing, simply because less time has been available for testing, testing which isn't typically required before engaging in mass exposure of the public.
There's also a connection between corporate self-interest in covering up safety risks (well documented in history, and presumably also occurring today as well) because synthetic compounds can be patented while natural compounds cannot.
In short the connection is real, but it's more subtle than your simple definition-based logic is giving credit for.