Edit. I should add that the humans are not completely one population because of genetic isolation and differential selection (especially over the last 10,000 years), but we are almost a single population. Like everything in genetics it gets fuzzy at the edges.
And the very definition of "intelligence" is incredibly complex and slippery, which is one of the reasons why I've always found trying to summarise such a complex property into a single numerical value such a silly exercise.
More on this here: http://slatestarcodex.com/2014/08/11/does-the-glasgow-coma-s...
NB I say that as someone who got a very high IQ test result - which didn't exactly convince me that IQ tests are a good idea....
Intelligence really isn't "multi-faceted" (read Gardner's own admission that his theory never panned out) and it isn't ill-understood (refer to the Nature or Nurture interview with Nancy Segal on YouTube).
There's two reasons people say that. One is, they fared badly on a test and want to dismiss it, and the other is, they fared well on a test and are bashful about it.
Also, IQ tests are meant to measure a person's intelligence, not to convince them that IQ tests are "a good idea" - for that you would have to study Psychometrics.