The frequency that you use the term "re-factor" over the term "factor" is often very telling about how you develop your systems. I worked a job one time where the guys didn't even know what factoring was.
Probably many people don't pick up on the word "to factor" something these days. They do not make the connection between the thing that mathematicians do and what that could relate to in terms of writing code. At the same time everyone picks up the buzzword "to refactor". It all depends on what ecosystems you expose yourself to. I think I first heard the term "to factor" something in math obviously, but in software when I looked at some Forth. Most people will never do that, because it is so far off the mainstream, that they have never even heard of it.