We've invented the equivalent of a "calculator for words" and we're going through the growing pains of discovering that putting words together is a separate activity from thinking. We've never needed to conceptualize them as separate activities until now so we don't have the conceptual distinction and language to even describe them that way.
I think proving theorems is a better analogy than rote calculation—like the writing process, it is creative.
While mathematicians don't sit down and smash random symbols together to concoct proofs, many of them will stress the importance of good notation. Chewing on notation a bit can help reveal connections in the hunt for a proof.
When people claim that "writing is thinking" they mean something similar. The free-associative process of writing out our thoughts can help bring an initial clarity that is difficult to achieve without an external medium. Most of the time, we ought to polish those thoughts and connections just as mathematician ought to polish his proof sketch, but there is some extent to which the actual activity does in fact make up a significant portion of the thinking process. This is why offloading all of that activity to LLMs is dangerous. You can judge the string of propositions produced by an LLM, but you haven't thought them.