Pakistani newspaper mistakenly prints AI prompt with the article

This has happened more than once, in different scenarios.

I wonder why is it that GhatGPT (and the rest) don't put the actual response in a box that is separate from the "hello" and "goodbye" part.

I once wrote a message to my landlord, and I asked ChatGPT to help me make the message better (being that english is not my mother tongue) and I included the "goodbye" part by mistake.

LLM's don't have any internal concept of "actual response" vs "surrounding words". Just like they don't have a internal concept of system prompt vs user input. Just like they don't even have an internal concept of what the LLM emitted vs what was given to it! It's all just one long sequence.

(Yes, it is possible to create tokens to represent category changes, but this is still in-band. the token is still just part of the sequence and the LLM isn't guaranteed to factor it in correctly)