If you're near a port of entry, you actually can be stopped at any time to be asked for proof of citizenship/visa/etc. See also
https://www.aclu.org/know-your-rights/border-zone There are plenty of border patrol stops along highways that never cross a border where you'll be routinely stopped.
In practice, that means "near the border with Mexico" where there are border patrol stations along highways well inside the US that do routine stops, but technically it applies to any port city as well.
_Technically_, if you're a US citizen, you're not required to have proof of citizenship on you. But a drivers license in the US does not show that you are a citizen, and they absolutely can stop you from leaving (at least long enough to mess up your day plans) if you cannot actually prove citizenship in practice. It's just a _lot_ easier to have a passport on you. Passport cards exist mostly for this purpose.
E.g. I lived in Texas for a long time and have a passport card for that specific reason. Driving down to the beach or just going on random trips involves passing through border control stations even though you never leave the US or even get particularly near the border. Your chances of just going to visit a friend in south Texas and winding up needing to pass through a border control checkpoint due to unplanned road closures/etc are pretty high, so it's best to have a card in your wallet at all times that counts. Most remote highways have them as soon as you get back near a town. Most of the time, they'll wave you through pretty quickly, but it's just a lot faster process if you have a passport card and not only a drivers license.
Side note: A friend of mine was a park ranger at Carlsbad Caverns. He picked me up at the El Paso airport and we were driving to Carlsbad. We went through a border control checkpoint. They threatened to arrest him on the spot because he had his park service uniform hanging up in the back, but no employee ID or other proof that he actually was a ranger. They said he was impersonating a federal officer. They made a big deal of it and made us wait while they called the park to verify he was employed (and that didn't work because it was about 8pm). We eventually got let go with a warning after he found random online photos of him giving cave tours and convinced them that he couldn't possibly have staged a ton of random folks on the internet to have him in the background of photos.... But still, it was kinda nuts...