←back to thread

Writing "/etc/hosts" breaks the Substack editor

(scalewithlee.substack.com)
603 points scalewithlee | 1 comments | 25 Apr 25 13:48 UTC | HN request time: 0s | source
Show context
netsharc ◴[25 Apr 25 14:24 UTC] No.43793903[source]▶
Reminds me of an anecdote about an e-commerce platform: someone coded a leaky webshop, so their workaround was to watch if the string "OutOfMemoryException" shows up in the logs, and then restart the app.

Another developer in the team decided they wanted to log what customers searched for, so if someone typed in "OutOfMemoryException" in the search bar...

replies(2): >>43793976 #>>43795364 #
PhilipRoman ◴[25 Apr 25 16:18 UTC] No.43795364[source]▶
Careless analysis of free-form text logs is an underrated way to exploit systems. It's scary how much software blindly logs data without out of band escaping or sanitizing.
replies(1): >>43796073 #
ycombinatrix ◴[25 Apr 25 17:05 UTC] No.43796073[source]▶
Why would someone "sanitize" OutOfMemoryException out of their logs? That is a silly point to make.
replies(3): >>43796243 #>>43796381 #>>43802850 #
teraflop ◴[25 Apr 25 17:32 UTC] No.43796381[source]▶
The point is not to sanitize known strings like "OutOfMemoryException". The point is to sanitize or (preferably) escape any untrusted data that gets logged, so that it won't be confused for something else.
replies(1): >>43796721 #
swyx ◴[25 Apr 25 18:02 UTC] No.43796721[source]▶
i think GP's point is how would you even sanitize the string "OutOfMemoryException" which presumably comes from a trusted system

i guess demanding "Structured logs for everything or bust" is the answer? (i'm not a big o11y guy so pardon me if this is obvious)

replies(2): >>43797095 #>>43797161 #
PhilipRoman ◴[25 Apr 25 18:36 UTC] No.43797095[source]▶
Low tech example: escape all newlines in user supplied strings, then add a known prefix to all user supplied data (let's say a double hashtag ##, but anything else works too). When you want to search logs for strings coming from your system, remove/ignore everything after the marker.

It all comes down to understanding whether the intersection of two grammars is empty.

replies(1): >>43797855 #
1. jethro_tell ◴[25 Apr 25 19:47 UTC] No.43797855[source]▶
The difficulty here is that in the example above, it's unlikely, given any amount of scale, that the two people were on the same team. They were doing different things with the same data and probably didn't know what the other was doing.

Sure you could add a convention to your 'how to log' doc that specifies that all user input should be tagged with double '#' but who reads docs until things break? convention is a shitty way to make things work.

There's 100 ways that you could make this work correctly. Only restarting on a much more specific string, i.e. including the app name in the log line etc . . . but that's all just reducing the likely hood that you get burned.

I've also written a OOM-Killer.sh myself, I'm not above that, but it's one of those edge cases that's impossible to do correctly, which is why parsing and acting on log data generally considered and anti-pattern.