←back to thread

379 points mobeigi | 2 comments | | HN request time: 0s | source
Show context
DanielHB ◴[] No.41869510[source]
I want to share a story in a somewhat related topic:

anti web-scraping techniques

The most devious version I ever seen of this, I was baffled, astonished and completely helpless:

This website I was trying to scrap generated a new font (as in a .woff file) on every request, the font had the position of the letters randomly moved around (for example, the 'J' would be in place of the 'F' character in the .woff and so on) and the text produced by the website would be encoded to match that specific font.

So every time you loaded the website you got a completely different font with a completely different text, but for the user the text would look fine because the font mapped it to the original characters. If you tried to copy-and-paste the text from the website you would get some random garbled text.

The only way I could think of to scrap that would have been to OCR the .woff font files, but OCR could easily prevent mass-scraping due to sheer processing costs.

replies(7): >>41869674 #>>41869684 #>>41869775 #>>41869796 #>>41869877 #>>41870330 #>>41871277 #
voldacar ◴[] No.41870330[source]
So it's a Caesar cipher, which is trivial to break. You don't need OCR or any computationally intensive solution.
replies(1): >>41870365 #
NoMoreNicksLeft ◴[] No.41870365[source]
You need OCR unless you're going to personally sit there and break it by hand so you can feed the tr/// translation yourself every time you need to scrape. And it's a bit more tedious than the puzzles we did as kids, likely the punctuation and lowercase/uppercase were mixed into the slop.
replies(1): >>41870387 #
1. connicpu ◴[] No.41870387[source]
If there's a part that doesn't change, eg a footer or something, you can get a head start and have it figure out the rest by deduction with a spellchecker
replies(1): >>41870416 #
2. NoMoreNicksLeft ◴[] No.41870416[source]
You might manage to cobble together frequency analysis too, but that would be challenging. If the ciphertext is very small, or is marketspeak without any sense to its message, then that's going to fall flat. And all this assumes just ascii rather than say a (even limited) unicode font. These assholes could be doing that just to have curly quotes or whatever.