←back to thread

Ask HN: How did the internet discover my subdomain?

287 points govideo | 2 comments | 06 Mar 25 22:34 UTC | HN request time: 0s | source

I have a domain that is not live. As expected, loading the domain returns: Error 1016.

However...I have a subdomain with a not obvious name, like: userfileupload.sampledomain.com

This subdomain IS LIVE but has NOT been publicized/posted anywhere. It's a custom URL for authenticated users to upload media with presigned url to my Cloudflare r2 bucket.

I am using CloudFlare for my DNS.

How did the internet find my subdomain? Some sample user agents are: "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7; en-us) AppleWebKit/534.20.8 (KHTML, like Gecko) Version/5.1 Safari/534.20.8", "Mozilla/5.0 (Linux; Android 9; Redmi Note 5 Pro) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.89 Mobile Safari/537.36",

The bots are GET requests which are failing, as designed, but I'm wondering how the bots even knew the subdomain existed?!

Show context
parliament32 ◴[07 Mar 25 00:15 UTC] No.43286370[source]▶
Certificate Transparency logs, or they don't actually know the domain name: just port-scanning[1] then making requests to open web ports.

[1] Turns out you can port-scan the entire internet in under 5 minutes: https://github.com/robertdavidgraham/masscan

replies(3): >>43286494 #>>43287497 #>>43287503 #
andix ◴[07 Mar 25 00:37 UTC] No.43286494[source]▶
Port scanning usually can't discover subdomains. Most servers don't expose the of the domains they server content for. In case of HTTP they usually only serve the subdomain content if the Host: request-header includes it.
replies(4): >>43286515 #>>43286516 #>>43286524 #>>43292643 #
1. benfortuna ◴[07 Mar 25 00:44 UTC] No.43286516[source]▶
I could be wrong, but the Palo Alto scanner says it's using global ipv4 space, so not using DNS at all. So actually the subdomain has not been discovered at all.
replies(1): >>43287624 #
2. reactordev ◴[07 Mar 25 05:46 UTC] No.43287624[source]▶
This is exactly what’s happening based on the log snippet posted. Has nothing to do with subdomains, has everything to do with it being on the internet.