←back to thread

Show HN: 4B+ DNS Records Dataset

(www.merklemap.com)
90 points Eikon | 1 comments | 15 Oct 24 19:26 UTC | HN request time: 0.203s | source

Hi HN,

I've been working on building a pipeline to create a DNS records database lately. The goal is to enable research as well as competitive landscape analysis on the internet.

The dataset for now spans around 4 billion records and covers all the common DNS record types:

    A
    AAAA 
    ANAME
    CAA
    CNAME
    HINFO
    HTTPS
    MX
    NAPTR
    NS
    PTR 
    SOA
    SRV
    SSHFP
    SVCB
    TLSA
    TXT
Each line in the CSV file represents a single DNS record in the following format: www.example.com,A,93.184.215.14

Let me know if you have any questions or feedback!

Show context
genmud ◴[16 Oct 24 00:34 UTC] No.41854522[source]▶
Neat! How is this different than domaintools/farsight [1]?

Passive DNS [2] has been in my toolbox for 15+ years, and is invaluable for security research / threat intelligence. Knowing historical resolutions to something are so helpful in investigations.

For anyone interested, they should check out the talk by one of the DomainTools people [3] on how it can be utilized for investigation.

Are you passively collecting this data, or actively querying for these records?

[1] - https://www.domaintools.com/products/threat-intelligence-fee...

[2] - https://www.circl.lu/services/passive-dns/

[3] - https://www.youtube.com/watch?v=oXmapqLkZd0

replies(2): >>41855606 #>>41861727 #
Eikon ◴[16 Oct 24 04:29 UTC] No.41855606[source]▶
From what I understand [1] is just tlds, not subdomains?
replies(1): >>41855837 #
genmud ◴[16 Oct 24 05:17 UTC] No.41855837[source]▶
That would be incorrect, they get subdomains for passive dns feeds.
replies(1): >>41857058 #
1. Eikon ◴[16 Oct 24 09:08 UTC] No.41857058[source]▶
Ok, it'd be interesting to know how big is their datasets compared to mine and how much they overlap.