←back to thread

Show HN: 4B+ DNS Records Dataset

(www.merklemap.com)
90 points Eikon | 4 comments | 15 Oct 24 19:26 UTC | HN request time: 0s | source

Hi HN,

I've been working on building a pipeline to create a DNS records database lately. The goal is to enable research as well as competitive landscape analysis on the internet.

The dataset for now spans around 4 billion records and covers all the common DNS record types:

    A
    AAAA 
    ANAME
    CAA
    CNAME
    HINFO
    HTTPS
    MX
    NAPTR
    NS
    PTR 
    SOA
    SRV
    SSHFP
    SVCB
    TLSA
    TXT
Each line in the CSV file represents a single DNS record in the following format: www.example.com,A,93.184.215.14

Let me know if you have any questions or feedback!

1. 35mm ◴[16 Oct 24 05:26 UTC] No.41855885[source]▶
How often is it updated?

Does it include expired domains?

replies(1): >>41857250 #
2. Eikon ◴[16 Oct 24 09:45 UTC] No.41857250[source]▶
> How often is it updated?

I plan to do 2 releases a month for now, goal is one a day.

> Does it include expired domains?

Yes.

replies(1): >>41862029 #
3. mh- ◴[16 Oct 24 18:18 UTC] No.41862029[source]▶
This is fantastically valuable, especially if you can add the first/last-seen as requested by another commenter. Thanks for doing this.
replies(1): >>41862091 #
4. Eikon ◴[16 Oct 24 18:23 UTC] No.41862091{3}[source]▶
Thanks.

That's quite a fun project!