←back to thread

ArchiveBox is evolving: the future of self-hosted internet archives

(docs.sweeting.me)
663 points nikisweeting | 3 comments | 16 Oct 24 16:18 UTC | HN request time: 0s | source

We've been pushing really hard over the last 6mo to develop this release. I'd love to hear feedback from people who've worked on big plugin systems in the past, or anyone who's tried our betas!
Show context
toomuchtodo ◴[16 Oct 24 16:53 UTC] No.41861236[source]▶
https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to enable a distributed archive. CDX indexes can be used for rapidly querying the underlying WARC archives.

You might support cryptographically signing WARC archives; Wayback is particular about archive provenance and integrity, for example.

https://www.loc.gov/preservation/digital/formats/fdd/fdd0005... ("CDX Internet Archive Index File")

https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... ("WARC, Web ARChive file format")

https://github.com/internetarchive/wayback/tree/master/wayba... ("Wayback CDX Server API - BETA")

replies(3): >>41861288 #>>41861743 #>>41861951 #
nikisweeting ◴[16 Oct 24 16:58 UTC] No.41861288[source]▶
I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ.

ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd...)

replies(4): >>41861315 #>>41861428 #>>41864927 #>>41880475 #
toomuchtodo ◴[16 Oct 24 17:13 UTC] No.41861428[source]▶
Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.
replies(1): >>41861514 #
1. nikisweeting ◴[16 Oct 24 17:23 UTC] No.41861514[source]▶
The idea with TLSNotary is that you can have several universities or central agencies running signing servers but you dont have to share the cleartext content of your archives with them to get it signed.

This dramatically changes what is possible with signing because previously to get ArchiveTeam's signature of approval, they would have to see the content themselves to archive it. With TLSNotary they can sign without needing to see the content/access the cookies/etc.

replies(1): >>41862865 #
2. viraptor ◴[16 Oct 24 19:27 UTC] No.41862865[source]▶
Isn't that already possible with any kind of notary by giving them a sha256 of the content only? Or am I missing some distinction?
replies(1): >>41863381 #
3. nikisweeting ◴[16 Oct 24 20:15 UTC] No.41863381[source]▶
You can do that but it proves nothing because TLS session keys are symmetric, so the archiver can forge server responses and falsely attest that the server sent them.

Look up "TLS non repudiation"

A real solution like TLSNotary involves a neutral, reputable third party that can't see the cleartext attesting to the cyphertext using a ZK proof.

The neutral third party doing attestation can't see the content so they can't easily tamper with it, and attempts to tamper indiscriminately would be easily detected and ding their reputation.