←back to thread

ArchiveBox is evolving: the future of self-hosted internet archives

(docs.sweeting.me)
663 points nikisweeting | 5 comments | 16 Oct 24 16:18 UTC | HN request time: 0s | source

We've been pushing really hard over the last 6mo to develop this release. I'd love to hear feedback from people who've worked on big plugin systems in the past, or anyone who's tried our betas!
Show context
bravura ◴[16 Oct 24 17:57 UTC] No.41861830[source]▶
@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable!

I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026

You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen many cases in the community where solo devs get burned out or have personal challenges that take priority etc.

It's hard for us users to lean in on ArchiveBox when after a happy month of archiving, things start break and you're left with maintaining a branch of your own fixes that aren't in main. Meanwhile, your solution of soliciting one time donations just makes the whole project feel more rickety and fly-by-night. How about thinking bigger?

We NEED ArchiveBox to be a real thing. Decentralized tooling for archiving is SO IMPORTANT. I care about it and I suspect many people do. I'm posting this so other people who care about it can also comment and chime in and suggest how it can become something we can rely on. Because archiving isn't just about the past, it's about the future.

Maybe it needs to be a dev org of three committed part-time maintainers, and a small foundation that people recurrently support is what grants it? IDK. I'm not an expert at how to make open source resilient. There have been discussions about this in the past, but I think it's worth a serious look because ArchiveBox is IMPORTANT and I want it to work any month I decide to re-activate my interest in it. I invite people to discuss ways to make this valuable project more sustianable and resilient.

replies(1): >>41862232 #
nikisweeting ◴[16 Oct 24 18:34 UTC] No.41862232[source]▶
Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space.

It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history.

(I should also mention that if you're a commercial entity relying on ArchiveBox, you can hire us for dedicated support and uptime guarantees. We have a closed source fork that has a much better test suite and lots of other goodies)

replies(4): >>41863824 #>>41863888 #>>41864122 #>>41864323 #
manofmanysmiles ◴[16 Oct 24 21:41 UTC] No.41864122[source]▶
I love this project. I "independently" "invented" it in my head the other day, and happy to see it already exists!

I'd love to see blockchain proof/notary support. The ability to say "content matching this hash existed at this time.

I'm exceptionally busy now but that being said, I may choose to contribute nonetheless.

I'd love to connect directly, and will connect to the Zulip instance later.

If we align on values, I may be able to connect you with some cash. People often call me an "anarchist" or "libertarian", though I'm just me, not labels necessary.

replies(1): >>41865806 #
nophunphil ◴[17 Oct 24 02:15 UTC] No.41865806[source]▶
Can you please explain what you mean by “blockchain proof/notary support”?
replies(1): >>41866018 #
manofmanysmiles ◴[17 Oct 24 02:56 UTC] No.41866018[source]▶
Motivation: Have evidence that some content existed at a particular time. For example, let's say a major website publishes an article, and later they remove it, and there is no record of it ever existing. If I host an ArchiveBox, I can look at it and see "Oh here is that article. Looks line it was published after all." However, why should you believe me I didn't just make it up?

If when I initially archived it, I computed a cryptographic hash of the content and posted that on a blockchain, then at a future date I can at least claim "As of block N, approximately corresponding to this time UTC, content that hashes to this hash exited."

If multiple unrelated parties also make the same claim, it is stronger evidence.

Is this sufficient explanation? I can expand on this more later.

replies(1): >>41866130 #
1. jazzyjackson ◴[17 Oct 24 03:21 UTC] No.41866130[source]▶
There's no reason to believe that the hashed and timestamped content was hosted at a particular domain, however (unless the content was signed by the author of course, then there's no Blockchain necessary). sure multiple peers could make some attestation that they saw it at that URL, but then you're back at square one of the reputation problem

Internet archive as an institution with a reputation that holds up to a judge is actually more valuable than a cryptographic proof that x bytes existed at y time

replies(1): >>41866418 #
2. manofmanysmiles ◴[17 Oct 24 04:26 UTC] No.41866418[source]▶
No, definitely not. I have no inherent reason to trust the people working at the Internet Archive over let's say close friend. For me trust is always a human to human concept, and no amount of tech or institutions will change that.

The more people I hear making a claim, the more I'm likely to deem the claim(s) as true. This is even true regarding the claims that cryptographic algorithms have the properties that make them useful in these contexts. I say this as someone who has even taken graduate level classes with Ron Rivest.

I'm not sure what will happen in a court. I imagine the more people that start making claims using cryptography as part of the supporting evidence, the more likely people will start to trust cryptography as a useful tool for resolving disputes about the veracity of claims.

So you would not get any value from multiple people making such claims?

replies(2): >>41867064 #>>41871800 #
3. nikisweeting ◴[17 Oct 24 06:52 UTC] No.41867064[source]▶
I think the best solution is to have multiple people with reputation attest to the encrypted TLS content without being able to see the cleartext of it, that way they cant easily tamper with it.

See my comments on TLSNotary stuff below...

replies(1): >>41867177 #
4. manofmanysmiles ◴[17 Oct 24 07:14 UTC] No.41867177{3}[source]▶
Woah, cool, yes, exactly this!

I think I read a paper or blog post about this concept a while ago, but never saw it implemented!

5. jazzyjackson ◴[17 Oct 24 17:39 UTC] No.41871800[source]▶
Wow, thanks for sharing your perspective it's quite different from mine. For me reality is not democratic, number of people making a claim doesn't influence the truthiness of it.

I bring up judges because Internet archive captures have been used as evidence in court cases, the first one I pulled up [0] makes an interesting distinction on whether the archive's snapshots are merely hearsay:

  The hearsay rule does not apply to the document (so far as it contains the representation) if the representation was made:

  (a)    by a person who had or might reasonably be supposed to have had personal knowledge of the asserted fact; or ...
The archive's office manager submitted an affidavit to the court as someone who would have personal knowledge of the fact that the date and claimed availability of the content are accurate. There's no cryptography involved, just an individual and an institutions reputation - this carries much more weight than any number of anonymous individuals attesting to a cryptographic proof

[0] https://www.judgments.fedcourt.gov.au/judgments/Judgments/fc...