ArchiveBox is evolving: the future of self-hosted internet archives

I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too.

I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I don't use it for personal content.

I love ArchiveTeam warrior, it's such a good idea! We run several instances ourselves, and it's part of our Good Karma Kit for computers with spare capacity: https://github.com/ArchiveBox/good-karma-kit

There are a bunch of other alternatives like ReadDeck listed on our wiki too, we encourage people to check it out!

https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution.

https://addons.mozilla.org/en-US/firefox/addon/single-file/

I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.)

Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw

And, for comparison, here's the original: https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP

It'd be nice if I could edit out the extra junk near the top, but the important bits are all there.

Singlefile is excellent, Gildas is a great developer. ArchiveBox has had singlefile as one of its extractors built in for years :)

I was about to post a link to the same URL but archived using singleFile, which looks like the original at amazon. I didn‘t because I realized that I have absolutely no idea what additional information would be hidden in the file. In the worst case any component sent by Amazon and archived into the file may contain PII, even if I am “logged out“.

I‘m not saying that singleFile is bad in any way, I‘m using it a lot on multiple devices, but I‘m not sure whether sharing archives is a good idea™.

100%, this is the challenge of archiving logged in content.

It becomes un-shareable unless we use fake burner accounts for capture, or have really good sanitizing methods.

Even when I‘m logged out I expect at least information on my geographical location to seep into the archive via URLs addressing specific CDN endpoints or similar mechanisms.

Yup, this is why the ArchiveBox browser extension sends URLs to a separate server for archiving with an isolated burner profile.

I should write a full article on the security implications at some point, there aren't many good top-down explanations of why this is a hard problem.

I know it’s a lot of work but this would be great and it may give readers a deeper understanding into security in general.

Thank you so much Niki :). The P2P sharing is a great idea. I really hope this feature will get things moving in the archiving field.

I've just tried Readeck and it doesn't save a good quality copy of the pages using the Firefox extension. SingleFile and WebScrapBook do a much better job.

I prefer WebScrapBook because it saves all the assets as files under the original names in a directory rather than a zip file. This means that I can use other tools such as find, grep, and file managers like Nemo to search the archive without needing to rely on the application that saved the page.

Readeck saves a page as a zip file. It's not hard to open from the command line or file manager, just unzip and launch the index.html in the web browser.

But it strips out a lot of detail. Zipping it also means that it's hard to deduplicate. I use WebScrapBook and run rdfind to hardlink all the identical files.

How does it save pages that are only available when you are logged in such as social networking pages?

You set up a chrome profile for archiving that's logged into all the sites you want to save. I recommend using burner accounts dedicated to archiving, so you'd have to add them to any private pages/groups you want to archive.

It is possible to use your main account for archiving but there are security risks (you cant share the snapshots without leaking session headers).

That’s a very cool solution- gives the user explicit control