←back to thread

91 points jchanimal | 1 comments | | HN request time: 0.229s | source

Hi, HN! As a cofounder of Couchbase, I pioneered mobile sync, and I’ve always wanted to bring the speed and reliability of local-first data to the web, incubating PouchDB among other efforts. I learned the constraints of real world financial applications at McKinsey & Company FinLab, and Merkle integrity research at Protocol Labs taught me smart contract data structures. As part of the JavaScript community (and early hosting provider for NPM) I’ve been waiting, and now with the availability of APIs like Passkeys and Origin Private Filesystem, I’m happy to say the browser is ready to support embedded databases.

Front-ends are a lot easier to write when your database handles live sync for you, but the existing solutions rely on heavyweight cloud APIs instead of putting the smarts at the edge, where it belongs. I started from a different set of constraints, and arrived at a lightweight embedded database that uses a git-like data model to offer cryptographic causal consistency across browsers, edge functions, and anywhere TypeScript runs.

It’s designed to make building full-featured apps as simple as calling `db.put({ hello: "world" })` and syncing them as easy as calling `connect(db, remote)`. People are using Fireproof for AI character chat[1], personal finance[2], and hedge funds[3], and we aim to be simple enough for novice coders to build enterprise-critical apps. Fireproof makes product owners dangerous, because just a little bit of code can define an application’s workflow and data model. See the code sample below.

The reactive APIs[4] are designed for live collaboration so your user interfaces update automatically, making it an easy way to add query collaboration to legacy dashboards, or write new interactive tools for your team. Merkle CRDTs[5] provide multi-writer safety while maintaining tamperproof data provenance, conflict tracking, and deterministic merges. The storage engine writes content-addressed encrypted files that can be synced via commodity backends like S3 or Cloudflare[], without sacrificing data integrity.

Our contributors include legends like Damien Katz, Meno Abels, Mikeal Rogers, and Alan Shaw. Fireproof is open source (Apache/MIT) and we know there are rough edges, so we hope this post stirs up collaborators![6] Please `npm install @fireproof/core` and give us feedback[7]. We are on the stable side of beta, so it’s a great time for the adventurous to join. I’m excited to see all the apps people write now that it’s easy!

[1] https://github.com/fireproof-storage/catbot/tree/main

[2] https://fireproof.storage/posts/quickcheck:-print-checks-at-...

[3] https://fireproof.storage/posts/contributor-spotlight:-danie...

[4] https://use-fireproof.com/docs/react-tutorial

[5] https://fireproof.storage/posts/remote-access-crdt-wrapped-m...

[6] https://github.com/fireproof-storage/fireproof/issues

[7] https://discord.gg/DbSXGqvxFc

Show context
bosky101 ◴[] No.42190661[source]
The website and example, and usage looks clean. Kudos! I have some questions around what's happening under the hood that werent evident from an initial read of both your website as well as GitHub.

1. Does subscribe listen for new changes on a transient server(just a queue). Or from a more persistent store?

2. Where do the events persist? I didn't see a connector to postgres. I did see one for s3.

3. What is the default persistence layer you are advocating?

4. Let's say you run 3 instances of the self hosted server. And a random one of them gets a teacher. And 2 random other students gets load balanced to two other servers. How does the teacher get all messages? What's the thread in a distributed setting

5. How do you filter only messages. Eg: only since time T.

6. Pagination / limits to avoid any avalanche?

7. Auth? Custom auth/jwt?

8. REST API to produce?

9. Are consumers restricted to browsers? What about one in node?

10. BONUS: Have you tested if this works embedded as an iframe or embedded in an native/react native mobile app?

replies(2): >>42195875 #>>42196225 #
1. jchanimal ◴[] No.42196225[source]
I answered 1-5 in my other reply, hit save and so here's more:

6. There is a limit parameter on the query API, and the underlying data structures utilize async iterator patterns so we have a go-forward path to a streaming (data / query larger than memory) implementation. But for now the decrypt implementation is eager instead of lazy, so that's the first place we'd want to focus to make data > memory workloads no problem.

7. Other embedded databases don't have auth, but we are network-aware so it's a different ballgame. Our next step is read/write access control on a per-ledger basis.

UCAN capability delegation allows us to keep an embedded mindset here, in that authorization becomes a matter of data validity, not something that has to be fetched from a centralized resources. How it works: client device agents generate non-extractable keypairs (like Passkeys) and can link them to account principals via any signing endpoint Fireproof trusts (for starters just the one we run, to an end user it looks like clicking a validation link in an email.) Agents create a new cloud database clock register by locally generating an ephemeral keypair that signs itself over to the principal. Our centralized clock register endpoint only allows updates to the resource identified by the clock's public key ID, from agents which have a valid signed delegation chain to the ephemeral key.

To a developer it will look something like `db.share("bob@example.com")` and now Fireproof Cloud will let Bob read and/or write the db also.

What's cool about this is that access control changes are just data manipulations, so they can happen offline. And the valid delegations can be safely delivered over any channel. In fact there are no secrets in this system except for the non-extractable keypairs.

If you are thinking to yourself "what about revocation?" -- we are hiring.

8. The sync endpoint has the minimal blob k/v (no list) and register APIs. And can all be floated on top of any raw kv with check-and-set semantics if needed.

We have plans for a REST API in Fireproof Cloud, where if you allow the cloud to decrypt and process your data, we can give you raw queries instead of you replicating and then querying locally. I am thinking a CSV output here would be a good place to start.

9. Runs great anywhere JS runs. We have examples (like CatBot linked above) that subscribe to the ledger on the backend and operate locally, often responding the user events. So the DB is acting as an RPC bus... this is a common pattern in CouchDB so I made sure Fireproof works great like that.

To run in an edge function, you usually aren't gonna replicate to local filesystem, instead you can configure the database to read and write directly with the cloud store. Because of the eager decrypt we do, this is actually pretty fast and not that chatty.

10. The CodePen demo on our homepage is an iframe, works great. We have a contributor (I think I see in the thread here) who is working on React Native -- most of the heavy lift is done, but our gateway interface is only now settling down to where it makes sense to finalize the integration. I have also done Socket Supply for mobile and that works great.