←back to thread

Use DuckDB-WASM to query TB of data in browser

(lil.law.harvard.edu)
224 points mlissner | 1 comments | 31 Oct 25 17:37 UTC | HN request time: 0.222s | source
Show context
mlissner ◴[31 Oct 25 17:37 UTC] No.45774572[source]▶
OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser.

Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

replies(4): >>45775637 #>>45775690 #>>45775706 #>>45780371 #
timeflex ◴[31 Oct 25 19:15 UTC] No.45775637[source]▶
S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.
replies(3): >>45775880 #>>45777187 #>>45777948 #
1. 7952 ◴[31 Oct 25 22:02 UTC] No.45777187[source]▶
I think this approach makes sense for services with a small number of users relative to the data they are searching. That just isn't a good fit for a lot of hosted services. Think how much that TB's of data would cost on Algolia or similar services.

You have to store the data somehow anyway, and you have to retrieve some of it to service a query. If egress costs too much you could always change later to put the browser code on a server. Also it would presumably be possible to quantify the trade-off between processing the data client side and on the server.