←back to thread

420 points rvz | 1 comments | | HN request time: 0.259s | source
Show context
pfraze ◴[] No.41412758[source]
Copying over my latest backend status update; figure folks would find it interesting

Servers are holding up so far! Fortunately we were overprovisioned. If we hit 4mm new signups then things should get interesting. We did have some degradations (user handles entering an invalid state, event-stream crashed a couple times, algo crashed a couple times, image servers hit bad latencies) but we managed to avoid a full outage.

We use an event-sourcing model which is: K/V database for primary storage (actually sqlite), into a golang event stream, then into scylladb for computed views. Various separate services for search, algorithms, and images. Hybrid on-prem & cloud. There are ~20 of the k/v servers, 1 event-stream, 2 scylla clusters (I believe).

The event-stream crash would cause the application to stop making progress on ingesting events, but we still got the writes, so you'd see eg likes failing to increment the counter but then magically taking effect 60 seconds later. Since the scylla cluster and the KV stores stayed online, we avoided a full outage.

replies(9): >>41412984 #>>41413343 #>>41413506 #>>41413569 #>>41415242 #>>41415812 #>>41416225 #>>41417516 #>>41417547 #
geekodour ◴[] No.41413506[source]
hi pfraze, can u tell us a bit more about the golang event stream? does it also trigger the computation of the views periodically?

more precisely wanted to understand how do you generate the event stream from sqlite

replies(1): >>41413605 #
1. pfraze ◴[] No.41413605[source]
In my previous description, I avoided talking about atproto details for clarity, but this is all part of that (atproto.com). The "kv stores" are what we call data repos[1] and they use sqlite for storage, but can produce individual event streams. Those streams flow into the golang event stream, aka the "relay". View computation happens continuously.

1. data repos are actually signed merkle trees, which gives at-rest authentication of the data as it gets shipped across organizational boundaries