When imperfect systems are good: Bluesky's lossy timelines

(jazco.dev)

Show context

pornel ◴[19 Feb 25 22:25 UTC] No.43108545[source]▶

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).

When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.

replies(5): >>43108664 #>>43108812 #>>43109007 #>>43110207 #>>43113811 #

1. VWWHFSfQ ◴[20 Feb 25 01:59 UTC] No.43110207[source]▶

>>43108545 #

At some point they'll end up just doing the Bieber rack [1]. It's when a shard becomes so hot that it just has to be its own thing entirely.

[1] - https://www.themarysue.com/twitter-justin-bieber-servers/

@bluesky devs, don't feel ashamed for doing this. It's exactly how to scale these kinds of extreme cases.

replies(4): >>43112987 #>>43113751 #>>43113906 #>>43116827 #

2. stavros ◴[20 Feb 25 10:05 UTC] No.43112987[source]▶

>>43110207 (TP) #

Given that BlueSky is funded by Twitter, I'm assuming they know a lot more than us on how Twitter architects systems.

3. genewitch ◴[20 Feb 25 12:10 UTC] No.43113751[source]▶

>>43110207 (TP) #

I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot.

So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time.

I should make a blog post to record all the ones I can remember.

replies(1): >>43117216 #

4. bitbckt ◴[20 Feb 25 12:28 UTC] No.43113906[source]▶

>>43110207 (TP) #

We never actually had a literal “Bieber Box”, but the joke took off.

Hot shards were definitely an issue, though.

5. Imustaskforhelp ◴[20 Feb 25 16:38 UTC] No.43116827[source]▶

>>43110207 (TP) #

Its so crazy.

Thanks a lot for sharing this link.

6. k1t ◴[20 Feb 25 17:04 UTC] No.43117216[source]▶

>>43113751 #

TV Pickup aka the Half Time Kettle Effect.

https://en.wikipedia.org/wiki/TV_pickup

↑