When imperfect systems are good: Bluesky's lossy timelines

I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).

When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.

This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.

Why do they "insert" even non-celebrity posts into each follower's timeline? That is not intuitive to me.

This problem is discussed in the beginning of the Designing Data-Intensive Applications book. It's worth a read!

To serve a user timeline in single-digit milliseconds, it is not practical for a data store to load each item in a different place. Even with an index, the index itself can be contiguous in disk, but the payload is scattered all over the place if you keep it in a single large table.

Instead, you can drastically speed up performance if you are able to store data for each timeline somewhat contiguously on disk.

That's insightful. Keep up the good work!

At some point they'll end up just doing the Bieber rack [1]. It's when a shard becomes so hot that it just has to be its own thing entirely.

[1] - https://www.themarysue.com/twitter-justin-bieber-servers/

@bluesky devs, don't feel ashamed for doing this. It's exactly how to scale these kinds of extreme cases.

Think of it as pre-rendering. Of pre-rendering and JIT collecting, pre-rendering means more work but it's async, and it means the timeline is ready whenever a user requests it, to give a fast user experience.

(Although I don't understand the "non-celebrity" part of your comment -- the timeline contains (pointers to) posts from whoever someone follows, and doesn't care who those people are.)

Do you know the name of the problem or strategy used for solving the problem? I'd be interested in looking it up!

I own DDIA but after a few chapters of how database work behind the scenes, I begin to fall asleep. I have trouble understanding how to apply the knowledge to my work but this seems like a useful thing with a more clear application.

Given that BlueSky is funded by Twitter, I'm assuming they know a lot more than us on how Twitter architects systems.

I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot.

So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time.

I should make a blog post to record all the ones I can remember.

> and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline

I think then you still have the 'weird user who follows hundreds of thousands of people' problem, just at read time instead of write time. It's unclear that this is _better_, though, yeah, caching might help. But if you follow every celeb on Bluesky (and I guarantee you this user exists) you'd be looking at fetching and merging _thousands_ of timelines (again, I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users).

Given the nature of the service, making read predictably cheap and writes potentially expensive (which seems to be the way they've gone) seems like a defensible practice.

Yes, we used the Yahoo! “Feeding Frenzy” paper as the basis for the design of Haplocheirus (the timeline service).

You might mix the approaches based on some cut off point

We never actually had a literal “Bieber Box”, but the joke took off.

Hot shards were definitely an issue, though.

Just to be clear, you are a Bluesky engineer, right?

off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.

You're not less partisan if you prefer a slimmer range of political leanings.

There's nothing wrong with being partisan if you're partisan against fascists who want to destroy democracy and the rule of law.

Funny how you call trump administration fascist. (theoretically its anti fascist but its still bad ,

Taking from the description of the video since this was what immediately ringed when you said trump===fascism

The liberal theory of the rise of Trumpism and its supposed fascistic features is inadequate in both effectively analysing and offering solutions to the present situation. Liberals often personalise or individualise people like Donald Trump and Elon Musk, casting them as deviations, as opposed to manifestations of class society. Class analysis suggests that fascism was a unique response to growing anti-capitalist organisations, socialist and/or anarchist, gaining prominence and posing threats to the economic base. The owning class required a mass movement which enveloped otherwise disillusioned people into a political project which had the collectivist, anti-free market appeal that socialist and anarchist organisations had, but nonetheless committed to solidifying and strengthening the economic base and profit motive. In modern America, no such anti-capitalist threat exists. Neoliberalism has created significant disillusionment with mainstream social and political institutions and systems, but this disillusionment hasn’t been captured by anti-capitalist forces, but rather by the populist right. As such, the populist right doesn’t need to give up the economic game, i.e. free markets, deregulation, privatisation, austerity, etc (with the exception of tariffs), but can purely rely on minorities as scapegoats in a constructed culture war, such as immigrants, ‘wokeness’, transgender people, etc. Therefore, capital doesn’t need to be subordinated to the nation-state, like pursued by contemporary fascist governments. Rather, in this ‘inverted’ fascism, capital takes over and exploits the state in a rather oligarchic manner.

https://www.youtube.com/watch?v=pqdLwkyfLdM

This video is really great , I spent 10 minutes looking for this.

I am not a trump supporter , The title might be a little clickbaity (basically the opposite of what it really is) You might find it really great.

It is one of the best videos I have ever watched on politics.

Its so crazy.

Thanks a lot for sharing this link.

Yup. This is a well-tread philosophical problem: the Paradox of Tolerance. Greater minds have concluded "to protect tolerance, one has to be intolerant of intolerance."

And, as always, bsky is a place of business - it is not a public venue. They can decide not to admit individuals who would threaten their business.

Oh fully agreed. But there's a large contingent of folks that are well represented here who think that it's inherently more intelligent to act like/be a centrist, that "both sides have something to offer," which isn't strictly untrue, but in practice especially with American politics just results in mealy-mouthed acceptance of pretty brutal status quos.

Like even left and right in terms of the mainstream here is nonsense. We don't have a left party at all, we have a conservative party, and we have an authoritarian fascist party. As a lefty none of my values are represented at all, I just get to vote each election for the conservative party that doesn't want my friends dead.

TV Pickup aka the Half Time Kettle Effect.

https://en.wikipedia.org/wiki/TV_pickup

I have heard it much more aptly described as “enforcing the social contract”.

You agree to uphold the contract of tolerance with everyone that participates. If someone refuses to uphold the contract with others who do, then you have no obligation to uphold the contract with that individual.

I understand why some people vote for some parties and why they’re “voting on inflation” or “right to abortion” but I guess, for me, keeping checks and balances and democracy is the one value above ALL for me.

In the span of human history, not a lot of countries and civilizations have lasted long, marked by constant instability and uncertainty for the future. We have a boring and imperfect political system created by our founding fathers but at least it’s been stable for nearly 250 years. A lot of people have tried standing up their own political system… most fail and everyone suffers. Even the founding fathers completely failed once first.

I know times are tough now but, in the context of history, they can be much worse and I rather not lose what good we currently do have.

> I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users

Random sampling? It's not as though the user needs thousands of posts returned for a single fetch. Scrolling down and seeing some stuff that's not in chronological order seems like an acceptable tradeoff.

> we got 250 years so far without imploding

We may have arguably recovered from it, but we rather famously did not get 250 years without the union violently fragmenting. (Our best record on that is right around 160, currently.)

Trump refusing to accept the 2020 election results should've been the line for many voters, but sadly it wasn't. And the potential crimes he and some of his allies may have committed while trying to overturn it will now never be prosecuted.

I find communist analysis tiresome, especially when in this case the populist right under Trump seems to be motivated in part by anti-free market ideas. The communist kneejerk reaction to every single situation is "this can be explained by class analysis". It's them trying to shoehorn their pet theory into everything.

Maybe this would be helpful:http://daslab.seas.harvard.edu/datacalculator/

I like that, it's less paradoxical, and likely easier to explain to people with less developed critical thinking skills.

Exactly. Tolerance is an opt-in protection. If you don't opt-in by exercising it yourself, you don't get the benefits.

Or, as a meme: YA_GOTTA_GIVE!.gif

2024: > More than 155 million people cast ballots in the 2024 presidential election. It's second only in U.S. history to the 2020 election. Turnout in 2024 represented 63.9% of eligible voters, the second-highest percentage in the last 100 years, according to the University of Florida Election Lab. The only year that beat it – again – was 2020 when universal mail-in voting was more widely available.

2020: > More than 158 million votes were cast in the election

So 3 millions of Democrats suddenly decided to not go out to vote "to save democracy" against "fascism"?

While it’s true we came close during Civil War, we still decided to keep the same system of government. In the end, while the Civil War did result in some constitutional crises, the root of the problem was more that one half of the country completely disagreed with the other half… I don’t think any political system can really work with that level of division and yet we kept the same one. Obviously the Civil War did very much bring into the question of states’ rights but, for better or worse, the founders were a little vague on that so we can still keep most of the same system and quabble over the details for the rest of eternity…

The simpler and much more likely answer, my friend, is that people didn’t vote from a combination of disillusionment, assuming Kamala would win, and likewise factors.

I saw many people close to me not bother voting because they didn’t enjoy Biden’s presidency, despite voting for him in 2020.

So, I find that FAR more likely as a reason than supposed election fraud.

> The only year that beat it – again – was 2020 when universal mail-in voting was more widely available.

You answered your own question. Voting was made more difficult in 2024, so fewer votes were cast.

Perhaps I misunderstanding, I thought the actual content of each tweet was being duplicated to every single timeline who followed the author, which sounded extremely wasteful, especially in the case of someone who has 200 million followers.

From the linked article: "Additionally, a reference to your post is 'fanned out' to your followers so they can see it in their Timelines."

So not the content, just a sort of link to it.

I'm really confused how tech people shifted from "voting machines are inherently insecure" to simply ignoring the issue despite many political connections between Democrats and voting machine vendors. I'll stick with the results of my research into the matter. If you think you're well enough informed and that your sources actually care about the truth, let's agree to disagree.

This is one of the most investigated issues in American legal history. There was absolutely no indication of fraud. You've fallen for a conspiracy theory. It's now Pizzagate-tier.

(I still argue with Pizzagate adherents on a monthly basis. They think it's perfectly logical.)