When imperfect systems are good: Bluesky's lossy timelines

Note that all of this reflects design decisions on Bluesky's closed-source "AppView" server—any federated servers interacting with Bluesky would need to construct their own timelines, and do not get the benefit of the work described here.

What reason does Bluesky give for not opening up their AppView code?

Another notable component that is closed source is the discovery feed generator, where at least there is some reason.

what else? profit by means of doing work that benefits first and foremost the private proprietors of the closed source

if they gave it away (which used to be unfeasible until the digital era) they feel they’re loosing their valuable effort which they’re wont on concentrating, not diluting.

The App View frontend is open source: https://github.com/bluesky-social/social-app

Much of the backend is open source as well: https://github.com/bluesky-social/atproto/tree/main/packages

What is not are the extra services they run to provide a better and faster UX. Even if it was open source, it likely costs 10s of thousands to run per month (they have moved largely to "onprem" hardware instead of the cloud aiui)

when I read the spec it seemed like the operator of an AppView & Relay would be most in need of compensation for their hosting costs due to the amount of demand on those components so I believe the spec allows an operator to implement their own AppView & monetize it as that operator sees fit, so that they can afford to operate the service and maybe even make money off of it so that they can make it their full time jobs.

I asked this and got

> We did a backend rewrite from postgres to scylla and it has a bunch of deployment specific stuff, but is functionally identical to the open source postgres version. Its not really a "v2" in terms of new features, we just made it make use of our hardware really well[1]

[1]: https://bsky.app/profile/iame.li/post/3l7e3jfqit22s

This is not true. Third party PDSes are fully supported by our app view, and our app view generates timelines for all the users on those PDSes.

As others have noted, the appview is open source. The dataplane has two implementations, one in postgres and another in scylla. The scylla dataplane is closed, the postgres one is open.

The interesting next stage for the postgres implementation is to create a sync engine for partial syncs of the network, so that an appview can run affordably. We ran some benches on the current state of the postgres implementation and found we could index 300k users on a $100/mo vps. I think with a couple of weeks of optimization that could reach 1mm users.

It seems this way to me as well. ATProto fundamentally changes how monetization works in social media by removing lockin. It's going to be interesting to see what emerges from this design decision.

Another interesting way to view ATProto is that it could be a collection of headless features and network browsers that leverage those feature providers.

What does this have to do with third party app views?

That's the frontend code, it doesn't include the backend API services, which are closed source.

Which is what I said in the second sentence

the backend (the AppView) can be found here:

https://github.com/bluesky-social/atproto/tree/main/packages...

there are various supporting services written in Go as well

https://github.com/bluesky-social/indigo

Thanks, so are both the Postgres and Scylla versions maintained in terms of new features?

I wasn't aware that AppView v1 was open source, and the most recent info I'm aware of on the topic is https://alice.bsky.sh/post/3laega7icmi2q, https://github.com/bluesky-social/atproto/discussions/2961 and https://docs.bsky.app/docs/advanced-guides/federation-archit..., and everything I've heard about Bluesky was that open source appview is "still coming".

This is great to hear—my current understanding of the most recent state of the art on the topic is https://alice.bsky.sh/post/3laega7icmi2q which mentions that the self-hosted appview is not yet open source. So I'm glad to hear the situation has changed in the past 3 months.

that's not the appview, that's the client

AppView is a specific term of art within the Bluesky federation architecture: https://atproto.com/guides/glossary#app-view, you were incorrect in identifying the public frontend repo as the AppView.

App View is a bit fuzzy of a term. To me it seems like a combination of frontend, backend, custom lexicon, and supporting services. There isn't really another place in the spec or design where clients or browsers fit in, which do in fact provide a view of the network via an app.

A frontend is (can be) part of an App View. It is quite literally the app you view the network through. There can also be headless app views and app views which have no backend

You didn't specify what kind of federated servers you were thinking about

The statement "any federated servers interacting with Bluesky" is ambiguous, because Bluesky's federated model means there's many different types of servers, and one user's view of what a "federated server" could be vastly different from another.

Federated PDS-s (which is probably the closest to what people mean when they say they want to federate on bluesky) would not need to reconstruct timelines if their users use the bsky.app appview.

It was open source (except the Scylla database layer) from the beginning, AFAIK - that blog post just says that they haven't set it up yet, because that's the hardest part to run

It's not coming, it never went away… As I understand it, the "business layer" with all the logic is above the data later, shared by the Postgres and Scylla versions, and the data layer just makes queries to the database. I think they are using the Postgres version locally for development.

Thanks, that's a fair point that I was overlooking. When I say a "federated server", I don't just mean a self-hosted PDS, I mean a third party app that potentially has its own lexicon and design decisions. Creating a robust third-party app that can meaningfully interact with the Bluesky network is still a very difficult engineering challenge, which I think this article does a good job demonstrating—that was the tension I was trying to underscore in my comment. Bluesky may be solving those engineering challenges for those clients who are satisfied with Bluesky's frontend and AppView, but every single other app built on top of ATProto will have to resolve those same challenges. This is directly downstream from Bluesky's "global firehose" topology and various design decisions that stem from that.

My thinking has evolved on this topic significantly as of late. My current thinking is we should create a secure gossip network on top of the Bluesky API, and forgot about all the DAG-CBOR stuff that gets stripped from the Jetstream. Hash the posts on the gossip layer and if posts change then diff them. This is all prep for when X billionaire buys out Bluesky then we just pop some signing key crypto on top of this gossip layer and wow! It's distributed!

isnt that ssb?

reverse-ssb

this is not correct

"UI" is part of the definition they give in the glossary

https://atproto.com/guides/glossary#app-view

The glossary specifically calls out the UI as part of an app view. Can you explain why it is not according to you?