Reliability: It’s not great

Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want.

Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it.

However, at the same time the market is looking for a successor to Heroku. A zero dev ops PAAS with instant deployment, dirt simple managed Postgres, generous free level of service, lower cost as you scale, and a few regions around the world. That isn't what Fly set out to do... exactly, but is sort of the market they find themselves in when Heroku then basically told its low value customers to go away.

It's that slight miss alignment of strategy and market fit that results in maybe decisions being made that benefit the original vision, but not necessarily the immediate influx of customers.

I don't envy the stress the Fly team are under, but what an exciting set of problems they are trying to solve, I do envy that!

I agree - fly is so easy to use (when it works) that it’s hard not to be impressed. BUT what I’ve found is that we don’t need edge compute, since our customers aren’t that latency sensitive, so it’s lost on us. It’s only a few more milliseconds to us-east-1.

I’ve heard (on HN) of a dozen different companies vying for the heroku replacement spots and yet Fly seemed to capture the attention. I couldn’t name another one off hand.

What I truly want and probably lots of other people too is Flyctl (and workflow) for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.

> generous free level of service,

This is likely the biggest culprit for a lot of these companies. Too many of us have grown up in the culture of getting hosting and platform for "free", but at some point the companies providing it still have to pay the bills. There has to be a better pricing model that let's someone deploy their relatively small, low-traffic app for $10s/month or even $200 - $300 / year for the basics (e.g. - Heroku free tier type capabilities). It's not going to save these companies but it would limit excessive growth of their own costs from a free tier while at the same time still being affordable for 1 - 2 person teams who are trying to get something in front of users.

> What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.

Google Cloud. It is painfully easy to spin up managed postgres, super easy to deploy gcp cloud functions or gcp cloud run. It isn't expensive either and just works.

Render.com is another spiritual successor of Heroku. I'd love a world where Fly and Render are both very successful companies.

Yeah, distributed systems at the global scale are very very difficult - at least with the Heroku style problem, you'd be looking at scaling in a single datacenter I think - deployments to multiple datacenters wouldn't share dependencies.

I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives.

Simplicity is the key to reliability, but this isn't a simple product, so idk.

If someone is not already using the holy trinity (AWS/Azure/GCP) there is probably a reason.

Do you have a guide in mind?

If it's sorting and sifting and clicking a bunch of stuff in the console, that's not painfully simple. If it's some easy cli commands, I think that's in the ballpark...

I can second that I‘ve seen render.com mentioned very often, maybe even more so than fly.

> What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dallas.

Pardon the ignorance, is this not the Amplify CLI [1] ?

[1]: https://docs.amplify.aws/cli/

Not gonna happen. Both will get acquired because that’s how things work now

The CloudFlare folks wrote a good blog post on how they are seeing their customers use Edge compute — latency is far down on the list: https://blog.cloudflare.com/cloudflare-workers-serverless-we...

I'm not using gcp anymore because it's not worth risk losing access to my personal gmail account just to play around with pet projects.

I might be paranoid, but I just don't feel comfortable when there's so much in play.

No

Render has some great features like making a new sub domain for when a PR is opened so you can test it as a fully working API before you merge

  > Flyctl for AWS

Have you tried AWS Copilot? I’m having good success with it. Probably not quite as simple as flyctl, but still it’s only one command to deploy a container.

I would really like fly.io to overcome these hurdles. I bet they will.

Hmm, that post is almost three years old -- still accurate?

The US CLOUD Act means a EU customer cannot use a US cloud provider to host PII, even if the server itself is physically in the EU, because US law will still compel the provider to yield the data to US authorities. The European Commission is trying to paper over the cracks with a fig leaf of judicial review, but it's only a matter of time until a Schrems III decision from the CJEU invalidates that polite fiction.

I can second this. We were evaluating moving off Heroku and to Fly.io, but we didn't need all of the edge compute stuff. We just want a better Heroku without having to think about infrastructure and having to think about edge compute just got in our way.

I feel like Next.js is in a similar position. While their main vision is SSR, I wonder if they are missing out on a chunk of the market that simply doesn't want to think about infra. We use them because we just don't have to worry about webpack or fiddling with deployment and hosting. We could care less about SSR and in fact we disabled it app-wide.

Can you elaborate?

Create a new Gmail account?

Not sure why this is downvoted, it’s a valid point.

Unless a company is very explicit about this not being in the books, I tend to share this outlook.

From the perspective of a recent founder, it's downright spooky to build around any SaaS, considering how few of them have been around for 10+ years, when that is certainly what our business is aiming for.

I know (and share the feels): Devs tend to get excited about the new thing – but if Google Workspace shut down next month, we would be in so much operational trouble. When other peoples fancies stand in the way of the entire operation you are responsible for, it actually begs the question how much closed source SaaS you can allow before it starts to be quite frankly irresponsible.

We are not imagining things. SaaS of all sizes shut down all the time, and when you are heavily relying on them and building software around them to run a business the prospect is spooky as hell.

I agree. And I know this is unpopular, but I think none of these companies should be expected to have a free tier. A low-cost tier? Certainly. Perhaps even a free trial with a credit card? Great.

But our team, who has used Heroku for over a decade, got bit multiple times by Heroku having a free tier.

Why were we impacted by other apps? Because Heroku’s load balancers are shared amongst all their apps. That includes all the sketchy apps running on the platform.

If Heroku could somehow isolate us from everyone else? Great - and they offered that for awhile with a reasonably-priced Add-On supported by them called SSL Endpoint. It cost about $15/month and put us into a pool that was shared with other folks willing to spend that much per month to run their app.

I understand that’s not great for a hobby project. But for those of us trying to run a large product on Heroku and not have to spend multiple extra thousands of dollars every month for a Heroku Private Space, this was a great way of pooling: put a small fee in place for one pool of resources. Not many malware writers or other misbehaving app creators will probably want to spend that much per month.

But they axed that a few years ago. Only a couple months after when we were thrown back into the load balancer pool with all the other free apps, one of the IPs was marked as spam and we had to figure out a kind of janky solution.

Additionally, Heroku seemingly spent a ton of resources on free tier support, malware fighting, etc. I hope to see more features on Heroku since they’ve dropped that support… but I haven’t seen much evidence of that in roughly six months since they did that. But we’ll see.

The amount of EU companies following this law is exactly 0.

That’s supported on most PAAS these days, including Heroku.

There's a wonderfully blunt saying that applies here (too): you are not in the business you think you are, you are in the business your customers think you are.

If you offer data volumes, the low water mark is how EBS behaves. If you offer a really simple way to spin up Postgres databases, you are implicitly promising a fully managed experience.

And $deity forbid, if you want global CRUD with read-your-own-writes semantics, the yardstick people measure you against is Google's Spanner.

Why would they be missing out? Vercel can host static sites just fine, whether that’s one generated by Next or any other framework or written by hand

What are the limitations to heroku that people are going to Fly for? Maybe there's a standard article that would be useful to read about it?

Nice write up!

I wish I shared your enthusiasm for where Heroku could go but I have a few friends at Salesforce I've asked about how they see Heroku internally and it really doesn't seem like it is going to get much love. Hope to be wrong though.

> dirt simple managed Postgres

Heroku PostgreSQL is very simple, yes. But once you need non-trivial scale it's expensive and extremely non-performant. Even a medium-sized RDS will outperform Heroku's most expensive database offering by 20x in my experience. My company doesn't even run PG on Heroku anymore. We have a VPC/Private Space connection to AWS Aurora because the cost/performance difference is so extreme.

One of the key design choices of Next.js was to enable granularity on the runtime (Node.js or Edge[1]) and the rendering method (static or dynamic[2]) on a per-route basis. If you want a full SSR site, that's okay. If you want a full static site, that's also okay.

We often see folks wanting a mix of both. For example, maybe the /about page is static, but the home page is dynamic and personalized based on the visitor. You can do all of this with Next.js. Our future direction is adding even further granularity, enabling this decision at the data fetch level, allowing you to cache results across deployments[3].

[1]: https://beta.nextjs.org/docs/rendering/edge-and-nodejs-runti...

[2]: https://beta.nextjs.org/docs/rendering/static-and-dynamic-re...

[3]: https://vercel.com/blog/vercel-cache-api-nextjs-cache

This is spot on. I found myself using Fly for a project because it was super easy, not because I needed edge compute. TBH it's still actually unclear to me who needs edge compute? What apps require this sort of infra? It's not 99% of web apps right?

Yeah I like them both a lot, having tried deploying small projects on each. However, I’ve defaulted to render at the moment because I’ve found it painless for my current project, and edge compute is low on my list of priorities.

Though to be fair, even if render collapsed overnight, I think I’d still be equally satisfied after moving to fly.

I know I've personally spent a large portion of my time updating systems to be compliant in the last few years, in North American companies.

It’s not true. I know people who lost contracts because they were using Azure and the customer wanted to respect the law.

I don't know the details of how Heroku implements their hosted postgres service, but I'm _guessing_ that it's just a bunch of PG servers running on EC2 instances. There's probably a lot of CPU stealing "noisy neighbors" going on. But yeah, I've also experienced Heroku's PG databases being dog-slow compared to RDS for the same workloads.

I can attest that there are a lot more than zero in Germany.

Personally I see this as a 'why not, if it works' type thing.

Sure you don't need it for 99% of usecases, but if it just works using familiar architectures then it is also strictly better for 99% of usecases so you might as well, and people will naturally want it.

That 'familiar architectures' part is the hard bit, though.

Please tell the legal department of our uni. I’m stuck with a home-made Kubernetes cluster where I have to mail the admins for provisioning, SSL and domain management. Would love to switch to Fly or Render

I'm going to plug Coolify, an open source Heroku alternative (with Docker support too) that I'm using on a cheap $5 Hetzner server which is a lot cheaper than the equivalent Fly or Render etc service, and it really doesn't have much upkeep from me even if you add in the time setting up the server initially, which is like an hour, and afterwards, it Just Works™.

https://coolify.io

But it isn't better in 99% of use cases. Lots of use cases are rendering an API response or HTML page that involves multiple database requests. Therefore the distance between database and app server is more important than the distance between the client and the app server.

Edge compute can be helpful for static or quite cachable content. But often this is handled as well or nearly as well by a caching CDN.

So that leaves a few cases where edge compute is useful. Where you are globally distributing the data itself (and ideally moving the data around as your users travel or move) which is incredibly rare and expensive to build, and when you need pure computation that needs no request to your backend and if 50ms of latency is important for a pure computation most of the time you can just move it to the client. In my experience these tend to be rare. I would estimate that edge compute is actually helpful for 1-5% of projects, not 99%.

I'm guessing that downvotes come from those who see the macro environment changing. With increased rates, borrowing to purchase companies may make less sense.

You can take a look at www.qovery.com It provides an Heroku like experience but runs on your cloud account (aws, scaleway or digital ocean).

They build on existing tech that is already working, so it is more stable.

I think this whole category is interesting, from the next-gen PaaS to the cloud-native ecosystem. Totally empathize with how hard what fly is doing in terms of scale and reliability is.

At Coherence (withcoherence.com) we're focused on a developer experience layer on top of AWS/GCP. You might describe it as flyctl for AWS.

In a nutshell if you offer cloud services you need to be better than the MAG clan, Digital Ocean too. And people will want it dirt cheap. It’s still hard to be a profitable web host as it always was (MAG has the advantage that none of them were web hosts at first base)

Where does the misalignment between what the customer thinks they want, and what they actually want fit in to your philosophy? Google Spanner is a great example of this because who doesn't want instantaneous global writes? It's just that, y'know, there's a ton of businesses, especially smaller ones, that don't actually need that. The smarter customers realize this themselves, and can judge the premium they'd pay for Spanner over something far less complex. What I'm getting to is that sales is a critical company function to bridge the gap between what customers want, and what customers actually need, and for you to make money.

The first releases of EBS weren't very good and took a while to get to where we are. Some places still avoid using EBS due to bad experience back in 2011 when it was first released.

Save for a few "in-preview" features, Fly was stable too but then they started growing faster than they could keep up (a good problem to have!). Stability isn't a permanent state.

This simply isn't true. At least not for EEC(Norway).

I still think that in the next pendulum swing we'll end up with edge computing and (smaller) self-hosted backends. Everything old is new again, and we haven't entirely recreated Akamai from first principles yet.

MAG?

I coincidentally tweeted the exact same thing earlier today.

I selfishly hope Fly put all their focus toward becoming Heroku 2.0. I’m sure some people care about all the edge latency stuff but I don’t know many of them.

God, I'm glad someone else sees it as clearly as I do. I learned about Fly from them acquiring freaking Litestream! The SQLite replicator! The canonical database at the edge of the network! Of course that's what they want to do.

On their free tiers though?

It's more about Heroku dropping free and low-cost plans, which is them demonstrating that they don't currently care about three low end of the market, more than any specfic feature.

> And I know this is unpopular, but I think none of these companies should be expected to have a free tier.

Free tier is a GTM motion which makes sense for novel tech products like Fly because: https://en.wikipedia.org/wiki/Technology_adoption_life_cycle

These threads from mrkurt a few months ago seem relevant here -

https://news.ycombinator.com/item?id=32955520

If they are a multiplier for a whole portfolio, there's not much reason for any particular branch to purchase them.

(This post seems like some evidence they might actually be building the wrong thing, though.)

microsoft apple google

I'm a co-founder at Northflank. This is what we've spent 3+ years building. https://northflank.com.

I am sympathetic with much of Kurt's post. We spent a long time building solutions to several of the areas highlighted (managed PG, persistent volumes, secret management and service discovery).

Making radical changes to architecture on a live cloud platform is always a challenge.

On the front-end Northflank is a next-gen PaaS built for high DX, speed, and powerful capability (real-time UI, API, CLI, GitOps, IaC).

Our backend is built using Kubernetes as an OS: providing a huge amount of flexibility on service discovery, load-balancing, persistence/volumes and scale.

The benefit of using Kubernetes is a universal API across all major cloud providers. We can scale clusters and regions across EKS, GKE and AKS in seconds, either in our managed PaaS or inside our customer's own cloud account.

Our managed dataservices: MySQL, Postgres, Redis, Mongo, Minio are all built using Kubernetes Operators with a small but mighty team.

From a generous free tier to autoscaling to managed postgres and other advanced PaaS/DevOps automation workflows Northflank offers something unique.

(Render founder) I'd love to understand why you think this is the only outcome. Render has positive gross margin and a clear path to profitability based on both our growth so far and the tailwinds in this space. I'm also aware of other companies like ours that have grown all the way to IPO or are well on their way.

I'm very explicit both internally and externally that an acquisition is a failure mode for Render. We're building this for the very long term and plan to keep it that way.

Microsoft (azure) Amazon (aws) Google (gcloud)

I’m assuming Azure, AWS, Google Cloud, but it’s new to me too

From context, I'm assuming Microsoft / Amazon / Google, referring to Azure / AWS / Google Cloud respectively.

Google associates different accounts that are from the same owner when handling issues FYI. So if they think your account is doing something wrong on GCP, be wary of associated accounts.

Totally agree with this mindset. My digital life is on the line because Google refuses to separate services.

might well have been yak shaving. If a company is under US jurisdiction it simply cannot comply to EU data protection.

One of the big benefits of edge compute is that it’s geographically distributed. Doesn’t make a big impact across the US, but globally a lot of nations have specific data laws, so it’s important to host data in the required nation. Keep customer data in its nation of origin, but have a single control plane and platform for ever data center.

Separating concerns, isolating things that are not related, these are some basic tenets of good engineering. Yet we all keep rolling the ball of mud downhill and act shocked it keeps growing and swallowing everything.

Things that just work and are delight to use and the AWS Amplify CLI are not often mentioned together. The Amplify CLI is a growing collection of poorly thought out, poorly implemented functionality that looks good in demos, but falls apart under any close inspection.

Thanks! I have talked with two Heroku folks who say (to me, a paying customer of Heroku Enterprise) that Heroku is absolutely in active development.

I let them know they need to demonstrate that to me. They have a roadmap [1], but it seems to have barely anything moving forward, including some really important concepts like http/2 support.

[1] https://github.com/orgs/heroku/projects/130

I have run the experiment, and Crunchy Data’s Postgres servers are 4X more bang for the buck than Heroku’s.

I let some folks at Heroku know this who are product managers, and they are investigating it… but I would be shocked if Heroku gets a big performance improvement anytime in, say, 2023.

20X seems like a lot for RDS, though I’d be curious to learn more! We are switching to Crunchy because of that clear cost/performance difference you mention.

This is, indeed, the exciting part. As Heroku fans, we never really felt like it needed a replacement. And if it did, it seemed like Render was the natural Heroku v.next.

One thing we've noticed, though, is that people do actually want Heroku but close to users. It's not exactly edge compute. In some cases, it's "Heroku in Tokyo". In others it's "Heroku, but running in all the english speaking regions".

I think the thing that ate up most of our energy is also the thing that might actually make this business work. We built on top of our own hardware. That's the thing that made it difficult to build managed Postgres. We put way more energy into the underlying infrastructure than most nü-Heroku companies. The cost was extreme, but I'm like 63% sure this was the right choice.

If you don't need all that much, Oracle Cloud does offer a free tier for VMs. You get 2 AMD VMs or 4 ARM VMs and even a free Oracle DB, object storage, load balancing and monitoring.

https://www.oracle.com/cloud/free/

It's still just a free tier so you can't expect good support, but, it's there.

... Are those North American companies prepared to willingly break EU laws then? Because in my (amateur) understanding it’s logically impossible to satisfy both CLOUD Act requirements and EU data protection ones (not just GDPR, but general due-process rights the CJEU considers required for privacy violations and US courts deny noncitizens).

Well, no longer free on Heroku, but it was

The difference between (free) Gmail and Google workspace is that workspace is a paid product. If you're big enough to warrant an AM, you can get terms which include continuity of business planning if Google does happen to shut down Workspace. (They won't.)

> who doesn't want instantaneous global writes

I want to gently note since I see a lot of misunderstanding around Spanner and global writes: Global writes need at least one round trip to each data center, and so they're still subject to the speed of light.

Just partner with Neon or other similar companies in this space. Scale-to-zero distributed databases is well understood technology.

https://neon.tech/

Egress pricing, for one.

fly.io charges an outrageous 2 cents/GB. Google is over 4x that.

At fly.io rates, 1Gbps average over a month is $6400/mo. Google is tiered and you’re looking at over $10k/mo.

For comparison, a cheap managed switch that can handle 1Gbps costs about $100, maybe a bit more if you want a nice one. A nice router is more. You can rent an entire rack, including power, cooling, and an unmetered 1Gbps for $300-$1k/mo (with maybe some wiggle room on both ends). You can buy a pretty nice server, amortize the price over a week or two, and still come out ahead.

You certainly get considerable value from a major cloud provider, and a lot of their other services are reasonably priced, but, depending on your workload, the egress prices and the corresponding Hotel California factor may make using a major cloud provider a poor proposition.

Yes, we are partnering with many companies like Hasura and Replit to help with managed Postgres. Since Neon scales to 0 and also autoscales to your workload it very economical for the long tail of low usage customers.

Check out DO app platform. It’s literally exactly what you describe.

Is your argument that Workspace is a paid product and therefore won’t be shut down? If yes, let’s keep in mind that Stadia was paid-for too. My trust in the longevity of Google products has been damaged beyond repair.

Dokku is also nice and battle-tested: https://dokku.com/

And may I also plug Lunni, a self-hosted Docker Swarm-based PaaS I'm working on right now: https://lunni.dev/

Both work pretty well on $5 servers.

Digital Ocean gave me the PaaS replacement and managed PG and I couldn't be happier.

If anyone else is looking.

I guess I’m just default cynical these days seeing how much money’s still floating around and the scale of the cloud big 3. Apologies, it wasn’t personal. I admire your vision and hope it can work, money always seems to talk eventually though. We need more companies that have the nerve to hold on and develop on their own.

Lunni looks really interesting! Looks like a Coolify competitor, I'll definitely check it out. Do you have a Discord to join? Coolify has one and I found it great to discuss the project and talk directly to the creator.

I used to use Dokku but I personally liked the GUI from Coolify so I've been using that. Nice to see that you have a GUI as well, makes configuring apps a lot easier.

Macro makes it harder to raise funding too though - VC no longer as attractive given the risks and higher interest rates available

Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'.

It's a delicate balance based on the locations that rows are being read and written. In the case where a row being repeatedly written from only one location and not being read from different location, the writes can be significantly faster than would be naively expected.

> As Heroku fans, we never really felt like it needed a replacement.

If Salesforce kept investing in heroku, it might not. But there is a huge loss of confidence in heroku's future going on among heroku's customers right now, which is part of what you're seeing, as I'm sure you know. (Also I think to some extent you are being political/kind towards heroku... if heroku's owners were still investing in heroku for real, adding 'edge' functionality like fly.io is focusing on is what one would probably expect...)

And frankly... your tool seems more mature and... not to be rude to competitors but seems to have more of that certain `je ne sais quoi` of Developer Experience Happiness that heroku _used_ to have and other potential heroku competitors don't really quite seem to have yet. Does what you expect/need in a polished and consistent way.

I think work you put into the underlying infrastructure definitely shows there, and was the right choice. Tidy infrastructure helps with tidy consistent developer experience.

So I understand why people are looking to you as a heroku replacement. I am too! (And I don't really need the edge compute stuff; although I could potentially see using it in the future, and it shows you folks are on top of things).

And while I kept reading fly staff saying on HN comments that you didn't want to be a heroku replacement, so were unconcerned with the few places people were mentioning where you still felt short of it -- when I saw your investment in Rails documentation and tools (and contribs back to Rails), I thought, aha, i think they've realized this is a market looking for them, which they are only a couple steps from and it would make sense to meet.

When you mention in OP a "heroku exodus" to you... I'm curious if that was all people who left when heroku ended free tier stuff, and they've all come to you for your free tier stuff... becuase that does seem dangerous, such a giant spike in users who are not paying and don't bring revenue with them! I don't personally use very much heroku free tier stuff. I hope if that's a challenge, it's one you can get over. I don't think you are under any obligation to offer free stuff that can be used for real production workloads indefinitely -- although, as I'm sure you know, free stuff is huge for allowing people to try _before_ they buy, and whatever limits you put on it to try to prevent indefinite production use get in the way of someone's "try before you buy" too... and at this point, _reducing_ your free offerings is a lot harder PR-wise than having started out with less in the first place. :(

Yes, especially as compliance and regulatory frameworks continue to evolve and become more difficult to adhere to as mentioned elsewhere in the comments.

We're inherently faster than other "serverless" platforms due to the scale and homogeneous design of our network, and that network has presence in nearly 50% more cities than it did just 3 years ago. We were plenty fast enough then and we're even faster now.

Other things that customers (still) really care about: developer experience, ease of use, and cost. Nobody likes paying the AWS tax to move data around—they just want to use the best solution from the best cloud provider. Workers and the associated storage primitives allow them to pick and choose from the best that AWS, Azure, Cloudflare, GCP, et al. have to offer.

(Disclaimer: I'm a long time Cloudflare employee focused on App Sec, and I speak to customers regularly who look to Workers largely for compliance reasons, but I don't work on the Developer Platform business. Am sure my Dev Platform peers will chime in with more nuanced answers!)

I would be glad to be shown a company with AWS, Google Chrome, Google Search, Slack and all the usual suspects.

I have never seen a company without Google Search, Google Chrome, AWS, Microsoft 360 and the lot.

Which alternatives are they based on?

No experience with either, but how does Coolify compare to Dokku, the OSS Heroku alternative I've been hearing about until now?

I've talked with companies like that as well and they start with strict rules and end up allowing clouds because no solution is compliant anyway.

The difference is that Stadia was definitely losing money, whereas Google Workspace might be profitable.

They're probably using older or cheaper instance types. By not upgrading while charging the same or more over time, one can skim more profit.

> I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives.

In my experience people who ran Postgres distributed across a WAN tended to use obscure third-party plugins at best, more often a pile of dodgy Perl scripts. Using something designed from the ground up to be clustered seems to have a much better chance of working out than trying to make something that's been built as a single-instance system for decades work across the internet.

Lunni has got an interesting concept — and I can actually see some good uses for it!

Is the actual "production" workflow still pasting a Docker Compose file in? I would much rather have an automated deployment process that doesn't require human input, that way it can be scripted as part of CI/CD, etc.

Personally, I fell in love with `git push production` (naming a git remote `production`) to trigger a deploy. Ironically I didn't like this back when I first tried Heroku, but it's grown on me since. As of now, I have a custom git receive hook on my server (building a NAS from "scratch" using IaC on my home server) that triggers a redeployment using Docker Compose.

Also, you mention Swarm... what does Lunni bring with Swarm as opposed to simple Docker Compose? Does it distribute across multiple systems?

How do Coolify and Dokku compare? I've been aware of Dokku for a long time already, however I've never been confident enough to rely on these interfaces to deploy applications, specially because of their business model to keep things going. I'll have to try them both eventually though, I absolutely hate PaaS honestly, the prices are all just too high, but the convenience is really nice when managing a multitude of services simultaneously.

> Like most things, it's more complex than that,

Sure, no doubt. My point wasn't really about the particularities. It was around the mistaken idea that I see sometimes where people believe that TrueTime allows for synchronized global writes without any need for consensus.

Yeah, I guess devs first will need to learn better SQL before it will start to matter.

I shared your post with Render's engineering team and it got a lot of love because we know the struggle and can truly empathize because of our own Heroku-accelerated growth. What Fly and Render are doing is hard, but someone needs to do it.

If the market is big enough to support AWS/GCP/Azure as $N00B businesses each, it’s not a leap to imagine a future where both Fly and Render are incredibly successful, loved, and independent businesses spanning decades. Let's keep at it.

Also the absolute disaster with security they had just before dropping free tiers, and the awful response which took months to even acknowledge some kinds of data (such as pipeline keys) where affected. [0]

[0]: https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

That's a nice idea actually, thank you! Just launched one:

https://discord.gg/9EAne8g2Pq

(Bridged to Matrix: https://matrix.to/#/#lunni:matrix.org)

Lunni is actually pretty young in terms of community (just me and a few friends now :-), so just a room in Telegram was sufficient so far, but I think it's a good time to start something more official.

Dokku doesn't have a GUI which is the main reason I switched from Dokku which I used to use before.

Those would not contain PII from your users though, unless you have terrible policies about copying personal information in random Google Docs.

Thanks, just joined. One thing I noticed is that the CI/CD setup for GitHub/lab is still based on the CI files themselves. What I like about Coolify and didn't like about others is that I could simply install a GitHub app to my account and Coolify would automatically pull my repositories and even set up git push to deploy for me, no messing around with CI files needed.

https://lunni.dev/docs/deploy/from-git/#github vs https://docs.coollabs.io/coolify/sources

This is an old doc from fly so I’m not sure how much of it is still accurate, but it talks about some of the stuff Heroku didn’t have that they have: https://fly.io/docs/app-guides/speed-up-a-heroku-app/

> There's no support for a single dedicated IP address for your application. With Heroku, your application's CPU resources are mostly located in one datacenter. Heroku doesn't support HTTP2 or Brotli compression and it doesn't do Edge TLS termination. And it doesn't run your applications on dedicated MicroVMs. These are all things that Fly's Global Application Platform does.

The other comment that mentions Heroku dropping low cost plans is the reason for the explosion in growth as I understand it though.

Yes.

Whenever a US law and a foreign law conflict, the US law always wins when you are in the United States. Complying with US laws is also a perfectly valid defense if a European citizen or state ends up bringing action against you in a US court.

All of the forum posts and howtos on spinning up an AdGuard or similar service on Fly.io instead of a local Raspberry Pi probably didn't help things. While it does drive user growth, it isn't the type of users you want.

One of the hardest lessons every business needs to learn is how to say no to the users they don't need.

I'll start with the Swarm since it's a major point actually: Docker's Swarm mode is comparable to Kubernetes or Nomad: you can launch a cluster of servers and run your application there.

Unlike Kubernetes or Nomad though, it uses mostly the same concepts Docker Compose does, to the point that your development docker-compose.yml file will likely just work there (with some minimal tweaks). I love this website that talks more about it: https://dockerswarm.rocks/

Edit: As opposed to `docker compose up`, when running on a single server: not much. It will restart on server reboot by default, and allow you to run multiple replicas of a service (deprecated in Docker Compose), but that's it. Most important though, it would allow you to add more nodes later on, and it will then scale your services across the whole swarm – so you can start with just one server and scale to hundreds if needed.

> I would much rather have an automated deployment process that doesn't require human input, that way it can be scripted as part of CI/CD, etc.

This is almost doable with Lunni. This guide will walk through setting up a CI for a typical webapp that packages it in a Docker image and pushes to a registry: https://lunni.dev/docs/deploy/from-git/ (currently for GitLab CI and GitHub Actions only)

As for the continuous delivery, we're gonna have a webhook that you can call when your CI pipeline is finished. It's not exposed in the UI yet but I'll try to prioritize it (now that I remember I wanted to do it :')

`git push production` feels a bit easier, but I'm a bit concerned about bloat: for this to work, we'll have to bundle some sort of CI and container registry with Lunni itself. I think sticking with third-party CI is a more elegant approach here. What do you think?

I haven't thought of an app yet, but a custom GitHub Action was definitely on my radar. IIRC this would still be a one-click setup, but more integrated with the Actions UI, and will be easier to extend / override later, if needed. What do you think?

Sounds good. I recommend going through the Coolify install and set up process (and adding applications and services, as Coolify calls them) which would give you more insight as to how they set it up.

By GitHub app I meant that Coolify makes you install its own custom GitHub app that then allows you to git push to deploy.

I admire your sentiment, at the same time founding teams don't typically say no to US$XX,XXX,XXX,XXX acquisition offers that'd cash you out for at least a few billion to you personally.

Are there any examples where the capitalism bottom line is ignored and a company keeps growing with extremely premium generous acquisition offers on the table? I can't think of any, but there could be a few. However, I expect it's pretty rare.

For companies with such tremendous growth, the venture capitalist firms are primarily looking to make their <big-multiplier> return and push priorities accordingly (understandably).

The only constant in life is change, it's best to focus on what you can do right now, today, and only put out promises or commitments that you have the necessary influence to follow through on. Some things are bigger than each of us.

Best wishes and godspeed to you and fly.io!

I think overall Coolify is a bit more modern. It uses some different components (e. g. Traefik vs Nginx for reverse proxying), and includes a UI in its basic package (Dokku was CLI only for a long time, now they have Dokku Pro with a Web UI). Otherwise it looks like the architecture is pretty much the same.

Re: business model: both Coolify and Dokku are open source, so even if their development stops, you can continue to use them no matter what. (You do have to pay for your own servers though :-) So it's not a PaaS in the traditional sense (like Fly.io or Heroku), but more like “build your own PaaS” thing.

I'll give it a go, thank you so much!

You're assuming that the US doesn't respond to political pressure and come up with an agreement with the EC to enable the flows. The wiretap act already goes beyond the fourth amendment in protection.

Backend simplicity also means a more shallow moat. It makes it easier for Digital Ocean/Linode (Akamai)/Hetzner to offer a competing service with the same backend knobs to turn, should they decide they want to get into that market.

The goal should be to make the backend as simple as possible, but no simplier. Complexity here leads to operational burden and toil. But that's why you hire good SREs and treat them well. What's more important is frontend complexity, aka how difficult it is for customers to use. Backend and frontend complexity aren't necessarily linked, which, imo, fly.io achieves, downtime aside.

Dokku Maintainer here.

I don't really have a business model. I do take donations from Open Collective (and Github Sponsors, which funnels to OC) and there is Dokku Pro, but those don't collect anywhere near the funds I'd need to stop my dayjob (at least now. Maybe someday?).

My business model is that code releasing is something I'm pretty passionate about. Dokku isn't even originally my project (Jeff Lindsay started it, I just took it over), but I've been working on it for almost a decade. It's open source and fairly simple, so even if something happened to me, others could theoretically continue the project on as desired (or build on top of it if need be).

I'd be interested in hearing any of your other concerns though :)

Dokku maintainer here

For proxying requests, Dokku currently supports:

    - nginx on the host (default)
    - traefik (via docker labels)
    - caddy (via docker labels)
    - haproxy (via docker labels)

We'll also soon support nginx via docker labels, which will work around issues where Docker sometimes assigns random IP addresses (and unlock TCP/UDP proxying as well).

I can't say anything else about Coolify since I haven't used it in a while, but I'd be curious as to what other parts are more modern about Coolify than Dokku.

if you add Akamai (Linode ) or Alibaba Cloud - then it will be come MAAG

Dokku maintainer here.

Dokku doesn't have an _official open source_ UI. There are a few unofficial OSS ones (Ledokku is the latest) that I'm aware of.

There is have a commercial offering in Dokku Pro (https://pro.dokku.com). It's paid (one-time lifetime license) but only so that I can at least partially cover my development time on it. The project is enough work on top of Dokku that I feel it is justified, especially as there is nothing stopping others from doing so, OSS or otherwise.

Thank you so much for your tremendous work!

Wow, looks like Dokku got a lot of upgrades since last time I used it thoroughly. I'm wondering though, why support four different proxies?

About the modern part: that was my opinion based on the way I recall Dokku and Coolify, and a quick scroll through the docs of both, so I might be really wrong here! I definitely need to check out both Dokku and Coolify again sometime.

Does Neon support triggers and subscriptions?

I am willing to pay a little extra for a nice dev/ops experience and simple/easy solutions that doesn't require spending days reading docs and diving into dashboards with thousands of options.

Usually this results in me jumping on new platforms and then abandoning them once they add too much complexity.

Edge of what exactly? The software resides within networks they control. There is no edge in this scenario.

We support 4 different ones to give folks choice. Some folks want/need features that aren't available on one vs the other (traefik has a ton of features, caddy is simple to configure, nginx has a ton of documentation) so it made sense from that perspective. It was also easy to add once I had the pattern going (though the default has stayed nginx).

One of the main features of Dokku is it's extensibility. You can cut one part out and replace it with another quite easily, and proxying is an example of that. I think that flexibility allows folks to use it in more situations than one otherwise would, though at the cost of being more difficult to maintain (and harder to have cohesion between parts of the system at times).

Assuming best practices are followed, AWS would have have to crack into multiple systems to offer up data for EU residents from AWS machines in the EU. Is there any record of them being required to do so?

Can you please explain what the Vault related failure is about? Is this about timing out services failing to start within an acceptable time range?

I suspect, in general, acceptability (or desire) for complexity in the cloud solution, and budget are positively correlated in customers.

It is going to be apps that provide rich experiences that need to do a lot of server communication to deliver them. I am thinking of things like collaborative whiteboards, for example. If 2 people are in Europe, working on the same whiteboard, then it should be low latency. The edge nodes will be near each other (or next to each other).

MAGA?

> I'd love to understand why you think this is the only outcome.

I’m curious why you think it isn’t? On a long enough timescale all good things seem to be acquired by large megacorps for a fuckton of money.

Slack, Linode, Minecraft, the list goes on. Eventually they all make the thing less than it was before under the founders’ vision. At least from my perspective.

It won’t stop me from cheering them on, but I’m still very skeptical of them not being bought out in 10 years.

yes to triggers - it's full postgres. logical replication is not exposed just yet, but soon

I would be kind of shocked if heroku gets a big performance improvement ever again. It seems the owners have decided to basically freeze it.

I can only assume that if you’ve ever used the amplify cli it makes total sense.

At least that single ‘no’ contained a whole wealth of blood and tears to me.

will subscription work with scale to zero?

> if you want global CRUD with read-your-own-writes semantics, the yardstick people measure you against is Google's Spanner.

I’m trying to build more of an intuition around distributed systems. I’ve read DDIA and worked professionally on some very large systems, but I’m wondering what resources are good for getting more on the pulse of what the latest best practices and cutting edge technologies are. Your comment sounds like you have that context so any advice for folks like me?

Well, they’re owned by bigcorp now right? Everything probably takes 10 times as long for no good reason.

Subscriptions should still work with scale to 0. NOTIFY/LISTEN doesn't: https://neon.tech/docs/reference/compatibility.

We will have an option to not scale all the way to 0 to support this scenario.

You can set up a git hook on repositories that listens for the completion of the docker_build task and then redeploys the app while pulling new images?

Personally I use portainer for basically the same thing. My only real requirement was that I could easily copy-paste in docker-compose files and have it just work.

I use caddy as the proxy, since I found the traefik configuration absolutely incomprehensible. Now I use only 2 labels to proxy instead of 15.

Not sure it's what you are looking for, but how Spanner mitigated CAP to delivery a relational DB at scale is a really interesting read[1]

[1] https://research.google/pubs/pub45855/

Geez chill.

> generous free level of service,

Can't you make the argument that Heroku got out of this market on purpose? I know they were bought out by "ye old corporate greedy meanie overlord" or whatever but... I'm sure there is data that showed it made sense from a "make money business" perspective to not be in that market.

It sounds like you built the right product on the wrong technology stack, at the lower levels. For example, I have never heard a Nomad success story, but this might be colored by interviewing engineers desperate to escape it.

Something like linkerd on Kubernetes would be stronger, I suspect. But I don't know the exact nature of your problems.

Yeah, point taken, I wasn't thinking to cluster across the WAN - more like an api wrapping postgres in a single DC. But you pay the price of read latency I guess... it's a hard problem no doubt.

The ridiculously overwhelming complexity is stickiness.

Think it’s bad to potentially technically move your solution from $CLOUD vendor? Wait until you turn around and realize you have at least one full time hire who’s entire role is “$BIGCLOUD Certified Architect” (or whatever) and your entire dev staff was also at least partially selected for experience with the preferred cloud vendor. At any kind of scale you have massive amounts of tooling, technical debt, and institutional knowledge built around the cloud provider of choice.

Then there’s all of the legal, actually understanding billing (pretty much impossible but you’re probably close by now), etc elsewhere in the org. At this point you’ve probably utilized an outside service/consultant or two from the entire cottage industry that has sprung up to plug holes in/augment your cloud provider of choice.

After realizing their cloud spend has ballooned well beyond what they ever anticipated plenty of orgs get far enough to investigate leaving before they realize all of this. Most decide to suck it up and keep paying, or try to somehow negotiate or optimize spend down further.

Cloud platforms are a true masterclass in customer stickiness and retention - to the Oracle and Microsoft level (who also operate clouds).

It’s interesting here on HN because while MS and Oracle are bashed for these practices AWS and GCP (for the most part) are pretty beloved for what are really the same practices.

Not exactly related to the OP, but: I think I speak for a large number of folks when I say that we don't care. The EU keeps passing all sorts of absurd laws that require dedicated auditors to comply with. It's just not going to happen. If they decide to actively enforce these things, they'll just isolate themselves from the rest of the world.

Companies have to guess what is PII and what is not, the EU have no idea (other than they know which companies they want to punish)

As an EEUU resident, we also don't care. We can survive without youtube and instagram and the whole surveillance industry. Some of the laws place a heavy burden on giant tech companies, but for good reason.

I've just started using Render and it's great!

Goodbye Heroku. :(

People don't trust it any more, because Salesforce have been under-investing in it for years and recently rug-pulled on everyone who had ever used the free tier (after providing that free tier for 15 years already - long enough for people to reasonably expect it to continue).

Your margins are going to end up being a lot better than any other PaaS that's built on top of the big cloud providers.

Yeah, basically that. One of the servers in our Vault cluster failed and prevented Vault agents from receiving secrets. For Nomad apps, this showed up as "allocation failures" and failed deploys. Machine based apps took an abnormally long time to start and caused other issues.

(No the OP)

I like Amplify and use it often. However, it isn't well integrated with "normal" backends, so if you want to keep a backend and frontend deployed together you either have to use their Amplify backend API or work out your own deployment.

I use Swarm for my own little app and love it, you should set up a Twitter so I can follow along on progress.

> Most important though, it would allow you to add more nodes later on, and it will then scale your services across the whole swarm – so you can start with just one server and scale to hundreds if needed.

Have you in all honesty and with first hand experience, deployed and supported in prod on swarm over hundreds of servers?

Awesome!

Linode is not the same scale as the top 3. I believe even Digital Ocean is bigger than them (for now).

They should have gone with GAA

Never once heard of that.

There is a gcloud cli, but I just automated deployments in CI...

https://github.com/google-github-actions/deploy-cloud-functi...

Oh that's really cool to know, wasn't expecting the Dokku maintainer to read my comment LOL. From what I had looked at some time ago I though the project was run with the profits from Dokku pro, although upon further thought I understand that that's probably not enough to keep someone working full-time on it LOL.

I checked the repo and yeah, it checks out, Dokku _is_ pretty manageable with a decently small codebase. Having a low bus factor is really important for me. I'll check it out soon, and hopefully leave a donation to help you keep the project going too :)

They place a burden on everyone. A burden that's going to create a two-tier internet where service is immediately refused to EU citizens by every provider except the giant tech companies that can afford to comply.

Sadly I think you are likely correct.

Nah Dokku really is mostly a labor of love. Originally I started working on it to provide a Heroku alternative for a group of students that couldn't afford what they needed to on Heroku (this was like... in 2014) and I've since been using it to run all my own stuff and the occasional client install when I do freelance.

> lower cost as you scale

The cost aside, I'm wondering how fly or heroku support their customers when they grow to microservices ecosystems.

The problem shifts from deploying easily to deploying reliably meaning one release of a service should not break the other services. Other problems appear too, like service discovery, peer authentication, gateways, test and staging environments where there are downstream dependencies, etc.

Are customers supposed to leave when they grow to this level? Or are there solutions for these?

Spotify, Apple, Amazon, Facebook, Twitter (kinda), Valve all have not been acquired and have existed for a long timescale...

I tried running a docker app on Digital Ocean's app platform, the UI is nice and I can see it being acceptable for your average CRUD app. But I had to abandon it due to the latency just being too high and their built in monitoring interval not being configurable to the ~5-20ms range (this was for competitive Battlesnake last year).

That's for top 4 companies by market cap.

> I have never heard a Nomad success story

There's a lot of Nomad at , just won't get any publicity but that's different.

The GDPR is quite clear on defining PII, I don't understand why you would claim otherwise?

That's a nice theory, but it may not survive the next few decades of regulatory capture by the same type of company you believe it's intended to act against.

Most of these platform have reached their critical mass to stay in business (and time to attract the customers like you) only because of the free tier. Sorry but even a low cost tier is too much for most wanting to give a try to a new infrastructure/stack. Most of these adoptions come from hobby projects trying it first and then recommending to use it in a professional setting. In a professional setting yes. you can afford to pay to low cost to evaluate it but you cant afford the time to do so. So they always rightly offer a trade of time to evaluate for the free cost. This is what actually brings the initial customers.

> so they're still subject to the speed of light.

I giggled. Good witty comment, bravo.

oh no, not again.

All of these will absolutely contain PII every time.

European states simply sue in their own territory or in front of the European Union Court of Justice.

Times are now different money is not free anymore so those big acquisitions need to make real business sense.

I got burned by this. I spent a lot of time researching and planning for this, only to discover there is no demand for solving this problem (yet?).

You don't speak for me. I don't want to live without YouTube.

Where are you from that you use EEUU as an acronym?

That's clearly survivorship bias.

What you want to know is the probability of a small, independent, high quality provider remaining independent, high quality and not bankrupt.

It does seem to be rare in the tech space, especially in the US. Becoming one of the largest public corporations on earth is one way to do it, as you suggested, but the odds of that happening are miniscule.

I think a Microsoft shill might choose a less suggestive name

For obvious reasons I’m excluding the companies doing the acquiring.

Except Valve I guess, but that was never a public company that could be acquired to satisfy investors in the first place.

I would even go so far as to say free users aren’t customers in a platform like this. There’s no revenue model and if it’s “run without constraint” you get some fun scaling problems but those types of issues can sometimes lead to optimizations that keep the system online in the face of many small apps but doesn’t necessarily cover the case of a large resource app.

MAGA cap?

What I love about Next.js is not having to think about hosting, webpack, hosting, typescript, scss, and so on. It just works.

I initially fought to get SSR working, fixing hydration errors and making sure our code was isomorphic.

I later realized that I can just use the parts of Next.js we need and turned off SSR. It wasn't a big value add for our particular product.

But doing this wasn't straightforward. I hadn't even realized it was a possibility until I stumbled across a blog post.

I had to copy a NoSSR implementation of the internet. It wasn't just some flag I could toggle for a page.

I've also found myself recommending Next to folks saying "Use Next.js, but btw you don't need use SSR. Make sure the trade-offs make sense."

I'm curious if I'm in the minority of Next.js users. What percentage of them don't need SSR but value everything else?

Not a git hook, but you can do that as a part of CI workflow. So, you can have a script like:

    docker buildx build --push ...
    curl -X POST https://lunni.example.com/api/webhooks/c8aaa9b8-1bda-4a99-820c-36a75d31f8a7

that will rebuild a Docker image, then trigger the redeploy.

To clarify, I'm referring to folks who just need to write single page apps, but don't benefit from SSR.

(I didn't find disabling SSR straightforward.)

I wonder if Vercel is underestimating the size of the market that just wants a "Heroku for React".

Nope :') I do know a guy though, and I've heard good things. I'd love to hear about your experience too!

I know that it is possible to outgrow Swarm – I think that's a nice problem to have actually. We might include some tools for “graduating” from Lunni to something more serious like Kubernetes at some point.

I guess it works when you don't have any compliant competitor.

That is exactly the problem at hand.

It's a combination of low to no enforcement, competitivity-killing laws and unrealistic efforts for said companies to take on.

Interesting that "EEUU" from my knowledge mostly refers to the US (Estados Unidos) in a Spanish context. The abbreviation for European Union would be UE (Unión Europea) right.

Hetzner or OVH would be compliant.

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” — Upton Sinclair

Defense and government is a huge sector. You can live very well off it.

They are not going to skimp on the rules. A large part of banking won't, either.

The problem is the European Commission is not applying political pressure because it rolls over for every fig leaf the US offers. It then takes Max Schrems to sue and several years before the CJEU overturns the "compromise".

That said, the Biden administration's latest proposal might pass muster if the proposed redress mechanism were truly independent as part of the Judicial Branch of the United States as opposed to the current proposal which is still part of the Executive and thus conflicted in ruling against surveillance decisions of the Executive Branch and its agencies:

https://www.whitehouse.gov/briefing-room/statements-releases...

https://noyb.eu/en/open-letter-future-eu-us-data-transfers

That said, even US citizens don't enjoy meaningful protection against warrantless wiretapping that clearly violates the Fourth Amendment due to the deference the judiciary has given to the executive, so I am not optimistic.

Oops. Late-night brainfart, I'm a portuguese speaker and got things a bit mixed up :)

Yep. The real question is how long until we get one.

Scaleway seems to go in the right direction but still a bit of work needed

They are however far from service parity with AWS, Azure and GCP.

I can't speak for Hetzner but OVH has also availability issues.

Why exactly is it seemingly so expensive not to sell your customer data?

I'm waiting for a site that does comparison matrixes. It should have checkboxes for autoscaled compute, easy build/push, scheduled/queued tasks, WAF and CDN, object storage (wish Render had this specifically), emails, easy addons to other SaaS.

It's potentially a lot more for the big clouds. Anything in the network path has a charge - load balancers, NAT gateways, etc.

So there is nothing in eu laws preventing you from opting into using these services. What _is_ prohibited is having a EU based product/service where your users are not aware that by using a service their data will be stored under us jurisdiction.

That is not the same as using us based products

Nice bit of FUD you got there.

You can use Google Search and be 100% compliant, because Google doesn't see any customer data. Google chrome isn't even a service, I can't imagine how you'd manage to stick customer data in there.

And if you think there are no companies without AWS and Microsoft 360, you need to expand your horizon. I work for one such company, and so do many of my peers.

I haven't put much thought in this, but is a Frankfurt data center provided by Amazon Web Services EMEA SARL (a Luxembourg-based company) considered a US cloud provider or a EU one? I mean, being wholly owned by a foreign owner doesn't generally change your jurisdiction, and employees of that wholly owned subsidiary (including its directors) are not required to obey USA laws or court orders but are required to comply with EU legislation.

> giant tech companies that can afford to comply

Where does this sentiment come from? Cost of compliance for Facebook is many orders of magnitude higher than cost of compliance for a website for your hairdresser or a restaurant.

In my startup, GDPR was barely a blip on our radar. We had to delete website logs and that's about that. You have to keep record of customers/payment information for laws that supersede GDPR, and that's it if you run a legitimate business not reliant on stealing.

There are also lots of companies that use AWS etc. for everything but customer PII and keep that in some SAP system on-prem.

Google Chrome through telemetry and account history synchronisation which log PII in URLs and searched.

Google Search will see PII go by if your marketing team is researching leads on LinkedIn for example.

> And if you think there are no companies without AWS and Microsoft 360, you need to expand your horizon. I work for one such company, and so do many of my peers.

And that's great.

What is the services stack your company is implementing?

What kind of alternatives do you use for your email, browser, centralised data storage, etc. ?

Yep.

Because when I think reliable cloud infra, I think Azure.

Close, the giant tech companies may or may not comply but they surely can afford the fines that the various EU Data Protection authorities dream into reality by twisting an ever-changing body of interpretation of ambiguously written rules.

This is really an oversimplification. MS and Oracle have licensing that's explicit in the way that it wants to lock you in, although in different ways. AWS and GCP posting public pricing that can apply all the way until you reach an absurd spend goes a long way, and the ability to turn off a workload tomorrow incentivizes these platforms to provide a high quality of service.

When working at AWS, a large part of the convincing for an MS shop would be around showing that we can offer a lower price than the 'discounting' that MS provides. Oracle was all about contract expiry.

While there's some complexity around migrating a workload, regardless of where it's at, many places are going into cloud migrations hoping to remain relatively platform agnostic. I've seen many successful migrations to and from different vendors, and often at an SMB or ME scale, in weeks not years.

The cost of egress plus a gateway or two is fairly close to the cost of burning a DVD twenty years ago. And it appears to actually be cheaper to burn DVDs and mail them today than to send data from a major cloud.

This becomes very relevant for things like archiving data. If you generate data outside of a major cloud, you can pay a major cloud a very reasonable fee to archive it for you. But if you ever download your archive, it will cost you about half the price of buying an external disk to store it on.

(To be fair, object storage is rather more reliable than a single crappy external drive. But if you access the data more than once, maybe you should have a colo or on-prem copy too.)

Doesn't Vault self-promote in the case of single node failure?

I noticed you mention Vault lives in the US, I'm sure you've already heard of this pattern, but Vault (Enterprise) supports [multi-region clusters for performance and DR](https://developer.hashicorp.com/vault/tutorials/day-one-raft...)

Why does there need to be a successor to Heroku?

did not know heroku had that.

FWIW, not currently using Next.js, but I'm someone who values everything else but not necessarily SSR. I've been eyeing to use Next.js in SPA mode. There are dozens of us!

The speed of light in vacuum is a hard upper limit. Most signal paths will be dominated by fibre optics (about 70% of C) and switching (adding more delay).

But, yes TrueTime will not magically allow data to propagate at faster-than-light speeds.

Thanks!

My understanding is that the distinction hinges on whether the data is available to a US based employee. Can the NSA show up at a US address and tell the people there to hand over the data? Can this data transfer happen without an EU based person taking some action? If the answer to both questions is yes, the data handling is not compliant.

Of course, IANAL, do your own research, etc.

Do you have examples of companies in this space that actually reached break-even? Heroku never hit profitability as far as I remember and with the Salesforce acquisition the question of profitability is moot. AWS is a counter-example to using a free tier as a GTM strategy. AWS did not start with a free-tier offering for S3 or EC2, that only came years later. By then they already had significant traction in the market.

Thank you! No Twitter yet but if you're on Mastodon by any chance: https://fosstodon.org/@lunni/

This simply isn't true. Look at the absurdity of all the cookie banners just to support basic login functionality. I'm all for internet privacy, but these laws are so sweeping that it's impossible to be compliant without a dedicated function for it.

That's not the issue. I don't want to see personal data sold either. It's all the little rules. There are hundreds of pages just in GDPR. You need a banner and explicit opt-in just to support login/logout functionality.

Did you read the comments? My takeaway is that it was bs.

That's the beauty of the services I named. Super easy to roll the code to any other similar PaaS provider. There is no vendor lockin.

It is postgres + http handlers.

Depending on what I'm building, I tend to cache on the edge with something like CloudFlare in front of GCP. Lowers the egress charges significantly, with the benefit of speeding everything up too.

Caddy is nice, I've been playing around with it too and I love it. Perhaps we can port Lunni to use it, too.

Portainer is also cool – we're using it internally as an API, actually! I've been using it before starting Lunni and my only objection is the UI. Portainer is kinda like a Swiss army knife for containers, but with this power comes the complexity, too.

For example, to see service logs, you have to pick an environment, then go to Stacks, find your stack, find the service you need, open it and then you'll see the service logs button. In Lunni, you open your stack right from the dashboard and click logs button right beside your service name: https://u.ale.sh/lunni-screenshot-logs-button.png

I believe Netlify introduced this feature. It is now ubiquitous (as alexgrover said).

There was one person strongly challenging the story and plenty of back and forth. It hardly seems a settled case one way or the other.

No need for cookie banners for functionality like login. Ref: https://law.stackexchange.com/a/32157

Yup. Which is basically a no-op. You need a court having jurisdiction over the defendant to have any relief. Even if you receive a financial judgement, international law does not put much weight in absentia cases.

If you have customers in the EU than the court has jurisdiction.

If the company doesn't comply, fines will be directly taken from customer payments for example.

Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief.

If the destination bank account is outside the EU, they can't touch it without cooperation from the defendant countries courts - which requires you to file in the defendants venue. If an EU country unilaterally seized intra-bank remittance they would be cut off from the international banking system without hesitation.

You seem to really be grasping at straws here, but the EU is not some all powerful entity that can enforce its laws outside its jurisdiction.

I honestly can't tell if you're trolling or you said 'AWS' and 'Microsoft 360' and meant cloud and managed email.

> What kind of alternatives do you use for your email, browser, centralised data storage, etc. ?

There are plenty of browser alternatives (firefox, safari, vivaldi, even chromium).

There are dozens if not hundreds of email providers, and you can even provide your own.

You can 'centralize data storage' on disks on hardware you own, on premises or colocated. You could even use one of the dozens to hundreds of managed service and cloud providers.

I don't know if they have reached break-even. I know that they are in business for a long time and are clearly getting investments based on the usage and the potential. Usage that would not be there were it not for the Free time contributed to test-drive their offering and the free "evangelizing" done by those "free-loaders". AWS/Google/Microsoft are in a different league because they can afford the waiting game for actual Enterprise Evaluations or can afford massive sale forces (Azure case). Smaller players can not afford it and have to rely on "recommendations" based on past experiences.

> Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief.

Of course you can, you simply reach for assets within the border of said member country or the EU. As I mentioned in my previous comment, you can for example get the funds from outgoing payments by customers of said company. You can also freeze accounts, prevent ownership or investments by any citizen of that country as well.

> If the destination bank account is outside the EU, they can't touch it without cooperation from the defendant countries courts - which requires you to file in the defendants venue. If an EU country unilaterally seized intra-bank remittance they would be cut off from the international banking system without hesitation.

There is nothing unilateral about a country seising money as payment of a fine from a company. This is a standard tool that every countries' IRS equivalent agency have in their tool belt.

> You seem to really be grasping at straws here, but the EU is not some all powerful entity that can enforce its laws outside its jurisdiction.

I never said that EU is all powerful, however, if business is done within the EU, EU countries have the power to access any and all funds going to the US for companies that do not comply.

They can also decide to block said service as a punitive measure.

> I honestly can't tell if you're trolling or you said 'AWS' and 'Microsoft 360' and meant cloud and managed email.

I meant both clouds and managed email / storages services.

> safari

Don't both Firefox and Safari have telemetry and various ping back services?

> There are dozens if not hundreds of email providers, and you can even provide your own.

> You can 'centralize data storage' on disks on hardware you own, on premises or colocated. You could even use one of the dozens to hundreds of managed service and cloud providers.

Sure you can, I'm just saying that it is rarely if ever done in medium to large companies.

I get the impression that you think "still subject to the speed of light" is some kind of hyperbole or something, like if you were on a freeway and saw a sign that said "end speed limit" and thought to yourself "welp, still can't go faster than c".

But when you're working on distributed systems that span the planet (say multi-master setups where ~every region can read and even write with low latency), you start thinking of the distance between your datacenters not in miles or kilometers but in milliseconds. The east coast and west coast of the US are at least 14 milliseconds apart:

  % units "2680 miles" "c ms"
  2680 miles = 14.386759 c ms

and that's not counting non-optimal routing, switching delays, or the speed of light in fiber (only 70% of c). Half of the circumference of the earth (~12500 miles) is likewise 67 milliseconds away absolute best case (unless you can somehow make fiber go through the earth).

> Of course you can, you simply reach for assets within the border of said member country or the EU.

Which is exactly what I said. If the US company has an EU subsidiary you sue in that venue that can grant you relief. There are US tax implications of holding foreign assets, so the 1% of US companies with overseas interests create a foreign subsidiary, the other 99% have absolutely nothing within the reach of the EU.

> There is nothing unilateral about a country seising money as payment of a fine from a company.

Funds in transit belong to the sender until they arrive in the destination account. The EU would be seizing the funds of an innocent third party (the customer), and the target company would just shrug and say "your payment didn't arrive send it again." The EU cannot seize a transaction in flight and also compel the target company to honor it against their books.

> if business is done within the EU, EU countries have the power to access any and all funds going to the US for companies that do not comply.

See above. Taking money from random EU customers I guess is something they could do, but I imagine their citizenry would be none too pleased about it.

Let me try to simplify it for you: the EU cannot take what is not in EU jurisdiction without the cooperation of the foreign court. If a company says they were complying with their domestic law which violated EU law, they would likely not receive the cooperation of domestic courts to grant relief.

Let me make it simpler for you.

If say Google were to not follow the GDPR for example, even if they didn't have any European subsidiaries, the EU or a member country would simply make all Google customers pay their subscription fees to them instead of Google as fine payment for the fine. Customers would see no service disruption.

In your example Google would not receive the funds and credit the customers account. How would they differentiate an EU government stealing the money from a customer who just didn't pay and say they did?

Feel free to call up your credit card or power company and ask them what happens if you send them a payment but it gets seized by the government along the way. Their answer will be that you still owe them money.

In your example the EU customers would be out the money, not Google. With no EU nexus (in your hypothetical) they cannot compel Google to provide services they were not paid for.

Can you explain why you believe this to be the case? Let's say you log the user in. Yes, you need consent to store a login cookie, but that doesn't mean you need "a banner and explicit opt-in". You only need explicit opt-in, which you can do by... putting a "remember me" box next to your login form[1]. Is that really so hard?

[1] https://law.stackexchange.com/a/32157

> How would they differentiate an EU government stealing the money from a customer who just didn't pay and say they did?

Because they would have been notified by a court beforehand and the fine would constitute an outstanding debt linked to a lost lawsuit.

Once that happens, the national collection agencies would take over and use the tools at their disposal, like collecting from customers directly, which is the equivalent of garnishing wages but for companies.

They would then receive regular updates about the remaining debt and what was already paid and by whom.

> Feel free to call up your credit card or power company and ask them what happens if you send them a payment but it gets seized by the government along the way. Their answer will be that you still owe them money.

If Google then refused service to the customers who's payments were redirected to that country's collection agencies, then additional punitive measure would be taken by the country.

Some of the punitive measure could be:

- growing interests on the outstanding debt

- blocking the service within the country or EU

- advertise that Google is delinquent and is refusing to pay it's debt to financial institutions

- prevent banks and financial institutions from loaning money or investing in Google

- configure an embargo for imports and exports towards Google

- extradition requests for C-suite or adding them to Interpol and Europol wanted people list

- etc.

> In your example the EU customers would be out the money, not Google. With no EU nexus (in your hypothetical) they cannot compel Google to provide services they were not paid for.

They can't force Google to provide services but Google will also lose that market (for the EU that's 450M people) and increasing punitive measures.

Also, Google refusing to pay would probably discourage financial institutions anywhere from servicing Google in the future and other countries from authorising Google on it's national market.

The best practices are built on solid first principles, and you can get a pretty good grip on them from the High Scalability archives. Back when they posted actual tech articles, their content was some of the best available anywhere. Since you've read DDIA you will probably get quite a lot out of the archive. In fact, you should be able to identify at least some of the unstated problems.