←back to thread

Launch HN: Fly.io (YC W20) – Deploy app servers close to your users

626 points mrkurt | 2 comments | 18 Mar 20 14:15 UTC | HN request time: 0.41s | source

Hello Hacker News! We're Kurt, Jerome, and Michael from fly.io (https://fly.io/). We're building a platform to run Docker applications close to end users. It's kind of like a content delivery network, but for backend servers.

I helped build Ars Technica and spent the majority of my time trying to make the site fast. We used a content delivery network to cache static content close to anonymous readers and it worked very well for them. But the most valuable readers were not these, but the ones who paid for subscriptions. They wanted personalized content and features for interacting with the community – and we couldn't make those fast. Content delivery networks don't work for Ars Technica's best customers.

Running Docker apps close to users helps get past the "slow" speed of light. Most interactions with an app server seem slow because of latency between the hardware it's running on (frequently in Virginia) and the end user (frequently not in Virginia). Moving server apps close to users is a simple way to decrease latency, sometimes by 80% or more.

fly.io is really a way to run Docker images on servers in different cities and a global router to connect users to the nearest available instance. We convert your Docker image into a root filesystem, boot tiny VMs using a project called Firecracker (recently discussed here: https://news.ycombinator.com/item?id=22512196) and then proxy connections to it. As your app gets more traffic, we add VMs in the most popular locations.

We wrote a Rust based router to distribute incoming connections from end users. The router terminates TLS when necessary (some customers handle their own TLS) and then hands the connection off to the best available Firecracker VM, which is frequently in a different city.

Networking took us a lot of time to get right. Applications get dedicated IP addresses from an Anycast block. Anycast is an internet routing feature that lets us "announce" from multiple datacenters, and then core routers pick the destination with the shortest route (mostly). We run a mesh Wireguard network for backhaul, so in flight data is encrypted all the way into a user application. This is the same kind of network infrastructure the good content delivery networks use.

We got a handful of enterprise companies to pay for this, and spent almost a year making it simple to use — it takes 3 commands to deploy a Docker image and have it running in 17 cities: https://fly.io/docs/speedrun/. We also built "Turboku" to speed up Heroku apps. Pick a Heroku app and we deploy the slug on our infrastructure .. typical Heroku apps are 800ms faster on fly.io: https://fly.io/heroku/

We've also built some features based on Hacker News comments. When people launch container hosting on Hacker News, there's almost always a comment asking for:

1. gRPC support: apps deployed to fly.io can accept any kind of TCP connection. We kept seeing people say "hey I want to run gRPC servers on this shiney container runtime". So you can! You can specify if you want us to do TLS or HTTP for an app, or just do everything yourself.

2. Max monthly spend: unexpected traffic spikes happen, and the thought of spending an unbounded amount of money in a month is really uncomfortable. You can configure fly.io apps with a max monthly budget, we'll suspend them when they hit that budget, and then re-enable them at the beginning of the next month.

One of the best parts of building this has been seeing the problems that developers are trying to solve, often problems we didn't know about beforehand. My favorite is a project to re-encode MP3s at variable speeds for specific users (apparently the Apple Audiobook player has no option for playback speed). Another is "TensorFlow at the edge" — they trained a TensorFlow model to detect bots and run predictions before handling requests.

We're really happy we get to show this to you all, thank you for reading about it! Please let us know your thoughts and questions in the comments.

Show context
ignoramous ◴[18 Mar 20 16:26 UTC] No.22618368[source]▶
Congratulations on the launch! I've been following fly.io ever since I stumbled on it 2 years ago.

A few questions, if I may:

> We run a mesh Wireguard network for backhaul, so in flight data is encrypted all the way into a user application. This is the same kind of network infrastructure the good content delivery networks use.

Does it mean the backhaul is private and not tunneling through the public internet?

> fly.io is really a way to run Docker images on servers in different cities and a global router to connect users to the nearest avaible instance.

I use Cloudflare Workers and I find that at times they load-balance the traffic away from the nearest location [0][1] to some location half-way around the world adding up to 8x to the usual latency we'd rather not have. I understand the point of not running an app in all locations esp for low traffic or cold apps, but do you also "load-balance" away the traffic to data-centers with higher capacity? If so, is there a documentation around this? I'm asking because for my use-case, I'd rather have the app running in the next-nearest location and not the least-load location.

> The router terminates TLS when necessary and then hands the connection off to the best available Firecracker VM, which is frequently in a different city.

Frequently? Are these server-routers running in more locations than data centers that run apps?

Out of curiosity, are these server-routers eBPF-based or dpdk or...?

> Networking took us a lot of time to get right.

Interesting, and if you're okay sharing more-- is it that the anycast setup and routing that took time, or figuring out networking wrt the app/containers?

Thanks a lot.

[0] https://community.cloudflare.com/t/caveat-emptor-code-runs-i...

[1] https://cloudflare-test.judge.sh/

replies(2): >>22618782 #>>22619186 #
mrkurt ◴[18 Mar 20 17:01 UTC] No.22618782[source]▶
> Does it mean the backhaul is private and not tunneling through the public internet?

Backhaul runs only through the encrypted tunnel. The Wireguard connection itself _can_ go over the public internet, but the data within the tunnel is encrypted and never exposed.

> I use Cloudflare Workers and I find that at times they load-balance the traffic away from the nearest location [0][1] to some location half-way around the world adding up to 8x to the usual latency we'd rather not have. I understand the point of not running an app in all locations esp for low traffic or cold apps, but do you also "load-balance" away the traffic to data-centers with higher capacity?

This is actually a few different problems. Anycast can be confusing and sometimes you'll see weird internet routes, we've seen people from Michigan get routed to Tokyo for some reason. This is especially bad when you have hundreds of locations announcing an IP block.

Server capacity is a slightly different issue. We put apps where we see the most "users" (based on connection volumes). If we get a spike that fills up a region and can't put your app there, we'll put it in the next nearest region, which I think is what you want!

CDNs are notorious for forcing traffic to their cheapest locations, which they can do because they're pretty opaque. We probably couldn't get away with that even if we wanted to.

> Frequently? Are these server-routers running in more locations than data centers that run apps?

We run routers + apps in all the regions we're in, but it's somewhat common to see apps with VMs in, say, 3 regions. This happens when they don't get enough traffic to run in every region (based on the scaling settings), or occasionally when they have _so much_ traffic in a few regions all their VMs get migrated there.

> Interesting, and if you're okay sharing more-- is it that the anycast setup and routing that took time, or figuring out networking wrt the app/containers?

Anycast was a giant pain to get going right, then Wireguard + backhaul were tricky (we use a tool called autowire to maintain wireguard settings across all the servers). The actual container networking was pretty simple since we started with ipv6. When you have more IP addresses than atoms in the universe you can be a little inefficient with them. :)

(Also I owe you an email, I will absolutely respond to you and I'm sorry it's taken so long)

replies(2): >>22621585 #>>22621949 #
1. e12e ◴[18 Mar 20 22:23 UTC] No.22621949[source]▶
> Wireguard + backhaul were tricky (we use a tool called autowire to maintain wireguard settings across all the servers).

I'm guessing that's? https://github.com/geniousphp/autowire

Looks like it uses consul - is there a separate wireguard net for consul, or does consul run over the Internet directly?

replies(1): >>22622081 #
2. mrkurt ◴[18 Mar 20 22:34 UTC] No.22622081[source]▶
Consul runs over a different connections with mutual TLS auth. That's the project we use!