Tower Defense: Cache Control

Regarding stale-while-revalidate, the author calculates maximum load like this: “With the options available on hn.unlurker.com, there are only 10 * 12 * 8 * 2 or 1920 possible combinations, refreshed at most once every 15 seconds”

That math works until it doesn’t. If for some reason there’s a greater than 15 second response, the cache will open the floodgate and won’t return any cached response until the cache is full.

Similarly, errors need to be accounted for. If the server returns a 500, is it configured to be cached? Is stale-while-error configured so the error state doesn’t take the server down?

Where's the fucking game, I was teased with a potential of a vibecoded game!

Now I really want to somehow work out a way to drop that in to my next architecture diagram wholesale: https://www.jasonthorsness.com/_next/image?url=%2F_next%2Fst...

"Why did you choose Go for your backend?"

"Well, the lightning strikes it implements deliver critical elec-damage against the customers that get past the ice blast from our DATA CACHE-powered FUNC, striking them in their frozen-status for x3 before they can get to our vital CPURAM resources. The most important thing to have in a cloud-based deployment is a diverse mix of elemental attacks and crit-stacking bonuses."

Put nginx in front with proxy_cache_lock with a sufficiently high lock timeout. Even if your service can somehow be that slow, now you have bounded requests again.

Read your CDN docs very carefully on how it processes requests. Make sure that your CDN does request coalescing.

At my previous job, I became responsible for handling >350,000 requests/second and I dug so deep.

Now we have to figure out which game will, if you train an a dnn on it, will result in a network that solved technical problems best. Maybe a facorio-playing AI could design a good cpu, haha.

> Why Not Redis? I have a single VPS, so I can get by with a simple SQLite database. If I had many instances of my API on separate servers

Just to push against this a bit. Redis can be very low memory cost and is very easy to run (I give it 5mb). I have a small single server with a few instances of my API that let's me cache pretty much everything I need.

Huh, I never thought of running it just on the one same node. I guess that would better prepare for a scale-up later.

Also, wouldn't each CDN node maintain its own cache and thus its own state-while-revalidate timer? Or is the implication that this is using a tiered cache where there's only a single CDN node that connects to the origin server?

Good point. I guess this depends on Vercel's cache topology and whether it uses multiple layers. I would have thought they would cache the output of the origin function in a cache right at the origin but I can't find anything about this in the docs so likely not. As an alternative, they do allow caching the fetch to the API in their "data cache" so I'll probably need to enable that to protect the VPS in the way I intend.

I've always wondered if it would be worth the effort to display information to the user to indicate that their request was stale and that they'll need to refresh again? 15s is pretty quick, but how do caching schemes work when users need to know when to use their time refreshing and checking the results. Alternatively, I'd be interested to read about the architecture of a modern always fresh caching scheme.

You may even be able to scale horizontally with many nginx+redis nodes, if the cache does not need to be shared.

It's so obvious once I've heard someone else say it but wow that's really clever actually.

I've been running a handful of low traffic services on a $10 VPS from DigitalOcean for years. I'm currently in a situation where a thing I've made might blow up, and I had to look into CDNs just in case. It's just static content, but updates once per day (it's a daily logic puzzle).

I must admit I had no idea what goes into runnig a CDN. First I had a look at DO's spaces object storage which has CDN support. But it wasn't exactly the right tool for serving a static website apparently, but rather for serving large files. For example I couldn't make it serve css files with the correct mime type without some voodoo, so I had to conclude I wasn't doing the right thing.

Then I looked at DO's app platform. But that seemed like an overkill for just sharing some static content. It wants me to rely on an external service like GitHub to reliably serve the content. I already rely on DO, I don't want to additionally rely on something else too. Seems like I could also use DO's docker registry. What? To serve static content on CDN I need to create a whole container running the whole server? And what do I need to take into consideration when I want to update the content once per day simultaneously for all users? It's easy when it's all on my single VPS (with caching disabled for that url) but I actually have no idea what happens with the docker image once it's live on the "app platform". This is getting way more complex than I was hoping for. Should I go back to the spaces solution?

Right now I'm in a limbo. On one hand I want to be prepared in case I get lucky and my thing goes "viral". On the other hand my tiny VPS is running on 2% CPU usage with already quite a few users. And if I do get lucky, I should afford doubling my VPS capacity. But what about protection from DDoS? Anything else I should worry about? Why is everyone else using CDN?

And I don't even have caching! An article like this puts my problem into shame. I just want to serve a couple of plain web files and I can't choose what I should do. This article really shows how quickly the problem starts ballooning.

It sounds like you might be looking at the wrong place. There's services like bunny.net and cloudflare CDN (I'm not affiliated with either, but I use the former) that are really easy to set up and configure, if you've built your site properly (edit for clarification: if you have clearly defined static content, and/or you're using some build system). You don't want to 'run' a CDN, you want to use one.

Configuration depends a lot on the specifics of your stack. For Svelte, you can use an adapter (https://svelte.dev/docs/kit/adapters) that will handle pointing to the CDN for you.

Cloudflare's offering is free, bunny.net is also probably going to be free for you if you don't have much traffic. CDNs are great insurance for static sites, because they can handle all the traffic you could possibly get without breaking a sweat.

If it’s updated daily, maybe host it on GitHub pages (public repo) and have a job push an update on a daily basis?

If you want to be more fancy, you could do private GitHub repo+ Cloudflare pages enabling you to prepare puzzles in advance.

You can get quite far with both service’s free offering or Cloudflare’s $5/mo stuff.

Full disclosure: I work for neither companies but I have regularly been a happy paid customer of both products.

My blog is on a totally overprovisioned VPS and I use Cloudflare to cache responses for non-logged-in users. I'm using Mediawiki because I want people to be able to edit it and sometimes it can be extraordinarily slow but with Cloudflare it loads instantly if someone visited it some time recently. I really like Cloudflare's caching control.

When I last benchmarked Redis vs. PostgreSQL for a simple k/v cache it was about ~1ms for PostgreSQL to fetch a key, and ~0.5ms for Redis. Faster, but not really noticeably so. I haven't benchmarked SQLite, but I would be surprised if the numbers are substantially different.

Of course Redis can do other things than just a k/v cache, and at scale you just want to offload some load from your main SQL server. But for "small" use cases my conclusion was that Redis doesn't really add anything. OTOH it's also not especially difficult to run so it also not a big problem to use it, but by and large it seems superfluous.

> stale-while-revalidate is a relatively recent cache-control option

Varnish Cache have supported that since version 4.1, it was released 2015-09-30:

> Varnish will now use the stale-while-revalidate defined in RFC5861 to set object grace time.

https://varnish-cache.org/docs/trunk/whats-new/changes-4.1.h...