Go-Safeweb

Handling https in the project also adds tons of complexity in the long run though: tls/ssl library versions, cert handling. Instead of having one way to deal with all of them (at the proxy layer, or sometimes at network layer), I have to deal with individual software way of managing those

I think you’re vastly overestimating the complexity of pointing a TLS library to a CA.

I mean, tons of "old-school" services handle these things fine for the last two decades, at least. It can't be that hard. It's just a TLS library, and some files in a specific format at the end of the day.

In my experience, most people have an extremely hard time wrapping their minds around how to configure TLS/HTTPS services and fail completely at understanding how it works.

For a single application, it's not too bad. When you have dozens of applications that all have different mechanisms to install a CA, rotate certs, etc. And some of those don't have a good way to automate rotating the certs, then it becomes a pain.

Their point isn’t about the complexity of installing a certificate. It’s doing it successfully and securely at scale.

Everything is easy until you have to do it at scale.

I don't think so. We run a horde of machines, and run plethora of services, which are custom and/or has a very narrow install base due to the niche they serve.

99% of them use system-wide PKI store for CA and their certificates, which is under /etc. All of them have configuration options for these folders, and have short certificate lives because of the operational regulations we have in place.

At worst case, we distribute them with saltstack. Otherwise we just use scp, maybe a small wrapper script. Managing them doesn't take any measurable time for us.

...and we have our own CA on top of it.

Do that for a bunch of different applications and you hit interesting issues. For example the Java TLS stack, which doesn’t accept a PEM certificate on its own, but needs the full certificate chain. Kibana, however, requires the full certificate chain including the root certificate, which isn’t usually a part of the certificate itself, and Elasticsearch complains about an invalid certificate if you point it to the same one.

So even for two apps from the same vendor, which are commonly deployed together, you need bespoke TLS file variants. Scale that to more applications, and you’ll find out the hard way that you are vastly underestimating the complexity of operating a software ecosystem.

> 99% of them use system-wide PKI store for CA and their certificates, which is under /etc.

Consider yourself lucky then.

For self-hosted third party software, I've seen requirements to provide it in an environment variable, upload it to a web form over plain http on localhost, specify an AWS secret service secret that contains it, put it in a specific location that is bind mounted into a container, create a custom image (both VM and container), etc.

I’ve done it. What you’re describing is like an hour of work. Moving TLS outside of the application is possibly the dumbest reason to spend the resources and complexity on a side car.