←back to thread

Be Aware of the Makefile Effect

(blog.yossarian.net)
431 points thunderbong | 3 comments | 11 Jan 25 04:13 UTC | HN request time: 0s | source
Show context
solatic ◴[11 Jan 25 21:15 UTC] No.42668983[source]▶
> Think about CI/CD setups, where users diagnose their copy-pasted CI/CD by doing print-style debugging over the network with a layer of intermediating VM orchestration. Ridiculous!

I don't think the author understands the point of "CI/CD systems". And I don't really blame them, because workload orchestration systems have been abused and marketed to the point where we call them CI/CD systems instead. Sure, if you think the point of CI/CD is to just provide a unified build and deploy it somewhere, you can write that in whatever language you like, and not need to know a bunch of YAML-fu.

But the whole point of workload orchestration systems is to configure the flow of workloads between machines as they inherently execute on different machines. The status quo is to debug over the network because, fundamentally, different machines will be connected by a network and the workload orchestration system is figuring out which machine to put it on.

If you think you can just run your CI/CD on a single server without networking or virtualization, I have some very large, parallelized testing suites to show you.

replies(1): >>42669126 #
semi-extrinsic ◴[11 Jan 25 21:41 UTC] No.42669126[source]▶
> If you think you can just run your CI/CD on a single server without networking or virtualization, I have some very large, parallelized testing suites to show you.

Nowadays you can get a single server with 256 cores and several terabytes of memory. I would be interested to learn what kind of testing suites have actual needs beyond that.

Without virtualization though is definitely no problem. The whole docker/k8s/whatever shtick is mainly because devs think it's more fun to invent new things than to learn how to use old ones properly. At least as long as you're running your own code on your own hardware, there is not a single thing solved by virtualization that wouldn't be solved equally well (or better) with traditional tools like environment modules and Slurm.

replies(2): >>42669515 #>>42671732 #
1. upbeat_general ◴[11 Jan 25 22:40 UTC] No.42669515[source]▶
For a start, any suite that takes >X hours on a single node, especially compounded if you have a large team of developers.

> At least as long as you're running your own code on your own hardware

Assuming you keep a consistent env/OS across all nodes you will want to run said code. Which can be difficult, even just between two users on a single node.

Not to mention the fact that a lot of (most?) code needs to (A) interoperate with other people's code and (B) at least sometimes run on other hardware.

replies(1): >>42681233 #
2. semi-extrinsic ◴[13 Jan 25 08:15 UTC] No.42681233[source]▶
> For a start, any suite that takes >X hours on a single node, especially compounded if you have a large team of developers.

If your testing suite takes several hours to run on a 256 core server, and this is something you want to run on every commit by every dev, then you have a problem with your testing suite. Running it on k8s is just slapping a bandaid on it.

> Assuming you keep a consistent env/OS across all nodes you will want to run said code. Which can be difficult, even just between two users on a single node.

> Not to mention the fact that a lot of (most?) code needs to (A) interoperate with other people's code and (B) at least sometimes run on other hardware.

Yes, this is the problem that has been solved since the 90s by environment modules. This is how clusters and supercomputers work. There is no virtualization, just tcl and shell scripts.

replies(1): >>42686772 #
3. upbeat_general ◴[13 Jan 25 18:29 UTC] No.42686772[source]▶
Clusters and supercomputers are able to fully control the hardware & OS. environment modules are useful in that setting but do not solve OS differences, not to mention themselves need to be installed/maintained. lots of time developers do not work on your same cluster (say, for any open-source code) and environment differences can be significant.

I regularly use large clusters and even there docker/singularity are very helpful at times (very simple example, glibc requirements).

When you are talking about well-written code that only requires posix (and nothing beyond) and does not interface with hardware, etc. etc. then virtualization seems crazy.