Red Hat to contribute container tech (Podman, bootc, ComposeFS...) to CNCF

What took them so long?

Waiting for IBM to buy them? (half joking)

My take was they sort of dug in said "Docker isn't made right for Linux, we're reinventing it"

On Fedora w/ SELinux that led to quite a bit of compatibility issues for a while with lots of quirky things that didn't behave the same.

I think their implementations have gotten pretty stable and improved in compatibility since then

Does podman support docker compose files well? Devs love them for local environments.

Reading about Keycloak and how long it is taking to patch critical vulnerabilities, I wonder is CNCF becoming how Apache was - where abandoned open source software goes to die.

That's my take as well. Red Hat's design choices fit into Linux much more neatly. Docker has always been rubbish with late cgroups v2 support, punching holes in my firewall, no rootless, etc.

I've been using podman-compose, yes.

I’ve been using it on my Fedora server because I make myself. I think all functionality and syntax is covered. However, the user feedback and TUI of docker-compose is way nicer (interactive at least). Also, podman compose does seem to recreate containers that do not need to be recreated in more cases than I have noticed docker compose do.

> punching holes in my firewall

I teach various Linux training courses. One of which is Containers. It always shocks several people per-class how Docker just blatantly ignores and rewrites existing firewall rules. And there's no real option to prevent that unless you want to manually configure ALL network routing.

For me personally, that was one of the big issues the pushed me over to Podman.

Also, Docker's insistence on "forcing" and preventing the disabling of using the malware-ridden Docker Hub didn't help me appreciate their security practices.[]

[]

https://jfrog.com/blog/attacks-on-docker-with-millions-of-ma...

https://www.infosecurity-magazine.com/news/malicious-contain...

https://www.bleepingcomputer.com/news/security/millions-of-d...

https://www.bleepingcomputer.com/news/security/docker-hub-re...

https://sysdig.com/blog/analysis-of-supply-chain-attacks-thr...

... ETC ...

I use podman with docker-compose files for my day-to-day work; spinning up databases and other service dependencies for locally running or containerized webapps.

podman-compose never worked very well for me, so I'm running with the podman.socket systemd service and the standalone version of docker-compose. That is however working flawlessly.

What I really like about podman (and which to be fair docker might have since catched up on) is that rootless containers work so well. Gone are the days where bind-mounting a project folder into a container would mess with your file permissions.

In my experience podman also feels easier and less invasive to install, although I can't say if the latter is really the case.

I think that CNCF has better handle on abandonware, plus really good observably. https://devstats.cncf.io/

You can run the standalone version for docker-compose against podman. You just need to have the podman.socket systemd service running.

Try podman kube generate and podman kube play. Podman can generate a Kubernetes YAML, and then you can run it with kube play.

I use it together with systemd in my home lab. It's Kubernetes for single node and without the bloat. I love it!

https://www.redhat.com/en/blog/kubernetes-workloads-podman-s...

My only problems with Podman is the lack of up to date repos across systems, the fact that the latest raw binaries are managed by a maintainer out of the goodness of their heart, and that the VS Code extension ecosystem for managing pods is not integratable with the existing Docker stuff (and the replacement extensions are woefully underdeveloped).

Otherwise it honestly is great and preferable over Docker.

I want to switch to podman. What are the general gotchas and difficulties you could see in doing that for multi architecture+os builds/deployments?

Podman actually works really well. Out-of-the-box virtually-no-configuration-needed rootless containers. It's also usable via docker-compose with a single env variable. (podman-compose wasn't up to par for us)

We've been using it for a couple of years running and managing hundreds of containers per server - no feeling of flakiness whatsoever. It's virtually zeroconf and even supports GPUs for those who need it. It's like docker but better, IMO.

Hope it gets a popularity boost from CNCF. Rooting for it.

I won't deny that the outdated repos was a pain in the past, but ever since ubuntu got to version 4 it's been working flawlessly for me.

I think version 4 was where podman became a reliable tool, whereas I found it to be flaky and unreliable in previous versions.

I don't use vs code extensions for managing my containers, so I can't say much about those, but I wonder if many of them won't work fine with an alias for docker and maybe the podman-socket running.

For more information on compose files take a look at the Compose Spec [0], looks like podman compose supports the Compose Spec which Docker compose files use as well.

[0] https://github.com/compose-spec/compose-spec?tab=readme-ov-f...

I completely agree and have had the same experience as you with docker-compose working better than the alternatives.

Past versions of podman were flaky, but since version 4, which is now a couple of years old, I haven't had any issues whatsoever. I'd recommend anyone using containers on linux to try it out instead of installing docker out of habit.

Or The Onion!

The IO through fuse-overlay is performance limiting though. It's almost half the speed as overlay directly for layers with many tiny files.

Note that Linux allows you to mount overlay within a user namespace if you are root within the user namespace.

In other words, if you are root within a container; even though it is not root on the host; Linux accepte ton mount overlay filesystems (most filesystems are not allowed). `man user_namespace`

> docker-compose with a single env variable what is that env variable?

Yes it doesm even with rootless. We are using it in prod.

Hopefully the Keycloak thing will spur more competition. I looked at some alternatives and settled on Keycloak because it was "obviously" the mature and hardened solution.

Well, clearly not.

To all those interested in podman, this book by Daniel Walsh is a gem. Highly recommended and it is free.

https://developers.redhat.com/e-books/podman-action

If podman compose would parse env var strings correctly, then it would be on par. Not sure why that hasn’t been fixed but probably because it’s a stepping stone instead of a well thought out replacement.

Is CNCF new Apache foundation? Looks like everyone dumps their stuff there. Does not look promising. Am I missing something? Probably RedHat paid salary to podman developers, but who will pay salary to them now?

Strimzi is CNCF, Strimzi still has a full time team of devs in RH.

> It's also usable via docker-compose

Is that "docker-compose" (with a dash) or "docker compose" (with a space)?

Its broken with alias but what's this about the podman-socket? Do you know where I can take a look at that?

I only dislike Podman because some distributions used it as an alias for docker which made a lot of docker-compatible software to not work on that distribution unless some workarounds. I wouldnt normally blame the application for this but in this case they are both, application and distribution, from the same dev.

Agreed, the `podman` command is 95% drop-in compatible with the `docker` command, but those edge cases are annoying and I would rather just use the docker cli backed with podman running the containers.

Probably DOCKER_HOST[0][1]

[0]: https://docs.docker.com/reference/cli/docker/#environment-va...

[1]: https://podman-desktop.io/docs/migrating-from-docker/using-t...

Both should do exacly the same, they are just installed differently. docker compose is installed as docker CLI plugin (Linux only), and docker-compose is installed as standalone binary.

See ref: https://docs.docker.com/compose/install/#scenario-two-instal...

Podman has a docker frontend. On Fedora, it is packaged as podman-docker, I believe. I recently went through the pain of getting testcontainers working on Fedora 41 with Podman. After enabling the Podman socket and setting an environment variables, I was off to the races!

There are subtle differences between the two and not exactly the same.

I think you can use the VSCode Docker extension if you enable the Podman socket.

That would be news to me, as both are pointing to the exact same GitHub repository[0]. Can you name the differences?

[0]: https://github.com/docker/compose

+1, Podman is great. I have been running it for a while on NixOS.

But Compose doesn’t mesh well with the overall NixOS configuration system. So I ended up building a custom tool that can convert your existing Compose project into a NixOS config.

You might just be convincing me to switch, I generally love docker and compose but the firewall thing still blows my mind and that there still just is not a solution.

My workaround has been to bind all docker port forwards to localhost and only ever expose them externally via reverse proxy. Which is annoying because that means I can't run the reverse proxy itself in docker.

You may need to do

  podman system reset

The Linux kernel only gained unprivileged overlay recently. Kernel fuse and fuse-overlay are incompatible so you need to wipe everything.

You may need to set

  [storage]
  
  driver = "overlay"

in storage conf as well.

https://docs.podman.io/en/stable/markdown/podman-system-rese...

I vastly prefer it to Docker, especially buildah over buildx. Instead of inventing yet-another-dsl buildah allows you to simply use shell scripts (though it does also support dockerfiles). Another thing buildah is really good at is not doing much automatically: you can really optimize layers if you care to.

The Podman ecosystem has given me a strong disliking of the Docker ecosystem, so I'm also rooting for it.

I'm sure Red Hat will continue to pay Podman developers, just like they continue to pay developers for the other upstream projects that are hosted at CNCF (like Kubernetes).

I'm we can all think of some projects "abandoned" to foundations through the years, but in general, I'd call getting core infrastructure out of the control of a single company and into a place with more transparent and democratic governance a good thing.

Dumb question: is it rootless for users on something like macOS?

I'd love to get the benefits of Docker without the battery drain and the Docker software, but I'm not sure if Podman would help much with either.

Zitadel is worth a look, we're replacing Keycloak with it currently.

Yes indeed. I have that on other systems as well. But I try to keep both around to notice these kind of differences. (I’m working on tooling that relies on docker compose files so I like to see how it behaves in different setups.)

On macOS it creates a centos VM to run containers in. Rootless simply means that the root user in a container maps to the runner outside and not as the actual system root.

Edit: .. because the runner is not needing to run as root

Usually, when big orgs like that dump their projects to such a foundation (like Apache), it is that they are about to drop investing in support it soon.

In the same way as how the Kubernetes ecosystem is the new Enterprise Java ecosystem? Often even from the same companies as back in the late 90s/00s.

Look: I'm probably ignorant, but from the outside the similarities seem striking.

Please explain why I'm wrong. I'm humble on this one.

I'm not sure why you think that. Plenty of CNCF projects, if not all of them, are still developed by the same corporate teams that originally developed them, even though ownership has been transferred. Cilium is still being developed by paid Isovalent employees. I'm pretty sure Kubernetes is still maintained mostly by Google employees. Longhorn and k3s are maintained by SUSE employees. This isn't Red Hat's first foray. They bought CoreOS, which is the company that originally developed etcd, one of the oldest CNCF projects that even predates Kubernetes, and most of its maintainers are still employees of either Red Hat or Google.

Being able to play around with kube files and then just shove them into a my-seevice.kube and execute them via systemd is really neat. The documentation is pretty OK as well: https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

The only downside is that some things are not yet supported in quadlets, and that things don't map 1 to 1 between quadlets and the command line.

I think people are missing the contribution of bootc and composefs. This is a big part of what undergirds Red Hat's new 'image mode' means of deployment. They're using container-related tooling to deploy whole operating systems, and it's a large part of where they're headed.

I write this to say, "This is not them dumping abandonware." To me, it's them putting these technologies under the supervision of a neutral third party to encourage adoption.

Important to note, this book is from early 2023 and supports Podman version 4.1. It's missing newer features like quadlets.

https://www.redhat.com/en/blog/quadlet-podman

This is cool and all I just want to make sure podman and others are maintained and useful. I’m sure they will be it’s just that I use podman every day and depend on it.

I could go back to docker but why?

I think being a CNCF project is better than not being one. It gives it more visibility and structure and thus is less likely to be abandoned.

But sure, unfortunately if not enough different companies and individiuals are maintaining stuff it gets abandoned.

The podman team and my team, working in the virtualization side (Podman Machine), will continue doing so

Previous docker-compose was a separate program, written in Python if I remember correctly, people usually preferred to them as v1. Later docker incorporated it into the docker binary itself as a subcommand so that’s v2

> Rooting for it.

No root necessary :)

I've replaced keycloak with https://goauthentik.io/ 2 years on, no complaints

Hmm maybe worth switching from Docker Desktop to Podman Desktop...

Last I checked, Keycloak has increased in activity since joining CNCF...

https://keycloak.devstats.cncf.io/d/1/activity-repository-gr...

CNCF has probably 20x the funding of the ASF and is a different organization that spends millions of dollars on security audits, events and more, you can read about it in our annual report: https://www.cncf.io/reports/cncf-annual-report-2023/

Also we actively remove/prune projects that aren't active... we will probably archive ~10 this year https://www.cncf.io/project-metrics/

I think I might be the only one that prefers Docker for building Docker containers using CI.

I use Drone, but instead of using the Docker plugin I start a detached (background) Caddy server to work as a proxy to DOCKER_HOST. That lets me proxy to the local Docker socket to take advantage of caching, etc. while I'm iterating, but gives the option of spinning up docker-in-docker to get a clean environment, without any caching, and running a slower build that virtually identical to what happens on the CI server.

I find that having the daemon available solves a ton of issues that most of the CI provided builder plugins have. For example, with the builder plugins I'd always end up with a step like build-and-tag-and-push which didn't work very well for me. Now I can run discreet build steps like build, test, tag, push and it feels far more intuitive, at least to me.

> Rooting for it.

I wanted to say something funny about "rooting" and "rootless", but it's probably too silly. :)

Ruthlessly rooting for rootless!

Yes. Definitely already needs a second edition. I would happily buy it.

> It always shocks several people per-class how Docker just blatantly ignores and rewrites existing firewall rules.

Yeah. Many times I've mentioned that to people, and they just don't believe it's a thing which Docker does. Including here on HN. :/

Composefs has a totally crucial feature that sold me immediately, which is that if two containers use the same file, the kernel can serve that file from the same page cache. This means you can hypothetically launch a lot of containers, and if they share some layers/base images, the memory usage can still be quite reasonable. Nice. https://github.com/containers/composefs?tab=readme-ov-file#b...

Kubernetes is the new websphere. Yaml the new ANT.

The font on that book is so awful. As someone with bad sight i have to strain my eyes a lot to read even a single sentence.

v2 is still a separate binary, it can just be installed in different ways (on Linux). If GGGP was referring to v1, then that's legacy software since 2+ years and they probably shouldn't use it.

The one problem I previously hit was that the ansible integration used the Python version; I hope they've fixed it now but haven't looked lately.

Maybe open source foundations should be more selective with the projects. I'm thinking of Oracle dumping OpenOffice, although it was obvious that LibreOffice is the way to go.

Looks like v2 is available: https://galaxy.ansible.com/ui/repo/published/community/docke...

If you swap to a content-based object store for container images (like OSTree/Flatpak), you could potentially save a lot of disk space too because you'll no longer need to be careful about your Dockerfile layers.

If two OCI images share the same file, they'll be de-duplicated on disk and only be downloaded once.

Thank you

On the I've hand: that's great news!

On the other:

> The Docker compose CLI plugin has no stable output format (see for example https://github.com/docker/compose/issues/10872 ), and for the main operations also no machine friendly output format. The module tries to accomodate this with various version-dependent behavior adjustments and with testing older and newer versions of the Docker compose CLI plugin. Currently the module is tested with multiple plugin versions between 2.18.1 and 2.23.3. The exact list of plugin versions will change over time. New releases of the Docker compose CLI plugin can break this module at any time.

until you start to use zfs.. (completely doable with fuse overlay, but just painful)

Let me present you a full 212-page RHEL 9 docs PDF on everything Podman, updated as of this month:

https://docs.redhat.com/en-us/documentation/red_hat_enterpri...

yeah, basically verbatim `DOCKER_HOST=unix:///run/user/1000/podman/podman.sock`

AFAIK docker-compose is a an older utility, and `docker compose` is the new-way to do things.

For posterity, there have been some issues when destroying containers. Errors about "inconsistent state of container" or such. But these have always been about non-running containers, so the answer has been destroy/recreate, so no measurable impact for the business. After spawning and destroying thousands of containers in a high-load live environment(across half a dozen servers), I consider podman pretty stable.

And assuming my own comment is high up, this is the env variable we automatically load:

> DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

Honestly everytime i wanted to use podman i hit a bug, which is already fixed most of the time but i can't get the new version because they don't have any direct repos to get updates from, this is very helpful for docker.

Also networking in rootless containers still suck.

The end result is just go bakc to docker with less hassle and better stability.