Most active commenters
  • zelphirkalt(5)
  • preisschild(3)

←back to thread

1101 points codesmash | 38 comments | | HN request time: 0.436s | source | bottom
1. miki123211 ◴[] No.45139800[source]
I've been dealing with setting up Podman for work over the last week or so, and I wouldn't wish that on my worst enemy.

If you use rootless Podman on a Redhat-derived distribution (which means Selinux), along with a non-root user in your container itself, you're in for a world of pain.

replies(13): >>45139949 #>>45139952 #>>45140035 #>>45140041 #>>45140112 #>>45140315 #>>45140558 #>>45140561 #>>45140736 #>>45140993 #>>45141204 #>>45141405 #>>45142506 #
2. mixmastamyk ◴[] No.45139949[source]
Sounds like you need to grant the user sufficient permissions. What else might go wrong?
replies(2): >>45140038 #>>45140065 #
3. marcel_hecko ◴[] No.45139952[source]
I have done the same. It's not too bad - just don't rely on LLMs to design your quadlets or systemd unit files. Read the docs for the exact podman version you use and it's pretty okay.
4. prmoustache ◴[] No.45140035[source]
How so? I have been using exlusively podman on Fedora for the most part of the last 7 years or so.
replies(1): >>45140335 #
5. iTokio ◴[] No.45140038[source]
Mounting Volume and dealing with FS permissions.

They are many different workarounds but it’s a known pain point.

6. sigio ◴[] No.45140041[source]
I've setup a few podman machines (on debian), and generally liked it. I've been struggling for 2 days now to get a k8s server up, but that's not giving my any joy. (It doesn't like our nftables setup)
7. marcel_hecko ◴[] No.45140065[source]
It's mostly the subgid subuid mapping of ids between guest and host which is non trivial to understand in rootless envs. Add selinux in the mix....
replies(2): >>45140439 #>>45140697 #
8. Nextgrid ◴[] No.45140112[source]
I've never seen the benefit of rootless.

Either the machine is a single security domain, in which case running as root is no issue, or it's not and you need actual isolation in which case run VMs with Firecracker/Kata containers/etc.

Rootless is indeed a world of pain for dubious security promises.

replies(3): >>45140192 #>>45140341 #>>45141552 #
9. bbkane ◴[] No.45140192[source]
I see your point but I wouldn't let the perfect be the enemy of the good.

If I just want to run a random Docker container, I'm grateful I can get at least "some security" without paying as much in setup/debugging/performance.

Of course, ideally I wouldn't have to choose and the thing that runs the container would be able to run it perfectly securely without me having to know that. But I appreciate any movement in that direction, even if it's not perfect.

10. thyristan ◴[] No.45140315[source]
Yes, but the reason for that pain is SElinux. The first, second and third law of RedHat sysadmin work is "disable SElinux".
replies(1): >>45141180 #
11. goku12 ◴[] No.45140335[source]
That surprises me too. Podman is spearheaded by Redhat and Fedora/RHEL was one of the earliest distros to adopt it and phase out docker. Why wouldn't they have the selinux config figured out?
replies(1): >>45140721 #
12. mbreese ◴[] No.45140341[source]
One of the major use cases was multi-user HPC systems. Because they can be complicated, it’s not uncommon for bioinformatics data analysis programs to be distributed as containers. Large HPC clusters are multi-tennant by nature, so running these containers needs to be rootless.

There are existing tools that fill this gap (Singularity/Apptainer). But, there is always friction when you have to use a specialized tool versus the default. For me, this is a core usecase for rootless containers.

For the reduced feature set we need from containers in bioinformatics, rootless is pretty straightforward. You could get largely the same benefits from chroots.

Where I think the issues start is when you start to use networking, subuids, or other features that require root-level access. At this level, rootless because a tedious exercise in configuration that probably isn’t worth the effort. The problem is, the features I need will be different from the features you need. Satisfying all users in a secure was may not be worth it.

replies(1): >>45145952 #
13. galangalalgol ◴[] No.45140439{3}[source]
What actual issues do you run into? We have selinux and rootless and I didn't notice the transition from docker as a user.
14. Insanity ◴[] No.45140558[source]
We went through an org wide Docker -> Podman migration and it went _relatively_ smooth. Some hiccups along the way but nothing that the SysDev team couldn't overcome.
15. jwildeboer ◴[] No.45140561[source]
Sure. Constructing the case to shoot yourself in the foot is not a big problem. But in reality things mostly just work. I’m happily running a bunch of services behind a (nginx) reverse proxy as rootless containers. Forgejo, the forgejo runner to build stuff, uptime-kuma and more on a bunch of RHEL10 machines with SELinux enabled.
replies(1): >>45141156 #
16. strbean ◴[] No.45140697{3}[source]
> subgid subuid mapping

trigger warning please D:

17. znpy ◴[] No.45140721{3}[source]
They have.

Most likely gp is having issues with volumes and hasn’t figured out how to mix the :z and :Z attribute to bind mounts. Or the containers are trying to do something that security-wise is a big no-no.

In my experience SELinux defaults have been much wiser than me and every time i had issues i ended up learning a better way to do what i wanted to do.

Other than that… it essentially just works.

replies(1): >>45147689 #
18. znpy ◴[] No.45140736[source]
Your issue is selinux then, not podman. It’s not correct to blame it on podman.
replies(2): >>45145927 #>>45147716 #
19. zamalek ◴[] No.45140993[source]
As a huge fan of podman this is definitely one of my disappointments. In the event that you're still struggling with this, the answer is using a --user systemd quadlet. You'll need to use machinectl (machinectl shell <user>@.host) for systemd commands to work, and you'll want to enable linger for that user.

One thing which just occurred to me, maybe it's possible to have a [container] and a [service].user in a quadlet?

20. preisschild ◴[] No.45141156[source]
Do you do OCI/container builds inside your forgejo-runner container?
replies(1): >>45141390 #
21. preisschild ◴[] No.45141180[source]
> The first, second and third law of RedHat sysadmin work is "disable SElinux".

Must not be a good sysadmin then. SELinux improves the security and software like podman can be relatively easily be made to work with it.

I use podman on my Fedora Workstation with selinux set to enforce without issues

replies(1): >>45147732 #
22. YorickPeterse ◴[] No.45141204[source]
Meanwhile it works perfectly fine without any fuzz on my two Fedora Silverblue setups. This sounds less like a case of "Podman is suffering by definition" and more a case of a bunch of different variables coming together in a less than ideal way.
23. mfenniak ◴[] No.45141390{3}[source]
People having trouble getting this configured is a common issue for self-hosting Forgejo Runner. As a Forgejo contributor, I'm currently polishing up new documentation to try to support people with configuring this; here's the draft page: https://forgejo.codeberg.page/@docs_pull_1421/docs/next/admi...

(Should live at https://forgejo.org/docs/v12.0/admin/actions/docker-access/ once it is finished up, if anyone runs into the comment after the draft is gone.)

replies(1): >>45142116 #
24. ThatMedicIsASpy ◴[] No.45141405[source]
SELinux has good errors and all I usually need is :z and :Z on mounts
replies(1): >>45141515 #
25. gm678 ◴[] No.45141515[source]
Can confirm, have been doing exactly what GP says is a world of pain with no problems as soon as I learned what `:z` and `:Z` do and why they might be needed.

A good reference answer: https://unix.stackexchange.com/questions/651198/podman-volum...

TL;DR: lowercase if a file from the host is shared with a container or a volume is shared between multiple containers. Uppercase in the same scenario if you want the container to take an exclusive lock on the volumes/files (very unlikely).

replies(1): >>45147704 #
26. pkulak ◴[] No.45141552[source]
Rootless is nice because if you mount some directory in, all the files don't end up owned by root. You can get around that by custom building every image so the user has your user id, but that's a pain.
27. preisschild ◴[] No.45142116{4}[source]
Im not hosting a Forgejo instance (yet), but self-hosted Gitlab with gitlab-runner in Kubernetes, so I was wondering how you solved this.

I'm using dind too, but this requires privileged runners...

28. jimjimwii ◴[] No.45142506[source]
My anecdote: I've been using rootless podman on Ubuntu in production environments in multiple organizations (both startup and enterprise) for years without encountering a single issue related to podman itself.

I'm sure what you wrote here is true but i cant fathom how. Maybe its a rh specific issue? (Like how ubuntu breaks rootless bwrap by default)

29. master_crab ◴[] No.45145927[source]
It’s always selinux. I’m surprised parent didn’t figure that out
30. lmm ◴[] No.45145952{3}[source]
> Large HPC clusters are multi-tennant by nature, so running these containers needs to be rootless.

I can't see how any kind of sensible security evaluation process would reach that conclusion. If you trust your users you don't need rootless, if you don't trust your users rootless containers aren't good enough. I suspect people do rootless because it seems easy and catches a few accidental mistakes rather than it being a legitimate security measure.

replies(1): >>45146490 #
31. mbreese ◴[] No.45146490{4}[source]
Think R1 research university or government lab level HPC clusters…

These are almost always multi-tennant with differing levels of trust and experience between users. The data processed here can often have data access agreements or laws that limit who can see what data. You can’t have a poorly configured container exposing data, for example. So, the number of people who have root access is very limited. Normal users running workflows would all be required to run code rootless.

32. zelphirkalt ◴[] No.45147689{4}[source]
I personally like the verbose notation for docker volumes in docker compose files, where source and target are separate attributes in the YAML file. Not all munged into one long string, and unable to specify the type of mount explicitly. But that notation does not support stating the :z or :Z. I am running a Debian most of the time to develop and had no issue with the docker bind mounts, but on Fedora Selinux messed things up and I would get strange permission denied errors in the container for bind mounted config files. So I would have to change my docker compose file just for Fedora and Selinux. I think I even tried it with one of z: or Z:, but still Selinux interfered. At some point I had the choice of burning many more hours into configuring Selinux, disable Selinux, or reinstall docker as root. Since the Fedora OS is merely a VM, I chose to install Docker as root.

My point is: If figuring things out with podman is similar to my experience, I understand why people don't want to do that. Do they have a definitive page dedicated to setting up Selinux for podman, that is well maintained and guaranteed to solve all Selinux issues, and allows me to use bind mounts with readonly permission?

33. zelphirkalt ◴[] No.45147704{3}[source]
How do I make it :ro then? For example it is a good practice to mount config files as readonly. But if I have to use :z, I think I cannot use :ro?
replies(1): >>45147887 #
34. zelphirkalt ◴[] No.45147716[source]
Although it would be podman's job to explain how to set up Selinux, to avoid issues with it (and not just "disable it" as the answer). That is, if they list themselves as available for OS with Selinux.
35. zelphirkalt ◴[] No.45147732{3}[source]
And now comes the part, where you link your guide how you set it up, please! I would like to try exactly that setup and OS. Have a Fedora VM here where I recently struggled with docker and Selinux.
replies(1): >>45151739 #
36. gausswho ◴[] No.45147887{4}[source]
:ro,z
replies(1): >>45147945 #
37. zelphirkalt ◴[] No.45147945{5}[source]
Ah, I never knew this is possible. Thank you!
38. thyristan ◴[] No.45151739{4}[source]
Docker != podman. Entirely different.

With podman, RedHat made an effort to make SElinux work. With Docker, as third-party-software, no proper SElinux config was ever written. With Docker, there is no hope at all that you'd get SElinux to work.

With podman, there is hope, as long as all your containers and usecases are simple, "well-behaved" and preferrably also RedHat-based and SElinux-aware. In the easy cases, podman + SElinux will just work. But unfortunately, containers are the means to get crappy software running, where the developers were too lazy to do proper packaging/installation/configuration/integration. So most cases are not easy and will not work with SElinux, if you don't have infinite time to write your own config...