QubesOS – A reasonably secure operating system

I really like QubesOS, but you cannot run VMs inside a qube, or other things that require VMs like Docker Desktop for Linux, because the xen hypervisor does not support nested virtualization.

Double whammy, because sadly you can't run it in a VM either. At least that's my experience.

> nested virtualization

Such abstraction is very unstable. You can always VNC into a machine from a browser though, so 'vmception' can be achieved.

A qube _is_ a VM though, no? So if you wanted to create a VM, you can create a QubesOS template and instantiate that?

Let me know how it works for you

If your hypervisor supports nested virtualization, it should work. It'll probably grumble about the lack of IOMMU and various other things, but you should be able to run it in VMWare or KVM, if you're willing to jump through some hoops.

Not sure why you'd bother, though. If you're already spinning up VMs, just use that capability. Qubes makes "spinning up a ton of VMs on the iron" a lot easier and more usable.

> Not sure why you'd bother

Because some software expects to run a VM itself such as Docker Desktop which I said in my original comment.

You can. It's just neither recommended nor enabled by default.

https://forum.qubes-os.org/t/nested-virtualization/14790

Poke around /etc/libvirt/libxl and your particular VM's config file. You'll find some lines like:

<feature name='vmx' policy='disable'/> <feature name='svm' policy='disable'/>

Enable it, and you should have working nested virtualization.

And if you'll note, under your original comment I added some detail on how to enable nested virtualization in Qubes.

My understanding is that this is how Qubes itself is tested.

I did that very thing about a year ago when I still had QubesOS installed, and it did not work. There seems to be a lot of misinformation about this swirling around the web. It simply does not work. There is a post somewhere that confirms it but I don't have the link. Unless the QubesOS devs/maintainers made a 180 degree turn since I tried it and decided to start compiling QubesOS with xen nested virtualization enabled, but I doubt it because their reason was that xen's nested virtualization feature is basically broken anyway.

Not a Qubes user, just a curious reader; what is the issue with this? It sounds like you’re implying that what parent said isn’t practical. I’m curious why?

Replied

Shoot, as soon as I hit reply some neurons lit up and now I remember I was actually able to enable nested virtualization in QubesOS, and the relevant options in the VirtualBox preferences inside a qube became enabled once I did that, but whenever I tried booting any VM the whole system hanged. The same system and BIOS settings worked in Ubuntu to boot a nested VM in VirtualBox, so I think I had the BIOS settings correct. Anyhow, it seemed like a dead-end, so I abandoned it.

The issue with trying to hack Docker Desktop and similar tools to boot a qube instead of a VM and somehow hack QubesOS to not isolate those qubes from each other seems self-evident to me and I don't care to explain further.

I'll have to look at it more. I mostly use AMD systems these days, which don't support nested virt in Xen, as I understand it, but it looks like it should work on Intel.

Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux?

And it should be quite lightweight as it’s just a container…

It’s not that I don’t believe you but I don’t understand it… why would you need VM on Linux for Docker?

edit: huh

https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Dock....

that’s… a bit stupid in my opinion. But you can always just use the default daemon so, eh. whatever. maybe I’m wrong. there are reasons I guess

It's a good question - docker doesn't require a VM on Linux, but Docker Desktop does. I assume it's to make it basically the same experience as on Docker Desktop on Windows and macOS, but I'm not totally sure. You can install docker the same way one would on a server in a qube in QubesOS and it works fine, I think I tried that once just to be sure, I just wanted to be able to have Docker Desktop. I also didn't want to paint myself into a corner in case I should need to run something else that also expects to be able to run a VM.

I was on Intel when I tried. No worries though, not really planning on trying it again.

> Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux?

docker has been playing rather loose with security for quite a long time in the past, so I per-se wouldn't trust it to be secure

additionally there are (oversimplified) 3 ways to run it:

1. a "root" deamon, the default docker daemon setup in most distros and the way docker originally was designed to be used. This is needed for some advanced (non secure) features (which IMHO should not exist) but has pretty serve security issues IMHO, like either being in the docker group being a de-facto root/sudo or it forcing you to use sudo all the time and bugs in the docker deamon potentially leading to catastrophic exploits.

2. as a unprivileged user using cgroupsv2 like podman does by default, this has some limitations but works really well and as long as "linux containers" are secure it should be secure (oversimplified). Still the security of containers isn't perfect as they share the kernel. As a rule of thump only use if you get reliable kernel security updates.

3. starting VMs, much more overhead in many ways and needing virtualization, but works with old "outdated" kernels or hardened kernels(1) not allowing unprivileged cgroups

The 3rd method is the most secure and works across most dirstros/linux setups even if unusual setup and is the same kind of path docker takes on windows/mac. So no wonder a product like Docker Desktop uses it. Also the overhead of virtualization is today not "that" bad.

The the first method is still the default method for docker CLI on most linux distros is honestly far beyond my understanding. In the past pre reliable cgroups v2, maybe, but today omg no. It has some valid use cases, but the default should be VMs or unprivileged containers especially on desktop systems (whatever distro maintainers prefer).

----

(1): On instresting conflict is security hardening vs. unprivileged cgroups/containers. On one side unprivileged cgroups allow a lot of additional security. For example they allow running various sandboxes without needing a suid binary or using sudo for setting up the sandbox. Which is really grate. At the same time (oversimplified) it increases the security sensitive kernel interface by a non negligible margin, which isn't grate and hence why hardened linux often disables it by default.

EDIT: Sorry for the bad spelling, I need to go to bed.

As someone who often runs multiple entire virtualized datacenters on top of ESXi hosts which are actually virtual machines themselves running in ESXi, I highly disagree.

Nested virtualization is a game changer in any kind of lab scenario. It also runs great on Chromebooks. The ChromeOS Linux environment is actually a KVM, so launching virtual machines from there is nested but you would never know it (based on my experience with the (framework Chromebook edition). Having 64GB of memory doesn't hurt either of course.

Technically correct, but kind of a shitty answer, no?

Do you want an answer to that question or do you want an argument?