QubesOS – A reasonably secure operating system

this is my next OS to experiment with. However i'm worried my toy laptop with 16gb of ram (thinkpad t480) wont cut it.

Anyone want to chime in here?

16GB of RAM is adequate for Qubes, in practice you end up running a handful of Xen VMs, with most of the VMs (vault, disposable,...) using little RAM.

The system requirements are specified as:

minimum 6GB RAM; 16GB recommended

https://www.qubes-os.org/doc/system-requirements/

I used it when I worked as a hiring manager. For this task it is ideal. All the behavioral security measures, like only to open attachments from people you trust, break down when your job description is basically to figure out who you can trust.

Qubes comes with a "Convert to trusted PDF" out of the box. Joanna Rutkowska explained how it works under the hood pretty nicely[1]. The tldr is that it is very thorough. With Qubes it is convenient too.

I used Qubes to open the application mails and their attachments and converted the interesting ones to trusted PDFs which I then forwarded to the relevant people. All further communication was only with the trusted versions.

[1] https://blog.invisiblethings.org/2013/02/21/converting-untru...

Last time I tried to throw it on an old box it couldn't handle it due to the lack of IOMMU. Granted this was like... 5 or so years ago? I think my newer laptop could probably handle it, so maybe it would be worth giving another shot.

I really like QubesOS, but you cannot run VMs inside a qube, or other things that require VMs like Docker Desktop for Linux, because the xen hypervisor does not support nested virtualization.

Is QubesOS used in the security companies or community?

I know it’s used in Mullvad, and recommended by Snowden (but he isn’t a security specialist).

I have been playing with it, and it might work as a daily driver.

I've used this as my daily driver before when my workflow needed more crossplatform operability between windows and different varieties of linux. This was a long while ago on some old 2013 laptop with 16 gigs of ram and it worked fine.

The real thing you gotta watch out for is full compatibility with all the virtualization extensions it needs in the processor. Key in that old laptop working was that it was an i7.

I hadn't checked on this project in some time, and last I looked it seemed to be stagnant. Good to see it going strong again.

I just remember that, hardware wise, there wasn't a lot of things that could run all aspects of the OS and be 100% with it. The "certified hardware" on the site also looks like it's out of date.

I used it productively (see my other comment) a couple of years a ago on an average Dell Inspiron. It worked pretty well, no complaints.

Anybody here actually use it day to day? Or is this more of a server OS?

I love Qubes a lot, I daily drove it for a few years and still have it on a laptop. but i would not recommend it even to most technical people, mainly because you forfeit the ability to run things on bare metal. "dom0" is the same as on normal xen - it is a VM and has the associated overhead still. On top of that, the official Qubes dom0 runs a very outdated fedora version.

I am instead writing my own code to automate libvirt to replicate much of Qubes' functionality (ephemeral roots for vms, disposable vm support, GUI isolation) so i can still have Qubes security for most of my apps but fully exercise my hardware when I want.

There are also some minor criticisms i have of Qubes' default, mainly that appvms have passwordless sudo by default and less importantly, no MAC such as apparmor. Passwordless sudo arguably makes it somewhat easier to break out of the VM and no in-vm sandboxing means you have to run every in a separate VM unless you want to set that up yourself.

For example i don't really want my work thunderbird to have access to my work browser, so a single "work" domain isn't enough for me.

It's not often that the benefits QubesOS brings are what security companies (or the community) needs. If you do malware analysis, Windows VMs in isolation on a throw-away laptop is better. If you do penetration testing, Windows or regular Linux will be better, too. If you do extremely sensitive communication you're in a very niche group of people and chances are you're using a provided equipment by your superior (e.g., modified laptops).

Double whammy, because sadly you can't run it in a VM either. At least that's my experience.

See my top level comment about daily driving it.

> Or is this more of a server OS?

It's specifically designed for desktop use

https://www.qubes-os.org/intro/

It is explicitly not a server OS, it really only makes sense as a graphical workstation OS.

It’s meant to be used in a single-user laptop or desktop.

What kind of threat model requires someone to use Qubes? I know Snowden uses it and there's even a testimonial of him on the Qubes site recommending it. Is this for people on 'lists' or are high value targets because they visited the wrong site or said something the authorities didn't like and their machines are now being targeted?

To add, it works quite well if you can accept the caveats of no (easy/default) GPU access (so no gaming). It is not a lot slower if you have enough ram to keep your VMs booted, i mainly noticed a difference in i/o speed but that may have been my particular setup.

You aren't meant to mess with dom0 (so also your desktop) much, so ricers won't like it unless they are willing to break the security model somewhat.

It is also not meant to be a shared PC.

Windows (10) support is pretty mediocre but it does work. I was able to do a WFH job that required windows using it without pain - you don't get seamless windows though.

> nested virtualization

Such abstraction is very unstable. You can always VNC into a machine from a browser though, so 'vmception' can be achieved.

You can use something similar on macOS, Windows or Linux, based on Docker containers, see Dangerzone: https://github.com/freedomofpress/dangerzone

Honestly you don't need to be super paranoid to benefit from Qubes. Windows and Linux default sandboxing is still pretty mediocre at best, people can't be perfect with security all the time and sometimes even legit programs get compromised, so the sandboxing security model helps contain the damage.

That said you can still setup sandboxing without Qubes.

QubesOS is very cool but I've always thought it'd cool/better if it was a patchset or repo on top of an existing distro like Archlinux or NixOS. I think that would be useful so you could adopt features from QubesOS individually and swap out different components. For example, it'd be nice to use KVM (QEMU or even crosvm) instead of Xen or build a Wayland based system instead of X11.

Yeah. I started playing with it about 2 years ago, and I now caution people that "it's really a bit virus-like." It went from being on one "play" computer to being my daily driver OS on 4 machines in that time.

It's fine. You just have to be willing to operate on a "If I can't do it, then I must not need to do it" sort of basis with things like media. But that's no real problem for me, I hate video anyway.

There is actually a project which aims to do that: <https://spectrum-os.org>

Unrelated to QubesOS.

I would say, almost anyone running anything untrusted from time to time can benefit from the strong hardware virtualization.

Qubes is my daily driver by the way. My attempt of explaining it to a layman: https://forum.qubes-os.org/t/how-to-pitch-qubes-os/4499/15

I guess adding more variation to QubesOS systems would make it less secure as there is more room for bugs.

>Qubes OS is a free and open-source, security-oriented operating system for single-user desktop computing. Qubes OS leverages Xen-based virtualization to allow for the creation and management of isolated compartments called qubes.

What's wrong with containers? They are supposed to provide better performance than VMs. Are containers less secure?

It's fine for light to moderate use, just don't expect to run a ton of AppVMs at once.

I've got an X250 (2C/4T) 16GB gutless wonder as my Qubes laptop, and it's fine. Qubes has memory balancing/clawback from VMs, so you can in practice have more AppVMs open than you'd expect - they'll be using less RAM, but it works.

If you're on a short-RAM laptop, though, definitely reduce Dom0's memory allocation. It defaults to 4GB, and you're perfectly fine with 2GB or perhaps even 1GB - there's not much going on in it, and that RAM is better used for AppVMs.

If you're short on cores, you might also look at the "sched-gran=core" flag to Xen. This allows hyperthreading, but ensures that hyperthreads are only ever scheduled in the same VM (and as the threat model assumes "anything in a VM can read anything else in a VM," a hyperthread-based leak doesn't gain an attacker any access you wouldn't otherwise have). The performance gains on a laptop can be noticeable.

Don't expect great battery life, though. Xen's power management is "present and accounted for," at best. There's also an incantation to disable turbo that helps a lot when mobile.

I know what Theo says about (x86) virtualization[1], but I think it's still useful to virtually separate your random browsing the web from things like health and banking, or where you keep your ssh keys (if you don't use a Yubikey or similar to keep it off your laptop) -- or other secrets.

You can be a victim of a random drive-by, you don't have to be a person on a "list".

[1] https://marc.info/?l=openbsd-misc&m=119318909016582

Yes.

Containers generally assume a non-hostile workload and don't make assurances about certain kinds of tampering.

I used it for sifting through job applications (see my other comment).

I think it was worth it because even though we were a small boring company we received a fair share of drag-net style malware via the application channels.

Some of the attempts were pretty good also. I used to joke that the bad guys tailor their cover letters better than most real applicants. Which of course is a bit of a hyperbole, but there is a kernel of truth. It's not too hard to write a believable application letter that fits the archetypal IT position. Similar to not all fishing mails being full of spelling errors malicious application letters aren't either.

I don't know if we ever received a targeted attack, but I wouldn't know. I think bigger companies certainly will and I can only consider every HR department a high risk area.

Containers rely on the security of the operating system. If you can compromise the system you can potentially compromise your neighbors. VMs rely on the security of the hypervisor.

I highly recommend it. Since I tried it a couple of years ago I can't imagine working without it. Examples of things that I can't imagine doing in any other system:

  - curl|bash or similar 
  - pip install, npm install etc
  - run any random github project
  - sudo install the drivers of my Brother printer
  - install zoom
  - plug random cheap USB devices to eg update their firmware

In addition to that you can easily restart frozen VMs without restarting the whole system, keep backups of your VMs (very handy when restoring or changing PCs), reinstall some VMs if you mess up, etc.

Not suitable is for GPU intensive work, though (although if you have a dedicated GPU in theory you can assign it to a VM).

Yes, they are less secure. VMs can rely on HW features to ensure memory isolation.

Containers share the host kernel, thus the attack surface is as large as the kernel functionality that is exported to the container by the host (usually almost all syscalls).

In VMs as far as I know the attack surface is much smaller as the interaction between the guest and host kernel is limited.

Containers rely on the kernel to enforce separation. They're great for keeping trusted workloads from interfering with each other, but I don't trust them for potentially hostile workload separation.

If you can compromise the kernel (and kernel exploits aren't particularly expensive nor uncommon), then a container is like a door locked by a sign that says "Please do not open without permission." If you don't care to go through it, you won't. And if you want to get through it, it doesn't stop you. Once you're in the kernel, containers don't offer any meaningful separation.

Qubes uses hardware virtualization with a fairly stripped down Xen to provide the isolation, and that's a somewhat harder lock to crack open if you want to transit between silos.

A qube _is_ a VM though, no? So if you wanted to create a VM, you can create a QubesOS template and instantiate that?

Anything new from that release a month ago?

https://news.ycombinator.com/item?id=36178205

Runs fine on my framework laptop with 16gb of ram.

> They are supposed to provide better performance than VMs. Are containers less secure?

Yes, the performance advantage is from less isolation/more sharing, and that's also why they are less secure.

The threat model assumes that any code in a VM can, with some application of effort, access anything else in that VM. Given the relatively low cost of local root exploits and kernel exploits, this is reasonable. Passwordless sudo is simply a reminder that you can't rely on intra-vm separation of anything you care about separating.

If you wanted to add additional hardening within a VM, it's supported - create your own templateVM for it, and use it. It's just not the default, and I generally agree with it. If you trust the OS kernel and features to keep things separated, there's no reason to run Qubes in the first place.

I'd say everyone that has the ability to use it. Society is in a dark age where basically nothing else is fit for use; one bad click on most machines could compromise you for years, it is an insane situation. Even if you have other needs that can't be fulfilled with Qubes, you should have a Qubes machine for other tasks that are security-sensitive

> What kind of threat model requires someone to use Qubes?

"Not trusting modern software to be correct nor secure" is sufficient.

I do almost all my web browsing in disposable VMs with no access to interesting things like my password manager, email, SSH keys, etc. I also run JITless (disable Javascript JIT engine), because those are a common attack point on browsers.

If you compromise my browser from a random site, you get nothing of interest. Even if you pop the kernel. You still have to get through Xen to get to anything I consider of value.

This project was never stagnant, a lot of things are always happening here: https://github.com/QubesOS/qubes-issues/issues.

Concerning the certified hardware, few vendors try to make the certification, and also coreboot is required: https://www.qubes-os.org/doc/certified-hardware/#hardware-ce...

>If you trust the OS kernel and features to keep things separated, there's no reason to run Qubes in the first place.

Yeah i see the argument - thats why I would still call Qubes very secure as is - but i personally prefer defense in depth. Mainly it would be helpful on machines with limited ram that can only run a few domains at once.

Let me know how it works for you

If your hypervisor supports nested virtualization, it should work. It'll probably grumble about the lack of IOMMU and various other things, but you should be able to run it in VMWare or KVM, if you're willing to jump through some hoops.

Not sure why you'd bother, though. If you're already spinning up VMs, just use that capability. Qubes makes "spinning up a ton of VMs on the iron" a lot easier and more usable.

Whoa is this firefox containers but at an OS level. Can you SSH to each container without using the window environment? I wanted to use something like proxmox but decided not to because my computer was only 4 cores and 4 threads. If I can use this but "dynamically" allocate the CPUs, it would be perfect for my application.

I didn't know about that but that looks really nice. From a quick glance I understand that they can even utilize OCR to make the trusted PDF into more than an image container. Back in the day when I used Qubes it could not do that. (I haven't used it for a while so I don't know if it can now)

I still think security-wise Qubes is a bit better because it relies on VMs instead of containers.

The problem is that containers rely on the OS kernel to enforce separation, and kernel exploits are an awful lot less rare than anyone would prefer.

If someone is delivering targeted malware to a company through HR channels, it's safe to assume that if they can escape the document viewer, they can probably also try for a local root/kernel exploit and escape the container.

Containers are separation of convenience - not a hard security boundary.

The way I recommend thinking about it is containers work as a "convenient precaution" -

"dumb scripts" that just copy files/install something, encrypt files, etc. will be well contained in a container.

"smart scripts" are more rare - but essentially if you're trying to break out of a container you can, container breakout methods are not uncommon. These types of malware are usually more rare.

So if your threat model is "I want to run this program that I'm pretty sure I trust but I'm not 100% certain" then a container is most likely fine as a convenient precaution.

But if it's "I want to make sure nothing can break out (especially if you're running user's code) and compromise the full system" then you want VMs.

With the recent pytorch-nightly compromise in december, AFAIK a container would have protected you, just don't assume that will always be the case.

EDIT: I wish katacontainers was easier to use and was more widely used - I feel like it gives most of the usability benefits of containers with the security of VM's which is what everyone should really want for most things. VM overhead can be pretty small, with under 100ms "boot" time, etc.

> Not sure why you'd bother

Because some software expects to run a VM itself such as Docker Desktop which I said in my original comment.

> the official Qubes dom0 runs a very outdated fedora version

Why does it matter? You do not run anything in dom0: https://www.qubes-os.org/doc/supported-releases/#note-on-dom...

You can. It's just neither recommended nor enabled by default.

https://forum.qubes-os.org/t/nested-virtualization/14790

Poke around /etc/libvirt/libxl and your particular VM's config file. You'll find some lines like:

<feature name='vmx' policy='disable'/> <feature name='svm' policy='disable'/>

Enable it, and you should have working nested virtualization.

And if you'll note, under your original comment I added some detail on how to enable nested virtualization in Qubes.

If you have the skills to port the tools to KVM, please do so. There's a shortage of sufficiently paranoid low level sorts with the time and interest in the hacking on Qubes.

Getting it working on ARM is also of interest.

Yeah. He's probably right. When we first saw Meltdown/Spectre/etc, and he preemtively disabled hyperthreading out of an abundance of paranoia, turned out he was right...

It's all broken, all the way down. However, compromising a browser or kernel is still a lot easier than compromising a hypervisor. At least in terms of number of known exploits.

Qubes tends to make very limited use of the riskier parts of Xen anyway, though. A lot of the security notices for Xen don't apply to Qubes because of how they've configured things or what features they use.

My understanding is that this is how Qubes itself is tested.

There are ways to get it running without an IOMMU, mostly involving making your NetVM and such paravirtualized. You just lose basically all of the hardware isolation benefits there. But for playing around, especially on a trusted LAN, you should be able to get it working. If you can't figure it out, jump on the IRC channel - there are a few people in there pretty good at weird hardware hacking for Qubes.

And "anything untrusted" includes "any use of the internet."

Browser escapes are cheap enough and common enough that we've seen malware delivered through advertising networks before.

It has remained quite active since I started using Qubes in 2016. I think when Joanna left day to day work it's possible things slowed a bit but I didn't have enough context since that wasn't long after I started.

Based on hanging out in the support forums it seems like there have been some pretty large groups of new users, honestly I have no idea why many came at once but they did. Of course for a project this small, "large" is relative.

The current work includes making dom0 smaller by delegating the gui to a service Qube.

I did that very thing about a year ago when I still had QubesOS installed, and it did not work. There seems to be a lot of misinformation about this swirling around the web. It simply does not work. There is a post somewhere that confirms it but I don't have the link. Unless the QubesOS devs/maintainers made a 180 degree turn since I tried it and decided to start compiling QubesOS with xen nested virtualization enabled, but I doubt it because their reason was that xen's nested virtualization feature is basically broken anyway.

Not a Qubes user, just a curious reader; what is the issue with this? It sounds like you’re implying that what parent said isn’t practical. I’m curious why?

Replied

What do you personally want to run on bare metal? I've been using Qubes for 7 years and the only thing I've missed is running MAME since Qubes doesn't do nested virtualization. But I do everything else I need on there - software dev, office productivity, web browsing, email, video transcoding, Spotify etc etc. It's rare I would ever need bare metal. (I'm not a PC gamer though...)

He's been right more times that I can count. Abrasive guy for sure, but he has decided not to suffer idiots. And he does what he does for himself; we are lucky beneficiaries.

Agree wrt your arguments; it's also why I write this in a browser in a VM that is not used for anything else than this sort of thing, and periodically I will roll back to a recent snap shot with a clean browser.

(I do not use Qubes, but I do like their work.)

Shoot, as soon as I hit reply some neurons lit up and now I remember I was actually able to enable nested virtualization in QubesOS, and the relevant options in the VirtualBox preferences inside a qube became enabled once I did that, but whenever I tried booting any VM the whole system hanged. The same system and BIOS settings worked in Ubuntu to boot a nested VM in VirtualBox, so I think I had the BIOS settings correct. Anyhow, it seemed like a dead-end, so I abandoned it.

The issue with trying to hack Docker Desktop and similar tools to boot a qube instead of a VM and somehow hack QubesOS to not isolate those qubes from each other seems self-evident to me and I don't care to explain further.

I'll have to look at it more. I mostly use AMD systems these days, which don't support nested virt in Xen, as I understand it, but it looks like it should work on Intel.

Neither of my AM5 motherboards will allow QubesOS to install with Secure Boot on (and it's not going to be turned off) which really annoys me as I want to use it.

And container escape exploits are getting burned by sending them out via email? Doubtful.

Does this also count for Proxmox?

I've been daily driving it on my Thinkpad T580 with 16gb of ram for the past two years. I really enjoy it. Very occasionally it tells me that it's running out of RAM, but when that happens it's usually because I forgot to shutdown some VMs. Besides for that everything works great. I'd recommend using an SSD though with at least 512GB of storage, but better with 1TB.

I personally daily drove QubesOS for about half a year when in school, and personally, I loved it. When I first tried it out, I fully expected it to be a nightmare to use, given that's how it's usually advertised by non-users. But in using it, I really enjoyed its workflow and the seamless compartmentalization of applications on a computer.

Program isolation is honestly a feature that other distros should use more often. The idea that programs can only access networks, USB devices, files, and X windows of programs that it's been explicitly let to access is an extremely useful tool that isn't just for people who are worried about government surveillance.

I personally enjoyed having about 5 different Firefox apps that each led to their own VM with its own files, browser history, cookies, and extensions, and even networks that automatically put traffic through a VPN at times. Chrooting and Firefox profiles only help so much, if you can even set them up to be as seamless as Qubes.

I'm of course not getting into the security benefits and all that of Qubes, but its where I feel a lot of people don't realize the benefits of an OS like this. The workflow improvement is just as inspiring as the security improvement from the system.

It depends on who you're targeting and what you want.

But the history of computers security can largely be summed as:

"What? You're just paranoid. Nobody would possibly X!"

Someone gets their asses handed to them by someone Xing.

"What? Why didn't you tell us X was a risk we needed to be concerned about???"

Iterate.

Related. Others?

Qubes OS 4.2-rc1 is available for testing - https://news.ycombinator.com/item?id=36178205 - June 2023 (3 comments)

New user guide: How to organize your qubes - https://news.ycombinator.com/item?id=33396604 - Oct 2022 (15 comments)

What Is Qubes OS? - https://news.ycombinator.com/item?id=32036899 - July 2022 (82 comments)

Qubes OS: A reasonably secure operating system - https://news.ycombinator.com/item?id=30776103 - March 2022 (97 comments)

Qubes OS 4.1.0 has been released - https://news.ycombinator.com/item?id=30215210 - Feb 2022 (1 comment)

Ask HN: Qubes OS or just separate VMs for separating work and private files? - https://news.ycombinator.com/item?id=29537961 - Dec 2021 (6 comments)

Qubes OS 4.1-rc1 has been released - https://news.ycombinator.com/item?id=28856957 - Oct 2021 (5 comments)

Qubes OS 4.0 has been released - https://news.ycombinator.com/item?id=16699900 - March 2018 (39 comments)

Qubes OS: A reasonably secure operating system - https://news.ycombinator.com/item?id=15734416 - Nov 2017 (144 comments)

Reasonably Secure Computing in the Decentralized World - https://news.ycombinator.com/item?id=15566563 - Oct 2017 (44 comments)

Toward a Reasonably Secure Laptop - https://news.ycombinator.com/item?id=14743238 - July 2017 (100 comments)

“Paranoid Mode” Compromise Recovery on Qubes OS - https://news.ycombinator.com/item?id=14218504 - April 2017 (14 comments)

Qubes OS Begins Commercialization and Community Funding Efforts - https://news.ycombinator.com/item?id=13069615 - Nov 2016 (24 comments)

Qubes OS 3.2 has been released - https://news.ycombinator.com/item?id=12604417 - Sept 2016 (30 comments)

Security challenges for the Qubes build process - https://news.ycombinator.com/item?id=11801093 - May 2016 (17 comments)

Qubes OS 3.1 has been released - https://news.ycombinator.com/item?id=11260857 - March 2016 (44 comments)

Converting untrusted PDFs into trusted ones: The Qubes Way (2013) - https://news.ycombinator.com/item?id=10538888 - Nov 2015 (5 comments)

Intel x86 considered harmful – survey of attacks against x86 over last 10 years - https://news.ycombinator.com/item?id=10458318 - Oct 2015 (169 comments)

Qubes – Secure Desktop OS Using Security by Compartmentalization - https://news.ycombinator.com/item?id=8428453 - Oct 2014 (49 comments)

Introducing Qubes 1.0 ("a stable and reasonably secure desktop OS") - https://news.ycombinator.com/item?id=4472403 - Sept 2012 (59 comments)

Qubes: an open source OS with strong security for desktop computing - https://news.ycombinator.com/item?id=2645170 - June 2011 (16 comments)

Review: Qubes OS Beta 1 — a new and refreshing approach to system security - https://news.ycombinator.com/item?id=2504274 - May 2011 (1 comment)

The Linux Security Circus: On GUI isolation - https://news.ycombinator.com/item?id=2477667 - April 2011 (47 comments)

Qubes Beta 1 has been released (strong desktop security OS) - https://news.ycombinator.com/item?id=2439096 - April 2011 (3 comments)

Qubes Architecture - actual security-oriented OS - https://news.ycombinator.com/item?id=1796384 - Oct 2010 (1 comment)

Open source Qubes OS is ultra secure - https://news.ycombinator.com/item?id=1249857 - April 2010 (7 comments)

Introducing Qubes OS - https://news.ycombinator.com/item?id=1246990 - April 2010 (20 comments)

Sounds like you abandoned it after half a year? Would you mind to elaborate on the reason (s)?

Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux?

And it should be quite lightweight as it’s just a container…

It’s not that I don’t believe you but I don’t understand it… why would you need VM on Linux for Docker?

edit: huh

https://docs.docker.com/desktop/faqs/linuxfaqs/#:~:text=Dock....

that’s… a bit stupid in my opinion. But you can always just use the default daemon so, eh. whatever. maybe I’m wrong. there are reasons I guess

My laptop broke and I had to buy a new one. The new one was intel 12th gen, which was unsupported at the time. Why haven't I come back to it? I now often use programs that require hardware acceleration for optimal use, which is unsupported by Qubes due it being a potential security issue. If you mainly use your laptop for non-intensive tasks though, I still highly recommend.

It's a good question - docker doesn't require a VM on Linux, but Docker Desktop does. I assume it's to make it basically the same experience as on Docker Desktop on Windows and macOS, but I'm not totally sure. You can install docker the same way one would on a server in a qube in QubesOS and it works fine, I think I tried that once just to be sure, I just wanted to be able to have Docker Desktop. I also didn't want to paint myself into a corner in case I should need to run something else that also expects to be able to run a VM.

I was on Intel when I tried. No worries though, not really planning on trying it again.

Proxmox containers are just regular containers.

https://pve.proxmox.com/wiki/Linux_Container

Browsers have built-in sandboxes, plus sometimes wrapped around stuff like snap.

And yet...

Browser exploits are a thing, and reliably compromise systems. Apple just released a security update yesterday for "something in WebKit," and we see regular browser security updates.

The art of escaping browser sandboxes seems to exceed the art of building browser sandboxes. The Javascript JIT engine gains you a lot of attack surface, unfortunately (one of the reasons I run JITless with Javascript).

As for snaps, they're just containers - kernel separated. Unfortunately, I consider the value of that against actively malicious code to be "about zero" - local root/kernel exploits are fairly cheap. Containers (so snaps) are great for convenience - if you want to run code you trust without worrying about dependencies, this is fine. They're not fine if you want to isolate things you don't trust - such as a browser from "everything else."

Qubes gives you a much harder boundary around your VMs than containers and sandboxes do.

QubesOS is amazing, with a lot of interesting concepts.

While its dependency on Xen, a fairly bloated and thus unsafe hypervisor, is unfortunate, there's an effort[0] to remake Qubes around seL4.

0. https://trustworthy.systems/projects/TS/makatea

There's also Makatea[0], an effort to build a Qubes-like around seL4.

0. https://trustworthy.systems/projects/TS/makatea

>You still have to get through Xen to get to anything I consider of value.

It's not unthinkable, as Xen is huge, at hundreds of kLoCs. But there's an effort[0] to make a Qubes that uses seL4 in place of Xen.

0. https://trustworthy.systems/projects/TS/makatea

>For example, it'd be nice to use KVM (QEMU or even crosvm) instead of Xen

Or even better, seL4, for which an effort exists[0].

0. https://trustworthy.systems/projects/TS/makatea

I've been using Qubes for the past 2 years while going to school, and I found it really fun and helpful. A lot of professors had me download random closed source software from random websites during the pandemic, and it was easier to download it to a VM than to convince them about Free Software. More than that though it's been really helpful just for my own workflow. I can hit a keybind and start working from essentially a fresh linux install. It's easier to stay on task when each VM is designed to only do one kind of task. It's also nice having debian, fedora, windows, kali, and whonix all easily accessible on the same machine.

The main sticking point for me is that Qubes is reasonably secure from _myself_. I make mistakes. I first started using linux with an Ubuntu install that I broke a year later because I accidentally added in a space when typing `rm -rf ~/Arduino` which made it `rm -rf ~ /Arduino`. On Qubes I can `sudo rm -rf /` on the VM I'm using right now and not break a sweat. I have a keybind to spawn a disposable "airgapped" VM to deal with sensitive or untrusted data, and it helps knowing that even if I mess up with whatever I'm doing, the VM will keep everything reasonably contained.

Some cool things that Qubes has outside of just VMs are its features enabled by the communication between VMs. Notable ones are Split GPG (https://www.qubes-os.org/doc/split-gpg/) which let you use a VM as if it were a smartcard for GPG and Split SSH (https://github.com/Qubes-Community/Contents/blob/master/docs...) which let you isolate your private SSH keys from your VM running your SSH client.

There are some sticking points around Qubes. For instance, I use Tailscale to connect my computers to each other from anywhere. Tailscale's install scripts add their keys to my VM's package manager for updates and installs. The proper way to do this in Qubes is to clone a TemplateVM, run Tailscale's install script, update, install, and then base an AppVM off of it. But that creates an entire new OS taking up storage and requiring updates. You can hack a way around this in an AppVM which saves a considerable amount of space, but it takes a lot of upfront time to do and requires you to manually update it.

Another sticking point is hardware acceleration. The desktop environment has access to hardware acceleration, so it runs fine, but opening videos in AppVMs is all software decoded. I'm on a Thinkpad T580 and it can run 1080p videos, but the fans turn on and can't do 4K. When I want to game or do something GPU heavy I either stream from my tower or completely switch over.

Overall, I'm really happy with Qubes and I'm planning to stick with it on my laptops.

I use Qubes because of one exact feature: single Desktop Environment for windows from different Virtual Machines. And AFAIK there are no alternatives.

Use it for work, development, personal tasks for about 2 years so far.

But it has many restrictions, e.g. gaming is problematic because of this single "main" desktop and trucking cursors bug, recording screen when there are several monitors is bugging and problematic, streaming or working with graphics is problematic, development that requires other virtualization like android (as i heard) is problematic (though docker works, tested).

This OS is enough for my tasks, I like concept of separating activities, e.g. when i share desktop on skype, people see only skype's VMs windows. And single Desktop Env is killer feature.

But because of many restrictions this OS is definitely not for everyone.

Also remote desktop cannot be done at least by usual means. And this is big No if you need to work remotely sometimes.

Snap uses AppArmor, while flatpak uses bubblewrap. You need to have a zero day in these sandboxes, in addition to in the browser. Not so easy!

But definitely VMs provide a much better boundary.

>If you do malware analysis, Windows VMs in isolation on a throw-away laptop is better. >If you do penetration testing, Windows or regular Linux will be better, too.

I am going to firmly disagree with these two points.

I've been a red teamer at one of the three big cloud providers for over a decade, and a passionate hobbyist with both malware RE and offsec (CTFs, bug bounties, etc). I've reversed easily 2000+ different samples of malware, have more CVEs than I can count (several dozen), and can make my way through a corporate network quicker than almost any known APT group.

I use Windows qubes, both as sandboxes and for certain utilities that only run on Windows, and I do pentesting from Linux-based qubes.

There is precisely one restriction I face with Qubes in my entire line of work: even with two GPU's installed for isolation purposes, Nvidia GPU's (even with FLReset+) do not like being passed through by Xen. Older AMD cards like my RX 580 work fine.

That said, there are only two things I'd have any use out of a GPU for - hash cracking (obvious) and LLM workloads (code generation, to speed up PoC prototyping, tool development, etc).

Fortunately, I have access to a six figure rig dedicated to hash cracking at work, as well as effectively unlimited usage of a non-local code generation LLM.

There is no circumstance in which running Windows on bare metal is ideal or optimal in any way whatsoever for just about any kind of security work.

Linux is better in some ways, but even then, segregating workflows in ring3/userland is a must, and Qubes makes this painless, quick, and easy compared to spinning up a bunch of VMs in your distro of choice.

The USP of Qubes isn't that it does anything magic to make a level of security possible that isn't on other platforms, it's how it makes attaining and maintaining that level of security so effortless and seamless compared to other solutions.

The only alternative I've played with that comes close (imo) is Subgraph OS, but it's really not an exaggeration to say that project is absolutely still in alpha status of development, and I would not yet rely on that for sensitive workloads.

One aspect of what you said that I do agree with and wholeheartedly support is hardware isolation. Even with Qubes, hardware isolation is a fine solution to Xen HV exploits, for the tiny handful that have affected Qubes' Xen implementation.

Don’t forget that, the VMs in Qubes should still be secure.

Running ChromeOS Flex (or Android?) or kicksecure is good. Whonix is good too, but it seems it’s for privacy.

Daily drove qubes for 5.5 years. I liked it a lot.

Dangerzone is an implementation of a concept known as CDR (Content Disarm & Reconstruction), where you convert anything to an image inside a sandbox, and then convert the raw pixel data back into an image inside a different sandbox.

It is a common workflow inside the government or other places where you need to move data across airgaps, or view content that is highly untrusted.

Shameless plug, I wrote my own that supports over 200 file formats: https://preview.ninja/

> Huh… why does Docker require VMs on Linux? Isn’t the selling point of Docker that it uses the same kernel on Linux?

docker has been playing rather loose with security for quite a long time in the past, so I per-se wouldn't trust it to be secure

additionally there are (oversimplified) 3 ways to run it:

1. a "root" deamon, the default docker daemon setup in most distros and the way docker originally was designed to be used. This is needed for some advanced (non secure) features (which IMHO should not exist) but has pretty serve security issues IMHO, like either being in the docker group being a de-facto root/sudo or it forcing you to use sudo all the time and bugs in the docker deamon potentially leading to catastrophic exploits.

2. as a unprivileged user using cgroupsv2 like podman does by default, this has some limitations but works really well and as long as "linux containers" are secure it should be secure (oversimplified). Still the security of containers isn't perfect as they share the kernel. As a rule of thump only use if you get reliable kernel security updates.

3. starting VMs, much more overhead in many ways and needing virtualization, but works with old "outdated" kernels or hardened kernels(1) not allowing unprivileged cgroups

The 3rd method is the most secure and works across most dirstros/linux setups even if unusual setup and is the same kind of path docker takes on windows/mac. So no wonder a product like Docker Desktop uses it. Also the overhead of virtualization is today not "that" bad.

The the first method is still the default method for docker CLI on most linux distros is honestly far beyond my understanding. In the past pre reliable cgroups v2, maybe, but today omg no. It has some valid use cases, but the default should be VMs or unprivileged containers especially on desktop systems (whatever distro maintainers prefer).

----

(1): On instresting conflict is security hardening vs. unprivileged cgroups/containers. On one side unprivileged cgroups allow a lot of additional security. For example they allow running various sandboxes without needing a suid binary or using sudo for setting up the sandbox. Which is really grate. At the same time (oversimplified) it increases the security sensitive kernel interface by a non negligible margin, which isn't grate and hence why hardened linux often disables it by default.

EDIT: Sorry for the bad spelling, I need to go to bed.

The people who work on QubesOS are phenomenal at what they've accomplished. It's not everyone's cup of tea, not least of all because of the hardware requirements and battery drain. All the same, their commitment to developing a reasonably usable, but above all a highly secure OS clearly shows.

They did consider KVM initially; I don't know how much things have changed and if they've reconsidered. The reasoning was that KVM's means of virtualization is too closely coupled with the Linux kernel, whereas Xen's hypervisor and dom0 are more separable.

> In Xen, at no point does the execution path jump out of the hypervisor to e.g. Dom0. Everything is contained within the hypervisor. Consequently itʼs easier to perform the careful security code audit of the Xen hypervisor, as itʼs clear which code really belongs to the hypervisor.

From the original 0.3 spec

> I accidentally added in a space when typing `rm -rf ~/Arduino` which made it `rm -rf ~ /Arduino`

NGL I laughed out loud reading this.

I do feel sorry I laughed though.

Possibly to some extent

Intel x86 considered harmful – survey of attacks against x86 over last 10 years - https://news.ycombinator.com/item?id=10458318

I bought a Librem laptop for Qubes support.

Feels like home.

How is this different than an immutable distro like Fedora Silverblue?

well if you bother to send an email that breaks out of the container, you might as well make it retrospectively delete the email to hide the evidence :)

Stellar Work - Thanks Dan.

As someone who often runs multiple entire virtualized datacenters on top of ESXi hosts which are actually virtual machines themselves running in ESXi, I highly disagree.

Nested virtualization is a game changer in any kind of lab scenario. It also runs great on Chromebooks. The ChromeOS Linux environment is actually a KVM, so launching virtual machines from there is nested but you would never know it (based on my experience with the (framework Chromebook edition). Having 64GB of memory doesn't hurt either of course.

Is this really a good idea? Won't the pdf thumbnail generator pwn you by merely navigating into the folder that contains the infected file?

Technically correct, but kind of a shitty answer, no?

Most of Xen's vulnerabilities do not affect Qubes OS: https://www.qubes-os.org/security/xsa/.

You can't ssh into the Qubes VMs from dom0. It would break the security model. You do not run anything in dom0 at all. But you can create an AdminVM that manages other VMs.

Ashamed to say but I do all these things on my main OSs for ~15 years and I never had any problem (IIRC). Yes, I'm not huge on security and I realize what this could do, but I figure the chance is as big as getting it through a channel I don't expect.

The installer is likely not signed in a way Secure Boot trusts it. Verify the ISO yourself with the PGP key, disable Secure Boot, install it, re-enable Secure Boot.

Standalone (non-Qubes) Whonix runs on Kicksecure; in fact, both projects are from the same author.

Most vulnerabilities of anything do not affect all its users.

But it's bad enough if any do. (some do affect Qubes)

It is an architectural problem.

SeL4 is a good replacement, with excellent performance and strong formal proofs.

And what happened then?

Also this: https://news.ycombinator.com/item?id=36178205

SeL4 is great an all, but no one of those Xen vulnerabilities has led to an escape since forever, have they?

xfreerdp works just fine, however it requires some fiddling if you want to use it on multiple monitors.

Here's the bit that confuses me: no secure boot.

QubesOS does a lot of good things, but Secure Boot is sort of a nonnegotiable for a lot of security profiles because its one of the few ways you protect boot.

I'd be interested to hear from someone more in the know why they haven't implemented it yet.

Pretty much everything you listed, plus games. All of that minus games is still usable on Qubes but there is a noticeable speed decrease.

As i said in my comment I personally want to run (some) things on bare metal, and Qubes is not good for that, which is good for their security model but is a deal breaker for me.

Do you want an answer to that question or do you want an argument?

Added. Thanks!

Added. Thanks!

I think the optional anti-evil maid support includes secure boot: https://www.qubes-os.org/doc/anti-evil-maid/

> Anti Evil Maid is an implementation of a TPM-based dynamic (Intel TXT) trusted boot for dracut/initramfs-based OSes (Fedora, Qubes, etc.) with a primary goal to prevent Evil Maid attacks.

I moved to another team that didn't have a requirement for running Qubes. As soon as I go back, I'll be using Qubes again. Right now I'm on pop_os.

Is rdp server in dom0 or in VM? Connecting remotely to VM at most i was able to see opened VM's windows, but no keyboard/mouse interaction because dom0 is responsible for it.

That makes much more sense, thanks.

Still a bit confusing that its not on by default, but also much more trustable.

For gaming I would just use a different machine. Trying to use an OS like this for that is always a compromise.

In my view the secure boot support provided by mainstream Linux distributions is more about providing installability on systems with secure boot enabled, rather than providing real security benefits.

My reasoning is that while the bootloader and the kernel are signed, the initrd image loaded very early on in boot is not, because it is generated on device. So it provides a convenient way to compromise any system you have physical access to.

The anti evil maid implementation I linked to attempts to mitigate this hole using a TPM. I'm not sure why it isn't on by default but perhaps it's because the implementation has different options that require deciding on a threat model, e.g. setting a TPM password or using an external usb device to store a LUKS key. Here's a good blog post about the anti evil maid implementation that qubes uses(it also works with other distros like Fedora): https://blog.invisiblethings.org/2011/09/07/anti-evil-maid.h...

This blog post contains a good overview of the secure boot status quo along with another potential future fix: https://0pointer.net/blog/brave-new-trusted-boot-world.html

It is certainly not as robust as MacOS or Windows protections, but I have been told it does provide a block to certain malware targeting the boot (though it is imperfect as you have pointed out).

In my case, I am not expecting total security. I just want access to be extremely inconvenient for opportunistic attackers.

I doubt ssh or simple sh is a matter in this case. You can do simple sh:

qvm-run --pass-io {VM_name} -- ls

Security model is not to do anything non-admin related from dom0: if you "own" dom0 you can do everything.

Allocating amount of CPU cores is a feature of VM. Not sure about dynamic though. There is dynamic flag about memory, not CPU. In practice it looks like CPU is shared among all VMs, no need of special flag for it.