Run0, a systemd based alternative to sudo, announced

This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. I knew where the files were, what they were called. I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.

The init system was probably peak of systemd, after that they started reinveting things in a more complicated way. Do we really need journald, systemd-boot, machinectl, systemd-networkd, sd-bus, systemd-resolved, systemd-nspawn? Do people actually use it all? Are there any metrics to show how many systems have it installed and in use?

journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there.

> I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.

My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe, but finding the logs in the first place was always a challenge. Systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.

The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.

Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.

Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.

Which involves glob’ing. I assume it allows to monitor several services at the same time.

Can systemd do that?

systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar.

I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.

With regular logs, I can go to /var/log, ls the dir, and get a nice list of what's being logged. There will be, for example, an `apache.log` file that has all the logs relevant to apache.

You sure would have forgotten about it when you tried to get onto another machine without the magic set of aliases

I am also prepared for someone to chime in that "log egress tools exist" to move up the Maslow's Hierarchy of Log Needs

    # journalctl -f _PID=${your_pid}  # option 1
    # systemctl status ${your_pid}    # option 2

[2]: https://www.freedesktop.org/software/systemd/man/latest/syst...

And you don't need to worry about "is this service rotating logs appropriately?"

In the old ways, that was apparent because of the presence of the log file.

Granted, this isn't the problem it used to be for us, however, it would come up because we'd have our services named things like `foo-ws` and knowing that `foo-ws` existed on a given box might be tricky were it not for the log files.

It's much easier in the systemd world to see the state of the system, the state of the service, and the logs of a service because it enforces this consistency.

I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.

The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work.

I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!"

Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it.

systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.

    journalctl -f -u <TAB>

No they don't. They are good answers. If you want answers to a more general problem, then ask. This is not an emotive topic.

That said, when I'm in a hurry and going to do a thing that I've been doing every day of my life, I'm really not a big fan of having to stop what I'm doing and reexamine a tool I've used all my life to figure out how to use it. I learned this stuff decades ago and I'm not anxious to relearn how to do it for a fractional improvement, and I'm not even really 100% sure that this new "inflate and snap" shoe model is actually an improvement, except for very specific use cases.

Once you deal with that you’ll realize journald is just in the way and needs to be modified to output a human readable log by default.

They suck in that the proposed method for accomplishing the task is a suck-ass downgrade from the previous ways the new way proposes to displace.

I beg to differ about emotive, because absolutely the other side of this fence behaves every bit as butt-hurt when challenged as I just did. You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.

Very well: I didn't say the commands were good or great. If you attempt a little objectivity you'll see it.

Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends.

> --no-pager

To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.

You can then use `systemctl status <pid>` to identify the unit if you need to.

I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary.

EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well.

EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\

Before even going into the nature of the commands, I said that the answer does not match the question. The answers addressed details, while the details in the question were merely examples.

Missing the point is actually merely one of the at least two dimensions along which the answers suck. Thank you for reminding me about that.

No, the answers addressed the two scenarios in the question. That's why they don't "SUCK". They are good answers to the question as asked, and not the question as you dreamed it. If you spent ten words on asking the question of your dreams, rather than having a go at people for not answering it (whatever it is), you might have an answer.

With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal.

> let me cd into /var/log and see what filenames I can find

You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue.

> or grep everything until I can find a couple of words that make some sense so I can keep digging further

journalctl --grep, or more conveniently, journalctl and then / to search.

> The lack of explorability in journalctl

It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.

Can you filter on search? How do you prevent applications from writing to Systemd at all?

And as far as learning the options goes, learning the basics of systemd and journalctl is much easier than learning how to, say, use a source control system effectively.

(And it's definitely worth trying out new source control systems -- for example, I think most people should at least check out Jujutsu, and many people are going to like it more than Git. As simple as Jujutsu is to use, learning it takes a lot longer than learning journalctl.)

With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines.

I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.

If you don’t want a service to log to systemd, then don’t run it in systemd?

I like systemd and have found it extremely useful. If it doesn’t fill a need for you then use something else.

For how long?

I've been around for decades at this point with people telling me how the newest and greatest thing will obsolete everything that came before it.

The best way to look through logs is still to materialize them in as text in a files hierarchy and use find with grep to look for issues.

There should be a command like "journalctl --top" that would list all logging services with amount of logs, oldest and latest log times, and last few lines of each, to improve that experience. Maybe there is already a way to do something similar?

There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.

Debian switched to systemd about 12 years ago, so at least that long.

Perhaps it was available prior to then, but so were daemontools, upstart, and probably a variety of other alternatives.

a system user/admin has an intuition about files. saying that ‘journalctl -f -u’ (fu, indeed :) and whatever else is inherently undiscoverable, and is a.. basically orthogonal mechanism for handling what should be a simple task. i.e., viewing some logs. it’s far easier to compose and extend from files (what if i only care about the mtime of the log, for instance), than this.

look, i think systemd isn’t.. terrible. i also think it’s suffered a bit of complexity fetishisation, and it seems as though that this resulting complexity may have become invisible to you.

run0 doesn’t seem like a bad idea. but i am wincing a bit at the thought of unrestricted javascript determining access control.

Journald would be perfect if I didn't have to know a thing about it. Work in background, move stuff from stdin/stdout to /var/log/journald/service.log, move old stuff in /var/log/journald/archive/service/2024-04/15.log.gz. I'd be happy. Why do I need this cryptic CLI I don't understand. It brings nothing but pain.

It can largely do everything it replaces, but differently, in some cases with improvements. Every year it increases its scope of 'problems' its solved.

But the issue is it hasn't _ACTUALLY_ solved a problem for me in about a decade, its only introduced problems. Its replaced things that worked perfectly for me and millions of other users for years/decades with something that... also works... but differently. Now to accomplish the _SAME_ outcome without any personal benefit I have to re-learn their new and usually opinionated way.

There was a time I could log into a system, find logs, parse dates and do everything in these discussions reletave to logs, rotation. Regardless of service because the standards were close enough and the basic tooling (tail/find/tar/grep/awk etc.) worked everywhere always, and you never had to google even if the service was one you never worked on before.

For systemd its the opposite, its almost a guarantee you have to google something along the line when interacting with new systems and services. And as soon as their newly enforced opinions and way of doing things is getting comfortable they go and replace something else that was also working... perfectly fine.

If it actively solved problems for the majority of users we wouldn't hate it so much. Instead it requires more mental overhead than anything other than XORG to deal with and constantly manage than anything else in the linux ecosystem.

It does and we don't. The only "we" who are "hating it so much" is a tiny vocal minority.

I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.

I wouldn't call the opposition tiny, although the vocal portion of any group tends to be the smallest. This would be my first online comment on it so I'm certainly part of the non-vocal group you ignore, that in my experience is significantly larger than the vocal side of group you dismiss.

Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.