Show HN: Nerdlog – Fast, multi-host TUI log viewer with timeline histogram

journalctl is mentioned once in the landing page and it seems to imply that journalctl is not supported per se, as logs need to be stored plaintext to legacy syslog (?).

I do not want to store plaintext logs and use ancient workarounds like logrotate. journald itself has the built-in ability to receive logs from remote hosts (journald remote & gateway) and search them using --merge.

The article makes it sound like it uses various command-line tools (bash/awk/head/tail) to process the logs. So, I imagine it's not a huge leap to extend support to using journalctl to do that work instead.

That's true, as of today nerdlog doesn't use journalctl, and needs plain log files. There were a few reasons of that, primarily related to the sheer amount of logs that we were dealing with.

As mentioned in the article, my original use case was: having a fleet of hosts, each printing pretty sizeable amount of logs, e.g. having more than 1-2GB log file on every host on a single day was pretty common. My biggest problem with journalctl is that, during some intensive spikes of logs, it might drop logs; we were definitely observing this behavior that some messages are clearly missing from the journalctl output, but when we check the plain log files, the messages are there. I don't remember details now, but I've read about some kind of ratelimiting / buffer overflow going on there (and somehow the part which writes to the files enjoys not having these limits, or at least having more permissive limits). So that's the primary one; I definitely didn't want to deal with missing logs. Somehow, old school technology like plain log files keeps being more reliable.

Second, at least back then, journalctl was noticeably slower than simply using tail+head hacks to "select" the requested time range.

Third, having a dependency like journalctl it's just harder to test than plain log files.

Lastly, I wanted to be able to use any log files, not necessarily controlled by journalctl.

I think adding support for journalctl should be possible, but I still do have doubts on whether it's worth it. You mention that you don't want to store plaintext logs and using logrotate, but is it painful to simply install rsyslog? I think it takes care of all this without us having to worry about it.

can't you just read from stdin?

i use lnav in this way all the time: journalctl -f -u service | lnav

this is the ethos of unix tooling

One small hitch I found is that this kind of tools are fixes in what to process, so for example I can't use them for structured logging. If it has an escape hatch where I can supply my own pipe (for example `process = 'vector ....'`) then it will be enough.

I appreciate this response, and want to say I really like your tool's UI over something like lazyjournal. But like the above commentor, I would love to see journald support as well, just because it's the default these days on the distros I use, and seems like the direction the Linux system industry has headed in.

Thanks for the feedback. I'll see what I can do. But for now, do you think the workaround of having to install rsyslog is not good enough?

Not really, at least not yet, because nerdlog's focus is very different than that of lnav. There is a section about it in the article as well.

In fact nerdlog doesn't even support anything like -f (realtime following) yet. The idea to implement it did cross my mind, but I never really needed it in practice, so I figured I'd spend my time on something else. Might do it some day if the demand is popular, but still, nerdlog in general is not about just reading a continuous stream of logs; it's rather about being able to query arbitrary time periods from remote logs, and being very fast at that.

I think it will impact first-time users giving nerdlog a quick test/trial run, and cause them to bounce to another tool when it doesn't show them logs from journald out of the box. Users can be finicky and impatient with new tools ;-)

Example: I'm running an Arch-based Linux desktop. Installing ryslog took several minutes to build and install. If I wasn't highly motivated to try out nerdlog, I would have canceled the install.

Also, can the SSH requirement for localhost be bypassed? Most users won't be running an SSH server on their desktop, and this would improve nerdlog's use-cases and make it easier for new users to give it a quick local test run.

Final suggestion: add `go get` support to your repo, so that I can install nerdlog from a single command and not have to clone the repo itself.

Yeah you're right, agree with both of your points.

The `go get` one should be easy to solve though, and my bad for not thinking of it before, thanks. I'll look into it.

I first responded before your edit about ssh and localhost, so: yeah, as briefly mentioned in the article, as of today there's no shortcut even for localhost. I was debating whether I should implement this feature before open sourcing it, but I had to draw the line somewhere (I have TONS of ideas what could be implemented), and since reading local logs isn't the primary focus of nerdlog, I decided to skip it for now.

But yes the bypass for localhost can definitely be implemented.

Appreciate your receptiveness, and sorry about all of the edits... I was rethinking my thoughts in real-time ;-)

Yeah, I'm bouncing for now on the localhost requirement. Or, on a related issue of not parsing my .ssh/config, a Match directive, and not wanting it to parse it yet. I grep'ed for an env var to override, but only USER and SSH_AUTH_SOCK are pulled in.

I did go get install ...nerdlog/cmd/nerdlog-tui@latest just fine.

Thanks for hacking in the open, and releasing early.

Sorry to hear you're having issues. I'll try to reproduce and fix the issue with the Match.

Not sure if that "Thanks" for releasing early is sarcastic, but regardless, I appreciate the feedback.

I would say that their thanks is sincere, and that they're applauding you for releasing a new tool to a public/critical audience while also taking feedback in very constructive manner.

Fyi I was going to create a Github issue for the journald support, but apparently someone else filed it first: https://github.com/dimonomid/nerdlog/issues/7

Just posting it in case you want to subscribe to it. Looks like it's a popular demand indeed, so I'll at least poke it and see what kind of performance we can get out of it.

Fyi I've done a simple benchmark today, and journalctl is indeed a lot slower than simply reading log files. In case you're interested: https://github.com/dimonomid/nerdlog/issues/7#issuecomment-2...

I have no doubt that journald is slower, and I have no real preference for it. The request for journald support goes back to it being the default on most distros these days. I have to interact with a variety of different servers in different environments, most of which are not managed by myself, and hence I have to interact with was previously setup... which is usually the default config provided by the distro.

I see, interesting. If you don't mind me asking, is it a sysadmin kind of job? Just trying to understand the use case better.

Regardless, journalctl support is the single most requested feature, so yeah I'll at least try to make that happen; hopefully on the upcoming weekend if I'm lucky.

My role is classified as DevOps consulting, working for several different companies, each with their own unique setup and teams... but yeah, it's basically glorified sysadmin work ;-). On the plus side though, I get to see the realities of internal company tech stacks, and that the default settings often are chosen over better solutions.

Thanks again for considering journald, and at the same time, don't forget that it's your project at the end of the day... you can always disregard feature requests if it's not a direction you want to head in. Though in this case, I do believe journald support would get your tool more traction with a larger audience in the long term.

Thanks for sharing! Good to know that nerdlog turns out to be helpful not only for devs (the original use case), but also for DevOps :)

Fyi, support for journalctl was added to master, in case you wanted to try it out. I didn't yet add automated tests with the mocked journalctl, but my manual tests show that it's working fine.

If a system doesn't have either `/var/log/messages` or `/var/log/syslog`, nerdlog will now resort to `journalctl` by default.

It can also be selected explicitly by specifying `journalctl` as the file, e.g. `myserver.com:22:journalctl`.

Looking good so far! Now it just needs localhost-without-SSH enabled by default (so that as a new user I test out nerdlog immediately without having to think about which server to connect it to), and your initial onboarding/out-of-the-box experience will be ready for a very wide audience! :-)

Yeah, I agree. Gonna try and get it done on some weekend.

Thanks for trying it out, and for all the suggestions, very helpful!

And thank you again for being so receptive to feedback!

Hey mcint, fyi both of these issues are addressed: the localhost one is addressed for real, and a Match issue is worked around: while it's still not properly implemented, at least it doesn't prevent Nerdlog from starting now. Just in case you wanted to give it another try.

Cheers.