Seer: A GUI front end to GDB for Linux

Interesting.

My experience is the opposite: I see developers waste hours stepping through their code a line at a time when a few judiciously placed logs (printfs() are fine, but we can do better) would have told them exactly what they needed in a jiffy.

If you have a fairly shallow bug, that is a single point in your code that always behaves incorrectly, then I find debuggers reasonably effective.

But most of the bugs that I see aren't that shallow, with code misbehaving when the context is just so and perfectly fine otherwise. In those cases, I need to see lots of different invocations and their context. The debugger is like trying to drink the information ocean I need through a straw. A mostly plugged straw.

I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.

(And it doesn't mean I never fire up the debugger. But it is fairly rare).

I have more or less the same experience like you. Logging is a very resilient and adaptable technique – I can use it on my laptop or on remote HPC clusters, almost regardless of programming language (except maybe Haskell), it works fine on parallelized code, and so on, with very little configuration needed. It’s also important to me that it can be done “async”, since some of my larger codes can only be run on HPC clusters by putting a job in a process queue and waiting.

I’ve tried debuggers and see the appeal but I find it less useful than print debugging / logging.

I also rely heavily on unit tests when writing new code, so that also reduces the surface that I need to look for bugs based on the log. Moreover, most of my projects have 1-3 programmers and can largely “fit in my head” (<10,000 lines of code), so it’s probably different if you work at a FAANG company or something.

Not the OP but...

> programmers to master the debugging tools of their ecosystem. I've seen countless experienced developers use printf-based debugging and waste hourse debugging something which could've been easily figured out by setting a breakpoint and stepping through your code.

If you're wasting hours with printf-based debugging, I don't think you've 'mastered the debugging tools of the ecosystem'.

There are multiple ways to debug - step debugger tools, printf, logging to a file, etc. Each have their place.

If you're spending hours on any one approach, and perhaps that's the only approach you know, that's a red flag.

If you've spent hours going through printf, logging and step debugging and STILL don't have a good answer... bring in external eyes.

I've found/fixed bugs in a few minutes because of adding some log stuff first, because in those cases, it's the easiest approach. In other cases, running a debugger and setting a couple breakpoints is indeed the easier approach to start with, and I've done that.

Sometimes you find it with the first approach, sometimes you need to try the next approach.

> If you have a fairly shallow bug ... But most of the bugs that I see aren't that shallow

Oh come off it, debuggers shine the brightest when there are lots of unknown unknowns. With printf debugging you can peel back exactly one layer at a time (oops, need to log one more thing) whereas with a debugger you can slice through the Gordian knot.

I would guess longer compile would encourage breakpoints over printf and this would be programming language specific.

Being able to change breakpoints at runtime helps a lot when tracking down something more complex. Visual Studio breakpoints are great, and they’ve added conditional breakpoints which are even better. Previously I would approximate this by having code specifically branch to hit a breakpoint, ‘if (X) { breakHere();}’

I write a fair amount of native C++ code but only call it from either Python or dotnet so when I make a mistake here it’s usually a segfault / memory access issue which kills the process. There might be a way to debug the C++ from dotnet or Python but logging to std out helps me isolate the location of the issue which is sufficient. It’s not a big enough problem and I worry that either writing tests in C++ or learning a native debugger will pay off in time saved.

Another thing to consider and is important to me - logging objects and state isn't always so simple. It can often be easier for me to open the debugger to look at the state of an object which cannot easily be printed.

I agree with both of you. Printf is not enough, breakpoints are not enough. The solution lies between. Ability to gather rapidly relevant information to converge on wrong states.

ps: I wish I could work on a porcelain layer to manage the breakpoints in a more logical manner. Considering a problem you'd like to create different sets of breakpoints, run various tests and gather results to reviews. With the ability to add or remove layers rapidly. It's probably not too hard to do.

Try using a debugger to debug a globally distributed system that humans aren't given access to for security reasons in a post mortem and then "come off it" yourself

I have found combining these things to be useful: breakpoints that print stuff and auto-resume the program. Allows you to attach trace points at-will without requiring a recompile or losing state.

Out of interest - what sort of objects are hard to print in this way but easy to view in a debugger?

My grief with debuggers is due to C++ and template code (usually STL) and optimizations zeroing out values. I wish it had better training wheels to say "nope that's STL library code, you probably don't want to step any deeper". That's largely a criticism of C++ itself, though. But yes for this reason I prefer printfs despite 20 years in the game.

In my case it's basically everything since I work in java, jackson's object mapper can easily get stuck or deserialise something incorrectly if the class hasn't been annotated correctly. So it's simpler for me to pull up the debugger and I can see the "actual" data thats making up my object, it also lets me run "queries" against anything of interest (i.e. computed fields).

The default toString method I've found to be useless almost every time I wanted to inspect an object in our codebase since it just prints the type + "id" for the object

Yes losing state is a killer

I think you have a great point here. Debugging tools make you dependent on a particular environment. Printing based debugging can work pretty much everywhere. If you master printf programming you can solve any debugging task.

Yes exactly, and I'll probably say very generally that I usually use breakpoints when I am in the exploration stage of a significant state bug, and I'll usually use logging when I generally know where the bug should be but I need to pin point the exact place

It depends on the code as much as anything. I wrote a regex engine in Zig, and the instant I get a bug report I set breakpoints on a failing test and step through.

On the other hand, I'm working on an interactive application, and when I see a problem with it, I add more logging statements until I figure out what the problem is. Any time the logs have excessive detail as a consequence, I gate them behind an 'extra' flag on a per-unit basis, only removing the ones which amount to "got here".

If I had to pick one technique, it would be logs. I naturally think in terms of a trace through the execution pathway, rather than a step-by-step examination of the state of a small window into the code. It clearly works the other way around for some people.

One thing that makes this approach better for me is that debug logging is literally free, Zig uses a lazy compilation model so logging code which doesn't apply to a release compilation isn't even analyzed, let alone compiled, let alone included. In a language which doesn't work that way, there's motive to use printf-only debugging, and clean up after yourself afterwards, and that's extra work compared to firing up a debugger. So it shifts the balance.

gdb has a skip function and you can exclude STL headers and functions named in special ways so that reduces the noise quite a bit.

A good debugger can provide more than just stepping thru code.

In IntelliJ with Java, you can set conditonal breakpoints with complex evaluations, you can set filters (only hit a breakpoint depending from where it is being called), use exception methods that only hit on certain exceptions instead of a specific line code, you can also use logging breakpoints, that act like printf debuging, but you don't need to scatter your code with print statements all over the place.

You can group, add descripitons, disable, enable and add temporary breakpoints, they are pretty powerful! I just wish intellij had a time travel debbuger like Visual Studio Pro.

https://www.jetbrains.com/help/idea/2024.3/using-breakpoints...

Visual Studio has natstepfilter

https://github.com/ocornut/imgui/blob/master/misc/debuggers/...

Inserting a breakpoint is just as easy as a printf, and as long as you're still using a debugging build, you don't have to recompile. With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile. With a breakpoint you can inspect the contents of anything at that scope, and even see what the code flow is with that given state. You can even save a core dump to go back to later.

You can also script breakpoints to output the info you want and continue, giving you your information ocean.

Basically, a debugger is a more efficient and powerful tool. In the one situation where you're not skilled with a debugger feature, a printf can be quicker than having to learn, but it's objectively worse.

Is that the only kind of difficulty you can think of? Lol.

An object with many fields (in a language with no conveniences for it). An object tree with multiple levels of nesting. A list or dictionary of such objects.

In general, print-based debugging requires a greater degree of specificity. If you know exactly what you're looking for it's great.

If you are performing a more exploratory sort of debugging, a decent graphical debugger will save you a ton of time.

Yes, portability and simplicity are the best parts of printf.

> If you master printf

The skill ceiling is low. Printf only does so much.

You could rope in environmental optimization to the skill discussion -- the ability to isolate areas of functionality, replicate problems, reason about unknown state, and do the legwork so that you can quickly spin the increased amount of iteration required by a simpler debugging tool -- but by then you have thoroughly sacrificed both simplicity and portability and are far past the skill floor of a debugger.

If we assess this by looking for problems created by overcommitting to one approach or another, overcommitting to a debugger looks like burning time trying to get tooling to work on a problem that doesn't really need it while overcommitting to printf looks like spending way too much time iterating on tiny steps that could have been jumped over given better visibility. I've seen both, of course, but I tend to see more of the latter and more denial about the latter. When you're burning time fighting tools it's obvious. When you're burning time because you don't know how a tool could have saved you time, it's less obvious.

YMMV.

I've also used both GUI-based debuggers and printf, and I prefer printf. But the most important thing it to write your code so there aren't many bugs and when there are they are easy to find. I do this using modular code, unit tests and regression tests.

> the ability to isolate areas of functionality

This is the key. You need to be able to narrow down where the bug is.

I agree with this comment but you really didn't need the first 4 words.

You can insert and remove breakpoints while running. You can inspect variables the instant you realize they might be relevant.

During my long career, I’ve always been told “You should know you code well enough that a few well placed printfs is the most you’ll need to understand a bug”.

But, most of my career has been spent debugging large volumes of code written by other people. Code I’ve never seen before and usually will never see again.

A debugger making a 10X productivity difference for me is no joke.

> With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile

In some languages, such as Python, it's fairly easy to write a debug-print function that prints all the local variables (as well as the function name and line number it was called in).

I use profiling tools more than step-debugging, and printf()/var_dump()/IO.inspect/System.out.println/&c. much more than both, because most of the time I just need to see what the data looks like in a few locations to have a solution.

Sometimes the problem doesn't show up immediately in data and the code is too complex or uses a lot of wormhole techniques like particular forms of exception abuse, that's when I might fire up the debugger and browse frames instead.

That misses the mark. You can’t really compare a hackish ”print the world as a string“ function against a debuggers ability to stop time, walk around, pick things up, slice them open, put them somewhere else, and start time again.

That’s not just not the same league, it’s playing a whole different game.

I'm an OS developer and in my view using printf is like seeing only half the world at once. There's a whole world of platform specific decisions that are made at compile time and runtime that you can only see through the lens of a good assembly debugger.

You're also talking about debugging apps running comfortably in the idillic world created by the OS. It's much harder to debug foundational pieces with printf's when the program immediately panics or early printing isn't available.

In my opinion it's good to build habits that can be generalized to all sorts of software and not limit oneself to writing code in a highly structured environment where most of the work is done for you. I can trace through a program faster than someone can insert/remove printfs and recompile their program, and I don't need to think about what to print. I can look at everything at that point in time, covert data to strings, look at the stack, registers, etc. Very powerful stuff.

I've been, and still am, at the mercy of both of printf-debugging style and real debugger.

Long time ago, worked on a port of game from PC to Playstation 1. Since we had the Yaroze "devkit" (not really a devkit, rather amateur kit for doing games), printf debugging was the only thing available.

Things kind of worked, but when we #ifdef-out the printfs it was crashing (and no debugger). We somehow discovered that one of "printf" side effect was clearing the math errors.

Why must this be a mutually-exclusive situation?

You can have the source code debugger log messages to the output window without having to add logging statements and recompile the affected code.

This is the 21st century.

see visual studio's tracepoint functionality - works in native and .Net languages, https://devblogs.microsoft.com/visualstudio/tracepoints/

sure, if you want the logging messages available for perusal when deployed in production, then this won't help.

Even better - use Hot Reload and tweak your code in the debugger - https://learn.microsoft.com/en-us/visualstudio/debugger/hot-...

[edit] GDB's equivalent to tracepoint is mentioned elsewhere in this thread - https://news.ycombinator.com/item?id=42147372

I think unit tests and debug/verbose print statements are useful for debugging. Even if you're firing up the debugger, these provide a lot of information up front! But more importantly, they make your project more approachable for contributors who aren't already familiar with the debugger.

I trend more towards print debugging than breakpoints.

To me the beauty of print debugging is you can see the flow, and see it quickly in contrast to the debugger. Simply with the debugger, a lot of the time is spent stepping past (at the moment) superfluous breakpoints.

Step, step, step, step, …, step, step, BANG!

Versus a quick BANG preceded by a trail of debris I can then postmortem. I use both, naturally, but prefer the crashes with a debris field than walking on eggs to potential disaster.

From my (super limited) experience, debuggers shine when:

- You're using a dynamically typed language.

Something like Rust can eliminate most bugs that come from incorrect data types. For me, a lot of bugs used to come from types that were different from what I expect.

- It is super easy to run your program with a debugger attached.

If your code needs to run on a K8s cluster or a dedicated test machine instead of locally, logs are much easier to get hold of than a remote debug session. Some people aren't even aware that they can attach a debugger to a computer over the network, or inside a Docker container.

- Your environment.

If you don't use an IDE that supports a debugger, it's another friction point. I'm not sure if Vim has something similar to, say, PyCharm's debugger.

Similarly, if you're a junior, and you reach out to a senior and they tell you to debug using logs, you probably will never switch to using a debugger yourself.

You call it haskish but it's something I've done in the pasty and find useful.

I don't find debuggers all that useful, because I often find I'm spending more time thinking about how to use the debugger rather than how to fix the bug; since debugging is hard I want tools that I don't have to think about at all, as they distract me from thinking about the bug.

Maybe that's because I don't have enough experience with a particular tool. If I used a debugger more often it would come naturally to me. But I find most of my bugs are simple enough that that doesn't happen, because I write modular code and TDD.

> I just wish intellij had a time travel debbuger like Visual Studio Pro.

You might find our Java product interesting, it adds Time Travel Debug to IntelliJ - https://undo.io/products/java/

Undo captures everything the process does, below the JVM level, so you can reproduce / rewind any problem you record as many times as you want (and copy the recording out of production onto a dev machine to debug, etc etc).

Please get in touch if you'd like a free trial.

Interesting. I had a similar experience: once the debugging instrumentation with #ifdef macros was switched off, the code that worked before suddenly crashed. In my situation it had to do with the stack, because my debugging macros used some local/"automatic" variables, and that had concealed the bug before in the DEBUG build.

One thing I also noticed is that using "problem-oriented" languages like Python or Java changes where you spend your time trouble-shooting: ironically, not where the problem is (business logic) anymore, those parts of the code indeed tend to work better, but intead you waste time with libraries (Java: CLASSPATH, Python: packages, all:version conflicts). In Contrast, in C/C++ it was mostly memory management errors and bugs in the actual business logic (the former is also a great distraction, somewhat diminished by the introduction of smart pointers).

At Google, had to do Java on borg, and used few times the "Cloud Trace" debugger (not sure how it was called), but it allowed you to watch multiple instances of your binary in production, and then add some isolated set of java statements around code blocks, this way you can say (somehow) - if you end up on this line, then "inject" (somehow) something to log out... and then you can add whatever you want to be logged (like arguments, variables around, etc.).

But then later it got scrapped, or something like it.

Cloud "debugging" when you have multiple instances is one of the cases where there is no suitable enough debugger (yet).

If you use a time travel debugger (rr-project.org, undo.io - where I work, or Microsoft's WinTTD) you can generally just record an application to a file and debug it (with full fidelity) somewhere else.

And you don't need a full debugger setup on the target machine, just the recorder binary.

Gives you the possibility to have a proper debug experience without having to set up debugging that somehow works in a live k8s pod, or connects through special firewall holes or somesuch.

The big ones for me with log/printf debugging are:

- I can get a good idea of the temporal behavior of the program, and I can just scroll up to see earlier state, rather than having to restart the program in the debugger. (I know that "time travel" debuggers exist, but I've found them finicky.) I can scrub back and forth through time just by scrolling.

- I can compare runs by diffing the logs. Sometimes that alone is enough to show where things start going amiss. Or I can keep instrumented logs from baseline runs.

- If there's a personally useful set of printf statements in an area that I'm in a lot, I can save those off to a patch file or a local branch. I don't have to reapply my breakpoints / watchpoints in the debugger each time. Easy persistence.

(That said, I do like to start with a debugger when tackling reproducible crashes.)

Which is why a good reversible debugger is so powerful. (I use rr-project.org and Pernosco but I imagine undo.io is similar.) Don't worry about stopping at the right point. Go back to the critical points over and over again, and go backwards to what generated an input. If you like logs, the debugger has facilities for generating tailored ones. Or manually generate your own debugging session log trail, generating entries throughout the execution in any order, and see them in execution order.

> I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.

Debugger helps a lot when performing TDD. You will enjoy setting a breakpoint to check whether a line is hit as expected by a test case.

I mean sure, logging metadata is better than nothing. But that’s akin to saying you don’t need a car because your feet have never failed you at crossing distances. Yes, you need to learn driving first, and that can seem hard at the beginning, but I doubt you’d want to go back to walking after.

To each their own, but I wholeheartedly recommend learning about debuggers. It should be one of the core tools of every software engineer.

You can trivially add printf-like statements on the fly with debuggers; with GDB, one method I often use is the ability to attach commands to breakpoints, to be executed when the breakpoint is hit.

  break SourceFile.cpp:123
  command
    pp var1
    pp var2
    continue
  end

With that `continue` at the end there, this breakpoint will not pause execution (except to run the commands).

Debuggers can do printf debugging too with tracepoints.