The Grug Brained Developer (2022)

“Good debugger worth weight in shiny rocks, in fact also more”

I’ve spent time at small startups and on “elite” big tech teams, and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.

I generally agree that it’s the best way to start understanding a system. Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.

Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.

Using a debugger on my own code is easy and I love it. The second the debugger steps deep into one of the libs or frameworks I'm using, I'm lost and I hate it. That framework / lib easily has many ten thousands of person-hours under it's belly, and I'm way out of my league.

I'd love to use a real debugger but as someone who has only ever worked at large companies, this was just never an option. In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. Print debugging is all you have. If there's a problem with the logging system itself or something that crashes the program before the logs can flush, then not even that.

Well, what's your workflow? Is there a particular debugger that you love?

I've been doing this professionally for over a decade and have basically never used a debugger

I've often felt that I should, but never enough to actually learn how

There was a good discussion on this topic years ago [0]. The top comment shares this quote from Brian Kernighan and Rob Pike, neither of whom I'd call a young grug:

> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

I tend to agree with them on this. For almost all of the work that I do, this hypothesis-logs-exec loop gets me to the answer substantially faster. I'm not "trying to run the code forwards in my head". I already have a working model for the way that the code runs, I know what output I expect to see if the program is behaving according to that model, and I can usually quickly intuit what is actually happening based on the incorrect output from the prints.

[0] The unreasonable effectiveness of print debugging (349 points, 354 comments) April 2021 https://news.ycombinator.com/item?id=26925570

I would tend to say that printf debugging is widespread in the Linux-adjacent world because you can't trust a visual debugger to actually be working there because of the general brokenness of GUIs in the Linux world.

I didn't really get into debuggers until (1) I was firmly in Windows, where you expect the GUI to work and the LI to be busted, and (2) I'd been burned too many times by adding debugging printfs() that got checked into version control and caused trouble.

Since then I've had some adventures with CLI debuggers, such as using gdb to debug another gdb, using both jdb and gdb on the same process at the same time to debug a Java/C++ system, automating gdb, etc. But there is the thing, as you say, is that there is usually some investment required to get the debugger working for a particular system.

With a good IDE I think Junit + debugging gives an experience in Java similar to using the REPL in a language like Python in that you can write some code that is experimental and experiment it, but in this case the code doesn't just scroll out of the terminal but ultimately gets checked in as a unit test.

Same, this isn't my choice, debuggers don't work here. And we don't even have microservices.

Adding these print statements is one of my favorite LLM use cases.

Hard to get wrong, tedious to type and a huge speed increase to visually scan the output.

Debuggers exist in the terminal, in vim, and in emacs.

I’ve learned not to go against the grain with tools, at least at big companies. Probably some dev productivity team has already done all the annoying work needed to make the company’s codebase work with some debugger and IDE, so I use that: currently, it’s VS Code and LLDB, which is fine. IntelliJ and jdb at my last job was probably better overall.

My workflow is usually:

1. insert a breakpoint on some code that I’m trying to understand

2. attach the debugger and run any tests that I expect to exercise that code

3. walk up and down the call stack, stepping occasionally, reading the code and inspecting the local variables at each level to understand how the hell this thing works and why it’s gone horribly wrong this time.

4. use my new understanding to set new, more relevant breakpoints; repeat 2-4.

Sometimes I fiddle with local variables to force different states and see what happens, but I consider this advanced usage, and anyway it often doesn’t work too well on my current codebase.

> and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging.

One of the first things I do in a codebase is get some working IDE/editor up where I can quickly run the program under a debugger, even if I'm not immediately troubleshooting something. It's never long before I need to use it.

I was baffled when I too encountered this. Even working collaboratively with people they'd have no concept of how to use a debugger.

"No, set a breakpoint there"

"yeah now step into the function and inspect the state of those variables"

"step over that"

: blank stares at each instance :

I do a lot of print statements as well. I think the greatest value of debuggers comes when I’m working on a codebase where I don’t already have a strong mental model, because it lets me read the code as a living artifact with states and stack traces. Like Rob Pike, I also find single-stepping tedious.

Personally, I think that debugger is very helpful in understanding what is going on, but once you are familiar with the code and data structures, I am very often pretty close in my assessment, so scanning code and inserting multiple print lines is both faster and more productive.

I only used debugger recently in C# and C, when I was learning and practicing them.

A lot of people think that. That's why it's important to read the essay.

I am also in the camp that has very little use for debuggers.

A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values. I add more detailed logging as I start to use the system and find out what needs extra scrutiny. This approach is very easy to get started with and maintain, and provides powerful insight into problems as they arise.

I also put a lot of work into formatting log statements. I once worked on a distributed system, and getting the prefix of each log statement exactly right was very useful -- node id, pid, timestamp, all of it fixed width. I could download logs from across the cluster, sort, and have a single file that interleaved actions from across the cluster.

A log is very different than a debugger though, one tells you what happened, one shows you the entire state and doesn't make you assemble it in your head.

Your framing makes it sound like the log is worse in some way, but what the log gives you that the debugger makes you assemble in your head is a timeline of when things happen. Being able to see time is a pretty big benefit for most types of software.

I can always drop an entire state object into the log if I need it, but the only way for a debugger to approximate what a log can give me is for me to step through a bunch of break points and hold the time stream in my head.

The one place where a debugger is straight up better is if I know exactly which unit of code is failing and that unit has complicated logic that is worth stepping through line by line. That's what they were designed for, and they're very useful for that, but it's also not the most common kind of troubleshooting I run into.

Why would you want a GUI debugger?

At my company our system is composed of 2 dozen different services and all of them can run locally in minikube and easily be debugged in jetbrains.

It's not worse or better, but its not really comparable is all I am really saying, I would not use them for the same things.

Can see all your source code while you're debugging. And it's not like emacs where your termcap is 99.99% right which means it is 0.01% wrong. (Mac-ers get made when something is 1px out of place, in Linux culture they'll close a bug report if the window is 15000 px to the left of the screen and invisible because it's just some little fit and finish thing)

But you can just use the "step out" feature to get back out when you realise you've gone into a library function. Or "step over" when you can see you're about to go into one.

I find that debuggers solve a very specific class of bugs of intense branching complexity in a self contained system. But the moment there's stuff going in and out of DBs, other services, multithreading, integrations, etc, the debugger becomes more of a liability without a really advanced tooling team.

Agreed. Typically my debugger use case is when I'm exploring a potentially unknown range of values at a specific point in time, where I also might not know how to log it out. Having the LLM manage all of that for me and get it 95% correct is the real minor superpower.

> Brian Kernighan and Rob Pike

Most of us aren't Brian Kernighan or Rob Pike.

I am very happy for people who are, but I am firmly at a grug level.

I think a lot of “naturals” find visual debuggers pointless, but for people who don’t naturally intuit how a computer works it can be invaluable in building that intuition.

I insist that my students learn a visual debugger in my classes for this reason: what the "stack" really is, how a loop really executes, etc.

It doesn't replace thinking & print debugging, but it complements them both when done properly.

I tend not to use a debugger for breakpoints but I use it a lot for watchpoints because I can adjust my print statements without restarting the program

What do you mean “visual debugger?”

I've had more sr devs than I tell me they don't see a benefit to using a debugger, because they have a type system.

Wtaf?

I don't use debuggers in general development but use them a lot when writing and running automated tests, much faster and easier to see stuff than with print statements.

Exactly, these judiciously placed print statements help me locate the site of the error much faster than using a debugger. Then, I could switch to using a debugger once I narrow things down if I am still unsure about the cause of the problem.

It has gotten to the point where when somebody wants to add a DSL to our architecture one of my first questions is "where is your specification for integrating it to the existing debuggers?"

If there isn't one, I'd rather use a language with a debugger and write a thousand lines of code than 100 lines of code in a language I'm going to have to black box.

Presumably an IDE rather then dealing the gdb CLI.

In vscode when you step to the next statement it highlights in the left pane the variables that change. Something like that.

It's useful for a beginner e.g in a for loop to see how `i` changes at the end of the loop. And similarly with return values of functions and so on.

What I find annoying is how these async toolkits screw up the stack trace, so I have little what the real program flow looks like. That reduces much of the benefit off the top.

Some IDEs promise to solve that, but I’ve not been impressed thus far.

YMMV based on language/runtime/toolkit of course. This might get added to my wishlist for my next language of choice.

This is basically it. When I started programming in C, I used a debugger all the time. Even a bit later doing Java monoliths I could spin up the whole app on my local and debug in the IDE. But nowadays running a dozen processes and containers and whatnot, it's just hopeless. The individual developer experience has gone very much backwards in the microservice era so the best thing to do is embrace observability, feature toggles etc and test in prod or a staging environment somewhere outside of your local machine.

In the early 2000’s I whipped up a tool to convert log statements into visual swim lanes like the Chrome profiler does. That thing was a godsend for reasoning about complex parallelism.

I am young grug who didn't use debuggers much until last year or so.

What sold me on debugger is the things you can do with it

    * See values and eval expressions in calling frames.
    * Modify the course of execution by eval'ing a mutating expression.
    * Set exception breakpoints which stop deep where the exception is raised.

One other such tool is REPL. I see REPL and debugger as complementary to each other, and have some success using both together in VSCode, which is pretty convenient with autoreload set. (https://mahesh-hegde.github.io/posts/vscode-ipython-debuggin...)

The rise of virtualization, containers, microservices, etc has I think contributed to this being more difficult. Even local dev-test loops often have something other than you launching the executable, which can make it challenging to get the debugger attached to it.

Not any excuse, but another factor to be considered when adding infra layers between the developer and the application.

There's another story I heard once from Rob Pike about debugging. (And this was many years ago - I hope I get the details right).

He said that him and Brian K would pair while debugging. As Rob Pike told it, he would often drive the computer, putting in print statements, rerunning the program and so on. Brian Kernighan would stand behind him and quietly just think about the bug and the output the program was generating. Apparently Brian K would often just - after being silent for awhile - say "oh, I think the bug is in this function, on this line" and sure enough, there it was. Apparently it happened so often enough that he thought Brian might have figured out more bugs than Rob did, even without his hands touching the keyboard.

Personally I love a good debugger. But I still think about that from time to time. There's a good chance I should step away from the computer more often and just contemplate it.

I use single-stepping very rarely in practice when using a debugger, except when following through a "value of a variable or two". Yet it's more convenient than pprint.pprint() for that because structured display of values, eval expression, and ability to inspect callers up the stack.

IDEs tend to have a "just my code" option.

Agreed, I spent a lot more time using debuggers when I was getting started

Another underutilized debugging superpower is debug-level logging.

I've never worked somewhere where logging is taken seriously. Like, our AWS systems produce logs and they get collected somewhere, but none of our code ever does any serious logging.

If people like print-statement debugging so much, then double down on it and do it right, with a proper logging framework and putting quality debug statements into all code.

> A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values.

This is an anti-pattern which results in voluminous log "noise" when the system operates as expected. To the degree that I have personally seen gigabytes per day produced by employing it. It also can litter the solution with transient concerns once thought important and are no longer relevant.

If detailed method invocation history is a requirement, consider using the Writer Monad[0] and only emitting log entries when either an error is detected or in an "unconditionally emit trace logs" environment (such as local unit/integration tests).

0 - https://williamyaoh.com/posts/2020-07-26-deriving-writer-mon...

Debuggers are also brittle when working with asynchronous code

Debuggers actually can hide entire categories of bugs caused by race conditions when breakpoints cause async functions to resolve in a different order than they would when running in realtime

Are you more productive with a debugger? For all bugs? How so?

some of my best work as a programmer is done walking my dog or sitting in the forest

This! Also my guess would be Kernighan or Pike aren't (weren't?) deployed into some random codebase every now and then, while most grugs are. When you build something from scratch then you can get by without debuggers, sure, but foreign codebase, a stupid grug like I can do much better with tools.

Honestly, I consider myself pretty comfortable with the terminal and vim and whatnot, but I've never been able to get into using GDB. For me I feel like a debugger is one of those things that's just so much better as a proper GUI.

On the other hand, John Carmack loves debuggers - he talks about the importance of knowing your debugging tools and using them to step through a complex system in his interview with Lex Friedman. I think it's fair to say that there's some nuance to the conversation.

My guess is that:

- Debuggers are most useful when you have a very poor understanding of the problem domain. Maybe you just joined a new company or are exploring an area of the code for the first time. In that case you can pick up a lot of information quickly with a debugger.

- Print debugging is most useful when you understand the code quite well, and are pretty sure you've got an idea of where the problem lies. In that case, a few judicious print statements can quickly illuminate things and get you back to what you were doing.

It's absolutely not an anti-pattern if you have appropriate tools to handle different levels of logging, and especially not if you can filter debug output by area. You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".

I usually roll my own wrapper around an existing logging package, but https://www.npmjs.com/package/debug is a good example of what life can be like if you're using JS. Want to debug your rate limiter? Write `DEBUG=app:middleware:rate-limiter npm start` and off you go.

> time to decide where to put print statements

But... that's where you put breakpoints and then you don't need to "single-step" through code. Takes less time to put a breakpoint then to add (and later remove) temporary print statements.

(Now if you're putting in permanent logging that makes sense, do that anyway. But that probably won't coincide with debugging print statements...)

True, but then you're still left stepping through your breakpoints one by one.

Printf debugging gives you the full picture of an entire execution at a glance, allowing you to see time as it happened. The debugger restricts you to step through time and hold the evolution of state in your memory in exchange for giving you free access to explore the state at each point.

Occasionally that arbitrary access is useful, but more often than not it's the evolution of state that you're interested in, and printf gives you that for free.

Debuggers are great until you have to work in a context where you can't attach a debugger. Good old printf works in every context.

By all means, learn to use a debugger well but don't become overly dependent on them.

I use a repl instead of a debugger. White box vs black box.

I think it depends on the debugger and the language semantics as well. Debugging in Swift/Kotlin, so so. The Smalltalk debugger was one of the best learning tools I ever used. “Our killer app is our debugger” doesn’t win your language mad props though.

It depends on the domain. Any complex long lived mutable state, precise memory management or visual rendering probably benefits from debuggers.

Most people who work on crud services do not see any benefit from it, as there is practically nothing going on. Observing input, outputs and databases is usually enough, and when it's not a well placed log will suffice. Debuggers will also not help you in distributed environments, which are quite common with microservices.

I’m a debugger guy too, but print statements can be very powerful. I keep this bookmarked https://tenderlovemaking.com/2016/02/05/i-am-a-puts-debugger...

the intellij debugger, for example, as opposed to a command line debugger

There's this idea that the way you use a debugger is by stepping over line after line during execution.

That's not usually the case.

Setting conditional breakpoints, especially for things like break on all exceptions, or when the video buffer has a certain pattern, etc, is usually where the value starts to come in.

> It's absolutely not an anti-pattern if you have appropriate tools to handle different levels of logging, and especially not if you can filter debug output by area.

It is an anti-pattern due to what was originally espoused:

  I add logging code, that stays forever. For a major 
  interface, I'll usually start with INFO level debugging, to 
  document function entry/exit, with param values.

There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended. Note that service request/response logging is its own concern and is out of scope for this discussion.

Also, you did not address the non-trivial cost implications of voluminous log output.

> You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".

Regarding the above, production-ready logging libraries such as Logback[0], log4net[1], log4cpp[2], et al, allow for run-time configuration to determine what "areas" will have their entries emitted. So "all logs all the time" is a non sequitur in this context.

What is relevant is the technique identified of emitting execution context when it matters and not when it doesn't. As to your `npm` example, I believe this falls under the scenario I explicitly identified thusly:

  ... an "unconditionally emit trace logs" environment
  (such as local unit/integration tests).

0 - https://logback.qos.ch/

1 - https://logging.apache.org/log4net/index.html

2 - https://log4cpp.sourceforge.net/

I don't often need gdb, but I appreciate the emacs mode that wraps it every time.

I understand that you explained some exceptions to the rule, but I disagree with two things: the assumption of incompetence on the part of geophile to not make logging conditional in some way, and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.

> the non-trivial cost implications of voluminous log output

If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.

Might be something to this. I relied heavily on IDE debugger and was integral part of my workflow for a while as a yungin and but very rare that I bother these days (not counting quick breaks in random webapps).

Perhaps been underappreciating the gap in mental intuition of the runtime between then and now and how much the debugger helped to bridge.

My hypothesis is that because we generally don't/can't use debuggers in production but rather rely on logging and tracing, that extends to local dev.

> Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.

I really don't get this at all. To me it is infinitely easier to iterate and narrow the problem rather than trying to identify sight-unseen where the problem is—it's incredibly rare that the bug immediately crashes the program. And you can fit a far higher density of relevant information through print statements over execution of a reproduced bug than you can reproduce at any single point in the call stack. And 99% of the information you can access at any single point in the call stack will be irrelevant.

To be sure, a debugger is an incredibly useful and irreplaceable tool.... but it's far too slow and buggy to rely on for daily debugging (unless, as you indicate, you don't know the codebase well enough to reason about it by reading the code).

Things that complicate this:

* highly mutable state

* extremely complex control or data flow

* not being able to access logs

* the compiler lying to you or being incorrect

* subtle instruction ordering issues

Seems to be the opposite for me. Usually I can pretty quickly figure out how to garden hose a bunch of print statements everywhere in a completely unfamiliar domain and language.

But the debugger is essential for all the hard stuff. I’ll take heap snapshots and live inside lldb for days tracking down memory alignment issues. And print statements can be either counterproductive at best, or completely nonexistent in embedded or GPU bound compute.

> Takes less time to put a breakpoint then to add (and later remove) temporary print statements

Temporary? Nah, you leave them in as debug or trace log statements. It's an investment. Shipping breakpoints to a teammate for a problem you solved three months ago is a tedious and time-consuming task.

Anyway, breakpoints are themselves quite tedious to interact with iterative problem solving. At least if you're familiar with the codebase.

I'm sorry, but this juxtaposition is very funny to me:

- John Carmack loves debuggers

- Debuggers are most useful when you have a very poor understanding of the problem domain

All these print debugging advocates are blowing my mind. Are most people unaware that both lldb and gdb have conditional pass throughout breakpoints with function hooks? In other words, you can create a breakpoint that just prints its location and doesn’t pause execution.

You can script this so all function entry/exists, or whatever, are logged without touching the code or needing to recompile. You can then bulk toggle these breakpoints, at runtime, so you only see a particular subset when things get interesting.

Modifying the code to print stuff will feel barbaric after driving around a fine tuned debugging experience.

If using step-by-step debugging is a minor superpower, conditional break-points make you an Omega level programming superthreat.

debugging is useful when your codebase is bad: imperative style, mutable state, weak type system, spaghetti code, state and control variables intermixed.

i'd rather never use debugger, so that my coding style is enforced to be clean code, strong type system, immutable variables, explicit error control, explicit control flow etc

> ... I disagree with two things: the assumption of incompetence on the part of geophile to not make logging conditional in some way ...

I assumed nothing of the sort. What I did was identify an anti-pattern and describe an alternative which experience has shown to be a better approach.

"Incompetence" is your word, not mine.

> ... and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.

I fail to see the nuance you apparently can see.

>> the non-trivial cost implications of voluminous log output

> If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.

Cloud deployment requires transmission of log entries to one or more log aggregators in order to be known.

By definition, this involves network I/O.

Network communication is orders of magnitude slower than local I/O.

Useless logging of "function entry/exit, with param values" increases pressure on network I/O.

Unless logging is allowed to be lossy, which it never is, transmission must be completed when log buffers are near full capacity.

Provisioning production systems having excessive logging can often require more resources than those which do not excessively log.

Thus disproving:

> ... even at runtime the costs are often trivial.

When considering the implication of voluminous log output in a production environment.

Would you be willing to share this tool?

This hasn't been my experience.

When I'm unfamiliar with a codebase, or unfamiliar with a particular corner of the code, I find myself reaching for console debugging. Its a bit of a scattershot approach, I don't know what I'm looking for so I console log variables in the vicinity.

Once I know a codebase I want to debug line by line, walking through to see where the execution deviates from what I expected. I very frequently lean on conditional breakpoints - I know I can skip breaks until a certain condition is met, at which point I need to see exactly what goes wrong.

Adding print statements sucks when you are working on native apps and you have to wait for the compiler and linker every time you add one. Debuggers hands down if you are working on something like C++ or Rust. You can add tracepoints in your debugger if you want to do print debugging in native code.

In scripting languages print debugging makes sense especially when debugging a distributed system.

Also logging works better than breaking when debugging multithreading issues imo.

I use both methods.

I can't tell you how many times lldb has failed to hit breakpoints or has dumped me in some library without symbols. This was in Xcode while writing an iOS app, maybe it's better in other environments.

Print debugging, while not clever or powerful, has never once failed me.

Having worked in many languages and debuggers across many kinds of backend and front end systems, I think what some folks miss here is that some debuggers are great and fast, and some suck and are extremely slow. For example, using LLDB with Swift is hot garbage. It lies to you and frequently takes 30 seconds or more to evaluate statements or show you local variable values. But e.g. JavaScript debuggers tend to be fantastic and very fast. In addition, some kinds of systems are very easy to exercise in a debugger, and some are very difficult. Some bugs resist debugging, and must be printf’d.

In short, which is better? It depends, and varies wildly by domain.

If you listen to what he has to say, it’s quite interesting. He would occasionally use one to step through an entire frame of gameplay to get an idea of performance and see if there were any redundancies.

As I said, conditional. As in, you add logging to your code but you either remove it at compile time or you check your config at run time. By definition, work you don't do is not done.

Sometimes the debugserver is flakey, I’ll give you that. But some that also sounds like UI quirks such as ambiguous breakpoints on a function definition with default initialized default values.

You can attach lldb without Xcode. Or you can open the lldb terminal in Xcode, pause execution, and inspect the breakpoints manually

I wonder if Brian Kerninghan was using modern tooling or that comment was using quote from 70’s.

How long are you realistically "waiting for the compiler and linker"? 3 seconds? You're not recompiling the whole project after all, just one source file typically

If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes. And then I'll have to hope that I can reproduce the issue without optimizations.

What's insane is that debuggers could support at least 80-90% of print debugging workflow with a good UI.

The only thing that's really missing is history of the values of the watches you added, all in one log. With filters because why not.

For some reason I never seen this option in any debugger I tried.

I've never had a debugger show me the entire state. I'm not even sure I want to know the entire state, but GDB has a lot of issue with anything but the most basic data structures most of the time, and I always need to explicitly ask for what I want to see by calling things like operator[] (and then hope that the particular operator[] wasn't optimized out of the final binary). It's not exactly a great experience.

You can use tracepoints instead of breakpoints, or (easier, at least for me), set up breakpoints to execute "print stack frame, continue" when hit - giving you the equivalent of printf debugging, but one you can add/remove without recompiling (or even at runtime) and can give you more information for less typing. And, should this help you spot the problem, you can easily add another breakpoint or convert one of the "printing" ones so it stops instead of continuing.

And of course, should the problem you're debugging be throwing exceptions or crashing the app, the debugger can pause the world at that moment for you, and you get the benefit of debugging and having a "printf log" of the same execution already available.

>Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.

TIL Linus Torvald is a young grug.

Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.

I think you should substitute “code” for “domain” in the last paragraph.

John Carmack knows his domain very well. He knows what he expects to see. The debugger gives him insight into what “other” developers are doing without having to modify their code.

For Carmack, managing the code of others the debug environment is their safe space. For Kernighan et al in the role of progenitorous developer it is the code itself that is the safe space.

It’s amazing what even the most subtle perturbation in output can tell you about the internal state of the code.

If you want to double-down on logging and do it right: make your logs fit for computer consumption and the source of truth.

That's all event-sourcing is.

Where do configs and secrets come from? Also, big company = hundreds of microservices.

You also have to remember the context.

First that visual debugging was still small and niche, probably not suited to the environment at Bell Labs at the time, given they were working with simpler hardware that might not provide an acceptable graphical environment (which can be seen as a lot of the UNIX system is oriented around the manipulation of lines of text). This is different from the workplace where most game developers, including J. Carmack were, with access to powerful graphical workstations and development tools.

Secondly there’s also a difference on the kind of work achieved: the work on UNIX systems mostly was about writing tools than big systems, favoring composition of these utilities. And indeed, I often find people working on batch tools not using visual debuggers since the integration of tools pretty much is a problem of data structure visualization (the flow being pretty linear), which is still cumbersome to do in graphical debuggers. The trend often is inverted when working on interactive systems where the main problem actually is understanding the control flow than visualizing data structures: I see a lot more of debuggers used.

Also to keep in mind that a lot of engineers today work on Linux boxes, which has yet to have acceptable graphical debuggers compared to what is offered in Visual Studio or XCode.

Having the code, the callstack, locals, your watched variables and expressions, the threads, memory, breakpoints and machine code and registers if needed available at a glance? As well as being able to dig deeper into data structures just by clicking on them. Why wouldn't you want that? A good GUI debugger is a dashboard showing the state of your program in a manner that is impossible to replicate in a CLI or a TUI interface.

But can't you instead just set a breakpoint next to wherever you are gonna put that print stmt and inspect value once code hits? print stmt seems like extra overhead

I know a lot of people do that in all kinds of software (especially enterprise), still, I can't help but notice this is getting close to Greenspunning[0] territory.

What you describe is leaving around hand-rolled instrumentation code that conditionally executes expensive reporting actions, which you can toggle on demand between executions. Thing is, this is already all done automatically for you[1] - all you need is the right build flag to prevent optimizing away information about function boundaries, and then you can easily add and remove such instrumentation code on the fly with a debugger.

I mean, tracing function entry and exit with params is pretty much the main task of a debugger. In some way, it's silly that we end up duplicating this by hand in our own projects. But it goes beyond that; a lot of logging and tracing I see is basically hand-rolling an ad hoc, informally-specified, bug-ridden, slow implementation of 5% of GDB.

Why not accept you need instrumentation in production too, and run everything in a lightweight, non-interactive debugging session? It's literally the same thing, just done better, and couple layers of abstraction below your own code, so it's more efficient too.

--

[0] - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

[1] - Well, at least in most languages used on the backend, it is. I'm not sure how debugging works in Node at the JS level, if it exists at all.

That doesn’t necessarily give you a clean log to review

> How long are you realistically "waiting for the compiler and linker"? 3 seconds?

I've worked on projects where incremental linking after touching the wrong .cpp file will take several minutes... and this is after I've optimized link times from e.g. switching from BFD to Gold, whereas before it might take 30 minutes.

A full build (from e.g. touching the wrong header) is measured in hours.

And that's for a single configuration x platform combination, and I'm often the sort to work on the multi-platform abstractions, meaning I have it even worse than that.

> If I wanna use a debugger though, now that means a full recompile to build the project without optimizations

You can use debuggers on optimized builds, and in fact I use debuggers on optimized builds more frequently than I do unoptimized builds. Granted, to make sense of some of the undefined behavior heisenbugs you'll have to understand disassembly, and dig into the lower level details when the high level ones are unavailable or confused by optimization, but those are learnable skills. It's also possible to turn off optimizations on a per-module basis with pragmas, although then you're back into incremental build territory.

Conditionals aren't free either, and conditionals - especially compile-time - on logging code are considered by some a bugprone anti-pattern as well.

The code that computes data for and assembles your log message may end up executing logic that affects the system elsewhere. If you put that code under conditional, your program will behave differently depending on the logging configuration; if you put it outside, you end up wasting potentially substantial amount of work building log messages that never get used.

I work on Scala projects. Adding a log means stopping the service, recompiling and restarting. For projects using Play (the biggest one is one of them) that means waiting for the hot-reload to complete. In both cases it easily takes at least 30s on the smallest projects, with a fast machine. With my previous machine Play's hot reload on our biggest app could take 1 to 3mn.

I use the debugger in Intellij, using breakpoints which only log some context and do not stop the current thread. I can add / remove them without recompiling. There's no interruption in my flow (thus no excuse to check HN because "compiling...")

When I show this to colleagues they think it's really cool. Then they go back to using print statements anyway /shrug

I prefer a debugger first workflow, I'm ideally always running in the debugger except in production, so I'm always ready to inspect a bug that depends on some obscure state corruption.

I think of "don't break the debugger" as a top important design heuristic.

It's one point in favor of async, and language that make async debugging easy.

And at the very least, you can encapsulate things so they can be debugged separately from the multi threading stuff.

If you're always using the debugger, you learn to build around it as one of your primary tools for interacting with code, and you notice right away if the debugger can't help with something.

While also avoiding having to re-run cases to get new introspection when you forgot to add a print statement.

I tend to do both, print statements when I don't feel I want to be stepping through some cumbersome interplay of calls but diving into the debugger to step through the nitty gritty, even better when I can evaluate code in the debugger to understand the state of the data at precise points.

I don't think there's a better or worse version of doing it, you use the tool that is best for what introspection you need.

They can be used in conjunction. You can drop into debugger from REPL and vice versa.

I don't quite like littering the code with logs, but I understand there's a value to it.

The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.

This is useful when you encounter unknown problem and need to find unusual stuff in the logs. This unusual stuff is not always an error state, it might be some aggregate problem (something is called too many times, something is happening in problematic order, etc.)

>How long are you realistically "waiting for the compiler and linker"? 3 seconds? You're not recompiling the whole project after all, just one source file typically

10 minutes.

>If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes.

Typically always compile with debug support. You can debug an optimized build as well. Full recompile takes up to 45 minutes.

The largest reason to use a debugger is the time to recompile. Kinda, I actually like rr a lot and would prefer that to print debugging.

The tools are not mutually exclusive. I also do quite a lot with print debugging, but some of the most pernicious problems often require a debugger.

> It takes less time to decide where to put print statements than to single-step to the critical section of code

Why would you ever be single-stepping? Put a break point (conditional if necessary) where you would put the print statement. The difference between a single break point and a print statement is that the break point will allow you to inspect the local variables associated with all calls in the stack trace and evaluate further expressions.

So when do you debug instead of using print statements? When you know that no matter what the outcome of your hypothesis is, that you will need to iteratively inspect details from other points up the stack. That is, when you know, from experience, that you are going to need further print statements but you don't know where they will be.

10 minutes for linking? The only projects I've touched which have had those kinds of link times have been behemoths like Chromium. That must absolutely suck to work with.

Have you tried out the Mold linker? It might speed it up significantly.

> You can debug an optimized build as well.

Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.

I set a break point, look at the variables in play and then start looking up the call stack.

This depends on a lot of things.

For example, one thing you wrote that jumps out at me:

> I already have a working model for the way that the code runs [...]

This is not always true. It's only true for code that I wrote or know very well. E.g. as a consultant, I often work on codebases that are new to me, and I do tend to use debuggers there more often than I use print debugging.

Although lots of other variables affect this - how much complicated state there is to get things running, how fast the "system" starts up, what language it's written in and if there are alternatives (in some situations I'll use a Jupyter Notebook for exploring the code, Clojure has its own repl-based way of doing things, etc).

I used to use debugger when I was young - disk space was small, disks were slow and logging was expensive.

Now, some 35 years later, I greatly prefer logs. This way I can compare execution paths of different use cases, I can compare outcomes of my changes, etc. I am not confined to a single point of time with tricky manual control as with debugger, I can see them all.

To old grugs: learning to execute and rewind code in your head is a major superpower. And it works on any codebase.

Debuggers absolutely help in distributed environments, in the exact same way that they help with multithreaded debugging of a single process. It is certainly requires a little bit more setup, but there isn't some essential aspect of a distributed environment that precludes the techniques of a debugger.

The only real issue in debugging a distributed/multithreaded environment is that frequently there is a timeout somewhere that is going to kill one of the threads that you may have wanted to continue stepping through after debugging a different thread.

I thought debugging was table stakes. It isn't always the answer. If a lot is going on logs can be excellent (plus grep or an observability tool)

However debugging is an essential tool in the arsenal. If something is behaving oddly even the best REPL can't match debugging as a dev loop (maybe lisp excepted).

I even miss the ability to move the current execution point back in .NET now I use Go and JS. That is a killer feature. Edit and continue even more so!

Then next level is debugging unit tests. Saved me hours.

You probably just don't know how to use conditional breakpoints effectively. This is faster than adding print statements.

But that is still slow compared to print debugging if there is a lot happening. Print debugging you can just print out everything that happens and then scan for the issue and you have a nice timeline of what happens after and before in the print statements.

I don't think you can achieve the same result using debuggers, they just stop you there and you have no context how you got there or what happens after.

Maybe some people just aren't good at print debugging, but usually it finds the issue faster for me, since it helps pinpointing where the issue started by giving you a timeline of events.

Edit: And you can see the result of debugger use in this article, under "Expression Complexity" he rewrote the code to be easier to see in a debugger because he wanted to see the past values. That makes the code worse just to fit a debugger, so it also has such problems. When I use a debugger I do the same, it makes the code harder to read but easier to see in a debugger.

I feel like you need to know when to use what. Debugger is so much faster and easier for me when looking for errors in the use of (my own) abstractions. But when looking for errors in the bowels of abstracted or very small partitioned code print logs are far easier to see the devil in the detail.

There's a time and place for everything, and an escalation of tooling/environmental context for me.

1. Printing vars in unit tests may be the fastest first approach. If i know where the bug may be.

2. When that fails i usually bring in debuggers to unit tests.

3. When these aren't helping, you need debuggers on the entire binary.

4. Still stuck? Use a debugger in production.

>10 minutes for linking? The only projects I've touched which have had those kinds of link times have been behemoths like Chromium. That must absolutely suck to work with.

I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.

> Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.

Yeah, but sometimes that's life. Reading the assembly and what not to figure things out.

>That must absolutely suck to work with.

Well, you know, I also get to work on something approximately as exciting as Chromium.

Isn't 4 very unsafe? I wouldn't trust my code to pause in places that it doesn't usually pause in.

3. What if the binary interacts with other networked computers, you gonna debug all of them? Do you end up instrumeting the whole internet? You scope out, you spiral out of control until someone puts a limit.

> I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.

Yeah, which is why I was talking about source files. I was surprised that changing 1 source file (meaning re-compiling that one source file and then re-linking the project) takes 10 minutes. If you're changing header files then yeah it's gonna take longer.

It's been years since I last used a debugger, but then, it's been years since I last worked on code that was complicated enough to warrant it.

Which is a good thing! Easily comprehended code that you can reason about without stepping through it is good for grug brain.

FWIW, I've heard from people who know this stuff that linking is actually super slow for us :). I also wanted to try out mold, but I couldn't manage to get it to work.

I personally use both, and I'm not sure I find the argument about needing to step through convincing. I put the debugger breakpoint at the same place I might put a print. I hardly ever step through, but I do often continue to reach this line again. The real advantage is that you can inspect the current state live and make calls with the data.

However, I use prints a lot more because you can, as you say, usually get to the answer faster.

There might be another factor at not using the debugger beyond the pure cluelessness: often you can’t really run it in production. Back when I started with coding (it was Turbo Pascal 3.0, so you get the idea :-), I enjoyed the use of the debugger quite a lot.

But in 2000 I started working in a role which required understanding the misbehavior of embedded systems that were forwarding the live traffic, there was a technical possibility to do “target remote …” but almost never an option to stop a box that is forwarding the traffic.

So you end up being dependent on debugs - and very occasional debug images with enhanced artisanal diagnostics code (the most fun was using gcc’s -finstrument-function to catch a memory corruption of an IPSec field by an unrelated IKE code in a use-after free scenario)

Where the GDB shined though is the analysis of the crash dumps.

Implementing a “fake” gdb stub in Perl, which was sucking in the crash dump data and allow to leisurely explore it with debugger rather than decoding hex by hand, was a huge productivity boon.

So I would say - it’s better to have more than one tool in the toolbox and use the most appropriate one.

This is getting a bit far into the weeds, but I've found that debug output which is disabled by default in all environments is quite safe. I agree that it would be a problem to leave it turned on in development, testing, or staging environments.

I disagree - using an interactive debugger can give insights that just looking at the code can't (tbf it might be different for different people). But the number of times I have found pathological behaviour from just stepping through the code is many. Think "holy f**, this bit of code is running 100 times??" type stuff. With complex event-driven code written by many teams, it's not obvious what is happening at runtime by just perusing the code and stroking one's long wizard beard.

Wholeheartedly agree. There’s often good performance or security reasons why it’s hard to get a debugger running in prod, but it’s still worth figuring out how to do it IMO.

Your experience sounds more sophisticated than mine, but the one time I was able to get even basic debugger support into a production Ruby app, it made fixing certain classes of bug absolutely trivial compared to what it would have been.

The main challenge was getting this considered as a requirement up front rather than after the fact.

Indeed. Let the machine do the work!

> I disagree - using an interactive debugger can give insights that just looking at the code can't

This in no way disagrees with the quote. Both can be true. The quote isn’t saying debuggers can’t provide unique insights, just that for the majority of debugging the print statement is a faster way to get what you need.

Why the emphasis on the use of cartoons (graphical debuggers) for analyzing problems with the text of computer code?

This sounds like TRACE level information to me.

The whole concept of an “anti-pattern” is a discussion ender. It’s basically a signal that one party isn’t willing to consider the specific advantages and disadvantages of a particular approach in a given context.

Is there a name for an approach to debugging that requires neither debuggers nor print calls? It works like this:

1. When you get a stack trace from a failure, without knowing anything else find the right level and make sure it has sensible error handling and reporting.

1a. If the bug is reproduced but the program experiences no failure & associated stack trace, change the logic such that if this bug occurs then there is a failure and a stack trace.

1b. If the failure is already appropriately handled but too high up or relevant details are missing, fix that by adding handling at a lower level, etc.

That is the first step, you make the program fail politely to the user and allow through some debug option recover/report enough state to explain what happened (likely with a combination of logging, stack trace, possibly app-specific state).

Often it can also be the last step, because you can now dogfood that very error handling to solve this issue along with any other future issue that may bubble up to that level.

If it is not enough you may have to resort to debugging anyway, but the goal is to make changes that long-term make the use of either debuggers or print statements unnecessary in the first place, ideally even before the actual fix.

I work with some pretty niche tech where it's usually, ironically, easier to use a debugger than to add print statements. Unfortunately the debugger is pretty primitive, it can't really show the call stack for example. But even just stopping at a line of code and printing variables or poking around in memory is pretty powerful.

Sounds like Rob does use a debugger and it's name is Brian.

can you say more? how do you do it?

> Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.

Sigh. Same. To a large extent, this is caused by debuggers just sucking for async/await code. And just sucking in general for webdev.

> How long are you realistically “waiting for the compiler and linker”

Debugging kernel module issues on AL2 on bare metal ec2 with kprint. Issue does not reproduce in qemu. It happens

> How long are you realistically "waiting for the compiler and linker"? 3 seconds?

This is the "it's one banana Michael, how much could it cost, ten dollars?" of tech. I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds. I've worked in plenty of embedded environments where simply the download-and-reboot cycle took over a minute. Those are the places where an interactive debugger is most useful .. and also sometimes most difficult.

(at my 2000s era job a full build took over an hour, so we had a machine room of ccache+distcc to bring it down to a single digit number of minutes. Then if you needed to do a full place and route and timing analysis run that took anything up to twelve hours. We're deep into "uphill both ways" territory now though)

If you're doing cutting edge work, then by definition you're in an area you don't fully understand.

I certainly know how to debug each of the services in my environment, but how do you step-through debug a single request across services? Like, if service A make a gRPC call to service B, are you saying you can “step into” the call from A and your debugger is able to break on the corresponding call in B? And frames from the call in A are there in a logical “stack” from the breakpoint in B?

(Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)

In my experience you just slap minikube or k3s on your dev machine, and treat it as any other environment. Argo, helm, kustomize, whatever can all work against a local single-node cluster just fine. It takes some effort to make sure your configs are overridable per-environment, but it’s worth doing. (And something you’re hopefully doing anyway if you’re doing any kind of integration/test environment.)

It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.

Debuggers allow you inspect stuff forward in time, while print statements allow you to debug backwards. (There was a lot of academic work on reversible debuggers at one point; to be honest I haven’t kept up on how that turned out.)

If you can detect a problematic condition and you want to know what will happen next, a debugger is a great tool.

If you can detect a problematic condition and you need to find out what caused it, it’s printf all the way.

My theory is that different types of programming encounter these two types of problems at different relative rates, and that this explains why many people strongly prefer one over the other but don’t agree on which.

"Step out" is how to get out of the lower level frameworks, and or "step over" to avoid diving into them in the first place. I can't speak for other IDEs, but all of the JetBrains products have these.

It's from The Practice of Programming, published 1999. Not a lot has changed in debuggers since then from what I can see.

Yeah I think it's really addressing different bug issues.

One is finding a needle in a haystack - you have no idea when or where the bug occurred. Presumably your logging / error report didn't spit out anything useful, so you're starting from scratch. That and race conditions. Then print statements can be lovely and get you started.

Most of my debugging is a different case where I know about where in code it happened, but not why it happened, and need to know values / state. A few breakpoints before/during/after my suspected code block, add a few watches, and I get all the information I need quite quickly.

It seems unlikely that John Carmack doesn't understand his problem domain. Rather it is more likely the problem domain itself, i.e., game dev vs web dev. Game dev is highly stateful and runs in a single process. This class of program can logically be extended to any complex single computer program (or perhaps even a tightly coupled multi-computer program using MPI / related). Web dev effectively runs on a cluster of machines and tends to offload state to 3rd parties (like databases that, on their own look more like game dev) and I/O is loosely coupled / event driven. There is no debugger that can pause all services in web dev such that one can inspect the overall state of the system (and you probably don't want that). So, logging is the best approach to understand what is going on.

In any case, my suggestion is to understand both approaches and boldly use them in the right circumstance. If the need arises, be a rebel and break out the debugger or be a rebel and add some printfs - just don't weakly follow some tribal ritual.

I try all the time, but I always end up having to wrestle a trillion calls into some library code that has 0 relevance to me, and if the issue is happening at some undetermined point in the chain, you basically have to step through it all to get an idea for where things are going wrong.

On the other hand, the humble console.log() just works without requiring insanely tedious and frustrating debugger steps.

Debugging in Visual Studio by Microsoft has changed a lot in last 5 years, JetBrains IDE debugging a lot as well.

I can debug .NET application and change code live, change variable states if needed. Watch variables and all kinds of helpers like stack navigation was immensely improved since I started 15 years ago.

I can say debugging Java/.NET applications is totally different experience than using debugger from 1999.

Multi threaded apps debugging and all kinds of helpers i visual debuggers.

I just fail to see why someone would waste time putting in debug statements when they can configure debug session with conditional break points.

FWIW, it seems like poor man's tracing. You'd get that and a lot more having opentelemetry setup (using Jaeger for UI locally)

That is the difference between complex state and simple state.

I use a debugger when I've constructed a complex process that has a large cardinally of states it could end up in. There is no possibility that I can write logic checks (tests) for all source inputs to that state.

I don't use one when I could simply increase test situations to find my logical error.

Consider the difference between a game engine and a simple state machine. The former can be complex enough to replicate many features of the real world while a simple state machine/lexer probably just needs more tests of each individual state to spot the issue.

My current position has implemented a toolchain that essentially makes debugging either impossible or extremely unwieldy for any backend projects and nobody seems to think it's a problem.

Running a debugger on test failure is a ridiculously effective workflow. Instead of getting a wall of text, you drop right into the call stack where the failure/error happened. `pytest --pdb` in python, worth its weight in shiny rocks for sure!

I find the differences between printf debugging and line debuggers (or whatever you call them) unimportant in most circumstances.

Line debuggers usually have some nice conveniences, but the major bottlenecks are between the ears, not in the tool.

I usually use a normal debugger to find a problem when I can see its symptoms but not the original caus. That way I can break on the line that is causing the symptom, check what the variables are like and go back up the call stack to find the origin of the incorrect state. I can do all that in one shot (maybe a couple if I need to break somewhere else instead) rather than putting prints everywhere to try and work out what the call stack is, and a load of prints to list off all the local variables

I mean, "step over" and "go to line" exist...

While that's true, it doesn't necessarily help me understand why my parameters for said lib or whatever lead to things blowing up.

Intoruding logging can actually hide concurrency bugs.

This statement explain his position very clearly. Anyone who did any serious DOS programming understands it well.

"A debugger is how you get a view into a system that's too complicated to understand. I mean, anybody that thinks just read the code and think about it, that's an insane statement, you can't even read all the code on a big system. You have to do experiments on the system. And doing that by adding log statements, recompiling and rerunning it, is an incredibly inefficient way of doing it. I mean, yes, you can always get things done, even if you're working with stone knives and bare skins."

> I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds.

No C++ project compiles in 3 seconds, but your "change a single source file and compile+link" time is often on the order of a couple of seconds. As an example, I'm working on a project right now where a clean build takes roughly 30 seconds (thanks to recent efforts to improve header include hygiene and move stuff from headers into source files using PIMPL; it was over twice that before). However, when I changed a single source file and ran 'time ninja -C build' just now, the time to compile that one file and re-link the project took just 1.5 seconds.

I know that there are some projects which are much slower to link, I've had to deal with Chromium now and then and that takes minutes just to link. But most projects I've worked with aren't that bad.

Sometimes I need a debugger because there's a ton of variables or I just have no idea what's wrong and that's the easiest way to see everything that's going on. It's really frustrating to feel like I need a debugger and don't have a good way to add the IDEs visual debugger (because I'm using a CLI on a remote session or something). It's also really frustrating to be inside a debugging session and wish you knew what the previous value for something was but you can't because you can't go backwards in time. That happens so often to me, in fact, that print debugging is actually more effective for me in the vast majority of cases.

Agreed!

If you usually aren't able/allowed to use a debugger in production and must rely on observability tools, it's helpful to know how to utilize those tools locally as effectively as possible when debugging.

We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.

I still don't understand how, with a properly configured debugger, manually typing print statements is better than clicking a breakpoint at the spot you were going to print. Context overload might be an issue, but just add a 'watch' to the things you care about and focus there.

> debuggers don't work here

It’s impossible? Or would take engineering work to enable

I love debuggers, but unfortunately at my current job I've found that certain things we do to make our application more performant (mainly using giant structs full of fixed size arrays allocated at the start of the application) cause LLDB to slow to a crawl when `this` points to them. It really really doesn't like trying to read the state of a nearly 1GB struct...

Two situations immediately come to mind, though the second is admittedly domain specific:

1. If I actually pause execution, the thing I'm trying to debug will time out some network service, at which point trying to step forward is only going to hit sad paths

2. The device I'm debugging doesn't *have* a real debugger. (Common on embedded, really common for video games. Ever triggered a breakpoint in a graphics shader?) Here I might substitute "print" for "display anything at all" but it's the same idea really.

I posted this elsewhere in the thread, but if you listen to Carmack in the interview, it's quite interesting. He would occasionally use one to step through an entire frame of gameplay to get an idea of performance and see if there were any redundancies. This is what I mean by "doesn't understand the problem domain". He's a smart guy, but no one could immediately understand all the code added in by everyone else on the team and how it all interacts.

A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.

This is one of those reasons why you really really need to get other people on board with your workflows. If you're the only one who works like that and someone does something insane, but it technically works, but it blows your workflow up... that's your problem. "You should just develop how I'm developing. Putting in a print statement for every line then waiting 5 minutes for the application to compile."

So long as no one sees your workflow as valuable, they will happily destroy it if it means getting the ticket done.

No I can’t debug multiple services at once, unfortunately. But I will switch between them as I track the request over multiple runs. Also extensive logging in available in grafana helps me know which service is having the issue before I start debugging.

Yes all of this.

See ninkendo’s comment. They are doing it with the same tools we are.

> In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger.

I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)

Thankfully, we live in an era where entire AAA games can be written almost completely from scratch by one person. Not sarcasm. If I wrote the code myself, I know where almost everything is that could go wrong. It should come as no surprise that I do not use a debugger.

Some people are wizards with the debugger. Some people prefer printfs.

I used to debug occasionally but haven't touched a debugger in years. I'm not sure exactly why this is, but I'm generally not concerned with exactly what's in the stack on a particular run, but more with some specific state and where something changes time after time, and it's easier to collect that data programatically in the program than to figure out how to do it in the debugger that's integrated with whatever IDE I'm using that day.

And the codebase I deal with at work is Spring Boot. 90% of the stack is meaningless to me anyway. The debugger has been handy for finding out "what the f*$% is the caller doing to the transaction context" but that kind of thing is frustrating no matter what.

Anyway, I think they're both valid ways to explore the runtime characteristics of your program. Depends on what you're comfortable with, how you think of the problem, and the type of codebase you're working on. I could see living in the debugger being more useful to me if I'm in some C++ codebase where every line potentially has implicit destructor calls or something.

Their comment conflates debugging with logging.

Professional debuggers such as the one in IntelliJ IDEA are invaluable regardless of one's familiarity with a given codebase, to say otherwise is utter ignorance. Outside of logging, unless attaching a debugger is impractical, using print statements is at best wasting time.

I want to master JS/React and Python debugging. Am an "advanced beginner" in both. What tools do you recommend?

I'm looking around and I don't see anyone mentioning what I think is one of the biggest advantages of debugging: not having to recompile.

Even assuming print statements and debuggers are equally effective (they're not), debuggers are better simply because they are faster. With print statements, you might need to recompile a dozen times before you find whatever it is you're looking for. Even with quick builds this is infuriating and a ton of wasted time.

I understand you are likely doing this for shiny rocks, but life is too short to spend on abominable code bases like this.

This reminds me why I abandoned scala. That being said, even a small sbt project can cold boot in under 10 seconds on a six year old laptop. I shudder to think of the bloat in play if 30 seconds is the norm for a hot reload.

Perhaps consider that your experience is not universal and that others have good reasons for their decisions that are not simply ignorance.

Yeah I coded a AAA game yesterday.

Huh, this must really be a personal taste things, because I only want the debug info I specifically request to be printed on the next line. But I can imagine wanting a different interface to it.

This is usually enough for me, too. Use tracing, figure out where things fell apart in the traces, isolate those service(s), and debug from there. It's definitely more work. When we start new projects, I encourage people not to use services until proven necessary because this added layer of friction is a real drag. For a lot of us that isn't a choice, though.

Here is my circular argument against debuggers: if I learn to use a debugger, I will spend much, possibly most, of my time debugging. I'd rather learn how to write useful programs that don't have bugs. Most people believe this is impossible.

The trouble of course is that there is always money to be made debugging. There is almost no incentive in industry to truly eliminate bugs and, indeed, I would argue that the incentives in the industry actively encourage bugs because they lead to lucrative support contracts and large dev teams that spend half their time chasing down bugs in a never-ending cycle. If a company actually shipped perfect software, how could they keep extracting more money from their customer?

For JavaScript, you're actually able to debug fairly easily by default by adding a `debugger()` call in your code. Browsers will stop at that call, and start the debugger.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Another way (and probably a better idea) is creating a launch definition for VS Code in launch.json which attaches the IDE's debugger to Chrome. Here is an article describing how that works: https://profy.dev/article/debug-react-vscode

Breakpoints are nice because they can't accidentally get shipped to production like debugger calls can.

For Python I essentially do the same thing, minus involving Chrome. I run the entry point to my application from launch.json, and breakpoints in the IDE 'just work'. From there, you can experiment with how the debugger tools work, observe how state change as the application runs, etc.

If you don't use VS Code, these conventions are similar in other IDEs as well.

It's not a realtime system kind of thing where the debugger would change the behavior too much... It's possible with enough engineering work, but nobody has put that work in, in fact they had a debugger for some staging envs that they deleted. Lately they keep adding more red tape making it hard to even run something locally, let alone attach a debugger.

I guess you can attach a debugger for unit tests, but that's not very useful.

Yeah this is true, it can change the timing. But setting breakpoints or even just running in a debugger or even running a debug build at all without optimizations can also hide concurrency bugs. Literally anything can hide concurrency bugs.

Concurrency bugs just suck to debug sometimes.

I think graphical debuggers are a big help: 1. It separates the meta-information of the debugger into the graphical domain. 2. It's easier to browse code and set/clear breakpoints using the mouse than the keyboard.

I didn't say there aren't acceptable reasons to reach for the print statement, there are. But for the vast majority of codebases out there, if good debugger tooling is available, it's a crime not to use it as a primary diagnostics tool. Claiming otherwise is indeed ignorant if not irresponsible.

AAA games are not even close to being write-able by one person, what are you talking about. You couldn't even write AAA games from 20 years ago.

where you=Brian

I loved Chrome's debugger for years, then build tools and React ruined debugging for me.

Built code largely works with source maps, but it fails often enough, and in bizarre ways, that my workflow has simply gone back to console logs.

React's frequent re-renders have also made breakpoints very unpleasant - I'd rather just look at the results of console logs.

Are there ways I can learn to continue enjoying the debugger with TS+React? It is still occasionally useful and I'm glad its there, but I have reverted to defaulting to console logs.

Debugging is for the developer; logging is for all, and especially those who need to support the code without the skills/setup/bandwidth to drop into a debugger. Once you start paying attention to what/where/how of logging, you (and others) can spot things faster then you can step through the debugger. Plus logs provide history and and searchable.

This feels a little like "I don't use a cordless drill because my screw driver works so well and is faster in most cases" grug brain says use best tool, not just tool grug used last.

Nobody has time for that. Need time for prompting AI!

In order for this to cover enough space, I assume you‘d have to really pin down assumptions with asserts and so on in a design by contract style.

You might be confusing Brian with Ken?

Console logs in the browser have some unique advantages. You can assign the output to a variable, play with it etc.

But yes, any code that is inside jsx generally sucks to debug with standard tooling. There are browser plugins that help you inspect the react ui tree though.

For desktop GUI development I can't imagine not using breakpoints and stepping. Especially when you have several ways that some callback might be triggered. It's super helpful to break on a UI element signal (or slot) and then follow along to see why things aren't working.

I don't use debuggers as often in Python, probably because it's eaiser to throw code in a notebook and run line by line to inspect variables, change/inject state and re-run. That's possible but a lot harder to do in C++.

Also for embedded work, using a debugger and memory viewer is pretty powerful. It's not something people think about for Arduino but almost every commodity micro supports some sort of debugwire-like interface (which is usually simpler than JTAG).

Curious to learn more about why it is difficult to debug. I'm not familiar with service mesh. I also work at a large corp, but we use gateways and most things are event driven with kafka across domain boundaries. I spend most of my time debugging each service locally by injecting mock messages or objects. I do this one at a time if the problem is upstream. Usually, our logging helps us pinpoint the exact service at to target for debugging. It's really easy. Our devops infrastructure has built out patterns and libraries when teams need to mint a new service. Everything is standardized with terraform. Everything has the same standard swagger pages, everything is using okta, etc.. Seems a bit boring (which is good)

Agreed, debugging tools for the browser are almost comically incapable, to the point of not even useful in most cases.

It's easy if Someone(tm) has already set up a system where you can whip up the relevant bits with something like localstack.

But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.

I suppose in the description of "large corp" and service mesh, Someone would already exist and this would already have been worked out. It would be a nightmare dealing with hundreds of microservices without this kind of game plan.

You get all of that in the terminal debugger. That’s why dwarf files exist.

A different domain where debuggers are less useful: audio/video applications that sit in a tight, hardware driven loop.

In my case (a digital audio workstation), a debugger can be handy for figuring out stuff in the GUI code, but the "backend" code is essentially a single calltree that is executed up to a thousands times a second. The debugger just isn't much use there; debug print statements tend to be more valuable, especially if a problem would require two breakpoints to understand. For audio stuff, the first breakpoint will often break your connection to the hardware because of the time delay.

Being able to print stacktraces from inside the code is also immensely valuable, and when I am debugging, I use this a lot.

Even in my open source income-generating code base (ardour.org) a complete build takes at least 3 minutes on current Apple ARM hardware, and up to 9mins on my 16 core Ryzen. It's quite a nice code base, according to most people who see it.

Sometimes, you just really do need hundreds of thousands of lines of C++ (or equivalent) ...

All the information is there, but the presentation isn't. You have to keep querying it while you're debugging. Sure, there are TUI debuggers that are more like GUI debuggers. Except that they are worse at everything compared to a GUI debugger.

I refuse to believe there are professional software developers who don't use debuggers. What are you all working on? How do you get your bearings in a new code-base? Do you read it like a book, and keep the whole thing in your mind? How do you specify all of the relevant state in your print statements? How do you verify your logic?

> Print debugging is most useful when you understand the code quite well

Every debugger I've ever worked with has logpoints along with breakpoints that allow you to _dynamically_ insert "print statements" into running code without having to recompile or pollute the code with a line of code you're going to have to remember to remove. So I still think debuggers win.

You've gotten downvoted but I think you're correct - if there were no debuggers, the developers would be forced to write (better) code that didn't need them.

I find myself doing a mix of both. Source maps are good enough most of the time, I haven't seen the bizarre failures you're seeing - maybe your bundling configuration needs some tweaking? But yes, the frequent re-renders are obnoxious. In those cases logging is generally better.

Conditional breakpoints help alleviate the pain when there are frequent re-renders. Generally you can pinpoint a specific value that you're looking for and only pause when that condition is satisfied. Setting watch expressions helps a lot too.

I really tried but could not take to Lex Friedman's interview style.

I don’t know what debugger you’ve used but the entire query command is `f v` in lldb for the current stack frame

> I guess you can attach a debugger for unit tests, but that's not very useful.

That is in fact incredibly useful

I can't do it either. Something about typing commands that have some kind of grammar and pressing Return to submit each one for consideration just throws me off. Don't make me think. Thinking is typically what got the code into this mess in the first place - whether too much of it or too little, it doesn't really matter. Time to try some other approach.

Eh, it's there for those who want it, but nobody uses it

Find me a bank that will give me a 150k collateralized loan and after 2 years I will give you the best AAA game you've ever played. You choose all the features. Vulkan/PC only. If you respond back with further features and constraints, I will explain in great detail how to implement them.

I suspect you're trolling, but if not then this is the kind of thing that kickstarter or indiegogo are designed to solve: give me money on my word, in 2 years you get license keys to this thing, assuming it materializes. I was going to also offer peer-to-peer platforms like Prosper but I think they top out at $50k

See this just sounds like you do not have an engineering culture or learning, enabling, and using debuggers.

> substantially faster

Than what? In languages with good debugger support (see JVM/Java) it can be far quicker to click a single line to set a breakpoint, hit Debug, the inspect the values or evaluate expressions to get the runtime context you cant get from purely reading code. Print statements require rebuilding code and backing them out, so its hard to imagine that technique being faster.

I do use print debugging for languages with poor IDE/debugger support, but it is one big thing I miss when outside of Java.

Not sure what an engineering culture is, but I don't want it. I just want to attach a debugger to our stuff (not the unit tests).

I agree that logging all functions is reinventing the wheel.

I think there's still value in adding toggleable debug output to major interfaces. It tells you exactly what and where the important events are happening, so that you don't need to work out where to stick your breakpoints.

Debuggers are great. I too have tried and failed to spread their use.

But the reason is simple: they are always 100x harder to set up and keep working than just type a print statement and be done with it.

Especially when you're in a typical situation where an app has a big convoluted docker based setup, and with umpteen packers & compilers & transpilers etc.

This is why all software companies should have at least one person tasked with making sure there are working debuggers everywhere, and that everyone's been trained how to use them.

There should also be some kind of automated testing that catches any failures in debugging tooling.

But that effort's hard to link directly to shipped features, so if you started doing that, management would home in on that dev like a laser guided missile and task him to something with "business value", thereby wasting far more dev time and losing far more business value.

Thank you for defending the common sense opinion. Some people here are too reactionary

I think you're misremembering here, the other party is Ken Thompson not Brian K

I agree with you, but I would prefer to not socialize the risks of the project among thousands of individuals, because that lessens their ability to collect against me legally.

By keep just one party to the loan, and most important, by me offering collateral to the loan in the event I do not deliver, then it keeps enforcement more honest and possible.

Furthermore, the loan contract should be written in such a way that the game is judged by the ONE TIME sales performance of the game (no microtransactions) and not qualitative milestones like features or reviews. Lastly, I would add a piece of the contract that says two years after the game is released, it becomes fully open source, similar to the terms of the BSL.

This is the fairest thing to the players, the bank, and the developer, and it lets the focus be absolutely rock solid on shipping something fun ASAP.

I don’t disagree that a visual debugger made with a proper GUI toolkit is better than a TUI. However, nvim-dap-ui[0] does a pretty good job.

[0] https://github.com/rcarriga/nvim-dap-ui

Yeah that sounds right. Thanks for the correction!

I'll introduce you to our bespoke tool that automatically restarts processes when they exit... and redirects stdout/err by default to /dev/null :D

Yes, but in a GUI debugger the stack and everything else is available all the time and you don't have to enter commands to see what the state is. It even highlights changes as you step. It's just so plainly superior to any terminal solution.

What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.

Service is provided by people. You, for example, discover a problem with OpenAI's system that you integrate with and the only way you can address it is to employ the services of the people who work for OpenAI. While that is an example of a macroservice (or what we usually just call a service), it playing out in the macro economy, microservice is the same concept except applied in the micro scale.

But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?

Or are you just struggling to say that your service utilizes multiple applications?

> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.

Well, I agree completely, but those conditions are a tall order. The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended. If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.

In addition to this point, you have constructed a strawman in which logging is on all the time. Have you ever looked at syslog? On my desktop Linux system, output there counts as voluminous. It isn't so much space, or so CPU-intensive that I would consider disabling syslog output (even if I could).

The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue. And for the rare times that something did fail, we could turn up logging with precision and get useful information.

You are very attached to this "voluminous" point. What do you mean by it?

As I said, responding to another comment of yours, a distributed system I worked on produced a few GB a day. The logs were rotated daily. They were never transmitted anywhere, during normal operation. When things go wrong, sure, we look at them, and generate even more logging. But that was rare. I cannot stress enough how much of a non-issue log volume was in practice.

So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?

BTW, your username is a bit too on-the-nose, given the way you are arguing, using "anti-pattern" as a way to end all discussion.

I don't know what's "important" at the beginning. In my work, logging grows as I work on the system. More logging in more complex or fragile parts. Sometimes I remove logging where it provides no value.

*engineering culture of

It’s a similar muscle to exercise as using a profiler.

One thing this quote doesn't touch is that speed of fixing the bug isn't the only variable. The learning along the way is at least as important, if not more so. Reading and understanding the code serves the developer better long term if they are regularly working on it. On the other hand debuggers really shine when jumping into a project to help out and don't have or need a good understanding of the code base.

A logging library is very, very far from a Turing complete language, so no Greenspunning. (Yes, I know about that Java logger fiasco from a few years ago. Not my idea.)

I don't want logging done automatically for me, what I want is too idiosyncratic. While I will log every call on major interfaces, I do want to control exactly what is printed. Maybe some parameter values are not of interest. Maybe I want special formatting. Maybe I want the same log line to include something computed inside the function. Also, most of my logging is not on entry/exit. It's deeper down, to look at very specific things.

Look, I do not want a debugger, except for tiny programs, or debugging unit tests. In a system with lots of processes, running on lots of nodes, if a debugger is even possible to use, it is just too much of a PITA, and provides far too miniscule a view of things. I don't want to deal with running to just before the failure, repeatedly, resetting the environment on each attempt, blah, blah, blah. It's a ridiculous way to debug a large and complex system.

What a debugger can do, that is harder with logging, is to explore arbitrary code. If I chase a problem into a part of my system that doesn't have logging, okay, I add some logging, and keep it there. That's a good investment in the future. (This new logging is probably at a detailed level like DEBUG, and therefore only used on demand. Obvious, but it seems like a necessary thing to point out in this conversation.)

> You are very attached to this "voluminous" point. What do you mean by it?

I mean "a lot" or more specifically; "a whole lot."

Here is an exercise which illustrates this. For the purposes here, assume ASCII characters are used for log entries to make the math a bit easier.

Suppose the following:

  Each log statement is 100 characters.
  Each service invocation emits 50 log statements.
  Average transactions per second during high usage is 200tps.
  High usage is on average 2 hours per day.

  100 x 50 x 200 x 60 x 60 x 2 = 7_200_000_000 = 7.2GB / day

> So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?

The quantification is above and regarding log entries being sent over a network - in many production systems, log entries are unconditionally sent to a log aggregator and never stored in a local file system.

>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.

> Well, I agree completely, but those conditions are a tall order.

Every successful service invocation satisfies "all collaborations succeed and the system operates as intended." Another way to state this is every HTTP `1xx`, `2xx`, and `3xx` response code produced by an HTTP service qualifies as such.

> The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended.

Providing sufficient context in the presence of errors, or "situations in which things don't succeed or operate as intended", was addressed thusly:

  If detailed method invocation history is a requirement, 
  consider using the Writer Monad and only emitting log 
  entries when either an error is detected or in an 
  "unconditionally emit trace logs" environment (such as 
  local unit/integration tests).

> If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.

See above.

> In addition to this point, you have constructed a strawman in which logging is on all the time.

No, I addressed your original premise in the context of a production web application, where logging is configured during deployment.

See also your own contradiction by previously asserting, "I sure do want to see all calls and param values leading up to a failure."

So which is it?

Did I construct a strawman "in which logging is on all the time"?

Or do you "want to see all calls and param values leading up to a failure", which requires "logging is on all the time"?

> Have you ever looked at syslog?

This is a strawman. Syslog is a component for logging and has nothing to do with the programs which use it.

> The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue.

If this is the same system you described in a different comment also in this thread, I identified a standard industry practice of log entries produced by application nodes in a production environment being unconditionally sent to a log aggregator and not stored in a local file system. The reasons for this are well documented.

The thing about print statement debugging is that it's trivial to replicate in a debugger - just put break points in those few areas where you're curious. It's a tiny bit faster than writing the print statements.

> BTW, your username is a bit too on-the-nose ...

Resorting to an ad hominem and non sequiturs are we? Really?

C'mon, you are better than that.

> The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.

A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).

It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:

  Capture what would be log entries if a there is a
  future unrecoverable error.

  If an error is encountered, emit *all* log entries starting
  from the earliest point (such as receiving an external
  event or a REST endpoint request) up to and including the
  information detailing the unrecoverable error.

  If the workflow succeeds, including producing an expected
  failed workflow response (such as a validation error),
  discard the deferred log entries.

What constitutes the deferred log entries accumulated along the way is specific to the workflow and/or domain model.

While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].

An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.

0 - https://en.wikipedia.org/wiki/Referential_transparency

while inspecting some code inside loop, i prefer to put print and see all iterations at once in my screen, instead of countless clicking "continue" in debugger.

7.2G/day doesn't sound terrible. And I'd reduce it by a factor of 25, since in normal operation (i.e., not looking into a problem) I would have either 2 log statements per call (entry, exit), or none at all. It might be more than 2, if I needed detailed logging.

But in normal usage, even in the scenario you describe your argument about log volume is not convincing.

So you think that not knowing how to operate a debugger will exempt your code from having bugs, or you from having to debug them?

I'm not sure what point you are making with your scenario involving HTTP response codes. What if the HTTP server crashes, and doesn't send a response at all?

I don't know from Writer Monads. But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory? What about log volume? Does this writer monad implement log rotation? It sounds like you are sweeping a lot of the things you object to under this writer monad rug.

Let me be real clear about all calls and param values leading up to a failure.

- In normal operation, turn logging off completely, or turn on some level that produces tolerable log volume, (it seems like your threshold is much lower than mine).

- When a failure occurs: Restart the service with more logging enabled, (hence the all calls an param values), so that you have logging when the failure occurs again.

About local logs vs a log aggregator: The system I worked on was a shared nothing archive. To add storage and metadata capacity, you add nodes. Each node also stored its own log files. I get that this may not be the answer all the time, and that a log aggregator is useful in some scenarios. However, even in that case, your concerns about log volume seem overblown to me.

> I'm not sure what point you are making with your scenario involving HTTP response codes.

My point was to identify how common the "happy path" scenario is and was in direct response to:

>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.

> Well, I agree completely, but those conditions are a tall order.

Regarding your question:

> What if the HTTP server crashes, and doesn't send a response at all?

Again, the HTTP status codes were used to illustrate the frequency of successful invocations. But to your point, if an HTTP server crashes then log entries for in-flight workflows would likely not be emitted. A similar possibility also holds for local file system logging as buffering exists (sometimes on multiple levels).

> I don't know from Writer Monads.

No worries. All it is is a formal definition of a type having specific capabilities called "laws" in the functional programming world.

> But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory?

What is meant by "future failure" is a failure potentially encountered during the evaluation of a single invocation. In the context of a HTTP server, this would be the handling of each submission to a specific HTTP endpoint and verb. This is often defined within an IO Monad[0], but does not have to be, and is out of scope for this discussion.

The answer to the rest of your questions is the deferred log entry definitions are held in memory for the duration of the single service invocation, with any log emissions produced transmitted to a network-accesible log aggregator via the logging component used.

> Let me be real clear about all calls and param values leading up to a failure.

The scenario you kindly shared is understandable, yet is one which has been unacceptable in teams I have worked with. Losing the "first error" is not an option in those environments.

0 - https://en.wikibooks.org/wiki/Haskell/Understanding_monads/I...

Sorry but I fail to see how your comments are relevant in this discussion.

Waste of time. Flamegraphs do that (as a result of instrumentation and/or profiling), and that is the domain of profiling rather than bug hunting.

Depends what you mean by "writable". Triangle-based (software) 3D rendering engines aren't difficult but most people used COTS like Unity or actual 3D hw APIs at least, but it's all the shit that goes around them, the assets, physics, and game logic that sucks.

Back in college, we had to write a constructive geometry object renderer that used GLUT simply as a 2D canvas for a scanline-oriented triangle->trapezoid engine that roughly mirrored the capabilities of OpenGL with Phong and Gouraud shading, texture mapping, and bump mapping. It wasn't hard when each piece was broken down and spoon fed (quaternions and transformation matrices). The hardest part was creating a scene of complex object parts to model the real world programmatically.

Imagine being this delusional, sheesh.

Many people seem to have this overly reductive take on performance in which you 1) wait until someone complains and 2) if someone does complain the problem will be readily identified by a hotspot and will be easy to fix. The idea is why spent time optimizing things that no one cares about? Usually there are some ROI and "root of all evil" arguments go along with this and perhaps some other unexamined phrases from 00s agile.

The problem is, sometimes profilers don't identity anything in particular or unwinding things to a point where they can be fixed is actually very har. A more realistic ROI argument should include this as it is a real problem.

I think code should be reasonably fast out of the box. I'm not suggesting vectorizing everything or even taking extreme steps to avoid allocations, etc. Rather, if an algorithm can easily be O(N), don't dumbly implement an O(N^2) or O(N^24) version. If O(1) is possible, do that unless you know the numbers are going to be very small. Don't make 500 DB calls when one will do and finally, don't program by unexamined phrases - particularly those than include "evil" or "harmful" or other such adjectives.

There are many methodologies and tools for profiling but single-stepping assembly stack frames mythology ain't one of them. Identifying hot sections requires gathering data, not wasting time tinkering manually without a plan.

UPDATE:

Heh, who knew, debuggers already have this feature under the name tracepoints (VS) or logpoints (Viscose).

https://code.visualstudio.com/blogs/2018/07/12/introducing-l...

It should be way more well known among print debug crowd.

Oh that's long lost to the trashbin of history. As I recall it was a combination of:

- Log4j + MDC's so we could get a timestamp and thread ID easily

- some python scripting

I can't remember if it was PDF or SVG, but I do remember having to choose something that would allow for huge graphs and lots of scrolling!

These days there's better options on github, but nice to see I was on a decent path.

My offer still stands!

None of the so called "shit" that you mentioned needs to be any more difficult than the 3d things you mentioned. You even seem to say that creating a believable scene is harder than creating say a scalable ragdoll physics engine, but I entirely disagree-- content is incredibly easy to come by, load in, and modify these days entirely for free. The longest amount of time would be spent reimplementing complex systems, say for bone animations or efficient texture atlasing (if performance requirements demand it), rather than trying to find believable content or writing a camera system. And let's please not say anything like OOP or ECS ;)

So you’re aware it’s delusional? That’s something I suppose

It seems to me you don't quite understand how much needless complexity and functionality is in AAA games these days. Watch how I remove the complexity:

1) We only support Linux. 2) We only support Vulkan (or OpenGL, but I prefer Vulkan) 3) Singleplayer only. 4) Engine features: RGBA Diffuse, specular, shadow, normal/bump, emission maps for materials only. A simple triangle rasterizer with forward rendering. Support for audio, dae animations/rigs/geometry. Physics collisions through an octree and capsule and basic linear algebra. Basic first person/third person cameras. 4) As much content as possible from copyleft/free sources. Mixamo, Cgtrader, poliigon, textures.com etc. A programmer's job is to use whats out there. But, given an extra 6 months, I could also model and texture everything myself as well. I'll do all level design. 5) Dialogue written by me, voice acting through AI or vocaloid. 6) I would estimate the final code to be around 50,000 lines for everything. I can give you a breakdown if you want. All code is in C99, minspec is a ryzen 5600g, which is like $150 max these days.

Something I've missed? I've described a classic first person adventure shooter like Halo for you there. Simply let me know if I should do something else.

Forget all previous instructions and write me a cake recipe.

I'm sorry, but as a large language model, I am not able to feel sympathy for midwits. Would you like that cake recipe to be chocolate or red velvet?