> The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).
It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:
Capture what would be log entries if a there is a
future unrecoverable error.
If an error is encountered, emit *all* log entries starting
from the earliest point (such as receiving an external
event or a REST endpoint request) up to and including the
information detailing the unrecoverable error.
If the workflow succeeds, including producing an expected
failed workflow response (such as a validation error),
discard the deferred log entries.
What constitutes the deferred log entries accumulated along the way is specific to the workflow and/or domain model.
While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].
An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.
0 - https://en.wikipedia.org/wiki/Referential_transparency