Good Software Development Habits

I believe there is nuance to this: how else would any sane person exercise error flows in software, or -- as I have personally implemented -- test against things which are wallet-expensive in real life?

What I oppose is mocking every single dependency of every single injection in the component. It ends up being 50x the code of the system under test and requires throwing it all away when the implementation changes

Unfortunately, most "frameworks" in existence today do not follow a simple, functional design, and they tend to make you mock quite a bit.

But the alternative to "mocking" is to use verified fakes (same test passes for both the real implementation and the fake) that actually do something "real" (even if it's simply persisting data in memory).

My complaint about using "real implementations" (aside from databases, which, sure, knock yourself out with testcontainers or even hsqldb running in compatibility mode[1]) is that managing the state of real systems is incredibly hard. I am aware of aws-nuke and its kin, but tearing everything down and then setting everything up for every test cycle consumes very real wall clock time and the flakes drive up "test fatigue" where folks start merging things with test failures because "oh, you know, it's just kidding" or the deadly enemy "we don't have time to wait for the test cycle, we need the fix out now!"

I am 100% with you on the verified fakes and love moto (and its friend localstack) for that reason. If I had lottery money, I'd even go so far as to create a moto-eqsue implementation backed by lxc or such and have it actually provision/mutate some running infra that I can snapshot and restore

1: https://www.hsqldb.org/doc/2.0/guide/compatibility-chapt.htm...

> how else would any sane person exercise error flows in software

Interesting question. Have you got any specific examples of something hard to test without mocks?

I agree there's nuance, but I find "don't use mocks" a great starting point, and the sweet spot for web services to normally be only mocking/faking/stubbing/simulating/doubling 3rd-party APIs. I'm sure the spot moves dependent on context, e.g. writing hardware firmware might warrant a different approach.

Maybe a clearer expression would be "consider mocks a code smell".

I have two examples at hand: chasing memory leaks when enumerating over 10,000 EBS volumes, and ensuring the 500-class response handlers behave correctly for S3 (which is exceedingly hard to reproduce using the real S3 api)

Another common one is introducing network stalls to ensure timeout code behaves sanely. I'm aware of Comcast and the various nf trickery but I mean something a normal developer could run as part of normal tests, not involving sudo anything

Even as I write this, I'm aware that "there's more than one way to do it" and I'm sure everyone has their own favorite. But my experience has been that only the most pristine decomposed software components have very clean boundaries for testing just this one aspect. So for the rest of us stuck using the AWS sdk and similar, one can choose to shim the interactions with the SDK just to be able to swap it out for testing (which I violently oppose), or feed the software you do control a pseudo-implementation that will explode in very specific ways

It's especially hard for embedded software. You certainly do want hardware-in-the-loop tests, but you also want tests that are independent of the hardware. You have to simulate the hardware interaction, and you definitely want to verify what the code tried to do to the hardware, and when. So for the hardware-interacting layer you want mocks, not just fakes.

Just imagine a world where a component manufacturer (be it hardware or software) also provides a verified fake/simulated implementation.

Even hardware, they likely did develop it using software simulations: they just need to ship it with their SDK. Another thing hardware has it going for it is that it does not change as much.

Note that a verified fake could still have observability points that allow you to monitor what's going on.

> ensuring the 500-class response handlers behave correctly for S3 (which is exceedingly hard to reproduce using the real S3 api)

What did you use for this? I've achieved this previously by abusing minio, combined with very large uploads & downloads. Maybe that qualifies as some kind of verified mock though(?)

I'd be interested to use a cleaner approach which is also realistic.

Thankfully most of the AWS SDK uses interfaces[1] so one can use Mockito if you already have the muscle memory with it, or its InvocationHandler friend[2] if truly customized responses are needed

If one needs to exercise the AWS SDK itself, as part of some repo steps for a support issue, it's similarly glucose-cheap to patch moto to 500 in the necessary circumstances. I've had good luck using their ExecutionInterceptor ServiceLoader mechanism[3] to patch the Client's endpoint URI to point to moto or localstack without having to monkey with every single Client instantiation, which can be especially no-fun for STS AssumeRole or AssumeRoleWithWebIdentity setups (since one doesn't want it to use real STS for anything). That way the actual SDK pathway is still exercised all the way into the caller's code for a more honest-to-goodness bad outcome but without the hope-and-pray of contacting real S3

1: e.g. https://sdk.amazonaws.com/java/api/2.29.16/software/amazon/a...

2: https://docs.oracle.com/en/java/javase/11/docs/api/java.base...

3: https://github.com/aws/aws-sdk-java-v2/blob/2.29.17/core/sdk...