An overly aggressive mock can work fine, but break much later

The mock discussion still misses the real solution, which is to refactor the code so that you have a function that simple reads the file and returns json that is essentially a wrapper around open and doesn't need to be tested.

Then have your main function take in that json as a parameter (or class wrapping that json).

Then your code becomes the ideal code. Stateless and with no interaction with the outside world. Then it's trivial to test just like and other function that is simple inputs translated outputs (ie pure).

Every time you see the need for a mock, you're first thought should be "how can I take the 90% or 95% of this function that is pure and pull it out, and separate the impure portion (side effects and/or stateful) that now has almost no logic or complexity left in it and push it to the boundary of my codebase?"

Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs.

I agree with you, however convincing an entire team of devs to explicitly separate the interface of impure parts of code is very difficult.

If you introduce a mocking library to the test portion of the codebase, most developers will start to use it as a way to shortcut any refactoring they don't want to do. I think articles like this that try to explain how to better use mocks in tests are useful, although I wish they weren't necessary.

> Then the complex pure part you test the heck out of, and the stateful/side effectful impure part becomes barely a wrapper over system APIs.

In practice the issues I see with this are that the "side effect" part is usually either: extensive enough to still justify mocking around testing it, and also intertwined enough with your logic to be hard to remove all the "pure" logic. I rarely see 90-95% of functions being pure logic vs side effects.

E.g. for the first, you could have an action that requires several sequenced side effects and then your "wrapper over APIs" still needs validation of calling the right APIs in the right oder with the right params, for various scenarios. Enter mocks or fakes. (And sometimes people will get clever and say use pubsub or events for this, but... you're usually just making the full-system-level testing there harder, as well as introducing less determinism around your consistency.)

For the second, something like "do steps I and J. If the API you call in step J fails, unwind the change in I." Now you've got some logic back in there. And it's not uncommon for the branching to get more complex. Were you building everything in the system from first principles, you could try to architect something where I and J can be combined or consolidated in a way to work around this; when I and J are third party dependencies, that gets harder.

"sans-I/O" is one term for that style. I like it a lot but it's not a free lunch.

I always liked the phrase 'Hoist your I/O' [1] but yes, you can only hoist it up so many times until its outside of your application completely (making it completely pure, and now someone else's responsibility).

[1] https://www.youtube.com/watch?v=PBQN62oUnN8

Funnily enough, I am preparing a simple presentation at work to speak about exactly that. The idea of separating "logic" from I/O and side effects is an old one and can be found in many architectures (like hexagonal architecture). There is plenty of benefit doing this, but testing is a big one.

It should be obvious, but this is not something that seem to be thought in school or in most workplaces, and when it is, it's often through the lens of functional programming, which most just treat as a curiosity and not a practical thing to use at work. So I started to teach this simple design principle to all my junior dev because this is something that is actually quite easy to implement, does not need a complete shift of architecture/big refactor when working on existing code, and is actually practical and useful.

Or "functional core, imperative shell".

The risk to that approach is that you end up writing code that cannot deal with the real world problems of I/O, such as timeouts, failed reads, jitter, and other weird behaviour.

Separating I/O from logic makes a lot of sense and makes tests much easier to write and code much easier to reason about, but you'll still need to implement some sort of mocking interface if you want to catch I/O problems.

> how can I take the 90% or 95% of this function that is pure and pull it out, and separate the impure portion (side effects and/or stateful) that now has almost no logic or complexity left in it

They addressed this concern already. These are not contradicting approaches.

This sounds great until one of your other functions calls that function.

You're just describing dependency injection, but if you say that, people won't want to listen cause doing that all the time sucks.

I think the disconnect is that the impure portion often has substantial amounts of inherent complexity. If you're working at a high level that's handled by a lower level (OS, framework, etc.) but somebody is writing that framework or OS or HAL. And they need to test too, and ultimately someone is going to have to make a test double of the hardware interface unless all testing is going to require hardware-in-the-loop.

You're right that it's not always easy to separate pure from effectful code. But you definitely can (model it as a state machine) and I think it's worth it, especially if those steps can take a long time, can fail, might be executed in parallel, etc.

For instance, I once worked on payment-related code at a large online retailer. The steps I and J from your example would have been calls to the payment gateway's API (payment initiation, actual payment request). There was also a step K (polling for payment confirmation) and even a step K' (a payment confirmation callback the gateway might or might not call before or after we get around polling for the payment status ourselves). And often there was even user interaction in between (the 3DS/3DS2 credit card payment scheme that's common here in the EU). Every single one of those steps could fail for a myriad of reasons, e.g. time out, be rejected, … and we had to make sure we always failed gracefully and, most importantly, didn't mess up our payment or order records.

Of course this was an old enterprise Java code base, created by people who had long left the company, and all this had been written just the way you imagine it. It was an absolute mess.

Every single time I worked on this code base I secretly wished one of the original authors had heard of state machines, pure vs. effectful code, and unit tests.

now you have N+1 tests!

I've had great success swapping in a in-memory database via dependency injection and just running 100% of the application, end to end.

in the ideal case my tests start by writing some randomised data using the external API, I then update it(if applicable) using the external API, and finally read it, also using the external API, and compare the actual result with what I expected.

I use randomised data to avoid collisions with other tests, which might cause flakiness and/or prevent running the tests concurrently. I avoid having seed data in the database if at all possible.

It's the only approach I've found that can survive a major refactor of the codebase. Anything short of breaking the external API, which is typically a no-no anyway, shouldn't break these tests.

Doing a refactor and being able to rely on the test suite for finding bugs and inconsistencies is amazing. Of course they won't find 100% of all bugs,but this way at least you know that a failing test means there's a problem in your production code.

Dependency injection frameworks can be a pain, but basic dependency injection (e.g. pass a DB handle to anything that needs the DB) is a must. What's the alternative? Having everyone independently create their own DB connections?

I think the important thing is that the code look pure from a testing perspective.

Say you've got a function that accesses a key-value store. Ideally, you can factor out the i/o so that you do all your reads up front and all your writes at the end, leaving a pure function in the middle. But if the code is too tangled up in its side effects for that, the next best thing is to create a fake KV store and then wrap the function like this:

  def doTest(input, initialState):
    kv = makeFake(initialState)
    result = doThing(kv, input)
    return result, kv.dumpContents()

doThing isn't a pure function, but doTest is. Now you can write tests like this:

  result, outputState = doTest(input, initialState)
  assert (result, outputState) == expected

I guess you could call that "imperative core, functional shell," lol.

Your just moving the problem one layer up.

You still need to write a test for how it all comes together and you should write tests for your error handling. You need a mock to respond with an error.

Thou shalt never mock the database, for thou shalt anger the database when thou moketh it.

In all seriousness, I have found this to be a useful suggestion, because the purpose of a test is to make sure invariants don't break in real code. When you mock the database, you're excluding large amounts of real code from test.

Not necessarily. If your error handling looks like this:

  value, err := externalLibraryFunctionFoo(a, b)
  if err != nil {
    return nil, fmt.Errorf("calling foo: %w", err)
  }

then you probably don't need to test it. All you're doing is bubbling up the error handling from other libraries' functions.

It's my preference to a point. I think you can over-do it. I've worked in systems that didn't use it and worked just fine. I would bet most folks err on the side of too little, unless they have some kind of framework doing the heavy lifting.

Tests help make sure that the intent is to just bubble up the error.

What if someone in the future is comes in and modifies that to add some complexity somehow or changes it to log and continue. Tests will catch that behavior

It's rarely possible to future-proof a function by testing its entire range of possible inputs, so I'm not particularly concerned about trying to anticipate trivial cases that might become non-trivial in future—that just feels like a special case of future-proofing.

I think it's more important to ensure that, after each new commit, every non-trivial code path in that function is exercised. If someone adds new logic, they're responsible for adding new tests to cover that logic; similarly, if they turn a trivial code path into a non-trivial one, they're responsible for adding tests to cover that too.