Unit tests as documentation

Getting all your teammates to quit giving all their tests names like "testTheThing" is darn near impossible. It's socially painful to be the one constantly nagging people about names, but it really does take constant nagging to keep the quality high. As soon as the nagging stops, someone invariably starts cutting corners on the test names, and after that everyone who isn't a pedantic weenie about these things will start to follow suit.

Which is honestly the sensible, well-adjusted decision. I'm the pedantic weenie on my team, and even I have to agree that I'd rather my team have a frustrating test suite than frustrating social dynamics.

Personally - and this absolutely echoes the article's last point - I've been increasingly moving toward Donald Knuth's literate style of programming. It helps me organize my thoughts even better than TDD does, and it's earned me far more compliments about the readability of my code than a squeaky-clean test suite ever does. So much so that I'm beginning to hold hope that if you can build enough team mass around working that way it might even develop into a stable equilibrium point as people start to see how it really does make the job more enjoyable.

If anything, in this scenario, I wouldn't even bother printing the test names, and would just give them generated identifier names instead. Otherwise, isn't it a bit like expecting git hashes to be meaningful when there's a commit message right there?

What do test names have to do with quality? If you want to use it as some sort of name/key, just have a comment/annotation/parameter that succinctly defines that, along with any other metadata you want to add in readable English. Many testing frameworks support this. There's exactly zero benefit toTryToFitTheTestDescriptionIntoItsName.

Readability of the code makes a lot of it's quality. A working code that is not maintainable will be refactored. A non working cofe that is maintainable will be fixed.

I don't know if any test tools that work like that though.

Also, are you a fan of nesting test classes? Any opinions? Eg:

Class fibrulatatorTest {

  Class highVoltages{

      Void tooMuchWillNoOp() {}
      Void maxVoltage() {}

I've been wishing for a long time that the industry would move towards this, but it is tough to get developers to write more than performative documentation that checks an agile sprint box, much less get product owners to allocate time test the documentation (throw someone unfamiliar with the code to do something small with it armed with only its documentation, like code another few necessary tests and document them, and correct the bumps in the consumption of the documentation). Even tougher to move towards the kind of Knuth'ian TeX'ish-quality and -sophistication documentation, which I consider necessary (though perhaps not sufficient) for taming increasing software complexity.

I hoped the kind of deep technical writing at large scales supported by Adobe Framemaker would make its way into open source alternatives like Scribus, but instead we're stuck with Markdown and Mermaid, which have their place but are painful when maintaining content over a long time, sprawling audience roles, and broad scopes. Unfortunate, since LLM's could support a quite rich technical writing and editing delivery sitting on top of a Framemaker-feature'ish document processing system oriented towards supporting literal programming.

People grab the first word they think of. And subconsciously they know if they obsess about the name it’ll have an opportunity cost - dropping one or more of the implementation details they’re juggling in their short term memory.

But if “slow” is the first word you think of that’s not very good. And if you look at the synonyms and antonyms you can solidify your understanding of the purpose of the function in your head. Maybe you meant thorough, or conservative. And maybe you meant to do one but actually did another. So now you can not just chose a name but revisit the intent.

Plus you’re not polluting the namespace by recycling a jargon word that means something else in another part of the code, complicating refactoring and self discovery later on.

Nine times out of ten this is the only test, which is mostly there to ensure the code gets exercised in a sensible way and returns a thing, and ideally to document and enforce the contract of the function.

What I absolutely agree with you on is that being able to describe this contract alongside the function itself is far more preferable. It’s not quite literate programming but tools like Python’s doctest offer a close approximation to interleaving discourse with machine readable implementation:

  def double(n: int) -> int:
    “””Increase by 100%

    >>> double(7)
    14
    “””
    return 2 * n

It claims that, in order for tests to serve as documentation, they must follow a set of best practices, one of which is descriptive test names. It says nothing about failing tests when the name of the test doesn't match the actual test case.

Note I'm not saying whether I consider this to be good advice; I'm merely clarifying what the article states.

The quality of the tests.

If we go by the article, specifically their readability and quality as documentation.

It says nothing about the quality of the resulting software (though, presumably, this will also be indirectly affected).

Identifiers matter.

https://go.dev/wiki/TableDrivenTests

So it's common to see unit tests like

  @Test
  fun `this tests something very complicated`() {
    ...
  }

It looks, feels, and reads much better.

About your other point: I have experienced exactly the same. It just seems impossible to instill the belief into most developers that readable tests lead to faster solving of bugs. And by the way, it makes tests more maintainable as well, just like readable code makes the code more maintainable anywhere else.

Just like normal code, test methods should indicate what they are doing. This will help you colleague when he's trying to fix the failing test when you're not around. There are other ways of doing that of course which can be fine as well, such as describing the test case with some kind of meta data that the test framework supports.

But the problem that OP is talking about, is that many developers simply don't see the point of putting much effort into making tests readable. They won't give tests a readable name, they won't give it a readable description in metadata either.

    describe('The foo service', () => {

      describe('When called with an array of strings', () => {

        describe('And the bar API is down', () => {

          it('pushes the values to a DLQ' () => {
            // test here
          })

          it('logs the error somewhere' () => {
            // test here
          })

          it('Returns a proper error message`, () => {
            // test here
          })
        })
      })
    })

I would be honored by anyone checking it out: https://github.com/linus/testy

It is really valuable when they are named well.

I’ve found this is where LLM can be quite useful, they’re pretty good at summarising.

Someday soon I think we’ll see a language server that checks if comments still match what they’re documenting. The same for tests being named accurately.

You can do better than "testTheThing".

Have your team (or a working group composed of your team, if your team is too big) put together a set of guidelines on naming conventions for unit test methods. Have your team agree to these conventions (assumption is that the working group would have consulted with rest of team and incorporated their feedback).

Then make that part of the code review checklist (so you aren't the one that is actually enforcing the policy). Do spot checks for the first little while, or empower some individuals to be responsible for that - if you really want to. Do a retrospective after a month or 2 months to see how everyone is doing and see how successful this initiative was.

If you're writing the test after then yeah, maybe it's hard, but that's one of the many reasons why it's probably better to write the test before and align it with the actual feature or bugfix you're intending to implement.

    describe("foo", () => {
      describe("called with true", () => {
        it("returns 1", () => {
          assert(foo(someComplicatedThing, true) === 1)
        })
      })
      describe("called with false", () => {
        it("returns 12", () => {
          assert(foo(someOtherIndecipherableThing, false) === 12)
        })
      })
    })

Tests with such names are essentially specifying the function's behaviour as "exactly what it did when first written", which is ignoring (a) that the code may have bugs and (b) that most codebases are in flux, as new features get added, things get refactored, etc. They elevate implementation details to the level of specification, which hinders progress and improvement.

Anyways, as already mentioned earlier: unit tests are code and all quality criteria that apply to any other code apply to unit tests too. We expect identifiers used in code to help understand the code no matter if it's the name of a unit test or any other entity in our program.

NB. To me this argument seems as bizarre as disputing washing your hands after using the bathroom. Why would anyone think that they should get a pass on code quality standards when writing unit tests? This just doesn't make sense...

Sometimes people are pretty bad a coming up new names, but selecting a good name given some options generally isn't a big problem. So maybe we should create a kind o LLM linter for this situation ?

The prompt could be along the lines:

"Given this function: <FUNCTION>

This unit test: <UNIT TEST>

And these naming considerations: <NAMING GUIDE>

Is the current test name a good option?

What would be some better options?"

I did some quick testing and it seems work to reasonably well. It doesn't create anything mind-blowing but at least it seems to provide some consistent options.

Naming things is one of the 2 hardest problems in computer science. The other one being cache invalidation and off by one errors.

Right now it just prints the prompt in the terminal.

[0]: https://github.com/catchorg/Catch2

Not saying it’s always the case, but it could be. Higher standards are not always better, they have diminishing returns.

Same. This is a good 80/20 in my experience.

Testing the happy paths is already very rewarding.

But indeed it tended to lead to frustrating social dynamics in stead of happy romcom scripts.

So I gave up on most of it.

That said, my opinion about test code is, it exists to find bugs or steer away from regression. API descriptions, together with a design with some graphs, should be formal and clear enough to understand code usage. I don't want to figure out the usage of fwrite() by going through its elaborate test suite.

Of course some test libraries make this approach difficult.

There could be some sort of formula to explain this better to determine how much effort to spend on tests vs features and product quality and importance of quality compared to that.

Not at all. Those kinds of names are like a de-facto standard for the people that try to push this kind of practice. Obviously the example I used is not related to any real test.

> This will help you colleague when he's trying to fix the failing test when you're not around.

Really? Encoding what a test function does in it's name is your recommendation for helping someone understand what the code is doing? There are far better ways of accomplishing this, especially when it comes to tests.

> There are other ways of doing that of course which can be fine as well

'Can be fine as well'? More like 'far superior in every possible way'.

> But the problem that OP is talking about, is that many developers simply don't see the point of putting much effort into making tests readable.

Not at all, making a test readable and trying to encode what it does into it's name are completely separate things.

Realistically, many unit tests are far more complicated (in terms of business logic) than functions where names actually matter, like 'remove()', 'sort()', 'createCustomer()', etc. I've worked in several places where people aggressively pushed the 'encode test description in test name' BS, which invariably always leads to names like 'testThatCreatingACustomerFromSanctionedCountryFailsWithErrorX'. It's completely absurd.

> Also, are you a fan of nesting test classes? Any opinions?

It really depends on the framework you're using, but in general nesting of tests is a good thing, and helps with organizing your tests.

[0] https://news.ycombinator.com/item?id=41871629#41877015

Very insightful, thanks.

Encoding such information into the name makes about as much sense as encoding constraints into SQL column names.

Your test description/documentation should be sentences, but there is absolutely zero reason to try to encode that into the name of your test function. Not to mention this article then suggests using another tool to decode this function name into a proper sentence for reporting... ok now you completely lost the ability to ctrl+f and jump to the function... terrible advice all around.

Why not just use a testing framework that actually supports free-form sentence descriptions/documentation for your tests?

Documenting your requirements by writing the tests in advance is of course painful because it forces you to think more upfront about the edge cases in advance. But that's precisely why it saves time in the long run, because then it makes it a lot more likely you can write the function correctly from the start.

The article is discussing the quality of the tests, not quality in general and not the quality of the resulting software.

That was my point.

There are three hard problems in computer science:

1) Naming things

2) Cachoncurr3)e invalidation

ency

4) Off-by-one errors

I think what you're focusing on is just syntax sugar. Those examples with the 'describe'/'it' pattern are just another way to provide names to test cases, and their goal is exactly the same. If you didn't have this syntactic support, you'd write the function names representing this.

It's exactly the same thing: documenting the test case in the code (so not a separate document), with its name.

The distinction between "comment" and "function name" becomes less relevant once one realizes a function's name is just another comment.

I've worked at companies that required this style naming for tests and it was an unholy mess, and it only works if the unit test is small enough that the name is still a reasonable length which at that point the code should be clear enough to understand what is being tested anyway.

> It doesn't bother me all that much to have test functions with names like `test_transaction_fails_if_insufficient_funds_are_available()

I mean, that's one example where you have one outcome based on one parameter/state. Expand this to a 2-field outcome based on 3 state conditions/parameters and now you have a 100-character long function name.

The point I'm making (and I think you are agreeing with me) is that trying to stuff a test description into a test function name is cumbersome and pointless. There are far better ways of adding descriptions/documentation for unit tests and pretty much every major language/testing framework supports these, nowadays.

The goal may be the same/similar, but one of the approaches is clearly superior to the other for multiple reasons (as stated by me and other many times in this comment tree). Also, I don't think you quite understand what 'syntactic sugar' means.

> If you didn't have this syntactic support, you'd write the function names representing this.

It's not any kind of 'syntactic support'. Pretty much every modern language/testing framework supports adding free-form test descriptions and names through various means.

> It's exactly the same thing: documenting the test case in the code (so not a separate document), with its name.

It's very clearly not the same at all lmao. And a test name, test description, other useful test documentation/metadata are also not the same.

> The distinction between "comment" and "function name" becomes less relevant once one realizes a function's name is just another comment.

Huge differences between a function name, a comment, and an annotation. HUGE. Read the other comments in this thread to understand why. If you actually worked in an environment where stuffing a test description into a test name is the preferred approach for a non-trivial amount of time, you'd know that once you get past a certain level of complexity your test names explode to 100+ character monsters, if only to differentiate them from the other tests, testing a different combination of states/inputs and outputs, etc.

> The article is discussing the quality of the tests, not quality in general and not the quality of the resulting software.

All of my comments in this thread are about unit tests and test quality, not general software quality.

> That was my point.

I still don't see any valid point being made.

Good luck with that!

Sorry this whole thing seems to upset you so much. Chill!

Tone is generally completely independent of good faith.

Go heal that ego and try again.

No need to get snarky about it and project your own feelings onto others though.

I was actually surprised you reacted with sarcasm.

Why are you trying to pick a fight?

The uninviting tone discourages further discussion. I really appreciated where this convo was going until..

> Read the other comments in this thread to understand why.

This could be restated in a non aggressive way. Eg: 'Other comments go into more details why'

> If you actually worked in an environment

Presumptive statements like this are unhelpful. We need to remember we do not know the context and experiences of others. Usually better to root discussion from ones own experience and not to presume the others perspective.

But I don't get the argument. Perhaps we're talking about different things (or have different mental anchors in mind)?

Giving python unittest syntax as an example, how does having a test called

  def test_myMethod_caseWhereZeroInputsAreGiven(): ...

  def test_case000135():
      """
      Tests: myMethod
      Date: 2024-10-23
      Commit base: edb78go9
      Summary: Tests case where zero inputs are given.
      Details: ...
      """

I don't think the bathroom analogy is good here either. I'm not arguing for sloppiness or bad code hygiene. A "better" analogy would be someone complaining about people not using the provided towels like people have always been doing, when the initial argument is that the towels in this particular room seem dirty and there's a perfectly fine airdryer "right there". Hence the answer "why does anyone think they get a pass washing their hands on a towel" just sounds bizzare to me, when the topic is appropriateness of sanitation method, not cleanliness itself.

(note: not being argumentative, I hope my tone does not come across as such; if you do bump onto this again and think I'm misinterpreting or missing some finer point entirely, I'd be interested to hear it)

No, I'm focusing on the most realistic and common ways this kind of pattern actually exists (based on my experience).

> If you name tests according to business domain concepts rather than tying it to individual parameters through some formulaic rubric,

While you say I'm focusing on 'the most obtuse possible way...', this kind of comment makes it seem like you haven't focused on any actual way at all. You're speaking in very ambiguous and vague terms, which actually can't be applied and enforced in practice. If you're actually trying to write a suite of unit tests around, say a function, with 3 parameters and multiple possible outcome states - you can't name your function the same for the different combinations of inputs/outputs, and you can't just handwave a 'business domain concepts' name into existence to cover each case - that just turns into an exercise of finding synonyms, abbreviations, and vague generalizations - it doesn't solve the fact that you still need all of the same test cases and they all still need to have unique function names.

You haven't actually thought through what you're proposing here.