←back to thread

Delete tests

(andre.arko.net)
125 points mooreds | 8 comments | | HN request time: 0s | source | bottom
Show context
recursivedoubts ◴[] No.45071410[source]
One of the most important things you can do is move your tests up the abstraction layers and away from unit tests. For lack of a better term, to move to integration tests. End-to-end tests are often too far from the system to easily understand what's wrong when they break, and can overwhelm a development org. Integration tests (or whatever you want to call them) are often the sweet spot: not tied to a particular implementation, able to survive fairly significant system changes, but also easy enough to debug when they break.

https://grugbrain.dev/#grug-on-testing

replies(11): >>45071535 #>>45071726 #>>45071751 #>>45071944 #>>45072117 #>>45072123 #>>45072158 #>>45072321 #>>45072494 #>>45074365 #>>45080184 #
RHSeeger ◴[] No.45071726[source]
Integration tests and Unit tests are different tools; and each has their place and purpose. Using one "instead" of the other is a mistake.
replies(8): >>45072079 #>>45072176 #>>45072722 #>>45072873 #>>45073135 #>>45074394 #>>45080460 #>>45093392 #
1. troupo ◴[] No.45073135[source]
In the absolute vast majority of cases unit tests are useless. Because your end result should be a working system, not isolated working units with everything useful mocked out of existence.

Integration tests will cover every use case unit tests are supposed to cover if you actually tests the system behavior.

replies(1): >>45073228 #
2. EliRivers ◴[] No.45073228[source]
At my last employer, some of our customers used our software to run systems of which the hardware value alone exceeded the entire market cap of my employer. They ran systems with a physical footprint many many times the area of my employer's leased offices and workspaces. The physical hardware devices that our software ran alone exceeded the value of my employer's market cap; just buying one of each would have been an enormous expense, ignoring that many of them were no longer available new and some of them were decades old, sitting alongside hardware that was made in the last six months. All the physical hardware working in synchronicity, linking up with similar sites in two other locations around the world, handing off to each other to follow the sun.

It would be nice to fully test system behaviour, but to do so would have bankrupted the company long before even coming close.

replies(1): >>45073237 #
3. troupo ◴[] No.45073237[source]
How were you making sure that your system actually works?

So you have unit tests testing things in isolation? Did you not test how they work together? Did you never run your system to see it actually works and behaves as expected? You just YOLO'd it over to customers and prayed?

replies(2): >>45073283 #>>45073293 #
4. EliRivers ◴[] No.45073283{3}[source]
I think the record uptime for a customer before they shut down one process to upgrade the software (leaving a dozen other such processes running, taking over the work in turn as each was upgraded) was on the order of six years. This was a set of 24 hour broadcast channels.

"How were you making sure that your system actually works?"

Good design and good software engineering goes a long way.

When you know that you cannot test by simply doing everything the customer will do, you have to think about what tests you can do that will indicate how the system will operate under a load that's orders of magnitude greater than what you can do yourself, with hardware you've never even seen. You have to think about how to write software that is likely to be high quality even if you can't test it how you'd like to.

For example, one can design the architecture in such a way that adding more load, more devices, will only linearly increase the demands on resources, and then from testing infer what the loads will be on actual customer sites. Any non-linearity in those regard were identified, if not at the design stage, in the unit-testing thereof.

One can design the code in such a way that the internal mechanisms of how devices work are suitably abstracted away, leaving as best one can manage common interfaces, and then rather than have to test with the exact arrangements of hardware customers will have, test with devices that to the extent possible, simulate the interactions our software will see. In this regard, it turns out that many devices that purport to meet standard protocols actually meet "variations about the theme" of protocols. But this too can be mitigated and handled to a degree by careful design and thought in the software engineering. The learning from doing this with one set of devices and protocols spreads to significantly different devices and protocols; every subsequent fresh design for the next iteration or generation of hardware is better and more resilient. A software engineering organisation that learns and retains knowledge and experience can go long way.

One can recognise that running live on customers sites is itself an opportunity. Some customers would never say a thing, for years on end. Some would want to be involved and would regularly talk about things they'd seen, unexpected things that happened, loads and events and so on; one can ensure that all that information is gathered by the sales people, the support reps, anyone and everyone who talks to the customers, and passed back effectively to have the results of that testing applied. For doomsday scenarios, such as crashes, resource exhaustion, pathological behaviours and so on; good logging and live measurements and dump catching etc can at least feed back so that this situation (which we would never be able to truly test ourselves) is not wasted, and gets fixed, and the lessons of it applied forwards into design and development. Harsh for the customer who finds an issue, but great for the hundred customers who will never hit it because it was tested by that unlucky customer. We'd be fools not to gather as much information as we could from poor customer experiences.

One can get hold of cheap, twenty year old devices that in theory match the same protocol, and go to town on them (some customers will actually be using that exact device and contemporaries - some customers will have brand new hardware that costs a tenth of my employer's market cap). From that, get an idea of how the software performs. Get another cheap device from ebay that is a decade old, and test it; see where it fails, but don't just fix those failures. From them, and similar repeats of the process, learn at a more fundamental level how devices differ and develop more general solutions that will either then be resilient to some new piece of hardware that hasn't even been made yet, or at least will not go wrong in such a way that the whole system is taken out and the poorly-supported brand new hardware is clearly seen by the software and reported on.

There's more. There's so much more, but once you have no choice but to come up with cheap, fast testing that nonetheless give a good indication of how the system will work when someone spends tens of millions on the hardware, software engineers can really come up with some smart, reliable ideas. It can also be really fun and satisfying to work on this.

"You just YOLO'd it over to customers and prayed?"

Absolutely not. It was all tested, repeatedly, over and over, and over the course of about fifteen year became remarkably resilient, adaptable, resource light, and so on. All the good things one would hope for. In a pinch, a small system could be run from someone's laptop; at the top end, banks and banks of servers with their fans banshee wailing 24 hours a day, with dozens of the principal processes (i.e. the main executable that runs) all running, all talking to each other across countries and time zones, handling their own redundancy against individual processes turning off. Again, when you begin knowing that the software has to deliver on such a range of systems, where one customer is two college kids in a basement and one customer is valued in the tens of billions (although doing a lot more, of course, than just what our software let them do), design and good software engineering goes a very long way.

replies(1): >>45077939 #
5. yakshaving_jgt ◴[] No.45073293{3}[source]
It’s a terrible tragedy isn’t it, that we can only choose one or the other.
replies(1): >>45078047 #
6. troupo ◴[] No.45077939{4}[source]
How do you "design a good system" without testing it? Oh wait:

> It was all tested, repeatedly, over and over, and over the course of about fifteen year

So, you do test how your system actually works, and not just isolated unit tests.

> Again, when you begin knowing that the software has to deliver on such a range of systems, where one customer is two college kids in a basement and one customer is valued in the tens of billions (although doing a lot more, of course, than just what our software let them do), design and good software engineering goes a very long way.

Indeed. And that good engineering would include a simple wisdom "unit tests are useless without integration and E2E tests, otherwise you wouldn't be able to run your software anywhere because units just wouldn't fit together".

And once you have proper integration tests, 99%+ of unit tests become redundant.

replies(1): >>45085573 #
7. troupo ◴[] No.45078047{4}[source]
You don't have to.

Unit tests work well for well-defined, contained units and library-like code.

E.g. you have code that calculates royalties based on criteria. You can and should test code like that with unit tests (better still, with property-based testing if possible)

Such code is in a tiny minority.

What you really want to do, is test that your system behaves as advertised. E.g. that if your API is called with param=1 it returns { a: "hello" }, and when with param=-1, it returns HTTP 401 or something.

The best way to do that is, of course E2E tests, but those are often quite difficult to set up (you need databases, external services, file systems etc.)

So you go for the middle ground: integration tests. Mock/stub unavailable external services. Test your full code flow. With one test you're likely to hit code paths that would require multiple unit tests to test, for a single scenario. You'll quickly find that easily 99%+ of your unit tests are absolutely redundant.

---

Offtop/rant/sidetrack.

This is especially egregious in "by the book" Java code. You'd have your controller that hits a service that collects data from facades that each hit external services via some modules. Each of those are tested in unit tests mocking the living daylight out of everything.

So for param=1 you'd have a unit test for controller (service mocked), service (facades mocked), each of the facades (if there are more than one, external services modules mocked), each of the external service modules (actual external services mocked).

Replace that with a single integration test where just the external service is mocked, and boom, you've covered all of those tests, and can trivially expand it to test external service being unavailable, timing out, returning invalid data etc.

8. EliRivers ◴[] No.45085573{5}[source]
We don't have those integration tests. They happened by chance when customers did things. Do we really have tests if we cannot define those tests, if we cannot run those tests? If we make a significant change that could feasibly affect the outcome of those tests, and we are unable to run those tests, do we have those tests?

"How do you "design a good system" without testing it?"

We must be coming at this through such widly different contexts. To me, it is simply obvious and normal that it's possible to create a good design for something, and that good design can exist before any tests have ever been created or executed.

To me, that you asks that question suggests that we have such different contexts that we might as well be speaking different languages. I would be horrified that people would churn out a rubbish design and just let tests handle all the crap and force it into a good design; but I do gather that's normal procedure in some industries.