Most active commenters
  • coryrc(7)

←back to thread

172 points yatrios | 12 comments | | HN request time: 0.641s | source | bottom
Show context
0xbadcafebee ◴[] No.42184298[source]
For those not aware, Shift Left[1] is (at this point) an old term that was coined for a specific use case, but now refers to a general concept. The concept is that, if you do needed things earlier in a product cycle, it will end up reducing your expense and time in the long run, even if it seems like it's taking longer for you to "get somewhere" earlier on. I think this[2] article is a good no-nonsense explainer for "Why Shift Left?".

[1] https://en.wikipedia.org/wiki/Shift-left_testing [2] https://www.dynatrace.com/news/blog/what-is-shift-left-and-w...

replies(12): >>42185611 #>>42186878 #>>42187331 #>>42187375 #>>42187393 #>>42187419 #>>42187454 #>>42187463 #>>42187501 #>>42188834 #>>42192801 #>>42194403 #
1. coryrc ◴[] No.42186878[source]
No evidence most of the activities actually save money with modern ways of delivering software (or even ancient ways of delivering software; I looked back and the IBM study showing increasing costs for finding bugs later in the pipeline was actually made up data!)
replies(4): >>42186930 #>>42187047 #>>42187893 #>>42187928 #
2. coryrc ◴[] No.42186930[source]
To be more specific, let's say I can write an e2e test on an actual pre-prod environment, or I can invest much development and ongoing maintenance to develop stub responses so that the test can run before submit in a partial system. How much is "shifting left" worth versus investing in speeding up the deployment pipeline and fast flag rollout and monitoring?

Nobody I've worked with can ever quantify the ROI for elaborate take test environments, but somebody made an okr so there you go. Far be it we follow actual research done on modern software... http://dora.dev

replies(3): >>42187551 #>>42187771 #>>42188365 #
3. coryrc ◴[] No.42187047[source]
Last rant: everybody is testing in production, but only some people are looking at the results. If you aren't then there's better ROI to be found than "shifting left" your e2e tests.
4. cogman10 ◴[] No.42187551[source]
In fact, in my experience, these elaborate test environments and procedures cripple products.

I'm firmly of the opinion that if a test can't be run completely locally then it shouldn't be run. These test environments can be super fragile. They often rely on a symphony of teams ensuring everything is in a good state all the time. But, what happens more often than not, is one team somewhere deploys a broken version of their software to the test environment (because, of course they do) in order to run their fleet of e2e tests. That invariably ends up blowing up the rest of the org depending on that broken software and heaven help you if the person that deployed did it at 5pm and is gone on vacation.

This rippling failure mode happens because it's easier to write e2e tests which depend on a functional environment than it is to write and maintain mock services and mock data. Yet the mock services and data are precisely what you need to ensure someone doesn't screw up the test environment in the first place.

replies(2): >>42187642 #>>42187699 #
5. coryrc ◴[] No.42187642{3}[source]
There are many reasons you want to be able to turn up your whole stack quickly; disaster recovery is just one of them. And if you can turn up your environment quickly then why not have multiple staging environments? You start with the most recent of yours and everyone else's prod version, then carrots combinations from there

Obviously this is for large-scale systems and not small teams.

6. jeltz ◴[] No.42187699{3}[source]
You are not wrong but I have had many experiences where mock services resulted in totally broken systems since they were incorrectly mocked. In complex systems it is very hard to accurately mock interactions.

Personally I think the real issue is not the testing strategy but the system itself. Many organizations make systems overly complex. A well structured monolith with a few supporting services is usually easy to test while micro service/SOA hell is not.

7. wrs ◴[] No.42187771[source]
No argument, but there can be limitations on how much you can speed up the deployment pipeline. In particular, the article is about integrated circuit development (actually about systems made of many ICs), where a “production deployment” takes months and many, many millions of dollars, and there’s not much you can do about it.

I heard a story decades ago about a software team that got a new member transferred in from the IC design department. The new engineer checked in essentially zero bugs. The manager asked what the secret was, and the new engineer said “wait, we’re allowed to have bugs?”

replies(1): >>42189565 #
8. 1propionyl ◴[] No.42187893[source]
Are you referring to the IBM Systems Science claims (likely apocryphal) in the Pressman paper, or Barry Boehm's figure in "Software Engineering" 1976 paper which did include some IBM sourcing (literally drawn on) but was primarily based on survey data from within TRW?

It baffles me that anyone would continue to promulgate the Pressman numbers (which claim ~exponential growth in cost) based on... it's not entirely clear what data, as opposed to Boehm's paper which only claims a linear relative cost increase, but is far more credible.

replies(1): >>42189508 #
9. crabbone ◴[] No.42188365[source]
Shift-left comes from the world before everyone was selling services. In that world shipping service packs was the way to fix problems discovered after a release, and releases were years apart.

From a QA perspective, I greatly regret that the world of infrequent releases is mostly gone. There are few kinds of products that still hold onto the old strategy, but this is a dying art.

I see the world of services with DevOps, push on green etc. strategies as a kind of fast-food of software development. A kind of way of doing things that allows one to borrow from the future self by promising to improve the quality eventually, but charging for that future improved quality today.

There are products where speeding the rollout is a bad idea. Anything that requires high reliability is in that category. And the reason is that highly reliable systems need to collect mileage before being released. For example, in storage products, it's typical to have systems run for few months before they are "cleared for release". Of course, it's possible to continue development during this time, but it's a dangerous time for development because at any moment the system can be sent back to developers, and they would have to incorporate their more recent changes into the patched system when they restart the development process. I.e. a lot of development effort can be potentially wasted before the system is sent out to QA and the actual release. And, to amortize this waste, it's better to release less frequently. It's also better to approach the time when the system is handed to QA with a system already well-tested, as this will minimize the back-and-forth between the QA and the development -- and that's the problem shift-left was intended to solve.

NB. Here's another, perhaps novel thought for the "push on green" people. Once it was considered a bad idea for the QA to be aware of implementation details. Testing was seen as an experiment, where the QA were the test subjects. This also meant that exposing the QA to the internal details of the systems or the rationale that went into building it would "spoil" the results. In such a world, allowing the QA to see a half-baked system would be equivalent to exposing them to the details of the system's implementation, thus undermining their testing effort. The QA were supposed to receive the technical documentation for the system and work from it, trying to use the system as documented.

replies(1): >>42189545 #
10. coryrc ◴[] No.42189508[source]
Pressman i.e. https://www.theregister.com/2021/07/22/bugs_expense_bs/

In a waterfall, single-deliverable model it wouldn't surprise me there is some increase in costs the later a bug is discovered, but if you're in that world you have more obvious problems to tackle.

People still use the Pressman numbers. So much for "data-driven decision making"...

11. coryrc ◴[] No.42189545{3}[source]
Companies won't pay enough for quality QA people, so you can't get good people to do it instead of a more lucrative dev position. So now everybody has to do testing, except they only want to do the bare minimum and they aren't as practiced at it so they aren't as good as someone with equivalent talent but more experience writing comprehensive tests.

Start paying "QA" more than their dev partners consistently and with better promotion opportunities and you can get better testing, but everybody seems to be making plenty of money without it.

12. coryrc ◴[] No.42189565{3}[source]
ICs are about the only exception, but semiconductor engineers are no dummies and have been building the appropriate simulation tools for a long time now.

If you could spin a chip every day that would mostly be a huge waste of time.