Splitting engineering teams into defense and offense

I once worked for a company that required from each engineer in the team to do what they called “firefighting” during working hours (so not exactly on-call). So for one week, I was triaging bug tickets and trying to resolve them. These bugs belonged to the area my team was part of, so it affected the same product but a vast amount of micro services, most of which I didn’t know much about (besides how to use their APIs). It didn’t make much sense to me. So you have Joe punching code like there’s no tomorrow and introducing bugs because features must go live asap. And then it’s me the one fixing stuff. So unproductive. I always advocated for a slower pace of feature delivery (so more testing and less bugs on production) but everyone was like “are you from the 80s or something? We gotta move fast man!”

This is interesting because it’s what I imagine would happen if we scaled this system to a larger team - offense engineers would get sloppy, defensive engineers would get overwhelmed, even with the rotation cycles.

Small, in-person, high-trust teams have the advantage of not falling into bad offense habits.

Additionally, a slower shipping pace simply isn’t an option, seeing as the only advantage we have over our giant competitors is speed.

I think we’ve worked for the same org

> offense engineers would get sloppy

Wouldn't they be incentivized to maintain discipline because they will be the defensive engineers next week when their own code breaks?

I suspect as the company gets larger time between defensive sprints will get longer, but yes, for smaller teams this is what keeps quality high, you’ll have to clean up your own mess next week.

This sort of thing is introduced when the number of bugs in production, especially bugs that aren't user-facing or a danger to data (eg 'just' an unhandled exception or a weird log entry), gets to a peak and someone decides it's important enough to actually do something about it. Those things are always such a low priority that they're rarely dealt with any other way.

In my experience whenever that happens someone always finds an "oh @#$&" case where a bug is actually far more serious than everyone thought.

It is an approach that's less productive than slowing down and delivering quality, but it's also completely inevitable once a team/company grows to a sufficient size.

I honestly don't really like the "let's slow down" approach. It's hard for me to buy into the idea that simply slowing down will increase product quality. But I think your comment already contains the key to better quality: close the feedback loop so that engineers are responsible for their own bugs. If I have the option of throwing crap over the wall, I will gravitate towards it. If I have to face all of the consequences of my code, I might behave otherwise.

Slow is smooth, smooth is fast?

Slowing down doesn't mean going slow. There's more to software development than vomiting out lines of code as quickly as possible.

>Slowing down doesn't mean going slow. There's more to software development than vomiting out lines of code as quickly as possible.

Tell that to seemingly every engineering manager and product manager coming online over the last 8-10 years.

I first noticed in 2016 there seemed to be a direct correlation between more private equity and MBA's getting into the field and the decline of software quality.

So now you have a generation of managers (and really executives) who know little of the true tradeoffs between quality and quantity because they only ever saw success pushing code as fast as possible regardless of its quality and dealing with the aftermath. This lead them to promotions, new jobs etc.

We did this to ourselves really, by not becoming managers and executives ourselves as engineers.

Greetings, Phil Dunphy