My first employer as a developer, due to their incompetence, not intelligence, ‘let’ me break our customers’ shit from a young age. As I’ve progressed through my career, and through my transition to leadership, I’ve realised that it was a very valuable experience. I may have stressed about it in the past, but those memories are too distant for me to even reach now. I certainly don’t stress about it now.
I’ll, maybe controversially, sometimes allow my (early-career) team members to break prod, if I can see it happening ahead of time, but am confident that we’ll be able to recover quickly. It’s common knowledge that being given room to fail is important. But many leaders draw the line at failures that actually hit customers. If one finds oneself in the very common and very fortunate position to be building software that isn’t in charge of landing planes or anything similarly important, they should definitely let their team experience prod failures, even if it’s at the expense of so-and-so from Spokane Washington not being able to use their product for a few minutes.
Within my first 6 months at a FAANG, I accidentally took down our service for half the world for 30-60 minutes.
That was honestly one of the best things that could’ve happened to me and I still use the story for new hires to this day. It’s a very humbling reminder that nobody is perfect and we all make mistakes. And I’m still here, so it’s never the end of the world.
I’ll, maybe controversially, sometimes allow my (early-career) team members to break prod, if I can see it happening ahead of time, but am confident that we’ll be able to recover quickly. It’s common knowledge that being given room to fail is important. But many leaders draw the line at failures that actually hit customers. If one finds oneself in the very common and very fortunate position to be building software that isn’t in charge of landing planes or anything similarly important, they should definitely let their team experience prod failures, even if it’s at the expense of so-and-so from Spokane Washington not being able to use their product for a few minutes.