I was once the primary party responsible for an accounting bug that would have cost Amazon around 100 million dollars in unselable inventory. I stayed up for two nights writing the most heinous regexes to scrape logs(1) and fix the months-long discrepancies. Ended up fixing the problem by writing a script that re-submitted an event for every inbounded shipment to every Amazon warehouse for the last few months. I took the next week off to recover physically and mentally.
Afterward I made a promise to myself to never get into such a panic again. But funny enough, when I feel like panic is immanent I just remind myself that whatever problem I have is absolute peanuts compared to that event. So for me at least having one truly bad event helps me put things into context.
(1) the logs had JSON-escaped JSON and I am slightly proud of those regexes because it is the only time I hopefully ever have to use 8 backslashes in a row https://xkcd.com/1638/
Afterward I made a promise to myself to never get into such a panic again. But funny enough, when I feel like panic is immanent I just remind myself that whatever problem I have is absolute peanuts compared to that event. So for me at least having one truly bad event helps me put things into context.
(1) the logs had JSON-escaped JSON and I am slightly proud of those regexes because it is the only time I hopefully ever have to use 8 backslashes in a row https://xkcd.com/1638/