Crowdstrike Incident | IM 70.3 YYC Tomorrow

Really Erwin?? Using the biggest IT incident in history to talk about Teri? You must be out of ideas about what to write about...

That's not true. There's always so much to write about, but it's been super busy again because of my upcoming race (tomorrow). Thankfully I've run out of things to do for the race so I have some downtime to catch up on blogging until dinnertime.

The Crowdstrike Incident @ Orora

For IT professionals like me, the 2024 Crowdstrike Incident will be a day in our professional careers that none of us will ever forget. The impact this incident has on the world is unbelievable. Many of my friends that didn't even work that day were impacted. One friend couldn't get his usual starbucks (coffee is important to people). Another friend couldn't fly home in the morning and was stuck at the airport all day, but thankfully was able to get on a Southwest flight at night. Sometimes not upgrading your systems like Southwest can be a blessing in disguise.

Major organizations are still recovering from the event. 8.5 million endpoints. Congress is involved. The economic impact is over 5 billion. That's a lot of money.

Thankfully I had the easy job on Friday. I wish I could have done more. My day started with a 6AM conference call with the IT leaders and the executives to level set on what was going on:

While on the call, I caught up with all the messages that were going back and forth with the IT team since ~11PM the night before. Since I don't sleep by my phone, I didn't realize about the severity of the incident until the call.

I was actually on my way to my 6AM gym class. It was clear that today was one of those days that the gym/cold plunge/morning routine will have to wait until tomorrow. All good.

After the call, the IT team had to get to work. My team (IT Operations) was responsible for remediating the ~2000 endpoints in the field that had the BSOD, and the IT Engineering team was responsible for remediating all 117 servers.

Usually we all WFH on Friday. Instead, most of the team went to the Buena Park office and hung out in the conference room. It was fun and the best part was when my colleague ordered an early lunch for us, just as I was getting hungry.

Here's some exciting photos of us in the late morning, and then towards the end of the day:

Overall, the team handled the incident very well. It was quite the team building event for us. The majority of our servers/applications back to being fully functional by around noon. We also had to write very clear procedures to deploy to the field to remediate all the endpoints. After that was completed, the ~48 Orora sites and remote users were able to self-remediate and get back to work. Great stuff.

One outcome I was surprised about: Every two hours, we would do check in-calls with the executives. Since everything got fixed rather quickly (relative to other companies), we didn't even need a 5PM check in call. I was even able to go home and have dinner. I assumed at the beginning of the day that we would be there all evening and possibly over the weekend. It's all thanks to a great IT team that all stepped up to do what needed to get done.

Another observation: According to my watch, my average stress level on July 19 was 34 out of 100, which is surprisingly low:

My stress level since Jun 30 (according to Garmin)

I'd attribute the low stress to a few factors:

• Orora has a great IT team that got ahead of the problem the night before

• I didn't take my PM ADHD medication (just forgot about it)
  

• I've been through several of these major incidents

• I really get it now that very few things in life are worth stressing about

2018 Camping Trip & Outlet Shopping Cut Short :(

That day reminded me quite a bit about Teri and our lack of times together because work got in the way.

More specifically, the incident reminded me of the time I went camping with Teri and some friends from FCBC Walnut in July 2018 in the Santa Barbara area:

It was a nice time of fellowship and camping together. Super chill, good food, just like any camping trip. I didn't have to plan the event which was also a huge bonus. Teri and I were still also in the process of figuring out what church we should attend after we were married. Multi-day events like these help build relationships. We were glad we went. I hope to more camping trips, in the future with whomever is interested, especially to Muir Rock.

On our way home, we stopped by the Camarillo Outlet Mall to do some much needed outlet shopping. I love getting good deals. Right when we got there, I got a phone call from my boss (working at Safran Passenger Innovations at the time) of a major incident could have affected all of our in-flight entertainment equipment in the field. Not good.

I won't forget my conversation with Teri at the outlets after that phone call. It went something like this:

Erwin (with tons of anxiety): Teri, I have to leave, now. There's an incident at work and I need to go in. Even my boss had to cut his vacation short and is heading into the office. Teri: Seriously?? We just got here. Can't it just wait until tonight when we get home? Erwin: No. This is a major outage and most likely I'll need to work all evening. Teri (with tons of anger): This always happens with you. Last time it was a power outage. It never ends. Erwin: I'm so sorry about this. Since you want to shop, I will just take the train home. Please drop me off and then you can take all the time you want. Teri: Ok. This is not fair. I'm not happy.

So a few hours later I got to the office and got to work to figure out with the team about what's going on. It was a long day. We had to hire an external party to help us with the incident, and the whole ordeal lasted several days. Way more time consuming than the Crowdstrike incident.

That Sunday evening, Teri arrived at the office and was quite supportive of me working till pretty late. Here's a few photos of us hanging at night at the office:

Not sure why my office is so messy

Looking back, I don't really have any regrets regarding how I handled the incident and having to leave Teri. I do wish I was able to communicate with her with a lot less anxiety. It reminds me of this quote on Teri's Facebook profile that I've shared a few times here:

"Anxiety is Love's greatest killer. It creates the failures. It makes others feel as you might when a drowning man holds on to you. You want to save him, but you know he will strangle you with his panic." - Anais Nin

I think about anxiety a lot these days because I wonder why many people around me have so much anxiety. I wish I could just tell everyone to be "calm, quiet, and relax" like Teri told me too many times. Unfortunately, that just makes people feel even more anxious. I feel like after you lose a spouse and feel like you have experienced one of the worst things possible in life, you'll just look at life in a different light and most things that would give me a ton of anxiety aren't as bad as they used to.

IM 70.3 YYC - Race is tomorrow :)

Here we go again...another race for me. Lots of last minute training and dieting and trying new things I should not try before a race like nutrition. The usual fun stuff. What could possibly go wrong? I'm looking forward to tomorrow despite the hills and realizing that I signed up for a different course a few weeks ago than I did in 2017. Lesson of the day: Always do your research before you sign up for a race.

On the topic of anxiety, there's definitely nothing for me to be anxious about for tomorrow's race. I've done too many of these and overall I'm in a good place with training preparation, besides the swimming and running portions. Here's how the past month has went...

Jun 17 to Jul 21: 8800 swimming yards & 11.7 run miles. I AM READY.

Earlier this week, I had the opportunity to host Salina for a few days at my house since she needed to visit San Diego for a work conference. Although I ate way too much (not her fault), It was nice and relaxing. We kept getting really good parking spots at every place we went to. Edwin and Alex also hosted dinner on Wednesday night, that was a lot of fun:

Here is a short interview from a few days ago I voluntold Salina to do my backyard regarding the race and doing cold plunges:

Now we are in Calgary. We flew together on Westjet (not AA) on Thursday evening. I brought my bike. It has been non stop eating since I arrived:

We can do hard things - Team Terwin

What's your predictions for the race?

Tomorrow is going to be an exciting day!

I shared in the video that my cousin Lemuel assumed my goal was under 6 hours. That seemed slow a few days ago, but after doing more research on the course, I'd be fine with that. Well not really. My "A" goal is actually 5:50:

The Best Bike Split race predictor has me at 2:48, so I'll target 2:50:

2h 48min would be nice to be done with a hilly bike course

I may regret this later... but tomorrow seems like a great day to throw out the race plan above and just go a lot harder than I should on the bike and first lap of the run. I won't do that for my "A race" IM California later this year. I What about 5:40? Or 5:30?? The 5:50 goal just seem slow to me vs. my glory days in 2014-2016 before I met Teri. You never know right?? I'm also one of the few people with a disc wheel...

Why are there very few disc wheels at IM 70.3 YYC??

I don't get it. Usually I'll see a ton of them at every race. Maybe all the fast people aren't here yet? Here are two good reasons to consider:

• These people aren't super serious with triathlon

• Disc wheels don't help as much for this course (since it's so hilly)

Hopefully not the latter. Anyways, back to my real goals for tomorrow, lets get this race done as soon as possible:

• Swim

• Overbike

• Pray

Worst case, the cousins have to wait a bit longer for me to finish if I end up walking a lot on the run. All good.

Anyways, that's enough for today, family time starts now. Thanks for reading! Feel free to track me using the Ironman apps. I am bib 357. Have a great rest of the weekend!

Erwin