Westjet global IT outage causing delays

[Donald]

Alright, who hacked them this time?

https://www.cbc.ca/news/canada/british- ... -1.6642315

[UnionDrive]

Perhaps a software update/issue? Just like the rogers outage earlier this year.

[averageatbest]

Alright, who hacked them this time?

https://www.cbc.ca/news/canada/british- ... -1.6642315

From what I've heard, it's nothing nefarious. It was a failure that was bad luck.

I am sure that the IT team will be looking into what went wrong and ways to mitigate the risk of future occurrences.

[Dronepiper]

You gotta wonder how much a glitch like this actually costs a large operation like westjet. My guess is 5 million at a minimum in Overtime, Passenger compensation, hotels for passengers, etc

[Donald]

The actual $$ cost plus bad press is immeasurable.

Do you think they'd admit it if they were hacked though?

[co-joe]

The actual $$ cost plus bad press is immeasurable.

Do you think they'd admit it if they were hacked though?

No they definitely wouldn't. A smaller 705 that I know (not mine) of got hacked, they got a ransom demand they chose to ignore and then had to make paper flight plans and passenger manifests for a week while they had to completely rebuild their reservations system. I think our industry is especially vulnerable to attacks like this with staff constantly coming and going, it doesn't take much for someone to gain access to systems if they get a staff password. The company I mentioned, I knew a pilot from there who still had access to schedules, passenger loads, contact info for everyone, 5 years after quitting. He repeatedly emailed their IT department to let them know but they didn't ever reply.

[Bede]

Nothing to do with getting hacked.

On Saturday November 5, WestJet experienced a system-wide outage that impacted operations across our network due to a cooling issue in our primary data centre.

https://www.westjet.com/en-ca/news/2022 ... perating-o

Someone with way more IT knowledge than me speculated that it was the chillers that cool the CPU's in the servers (modern ones are liquid cooled). The chiller(s) failed and everything overheated and shut down.

[daedalusx]

Nothing to do with getting hacked.

On Saturday November 5, WestJet experienced a system-wide outage that impacted operations across our network due to a cooling issue in our primary data centre.

https://www.westjet.com/en-ca/news/2022 ... perating-o

Someone with way more IT knowledge than me speculated that it was the chillers that cool the CPU's in the servers (modern ones are liquid cooled). The chiller(s) failed and everything overheated and shut down.

The Boomer as spoken. He says has it was nothing to do with being hacked, Westjet themselves said so! So it must true.

Imagine thinking that a 5 billion dollar company *entire* flight OPS is at the mercy of a single point environmental failure in a datacenter.

Oh yeah, the cooling failed in our datacenter running the entire flight ops and reservation and we had to shut it down. That makes perfect sense.
As if there was no multigeo cloud load balancing of critical services ... yup everything was running fine on a single bare metal server tucked in the corner until Raphael spilled his coffee on the cooling system causing a full 12+ hour outage.

[rooster]

Nothing to do with getting hacked.

On Saturday November 5, WestJet experienced a system-wide outage that impacted operations across our network due to a cooling issue in our primary data centre.

https://www.westjet.com/en-ca/news/2022 ... perating-o

Someone with way more IT knowledge than me speculated that it was the chillers that cool the CPU's in the servers (modern ones are liquid cooled). The chiller(s) failed and everything overheated and shut down.

The Boomer as spoken. He says has it was nothing to do with being hacked, Westjet themselves said so! So it must true.

Imagine thinking that a 5 billion dollar company *entire* flight OPS is at the mercy of a single point environmental failure in a datacenter.

Oh yeah, the cooling failed in our datacenter running the entire flight ops and reservation and we had to shut it down. That makes perfect sense.
As if there was no multigeo cloud load balancing of critical services ... yup everything was running fine on a single bare metal server tucked in the corner until Raphael spilled his coffee on the cooling system causing a full 12+ hour outage.

Ok wise one, since you seem to be an expert at knowing what it wasn't, why don't you tell us what it was?

[daedalusx]

Nothing to do with getting hacked.


https://www.westjet.com/en-ca/news/2022 ... perating-o

Someone with way more IT knowledge than me speculated that it was the chillers that cool the CPU's in the servers (modern ones are liquid cooled). The chiller(s) failed and everything overheated and shut down.

The Boomer as spoken. He says has it was nothing to do with being hacked, Westjet themselves said so! So it must true.

Imagine thinking that a 5 billion dollar company *entire* flight OPS is at the mercy of a single point environmental failure in a datacenter.

Oh yeah, the cooling failed in our datacenter running the entire flight ops and reservation and we had to shut it down. That makes perfect sense.
As if there was no multigeo cloud load balancing of critical services ... yup everything was running fine on a single bare metal server tucked in the corner until Raphael spilled his coffee on the cooling system causing a full 12+ hour outage.

Ok wise one, since you seem to be an expert at knowing what it wasn't, why don't you tell us what it was?

You can believe what ever you want my friend but to say that it couldn't possibly be some form of DDOS/hacking/ransomware attack etc because the CEO said so in an open-letter is naïve and foolish but don't ask me, ask anyone who has actual IT experience running critical infrastructures you might find a little bit of skepticism at taking the company's own words for granted.

When Garmin was hacked a few years ago, did they go openly to the press and told their millions of customer that their personal information were at risk while they were currently negotiating a ransom or did they just kept quiet until the issue was resolved ? It's been 2 years and we still don't "officially know" if a ransom was paid or not.

[CpnCrunch]

Nope, no clouds in Calgary.

https://www.leveleng.ca/featured-projec ... ta-centre/

[Bede]

You can believe what ever you want my friend but to say that it couldn't possibly be some form of DDOS/hacking/ransomware attack etc because the CEO said so in an open-letter is naïve and foolish but don't ask me, ask anyone who has actual IT experience running critical infrastructures you might find a little bit of skepticism at taking the company's own words for granted.

When Garmin was hacked a few years ago, did they go openly to the press and told their millions of customer that their personal information were at risk while they were currently negotiating a ransom or did they just kept quiet until the issue was resolved ? It's been 2 years and we still don't "officially know" if a ransom was paid or not.

Every WJ employee gets IT status updates on system outages, the cause, and what is being done to restore serviceability. I happened to be watching them roll in as I was flying (actually grounded). It seems that Fareed in IT's real time comments on the cause seem to backup what the COO says. Who knows, maybe the 28 IT status updates we got were all made up and the real reason can only be divined by the talking heads on this forum.

[goldeneagle]

Someone with way more IT knowledge than me speculated that it was the chillers that cool the CPU's in the servers (modern ones are liquid cooled). The chiller(s) failed and everything overheated and shut down.

One of the data centers in Europe where I have a couple servers located experienced exactly this problem a couple years ago. The cooling system involves a large water supply, outside radiator, and pumps with lines over the racks and drops in each cabinet to the cpu installations. the issue they had, line broke on the overhead, flooded the rack.

The 'lesson learned' for them, in new installations, cooling water lines now run under the floor, not overhead the racks.

But this kind of thing is exactly why my company runs multiple redundant systems located in physically separate data centers, on different continents with automatic failovers in place.

On the subject of systems failing when the data center cooling system fails, been there, done that, got that t-shirt. I think we had 10 minutes of downtime as systems rolled over connecting to the other data centers.

For a company the size of WestJet that is so dependant on real time data to not have redundant systems in multiple locations for protecting against this type of failure shows pure incompetence in the IT department.

[digits_]

For a company the size of WestJet that is so dependant on real time data to not have redundant systems in multiple locations for protecting against this type of failure shows pure incompetence in the IT department.

Or it's the executives discovering that the cheap solution with a 99.9% uptime guarantee is not all that great

[rooster]

The Boomer as spoken. He says has it was nothing to do with being hacked, Westjet themselves said so! So it must true.

Imagine thinking that a 5 billion dollar company *entire* flight OPS is at the mercy of a single point environmental failure in a datacenter.

Oh yeah, the cooling failed in our datacenter running the entire flight ops and reservation and we had to shut it down. That makes perfect sense.
As if there was no multigeo cloud load balancing of critical services ... yup everything was running fine on a single bare metal server tucked in the corner until Raphael spilled his coffee on the cooling system causing a full 12+ hour outage.

Ok wise one, since you seem to be an expert at knowing what it wasn't, why don't you tell us what it was?

You can believe what ever you want my friend but to say that it couldn't possibly be some form of DDOS/hacking/ransomware attack etc because the CEO said so in an open-letter is naïve and foolish but don't ask me, ask anyone who has actual IT experience running critical infrastructures you might find a little bit of skepticism at taking the company's own words for granted.

When Garmin was hacked a few years ago, did they go openly to the press and told their millions of customer that their personal information were at risk while they were currently negotiating a ransom or did they just kept quiet until the issue was resolved ? It's been 2 years and we still don't "officially know" if a ransom was paid or not.

I will believe whatever I want, but that wasn't the point. I don't even have an opinion on it. All I was saying was you don't know what it was. Whether WJ IT was full of it or not, we won't know. Not even you, unless you definitively know who/what was responsible. Regardless, it doesn't mean it DIDN'T happen either. That's all.

[Eric Janson]

Imagine thinking that a 5 billion dollar company *entire* flight OPS is at the mercy of a single point environmental failure in a datacenter.

After 35+ years in this Industry I've seen plenty of cases where a single point failure has crippled Operations. It's usually someone with no detailed knowledge or understanding trying to save money. Almost always ends up costing multiples of what it would have cost to do it properly in the first place.

Disclaimer:- No knowledge about this incident