Major IT/ Tech outage affecting many countries

[EricF]

Luckily, TrekBBS has @EricF

Too kind. Well obviously we don't run Windows servers. Although it's not been trouble free Treking all the time, we've had our fair share of DDOS and so forth which are difficult to mitigate without going to a large network provider who can soak up the traffic. So I do feel a bit sorry for the poor folk on the ground trying to unpick the mess.

Actually I initially assumed there were two issues. I saw an alert about the Crowdstrike update screwup, but the news channels were reporting a "Microsoft outage" - so I assumed one of the Azure DCs was having major issues. Seems it was actually just the one problem badly reported.

Amusing that Crowdstrike had recently published this https://go.crowdstrike.com/rs/281-OBQ-266/images/report-2024-state-of-app-security-report.pdf

As ever the rule is if you can't afford the outage then plan for it. I have a lot more sympathy for the smaller companies impacted who can't afford to do things "properly". If you're systems are critical then well maybe they shouldn't autoupdate without having secondary systems available that are not autoupdating, but are not open to the same threat profile? That said we rely on third parties for some blocklisting for bad IP ranges, I do at least filter them against our own whitelists before deploying however so they are checked to some extent before being loaded (well actually a duff list can't be loaded so there is that).

 

[Asbo Zaprudder]

CrowdStrike Falcon is also supported for Linux and MacOS, apparently, but the BSOD problem has only been seen on Windows servers. (Linux's systemd supports systemd-bsod for boot failures in case an admin misses BSOD.) Some unsubstantiated rumours have it that the problem was caused by an antivirus update signed with an invalid certificate, but I don't know why that would abend the system.

 

[EricF]

Lets not get onto my thoughts on systemd

 

[Santaman]

^^

I don't think it will get much better in the future, software gets more complex and businesses do not want to, or can, spend money on backup systems.
Also, a LOT of Q&A has been delegated to A.I./automated systems, and with A.I. I mean Artificial Idiot, that's why Windows users are alpha/beta testers for M$ who can then prevent (some) of the f#ckups they make from reaching the money sphere (server licence slaves)

Not that Linux is f#ckup free, one of the more recent LTS kernels made it impossible for machines with Raven Ridge and other AMD APU/laptop chips to boot, however it is relatively easy to go back to a working kernel.

 

[Asbo Zaprudder]

Lets not get onto my thoughts on systemd

Thankfully, I'm retired so that particular brain fart of Lennart Poettering is something that I no longer need to worry about.

 

[Gingerbread Demon]

I feel sorry for smaller companies with over 40 computers and one person to admin all of that

 

[John Clark]

I feel sorry for smaller companies with over 40 computers and one person to admin all of that

And those where the main portion of staff work away from the office. Fortunately, at least some of them are IT themselves

 

[Disposable_Ensign]

Lets not get onto my thoughts on systemd

Bane of my existence.

 

[Marc]

technical discussion on the outage by Dave Plummmer who was a former programmer/developer at Microsoft.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

Couple of points
a) an OS will blue/black/pink screen on kernel mode errors (was was the case with crowdstrike) because the results could be far far worse.
b) Crowdstrike runs in kernel mode and as such has been through the WQHL process as device drivers. Problem is that takes time and when you're dealing with product the continually updates (the definitions) it a hassle so they do little endrun. The WQHL cert is intact because the overal program doesn't but they have a way of updating the definitions.
c) He speculates that the software didn't want h Windows running without thus bring everything down in a heap.

 

[Asbo Zaprudder]

His reasoning seems sound. A corrupt p-code file should have been caught in testing before it went live. Depending on the nature of the update file, it could presumably be a relatively easy way for a man-in-the-middle attack by a state-sponsored bad actor to cause massive disruption. Are these updates even encrypted and digitally signed? It might sound paranoid but Putin is just itching to get some revenge for the help being given to Ukraine.

 

[Marc]

His reasoning seems sound. A corrupt p-code file should have been caught in testing before it went live. Depending on the nature of the update file, it could presumably be a relatively easy way for a man-in-the-middle attack by a state-sponsored bad actor to cause massive disruption. Are these updates even encrypted and digitally signed? It might sound paranoid but Putin is just itching to get some revenge for the help being given to Ukraine.

it's my understanding from Dave's video is the updates aren't signed - instead they're inserted in a way that they don't have to go through the certification process because of the time take and don't change anything that would invalidate the security certificate.

And this is probably the same for any reputable AV/security because of the frequency of their updates.

But I guess it's challenge/balance between ensuring that updates go out quick and and things like zero-day exploits are closed off ASAP vs the risk that something could be snuck in that's unsigned and really bad shit happens.

 

[Asbo Zaprudder]

it's my understanding from Dave's video is the updates aren't signed - instead they're inserted in a way that they don't have to go through the certification process because of the time take and don't change anything that would invalidate the security certificate.

And this is probably the same for any reputable AV/security because of the frequency of their updates.

But I guess it's challenge/balance between ensuring that updates go out quick and and things like zero-day exploits are closed off ASAP vs the risk that something could be snuck in that's unsigned and really bad shit happens.

I understand why Crowd Strike took this route to avoid Windows Hardware Compatibility Program Certification* (which I'll abbreviate WHCPC), MicroSoft's certification process for drivers. I mean a different sort of certification. Handling signed updates could be a mechanism internal to CrowdStrike Falcon, Update files containing p-code or whatever is required could be digitally signed without requiring WHCPC. The file would then be (relatively) secure against tampering unless an adversary knew the private decryption key. It could also include a checksum to verify the decrypted file's integrity. It might already use this mechanism - I really don't know.

*I believe this is the current name for WHQL, see https://learn.microsoft.com/en-us/windows-hardware/design/compatibility/

Treat the above ramblings with an appropriate dose of NaCl.