Catastrophic Microsoft outage

[GrandSportC3]

Some security software that Microsoft is using cause millions of users to be able to log on this morning. Half of out staff is unable to log on to our remote desktop. I'm one of them..

[GTOguy]

Can you log on in your plaid while you enjoy some chicken nuggets?

[cptlo306]

Huge mess. This type of issue is one of many reasons I decided to work only for small businesses when I last changed jobs about 10 years ago.

[GrandSportC3]

Can you log on in your plaid while you enjoy some chicken nuggets?

Plaid is not running om microsoft..

[GrandSportC3]

Huge mess. This type of issue is one of many reasons I decided to work only for small businesses when I last changed jobs about 10 years ago.

I work for one of the world's largest banks and half of our employees can't log on this morning. I smell lawsuits coming. Huge loss of productivity from this issue.

[cptlo306]

I work for one of the world's largest banks and half of our employees can't log on this morning. I smell lawsuits coming. Huge loss of productivity from this issue.

Yep, someone is going to be paying some major $$ for this.

I don't envy all the IT folks having to work on fixes. It's going to be a long weekend and next week (maybe even longer) for them.

[DJ_Critterus]

Banks, airlines, and a lot of other places are shut down globally, too.

[Onebadcad]

Banks, airlines, and a lot of other places are shut down globally, too.

This BETTER NOT affect my nearby Arby's!!

[DJ_Critterus]

This BETTER NOT affect my nearby Arby's!!

Better go see if your coupons are still valid.

[KenHorse]

Some security software that Microsoft is using cause millions of users to be able to log on this morning. Half of out staff is unable to log on to our remote desktop. I'm one of them..

Oughta use a good VNC and not MS's Remote Desktop

[ratflinger]

MS huh? Didn't affect me at all. Those UNIX boxes still up?

[owc6]

Actually, Crowdstrike pushed a patch though to fix a Microsoft vulnerability, but the file was corrupted. It affects all Microsoft users that use Crowdstrike, not Microsoft users that do not use it.

[MadInNc]

Earlier today……

[BayouCountry]

What worries me is Crowdstrike seems to be a single point of failure.

[GrandSportC3]

Yep, someone is going to be paying some major $$ for this.

I don't envy all the IT folks having to work on fixes. It's going to be a long weekend and next week (maybe even longer) for them.

I start working at 6:30 am and now it is 11:30 am and I already lost 4 hours of productivity with this issue and still can't log on to work. Today was an office day for me but was sent home and told to keep trying to log on from home. Same issue so far. The amount of work hours lost from this incident is INSANE! My employer has a 6 figure number of employees and I assume that more than half are affected one way or another.

[GrandSportC3]

What worries me is Crowdstrike seems to be a single point of failure.

Indeed, Crowdstrike IS the single point of failure. How were they able to push out that update without it being tested sufficiently?

[GrandSportC3]

Actually, Crowdstrike pushed a patch though to fix a Microsoft vulnerability, but the file was corrupted. It affects all Microsoft users that use Crowdstrike, not Microsoft users that do not use it.

My home laptop works fine. It's just my work system that I can't access. I can't do any work without being connected.. 4 hours lost work and counting.

[Wathen1955]

Indeed, Crowdstrike IS the single point of failure. How were they able to push out that update without it being tested sufficiently?
Because you are the beta tester.

[GrandSportC3]

Because you are the beta tester.

It appears that one of the files got corrupted and pushed out to millions of microsoft desktops.
It's 5 hours now since my work hours started and still no ability to log on..

[The_Dude]

Someone is losing their job. Hopefully, they'll update their processes so that this can't happen again.

[Raazor]

Semi-Big company here. 6,600 employees so maybe mid-cap.

No issues that I'm aware of. But I didn't log in until just now. Maybe it was fixed?

[dvarapala]

Some security software that Microsoft is using cause millions of users to be able to log on this morning. Half of out staff is unable to log on to our remote desktop. I'm one of them..

My company uses Crowdstrike as well. The entire office is down, including our VPN. Good thing I'm WFH today. ;)

[dvarapala]

Semi-Big company here. 6,600 employees so maybe mid-cap.

No issues that I'm aware of. But I didn't log in until just now. Maybe it was fixed?

If you don't use Crowdstrike security products you won't be affected.

yyC0Y70RGhU

[GrandSportC3]

Semi-Big company here. 6,600 employees so maybe mid-cap.

No issues that I'm aware of. But I didn't log in until just now. Maybe it was fixed?

It only affected desktops that ran that update with the corrupted file.

[dvarapala]

It appears that one of the files got corrupted and pushed out to millions of microsoft desktops.
It's 5 hours now since my work hours started and still no ability to log on..

You can resolve the issue by booting into Safe mode and deleting the bad file. Literally a 5-minute fix.

Your IT department sucks. :yesnod:

[mrvette]

China got in and broke that new set of dishes.......


anyone care to BET it's a sabotage from them???

[Unsuspicious]

Boot Windows into safe mode or the Windows Recovery Environment.
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
Locate the file matching “C-00000291*. sys” and delete it.
Boot the machine normally.
Hire competent IT people.

[GrandSportC3]

You can resolve the issue by booting into Safe mode and deleting the bad file. Literally a 5-minute fix.

Your IT department sucks. :yesnod:

I can't do that as I work on a VDI (Virtual Desktop) logging in to a remote desktop. I'm in Florida and my actual session is on a computer in New York.
Only system admins can do what you are suggesting. My VDI can't connect to the remote session at all.

[The_Dude]

Semi-Big company here. 6,600 employees so maybe mid-cap.

No issues that I'm aware of. But I didn't log in until just now. Maybe it was fixed?

My time tracking plugin was down last night. Not sure if it's back today.

[KenHorse]

Like I said, why not use a private VNC and screw MS?

No single point of failure, plenty of security. Dumbass IT "professionals" and their reliance on an MS account...

[Frankie the Fink]

Huge mess. This type of issue is one of many reasons I decided to work only for small businesses when I last changed jobs about 10 years ago.

And why I don't use ANY "cloud" for ANYthing critical....don't have accounts on them nor store data on them...

[NEED-A-VETTE]

Some security software that Microsoft is using cause millions of users to be able to log on this morning. Half of out staff is unable to log on to our remote desktop. I'm one of them..

Same. Except I was able to log in. :rolleyes: :D

Some of the caregivers that report to me were not, though.

You really learn how robust your downtime procedures are when this happens. :funny:

[NEED-A-VETTE]

106600


106601


106602


106603

[cptlo306]

And why I don't use ANY "cloud" for ANYthing critical....don't have accounts on them nor store data on them...

Hosting stuff in the cloud has advantages and disadvantages. This issue is clearly a disadvantage but cloud hosting can be way better if done correctly.

[GrandSportC3]

I'm 20 mins from the end of my work day.. Still can't connect. Hopefully, it will be fixed by Monday. Monday is Work from Home day for me.

[Tikiman]

'

[Rikki Z-06]

I'm retired and could care less. :yesnod:

[KenHorse]

Wasn't Crowdstrike somehow involved in the HRC Russia collusion hoax?

[Steve_R]

I'm retired and could care less. :yesnod:

*couldn’t :slap:

[SteveOceanside]

Over this way, we're told that cafes had to shut today as they couldn't take card payments. Will people wake up? These cafe's used to have tills with a float of cash in them, but then decided that cash is "so old-fashioned".

I hope we have more of these failures, because only shock treatment will wake the sheeple up to the implications of a "cashless society".

[jw38]

My wife and I flew home from Amsterdam yesterday. We made it out of Schipol just before it was affected by the shitstorm!

[Steve_R]

My wife and I flew home from Amsterdam yesterday. We made it out of Schipol just before it was affected by the shitstorm!

I’m flying home today. It’s been interesting to put it mildly. I’m lucky I’ll make it home, lots of people not so lucky.

[MadInNc]

Even Snoop Dog pissed off

[lrobe22]

microsoft blows. bill gates blows. Mac for the win every day.

[Rikki Z-06]

*couldn’t :slap:

could not :kick:

[Frankie the Fink]

Hosting stuff in the cloud has advantages and disadvantages. This issue is clearly a disadvantage but cloud hosting can be way better if done correctly.
Right, apparently its working so well. I'm a degreed Computer Scientist with 40 years in the industry, for my personal, non-commercial work, I'll just keep everything on my own local disk farm, but thanks...

[cptlo306]

Right, apparently its working so well. I'm a degreed Computer Scientist with 40 years in the industry, for my personal, non-commercial work, I'll just keep everything on my own local disk farm, but thanks...

Good for you. I'm also a degreed Computer Scientist (Georgia Tech) but with only 34 years in the industry.

Cloud apps can work great but too many people just assume stuff in the cloud is safe and doesn't need backup or recovery planning.

Take email for example, very few companies or people host their own email servers now. It's just too much of a hassle these days.

To each his own.

[KenHorse]

Good for you. I'm also a degreed Computer Scientist (Georgia Tech) but with only 34 years in the industry.

Cloud apps can work great but too many people just assume stuff in the cloud is safe and doesn't need backup or recovery planning.

Take email for example, very few companies or people host their own email servers now. It's just too much of a hassle these days.

To each his own.

I despise the cloud for business programs (and CAD). It's getting tougher for those of us who aren't IT ignorant to not need the cloud.

[MadInNc]

My wife and I flew home from Amsterdam yesterday. We made it out of Schipol just before it was affected by the shitstorm!

You were one of the lucky few….


12hr. Time lapse of air traffic of CrowdPlow outage

[VatorMan]

Everyone talks about straws killing turtles, but putting a photo on OneDrive builds another 120MW server farm. **** EV’s-Server farms will kill the National electric grid.

A bit off the true topic, but you get the gist.

[NEED-A-VETTE]

You were one of the lucky few….


12hr. Time lapse of air traffic of CrowdPlow outage



https://x.com/marknagi/status/1814294015447023744?s=46&t=SgvQfLYNggwY7GXoXGPnsw

[ZipZap]

Even Snoop Dog pissed off

EA Sports College Football 25 dropped and nobody could play on the servers.:rofl:

[ZipZap]

Everyone talks about straws killing turtles, but putting a photo on OneDrive builds another 120MW server farm. **** EV’s-Server farms will kill the National electric grid.

A bit off the true topic, but you get the gist.

The sheer number of GPU's employed is silly. Since power is at the core of fast searching of large datasets, which is the basic AI building block, we'll never have enough power on the grid. I've heard estimates of about 50%.

[dvarapala]

Everyone talks about straws killing turtles, but putting a photo on OneDrive builds another 120MW server farm. **** EV’s-Server farms will kill the National electric grid.

:iagree:

Especially now with the explosion of AI. It won't be long before AI datacenters have their own nuclear power plants. :yesnod:

GQA0xfkWpqw

[VatorMan]

One of my best EE’s I deal with was building 180MW server farms before he saw that it would never end. I run a 560 acre, 55 building facility that maxes out at 30MW. Buildings have loaded/unloaded times.

Servers ? Nope. Continuous.

And no one is talking about this.

[Frankie the Fink]

My pool computer couldn't contact the server due to this... Just think how America's cyber enemies are noting how fragile our infrastructure is - not even counting other nations.

[Frankie the Fink]

My wife and I flew home from Amsterdam yesterday. We made it out of Schipol just before it was affected by the shitstorm!

Wife, daughter and granddaughter just made it under the wire flying back from Dublin a few hours before this mess.

[Frankie the Fink]

Good for you. I'm also a degreed Computer Scientist (Georgia Tech) but with only 34 years in the industry.

Cloud apps can work great but too many people just assume stuff in the cloud is safe and doesn't need backup or recovery planning.

Take email for example, very few companies or people host their own email servers now. It's just too much of a hassle these days.

To each his own.

The issue with the cloud is the same as having some rent-a-cop guarding your store. There are very good ones and seriously lame-ass ones. I toured a data center in Rockville, MD when the Dept of Justice was considering farming out their server needs. I was unimpressed with the security, staff expertise, the "hardening" of the site and management. DOJ took my recommendation to not migrate to that site. Their cybersecurity guy had the most ridiculous response I ever heard when I asked about APTs for Cripe's sake.

Not only did I not believe they passed the ISO standards, I guarantee my "black hat" team could have gotten in and "owned" the place in about 5 hours - tops.

[BayouCountry]

Good for you. I'm also a degreed Computer Scientist (Georgia Tech) but with only 34 years in the industry.

Cloud apps can work great but too many people just assume stuff in the cloud is safe and doesn't need backup or recovery planning.

Take email for example, very few companies or people host their own email servers now. It's just too much of a hassle these days.

To each his own.

Not a hassle. It just takes too much staff to maintain even a “simple” email system. I was backup admin for Exchange Server at a midsize company and it took a lot of my time; especially when the primary admin retired.

Try taking down two Hospitals and multiple clinics for an email system update. It’s not fun. Outsourcing to Google made perfect sense for our organization.

[BayouCountry]

My pool computer couldn't contact the server due to this... Just think how America's cyber enemies are noting how fragile our infrastructure is - not even counting other nations.

This!

[MadInNc]

History repeats itself

[Frankie the Fink]

Good for you. I'm also a degreed Computer Scientist (Georgia Tech) but with only 34 years in the industry.

Cloud apps can work great but too many people just assume stuff in the cloud is safe and doesn't need backup or recovery planning.

Take email for example, very few companies or people host their own email servers now. It's just too much of a hassle these days.

To each his own.
Meh, Hillary ran one out of a bedroom closet with a lame IT yokel and a few staffers. And the fact is that with those cloud based emails the estimate is that 7 people could read your email before you even receive it, if they so wished...

[DDSLT5]

I'm retired and could care less. :yesnod:

How much?

[Vince Clortho]

[QUOTE=DDSLT5;2297600]How much?QUOTE][/106762

[The_Dude]

Quoting is hard

[Frankie the Fink]

I guess after 40 years in IT I'm a bit taken aback that Soooo many outfits are using a narrow slice of vendor tech and that a glitch is so pervasive. I think I heard that Southwest was about the only airline that didn't get nuked by this debacle - they use some other IT infrastructure.

[SteveOceanside]

Debacle indeed. What about "testing"? Are they all DEI now? :rolleyes: Didn't affect me personally, but then I rely on few servers. If my bank went down I wouldn't know just yet.