Cloudflare is one of those companies most people never think about, yet on mornings like this it suddenly becomes the main character. Its network sits in front of millions of sites, speeding them up, filtering junk traffic, and quietly handling DNS and security for everything from tiny blogs to giant social platforms. When that layer… continue reading.
As mentioned in another topic, this impacted us for at least a half day, with ripple effects going on for several additional hours. I think that the fundamental components of the components we refer to as the Internet need a lot more redundancy and a lot less dependence on one to three large companies where some poorly tested change brings down the entire infrastructure.
Very bad choice of activities, but CloudFlare should not have such a controlling influence over a network that affects not only communications, but also tens, if not hundreds of billions of dollars of business each day.
Follow up from me: 4-5 days later, things from my end seem 100% back to normal. Based on seat of the pants observations, even 24 hours later, there were occasional blips or sites that had failed to reset or restore themselves following the incident.
On a probably unrelated matter, but one that REALLY seemed to expose our device, seven years ago we moved into our retirement community and we have a Spectrum cable box. I’ve been complaining to my wife how terrible the technology is, calling it twenty year old trash. Anyway, the CloudFlare incident seemed to do something else; it might be a coincidence, but those power issues may have brought the device close to final (but not 100%) end of life.
If there is a positive on this barely related topic, the Spectrum Internet/Cable vendor DOES have a new generation, much smaller, current generation box now available that supports HDMI and streaming connections. I want to get one of those devices as a replacement, and if/when I do, I’d be glad to write a completely separate article about it; meanwhile, the RELATIONSHIP of all of this to the CloudFlare issue is that the problems occurred BEGINNING with the CloudFlare incident!
On another note, guys, I’m thoroughly enjoying using Cloudflare’s custom rules. That said, if the security risk isn’t too high, we can temporarily pause Cloudflare when there are issues like this.
No chance to replace ISP router with your own device? I’ve always been a bit wary of ISP hardware. Especially if you have to pay a rental fee. If not, then it’s not as bad.
I’m sure that it’s not so true to say that “Cloudflare is one of those companies most people never think about”, as more and more, they seem to be used for some security process when opening links to general sites that use them, like recruitment sites.
Given how long it would take to handle this, from my experience, I’ve formed a negative impression of them - like it would take some minutes (at worst, half the time of my morning commute) just to get into a page (patchy internet access on trains not helping). Infuriating, but increasingly employed, and Cloudflare’s name is there on the screen all the time, along with a spinner.
Anyway, I struggle to imagine that a major service provider such as them would build a system with such a vulnerability to what seems to be a programming error, if that’s what “a bug in generation logic for a Bot Management feature file” amounts to. Don’t they have a ‘Go Back’ button for such situations, or deploy updates either fully tested or through a graded transition? It doesn’t sound too clever to me.
I got some more information about CloudFlare from a well respected Web administrator that I know. CloudFlare, despite the recent issue, is highly regarded for it’s work in blocking and protecting against many different Web bots. That’s why there was such a big disruption when it didn’t work properly; a LOT of places do use it and continue to use it. Even they are not perfect, but I’m sure they are working hard to make their solution more resistant to the recent issue they faced; meanwhile they still have a very good reputation as a protector of Internet network services.
Both of your takes are interesting, especially given how visible Cloudflare has become in day-to-day browsing. The outage highlighted exactly what you’re describing: most people don’t think about Cloudflare at all until something breaks, then suddenly it feels like half the internet is spinning.
Hearing different angles on it helps paint the full picture of how widespread and surprising it was.
