Cloudflare, a company providing performance and security to websites, is having network problems of its own this morning — and taking down a lot of its customers’ sites and apps in the process. Affected companies include podcast app Overcast, chat service Discord, managed hosting provider WP Engine, eCommerce hosting provider Sonassi, public web front-end CDN service CDNJS, and many others — including the sites that rely on the web hosting or who partner with Cloudflare for their CDN service.
According to Cloudflare, it identified a possible route leak that’s impacting some of the Cloudflare IP ranges, and its working now to resolve the issue.
The problems were first identified around 7:02 AM EST, says Cloudflare, and the problem was identified shortly thereafter.
Its status page has been providing continual updates.
The company said at 8:34 AM EST, “this leak is impacting many internet services including Cloudflare. We are continuing to work with the network provider that created this route leak to remove it.”
Update: The company at 12:42 AM UTC / 8:42 AM EST says the issue is resolved:
The network responsible for the route leak has now fixed the issue. We are seeing improvement and are continuing to monitor this before we consider this issue resolved.
Update 2: Cloudflare statement —
Earlier today, a widespread BGP routing leak affected a number of Internet services and a portion of traffic to Cloudflare. All of Cloudflare’s systems continued to run normally, but traffic wasn’t getting to us for a portion of our domains. At this point, the network outage has been fixed and traffic levels are returning to normal.
BGP acts as the backbone of the Internet, routing traffic through Internet transit providers and then to services like Cloudflare. There are more than 700k routes across the Internet. By nature, route leaks are localized and can be caused by error or through malicious intent. We’ve written extensively about BGP and how we’ve adopted RPKI to help further secure it.
Update 3 – Cloudflare CEO Matthew Prince threw shade on those to blame for this, via a tweet:
Update 4 – Cloudflare detailed the issue further in a blog post. Verizon, in particular, was held accountable for the internet’s “small heart attack” (the BGP routing leak.) Cloudflare pushed for better industry standards and routing security.
At Cloudflare, we wish that events like this never take place, but unfortunately, the current state of the Internet does very little to prevent incidents such as this one from occurring. It’s time for the industry to adopt better routing security through systems like RPKI. We hope that major providers will follow the lead of Cloudflare, Amazon, and AT&T and start validating routes. And, in particular, we’re looking at you Verizon — and still waiting on your reply.