Yesterday’s Internet Outage In Parts of U.S. and Canada You Didn’t Hear About

How a Tiny Error Shut Off the Internet for Parts of the US and Canada

Reposted from Wired

Lily Hay Newman13 hrs ago

© Joe Raedle

A year ago, a DDoS attack caused internet outages around the US by targeting the internet-infrastructure company Dyn, which provides Domain Name System services to look up web servers. Monday saw a nationwide series of outages as well, but with a more pedestrian cause: a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country.

Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. “Our network experienced a service disruption affecting some customers with IP-based services,” the company said. “The disruption was caused by a configuration error.” Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring “an external network issue” and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone.

 

© Downdetector.com 

The misconfiguration was a “route leak,” according to Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks, which monitors global internet operations. ISPs use “Autonomous Systems,” also known as ASes, to keep track of what IP addresses are on which networks, and route packets of data between them. They use the Border Gateway Protocol (BGP) to establish and communicate routes. For example, packets can route between networks A and B, but network A can also route packets to network C through network B, and so on. This is how internet service providers interoperate to let you browse the whole internet, not just the IP addresses on their own networks.

In a “route leak,” an AS, or multiple ASes, issue incorrect information about the IP addresses on their network, which causes inefficient routing and failures for both the originating ISP and other ISPs trying to route traffic through. Think of it like a series of street signs that help keep traffic flowing in the right directions. If some of them are mislabeled or point the wrong way, assorted chaos can ensue.

Route leaks can be malicious, sometimes called “route hijacks” or “BGP hijacks,” but Monday’s incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before.

“Folks are looking to tweak routing policies, and make mistakes,” Arbor Networks’ Dobbins says. The problem could have come as CenturyLink works to integrate the Level 3 network or could have stemmed from typical traffic engineering and efficiency work.

Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using “route filters” that check the IP routes their peers and customers intend to use to send and receive packets and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet and can have their own mistakes.

Monday’s outages reinforce how precarious connectivity really is, and how certain aspects of the internet’s architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.

Tinker, Tailor, Soldier, Spy: Largest Cyber Attack In History Slows The Internet

The last two days have seen one of the most interesting and disturbing examples of the ongoing problems with Internet security, and the potential of contending with anonymous groups and aggressive or governments who use the Internet for hostile purposes.  We have just experienced the largest Internet cyber attack in history.  The New York Times, The Guardian and host of other global media and technology news sources, and blogs have seized on this story.  Internet experts are pessimistic that anything can be done to defend against this situation, or any other similar attack, other than to find and prosecute the perpetrators.  Some experts have speculated that another attack on this scale could have grave consequences for the global banking and investment trading systems.

Read more: http://www.nytimes.com/2013/03/28/technology/attacks-on-spamhaus-used-internet-against-itself.html?pagewanted=all

The attack was restricted to simply slowing the Internet to a crawl in some places around the World.  But it has already been seen that such attacks can target the electrical grid, water systems, natural gas distribution: any essential infrastructure system attached to the Internet, even with state-of-the-art firewalls and other security measures.  This feels like the Cold War spy world of John Le Carre‘s George Smiley, and the current film Tinker, Tailor, Soldier, Spy, only fast forwarded to the cyber world of 2013.. It is a game of leap frog with no end in sight.

If any of my UBC Management students noticed a distinct slowing of their Internet traffic over the last two days, they were not imagining it.  This has been a dramatic foreshadowing of the George Orwell Brave New World we are entering, a dispute between two virtually unknown Internet companies and a group of hackers in Eastern Europe has led to the largest global Internet cyber attack in history.  The attack continues tonight, though it is apparently diminishing.  Internet security experts who have been monitoring global Internet traffic for the last two days, estimate that the attack is at least three times larger than any previous “distributed denial of service” attack observed.  A distributed denial of service or DDOS attack, occurs when someone or a group, creates small Internet “bots” (robot code) that are unleashed across the global Internet.  The bots enter our computers, unknown to us, and sit in our Internet devices, until they are ordered by their Master to simply “ping” or go to one specific Internet address.  This essentially overloads the Internet backbone, and the destination Web site, making it unreachable by anyone. The only solution is to take the Website completely offline and to wait for the storm of “bots” to diminish.  This kind of thing has been around for years, but it continues to be very difficult to defend against. Until a source is identified, all users must be denied access to the Website. Hence the name, “denial of service.”  This time, DDOS has risen to a new level.

The two companies involved in the original dispute were Spamhaus and a Dutch website known as Cyber Bunker. Anyone ever heard of either of them?  Spamhaus works to identify and block SPAM. When Spamhaus decided to include Cyber Bunker in its list of offending spammers, it appears the Cyber Bunker retaliated and recruited a group of criminal hackers in Eastern Europe to help them stage the largest DDOS attack in history. Some have described it as a “nuclear bomb dropped on the Internet.”

We only have more of this to look forward to, as Internet infrastructure experts do not have any silver bullet to offer us.