Destroy the Internet with a hacksaw?
Thursday, April 9th, 2009 at 10:54 am
This morning many people in Silicon Valley woke up without 911 service, Internet, cellular phones, and in some cases TV. Web sites were impacted and Internet traffic between a few major datacenters stopped flowing. Several of our employees were cut off from the Internet and phone service.
AT&T put out a press release stating that there was a fiber cut, but to make this happen, there had to be several cuts. According to several employees that work at AT&T, it may have been done by the very people that repair this stuff, the Communication Workers of America Union (CWA).
This of course is speculation on behalf of these individuals. The cuts could have also been framed to make it look like it was done by a competent group, someone that knows where the major fibers are sitting inside specific manholes. However, the CWA contract to do work for AT&T expired last Saturday night. According to various press releases from CWA, “five of CWA’s six agreements with various AT&T companies expire at midnight, April 4, 2009″. The cuts were clean, done apparently by a hacksaw. The first major cut went down around 2 AM in the South East Bay which isolated the city of Santa Cruz. Another cut around 4 AM took out the major Metromedia Fiber Network (MFN) in the San Francisco bay area as well.
Regardless of who did it - someone did it.
These areas are within driving distance of each other.
What’s more terrifying, the cuts were clean and easy to fix, but what would happen if they were mangled and more calculated? What happened if rather than going down into the manhole, the perpetrator poured many gallons of gasoline down the hole, and tossed a match on it? It could have melted all the fiber/glass/plastics together, causing complete mess making the problem much worse.
Fiber maps such as the one below show details on exactly where fiber runs are and how to locate perfect targets, found directly off of almost every network provider’s web site:
Now, what would happen if someone were to coordinate with a group of people and demolish key areas where fiber concentrations are very thick in major cities around the world. What if it were done all at the same time?
If that were to have happened, more than just Silicon Valley would have been impacted: Globally people would be waking up to find out their land lines don’t work, 911 calls fail, their email is down, and yes no Twitter.
Sound improbable? Well, three undersea cables: FLAG, SMW-3, and SMW-4 where all cut at the same time on January 5th 2009, which crippled communications to several countries. Traffic had to be re-routed and it really was a mess, but nobody was left in the dark.
Realistically it would be a few days of an outage, but it would not be great given society is depending more and more on the Internet for day-to-day things such as the DMV, voting, work, and daily life. Our society is becoming dependent on something very fragile and somewhat unprotected.
Tags: CDN, Content Delivery, Fiber Cut, Internet, San Francisco, Security
April 9th, 2009 at 11:09 am
So you can take out all of MFN’s fiber network with one of these?
http://www.ameriskurutflutningur.com/is/item/Maglite_AAA_Fibre_Optic_Extension.htm
April 9th, 2009 at 11:17 am
The fragility of fibre networks has always amazed me. Don’t know about the US (except FL), but in London, UK, all the manholes are really easy to identify (they all have the Company logo on them!) and open. I suspect you could easily take out large sections of London, such as Docklands, by taking a section of the fibre out. 2 minute job with basic tools. Most manholes have a hook underneath to enable them to be locked, I believe, but normally are not used. Attach a chain and anchoring device to that, toss some quick-dry cement in the hole, replace the manhole cover with anchor into the cement and, well, you get the picture.
April 9th, 2009 at 11:21 am
http://www.mercurynews.com/ci_12106300 has an article posted with hints to the CWA link.
April 9th, 2009 at 11:47 am
“DrFiber” lol
April 9th, 2009 at 11:59 am
Interesting… It would be nice if we had some backup forms for communication. People used to live decent lives prior to fiber optic technology.
April 9th, 2009 at 12:00 pm
We had to have seen this coming… right? Basic risk strategies would have vetted this!
We clearly understand the issues now (thanks for that) but the bigger question is what do these Fibre companies do from here? There aren’t any simple solutions here that I can find… it sounds like a redesign to me and could cost a crap-ton of money.
April 9th, 2009 at 12:41 pm
Union members to some crazy things….
i saw a newly installed PBX installed in a major NYC hospital dripping with OJ once.
April 9th, 2009 at 12:42 pm
At present, customers ask for cheaper, cheaper, cheaper.
Where’s the money for better and more resilient supposed to come from? Just because the services offered tend to be pretty important infrastructure doesn’t magically make the necessary funds to properly secure them materialize.
April 9th, 2009 at 1:11 pm
@Andrew Ross
Who needs terrorists when there are contractors about - http://www.theregister.co.uk/2009/04/08/bt_hole_hits_vodafone/ & http://www.flickr.com/photos/23919135@N00/3426407496/
April 9th, 2009 at 1:19 pm
I wonder if the vandals used eye protection.
April 9th, 2009 at 1:44 pm
All emergency response services should have a satellite backup link.
A 1.2m dish on ku band is adequate for this but it would alleviate alot of problems.
Also seriously freaked people can stock up on satellite phone facilities just in case.
Anyway this just goes to show that new technology doesn’t necessarily replace old tech. Satellite backup for major trunks is a must have.
April 9th, 2009 at 1:45 pm
Thats I love the old technology HF radio if it falls down u just string up another 1 in less than an hour
April 9th, 2009 at 1:55 pm
Your hypothetical at the end sounds so ominous, but I find the idea a little ridiculous. My guess is that if you could stage 10-20 attacks in a day, you could maybe get a single city partially offline for two weeks, with some services like 3G being brought back up very quickly. Spread out over the world it would be felt even less than that. What goals could a group hope to achieve that would limit them to data lines and not also buildings or people?
April 9th, 2009 at 1:57 pm
I’m not sure which is more pompous or arrogant:
a) Network providers that publish their fiber maps and assume they are invulnerable to vandalism, or
b) Annoying people who are not British that use the spelling “fibre”. Being British does not automatically make you smart. *Pretending* to be British does automatically make you useless.
-a
April 9th, 2009 at 2:40 pm
Alright, get the gas, it’s raiding time.
April 9th, 2009 at 2:46 pm
Wow. This may have been a joke of sorts… But a full scale breakdown of the nation, no mater how unlikely, is somewhat daunting.
April 9th, 2009 at 2:52 pm
Dave Peck said it.. this article was really written to be a little over-the-top and excessive, a sarcastic joke… but regardless of how you look at it, it is still disturbing.
April 9th, 2009 at 3:16 pm
Hey, thanks AT&T for giving terrorists ideas on how to take out our communications systems. You should have said it was faulty hardware or something. Stupid fools. Better start locking those manhole covers now. Be a little smarter next time will ya? We live in a different world now.
April 9th, 2009 at 3:29 pm
Considering everyone is still working sans contract at the
moment, I doubt it was a disgruntled Union worker. When they
do it, it normally takes place AFTER a strike is in progress,
not before. Unless it was meant as a warning that is. . .
As for giving ‘terrorists’ ideas on how to kill the comm
systems, why go for the small change when you can demolish
the entire Central Office ? Why whack a few fiber lines
when you can take out the entire hub ? Go for a few major
downtown offices if you want to get them all . . . lol
They’re not going to bother with small scale vandalism if
they ever decide to go that route I assure you.
Fiber cuts happen all the time and the majority of them
are caused by construction and the dreaded back-hoe. Since
most fiber rings are designed to protect against single
path failures, most folks don’t even realize they happen.
Few folks outside of the company will know where the primary
and secondary fibers lie for any given service.
April 9th, 2009 at 4:23 pm
I think that these networks need redundancy in available pathways. My comparison is not perfect but let look at water systems. The grid based systems allows for multiple flow paths to almost any point within the system. While managing water flow is simple compared to these large fiber networks but the principal of running a network with a higher fault tolerance would allow for higher reliability. This idea is my no means perfect, but it allows for higher tolerance of failure. Why don’t we see this principal applied at the same level as water systems? Cost & Complexity. Think about every point where your neighborhood connects to the larger network, this principal would force an additional connection point to the neighborhood.
April 9th, 2009 at 5:03 pm
Angry union workers in the past have cut telephone lines in order to rack up overtime right before a strike. Happened to SBC (now AT&T) before the last strike in 2004.
http://www.dslreports.com/shownews/43355
April 9th, 2009 at 7:54 pm
Um we’re kind of dependent on electricity too, have been for a while. It goes through similar conduits and when it’s out, I assure you, your precious Internets are gone too. Sometimes it’s out even for days if the fire does enough damage to your hood. What’s the worry? I’d much rather lose my fiber than my electricity.
April 9th, 2009 at 8:11 pm
I am the System Administrator for an ISP down in San Martin, CA. We were directly and completely affected by the fiber cut. Both of our upstreams, it turns out, run over that very fiber link. All of our 15,000+ customers have been out of service since the cuts. Land-lines in Gilroy, San Martin, and Morgan Hill could only call locally until around 5pm. Cell phones did not function with any stability until 5:30pm.
As a side note, I live about 3 blocks from the scene of the fiber cuts in South San Jose… Quite a scene over there; umpteen AT&T trucks, several television stations reporting from the scene, road that runs parallel to the railroad tracks that the fiber is buried next to is coned off….
We (myself and my co-workers) all speculate that the Union is responsible for the cut. AT&T is offering a $100,000 reward to the person who provides information that leads to the capture and prosecution of the individual(s) responsible for the cuts.
April 10th, 2009 at 12:19 am
I am a CWA telephone worker. Only the craziest among us would damage our own network. 1. It is a federal crime under the Patriot Act and 2. with the built in redundancy of most fiber rings, only isolates a small amount of customers.
It could have been some ass looking for copper. I have seen thieves steal the optical equipment out of the remote huts or take live wire from the local electrical utility and light themselves up.
In Detroit, Michigan the manhole covers are stolen right off the street. Makes one hell of a pothole experience.
None of this damage is caused by the Union. Take your anti-union bias and stick it!
April 10th, 2009 at 3:58 am
[...] How are we going to protect those big wires? Barrett Lyon asks whether it’s possible to destroy the network with a hacksaw. In a word: Yup. It happened yesterday. AT&T used Twitter—home of the Fail Whale—to [...]
April 10th, 2009 at 5:21 am
[...] I wish I’d seen this article before writing my own. I hadn’t know there were contract negotiations going on between [...]
April 10th, 2009 at 5:44 am
[...] extra drama with AT&T’s cable workers union the CWA, just as their contract expires, some yahoo cleanly cuts two cables in two different places. Fortunately, the repairs were relatively simple, and service was restored [...]
April 10th, 2009 at 7:13 am
[...] 3,000 people are following AT&T’s updates on the situation (which some speculate was an inside job by the Communications Workers of America (CWA) [...]
April 11th, 2009 at 3:30 pm
This vulnerability is not limited to communications networks alone. The truth is, *EVERY* major piece of infrastructure in our society is fragile and easily subject to sabotage by folks knowledgeable about how the system works. Everything said here also applies to the electrical grid (just to cite one example of another piece of mission critical infrastructure I’m somewhat knowledgeable about), for example. I have personally been in a position to disrupt vast swathe’s of the Internet at a couple different points in my life–ask me about the “security” at MAE-West in the 1990’s when half the traffic on the Internet flowed through it… or about how easy it would have been to “poison” the root server’s DNS cache at various points. I’m sure workers in other industries are equally well aware of the “soft” points in their systems, past and present.
The really amazing thing is not how vulnerable the systems are, but how rare it is that those who have the knowledge to severely disrupt them do so. A hundred determined individuals could probably reduce it to chaos, even without detailed knowledge of our infrastructure. It speaks volumes about the essentially decent nature of humanity that our society continues to function.
April 14th, 2009 at 5:33 am
First of all, a fiber ring is always constructed, so that one cut can’t harm it. 2 or more cuts on the same ring is an entire different story.
Also, some lad from an ISP stated above, that even though they have 2 upstreams, it turns out, they use the same fibers: bummer!!
Redundancy, seperate path, seperate media for that sake is always a possibility, if you plan and pay for it. You can for example with todays technology create short distance GigE radio or laser links, that can cater for connecting to a fiber-ring somewhere else, if necessary, or to bridge in case of a failure. Less bandwidth (150-300 mbit/s) can even be transmitted over distances over 50 miles, no problem.
The whole London scenario mentioned earlier is just a sign of, that BT did not invest into redundancy.
What happened to AT&T is just malicious and is a worst case scenario. Of course you can try to cover for that case, but at what price tag ? The consumers are not ready to pay for the infrastructure, so if you want a bullet proof network, please pay for it.
May 6th, 2009 at 5:55 am
At a more localised level, I came into work a couple of months ago to find that someone had pulled up the BT manhole cover outside our office and cut straight through all the phone and internet lines to our building…it was two days before we were back up and running. Since they had cut the lines as close to wall as they could BT had to call in a team to dig up the pavement before they could make the repairs. I found a site that specialises in locking manhole covers but BT refuse to install one….so I’m now living in fear of them coming back and doing it again, presumably to burgle the office. I’m not sure what the big picture solution is but just securing the manhole outside our office would at least make me feel a little more comfortable.