Remote Terminal
Home Up RT Generator Lessons Learned Outages Flagami Outage

 

My presentation made at session WS CP103 of the 23rd Florida Governorís Hurricane Conference, May 14, 2009:

Telephone Power Dependency

My presentation made at session 601 of the 17th World Conference on Disaster Management, July 10, 2007:

Telephone Power Dependency

This morning my neighborhood lost power. Unfortunately, the phones died also.

At about 7:45 this morning, May 3, 2001, my bedroom UPS woke me up. No big deal, the power's out. Then my alarm company called on my cell phone to say that they had lost contact with my alarm panel. "is your phone line dead?" Sure enough, no dial tone. This isn't good.  I take dial tone failures quite seriously. In addition to making house alarms useless, it also effectively shuts down 911 in the area. The local cell site may be out, or at least overloaded.   So started my search for information about phone network power failures.  

In the old days, each telephone line went to each home, each on it's own copper pair of wires. Everything was powered from the huge bank of batteries at the Telephone Company Central Office (CO). These batteries are charged all the time using rectifiers connected to the power company. When the power went out, the batteries could run the whole phone system for hours. But within seconds a huge generator kicks in and starts charging the batteries again. It's just like a huge UPS. That is why the phone network was considered one of the most reliable networks in the world.

Well, things are different now. Over the last 10-15 years, BellSouth has become very dependent on utility power in the field.

I luckily know where the BellSouth SLCs (Subscriber Line Carrier) are in my neighborhood. These are where they change the fiber optic lines from the Central Office into copper pairs to carry the phone lines into your home. They can be big beige cabinets above ground, or like a hatch on a small hill. Below this hatch is an underground Controlled Environment (Air conditioned, dry) Vault (CEV). All of this requires utility power. There is NO standby generator at these sites. That's something I want to change.

Here's a couple CEVs in a local park.  Believe it or not, there's a large room with air conditioning and a lot of high tech equipment just below the ground:

Here's are a couple of the above ground cousin of the CEV, an SLC Cabinet.  No Air conditioning, or sump pumps.  Fans keep everything cool.  Circuit boards are behind the ribbed doors.   You can see the power meter in the middle:

A couple Bell service techs were parked there, so I assumed the situation was well under control. Wrong. I asked them if they knew the phone system was down. No, they were there for other work. After I told them I thought the power outage had caused the SLCs to fail, they went down into the three of them and found alarms all over the place. The batteries that are SUPPOSED to run the equipment for 4 to 8 hours had failed on the racks that had the multiplexers. That meant that all connectivity toward the rest of the network was dead. The other batteries for the equipment toward the homes was working. Apparently, battery failure is quite common on these systems.

After the first service tech made some calls on his cell phone, the parking lot started to fill up with BellSouth vehicles. 10 at one point. There seemed to be some confusion about what to do. After some discussion, they ordered two generator trailers to come to the location. Before they arrived, FPL restored the power.    Just wondering... what would the BellSouth tech do if the cell phones weren't working?   There are no 2-way radios in the service vans.   Is there a copper delivered phone line at each SLC in case the SLC is down?

The Big Picture

Here's why I'm sharing this with you... in case of a large, prolonged power failure, this could be happening all over Miami-Dade County. There must be thousands of these SLCs around just Dade. Does anyone remember the day long power failures all over South Florida when one relay somewhere at FP&L failed? In my case today, the phone failure only affected maybe 1000 homes in West Miami Lakes. A few miles south, it would have affected a hospital. Maybe a 911 center a few miles East. It wouldn't be a bad idea to know where your neighborhood's SLCs are located.  Use the pictures above to find them in your neighborhood.   If you find these within a half mile or so, then there's a very good chance that the only thing between you and a phone line failure is a battery.  You might not be able to do much, but just confirming that the FPL power is out, by looking at the power meter, will help.  If it's an old meter, is the big disk stopped? If it's the type with the LCD display, is the display blank? If either is true, the SLC isn't getting FPL power. It may or may not be on battery power. Call 305-780-2355 (not 611) on your cell phone and report your line as being out, but also mention that the local SLC has no power. That MAY trigger the right people to get to your site faster.   And I guess there's nothing to prevent you from calling the power company yourself.   You'll need the meter number and/or the street address for the SLC.  The address is usually attached to the outside of the SLC enclosure, with reflective orange and black stickers.  

Here's a spec sheet for a CEV: http://www.hartfordconcrete.com/cev.asp Remember, most of it is buried. Just being there to point to the CEV the stopped meter could take an hour or more off of the repair time. Suggest that they order the generator NOW. If it's a huge outage, and the limited generators are already in use, and you know an electrician in your neighborhood, you might be able to work together to set up a temporary feed from a building with a back up generator. Lots of things become possible after a disaster.

Here's a power meter for an older CEV.  The meter is on the right, the connector for the generator on the left :

Here's a newer style.   This one feeds a CEV that has IFITL equipment:

 

At another SLC failure a few years ago, I actually went down in the CEV with the BellSouth service tech and bypassed a blown out circuit breaker panel with a few extension cords. That got the site going again. That time, even the generator being there didn't help. They didn't have an electrician available, so they needed help understanding what to do when the wiring in the CEV failed. Once they were stumped, they were very willing to accept suggestions from this sidewalk supervisor. One small quirk in the CEV. The AC plugs on the equipment are 20 Amp, not the 15 Amp that we're used to. That's means one of the blades is rotated 90 degrees. Most extension cords can't handle that. But most receptacles can. If you make your own extension cords or quad boxes, use the 20 Amp receptacles. The left slot will look like this: -|

Other issues to consider

As mentioned before, the batteries in a SLC are designed to supply power to the equipment for four hours.   Some sites may last longer.   However, the electronics at the SLC site are only part of the equation.   Air conditioning, blowers, lights and sump pumps operate only on AC power.  So, in a very tightly packed, very active CEV, will the equipment tolerate the higher temperatures until the back up generator arrives?   

Many people wrongly assume that cell phones are a good backup plan in the event of a phone network failure.   Some cell sites are linked with microwave.   However, there is a finite limit to the number of microwave channels that can co-exist in any area.   Many cell sites, especially the newer ones, are linked with Telco supplied T1 circuits, which are also derived from fiber, in the same SLC enclosures that carry your phone line.   SLC down?  Local cell site is down also.   

CLECS, or Competitive Local Exchange Carriers, like SupraTelecom, usually use the same exact facilities as the Incumbent Local Exchange Carrier, BellSouth in our area.   So just switching phone companies doesn't help a bit.   In some cases, the CLEC may have their own facilities, but these are usually available to businesses in high bandwidth circuits, like T1s.   

Disaster planning issues

So, what is the master plan in case of an disaster effecting power?   As it has been explained to me, BellSouth owns a fleet of generators on trailers that will be distributed to each SLC location having a power failure.   Questions I have that I think need review:

bulletHow many SLC sites are there in Miami-Dade County?  From my observations, my guess is 1000 or more.
bulletHow many SLC sites already have built in, permanent generators?
bulletHow many portable generators are available, right now, in our county?
bulletIf our county were hit by a massive power failure, or a large disasters like Hurricane Andrew, how long would it take to get a generator connected to each SLC?   This includes transport to the site from wherever they're stored and the set up.
bulletIf the answer to the question above is greater than four hours, what happens to the SLC sites that run out of battery power or overheat before a generator can be hooked up?   Has the decision been made about which sites get the generators first?   
bulletOnce set up, what procedure will be followed to prevent them from being stolen.  This has happened in other disasters.  
bulletIn the case of 911 centers, hospitals, and other critical locations, are there dedicated fiber SONET loops to the location that are totally independent of power anywhere other than the CO and the emergency site itself?   Otherwise, these locations may fail if any equipment in a loop loses back up power.   Some SLCs are in privately owned buildings and may not even be accessible to the Telco Techs in the event of a disaster.   
bulletis there data available about the reliability of the batteries used in SLCs?   information I have found on the internet indicates that batteries designed for 5 years of life are dying after only a couple years.   

Is all this something related to ham radio? Not really. But, to me, its just another piece of information we might need during a communication emergency. More on what to do when all the phones are dead is available in the page about dealing with Phone Outages

New as of 12/16/2004:

A very good article about the effect of the hurricanes in Florida.  Notice that the point out the weakness of T1s powered by batteries.   http://mrtmag.com/mag/radio_miracle/

BellSouth employees: Feel free to correct me on any of this. I know you know it better than I do. I want to get all of this on this page for future reference. I'll also add some pictures.   E-mail me here:  Ray J. Vaughan

More info on SLC: http://www.telco.com/products_solutions/WhitePapers/digital/page1.html

Personal Preparation:  If you have nothing but cordless phones at home, or any other type of advanced phone that needs AC power to work, no one can reach you in a power failure. Have at least one good old fashioned phone you can hear hooked up. Could be very important in a 'shelter in place' scenario.   

Here is some information I found on the Internet about SLC power failures.   It would appear that I'm not the only one with these concerns.  

From:  http://www.cctec.com/maillists/nanog/current/msg00606.html

To: nanog@merit.edu (nanog list)
Subject: Re: SLC power failures
From: David Lesher <wb8foz@nrk.com>
Date: Wed, 3 Jan 2001 22:52:21 -0500 (EST)
Delivered-To: nanog-outgoing@merit.edu
In-Reply-To: <20010104033803.14631.cpmta@c004.sfo.cp.net> from "Sean Donelan" at Jan 03, 2001 07:38:03 PM
Reply-To: wb8foz@nrk.com
Sender: owner-nanog@merit.edu

Unnamed Administration sources reported that Sean Donelan said:
>
> As power is being restored throughout the midwestern part of the US,  one interesting feature showed up. A lot of telephone companies are using Subcriber Loop Carriers (SLCs) to provide service. Some phone  companies are using local power sources with limited battery backup for some SLCs. With the extended power outages, central offices with generators maintained service; however, the remote equipment didn't have local generators. This resulted in people losing their local loops and telephone service as the power outages lasted for days.

Yep... The theory is, the RBOC has portable generators to deploy as needed. There's a male plug on each to receive same. That works fine for local failures (truck hits power line..). But in a widespread debacle, say Hurricane Andrew, earthquake, ice storm, etc... they have nowhere near enough, nor can they get them deployed fast enough if they did.

("re: Some" above; I've never seen a generator installation at a SLC...)

Subject: Re: SLC power failures
From: Chris Adams (cmadams@hiwaay.net)
Date: Wed Jan 03 2001 - 23:19:21 EST

Reply: Chris Adams: "Re: SLC power failures"

Once upon a time, Sean Donelan <sean@donelan.com> said:
> As power is being restored throughout the midwestern part of the US, one interesting feature showed up. A lot of telephone companies are using Subscriber Loop Carriers (SLCs) to provide service. Some phone companies are using local power sources with limited battery backup for some SLCs. With the extended power outages, central offices with generators maintained service; however, the remote equipment didn't have local generators. This resulted in people losing their local loops and telephone service as the power outages lasted for days.

I've noticed that all the newer BellSouth SLCs in this area (Huntsville, AL) appear to have natural gas hookups. I have been wondering if this is for some kind of backup power. Anyone else seen this? 

Other reading:

Here's some data about Sources of Failure in the Public Switched Telephone Network

Sources of Failure in the Public Switched Telephone Network, pp. 31-36

Nat'l Academy Press, Trust in Cyberspace (1999), Contents

Comprehensive Communications for Emergency Medical Vehicles

Information about premature battery failures in SLCs due to heat

An outdated FCC report about Telco reliability

Florida PSC

Since posting this page, I've had some feedback I would like to share with you:

5/14/2001

As an engineer involved in the design and installation of this equipment I heartily agree with your analysis. Furthermore cell phone site would overload in such a failure even with backup and the whole communications within an area would co down including maybe fire and police who rely on telephone circuits to control their radio systems.

Manufacturers do provide backup battery for estimated times. Telcos should get at least the minimum required by the manufacturer and this requirement must be enforced by both the FCC and the local public service commission.

<Ray> I know this works on a site by site basis, but I'm afraid that most areas could never handle losing power at all location. I don't see any practical way to deploy thousands of generators in 4 to 8 hours. But I'm willing to bet that the first 48 hours will be spent waiting for the rest of the generators to show up. Maybe I'm just a pessimist, but I just don't see how there can be enough generators ready to go. And I wouldn't put money on the extra ones you're planning on will actually be there. Highest bidder gets the power.

<Ray> Do you keep the required special connectors and cables on-hand for these third party generators?

Sure do.  Usually standard electrical supply depending on current requirements but the site must have its backup batteries in place and working and checked regularly. 4-8 hours is a good time usually more is not necessary and a backup generator could get to a site before the batteries go dead You would not need 1 for generators as outages usually are in a small area and most COs only have 2 or 3 remote sites as you show in your web page

Robert Schoenfeld

05/14/2001

Hi Ray, 

Your page is very interesting and hits home on quite a few notes. I work for a small ILEC in the North East. We see our share of power failures due to both natural and man made events. In use in our network are OPM's (Off Premises Modules) which are very similar to SLiC's. They consist of a pad mounted box (no vaults ) with equipment and batteries contained. We have a battery replacement schedule that allows all of these OPM's to have at least 8 hours of standby time under full load. In addition, we have enough portable generators to cover about 50% of these units ready to go at all times. We also have commitments from various local vendors to supply more as needed in an emergency situation. A cross section of other outside employees have been trained in deployment and operations of these units. We have letters of cooperation with electric companies serving our areas making the PSTN a priority in restoration. The equipment housed in buildings is being backed up by a combination of auto-start and manual generators. These buildings also have batteries with larger capacities than we can fit in the OPM's that can last in some cases several days. During the Y2K make ready, we estimated we could run the current backup system for about 2 weeks with out any outside help (except for gasoline) and months if necessary with outside help. I hope this shows you not all phone companies take this matter lightly. Maybe being as small as we are makes it easier logistically to deal with problems. Please feel free to contact me if you have any questions.

Name withheld  

An Australian Communications Authority (ACA) fact sheet about power and telephones.  Almost all of this information would apply in the US.   

http://www.aca.gov.au/consumer/fsheets/consumer/fsc46.htm  

BellSouth explains the need for local power for ISDN circuits:

Does ISDN need its own power back-up-  

A simple explanation about how regular phones might work in a power failure.

Why does the phone still work when the electricity goes out?

10/26/2006

Ray,

Great explanation. I'll keep that link filed away. Based on your email address, looks like we've both had lots of recent experiences with widespread phone outages caused by the loss of utility power.

I've mentioned it before, but I'll bring it up again. Even when BellSouth managed to get generators to the SLCs in 2004, the circuits would go down when the generators ran out of fuel or get stolen.

During Wilma, they did a better job of keeping the generators from being stolen, but thieves would poke holes in the fuel tanks to steal the gas or diesel.

We re-engineered our backup POTS lines to run on copper whenever possible, but some sites only had service that ran through SLCs.

SLCs make it possible to get DSL to places that would be dark otherwise because of distance limitations and they make it less expensive than copper to bring in lots of service to an area, but they aren't bulletproof.

We were given the option to move a bunch of our T1s off fiber and onto copper to avoid the utility issue. We debated it for a while: do we want something that may lose power once every few years when we get a hurricane or do we want something that bounces every time some dog lifts his leg in the wrong place? We opted to stay on the fiber.

Again, thanks, Ray for a great explanation, with pictures even.

--

Ken Steinhoff Telecommunications Manager

Palm Beach Newspapers, Inc.

ksteinhoff@pbpost.com

For my use later...

Somehow the wires got crossed
Communication's lost
Can't even get you on the telephone
Just got to shout about it
I'm losing you
I'm losing you

John Lennon "I'm losing you" 1980

Have anything you would like to add to this web page?   E-mail me, Ray J. Vaughan  Please let me know if I can add your information to this web page, and if I should use your name or not.

Thanks. 

Except for portions owned by others, Copyright: Ray Vaughan, 2008

Comments/Questions