Hurricane Wilma Restoration Info Blog

Florida Power & Light spend several hours yesterday removing the damaged poles that have been keeping us from utility provided power. They then installed new poles and brought the downed lines back up. The process is, actually, quite amazing and included about twenty-five contractors brought in to work on the storm damage (See Photos). Many of them were from Virginia and Canada.

In any case, utility-provided power was restored to our segment of the grid yesterday. We will remain on Generator Power, however, because Florida Power & Light has notified us that they will have to take this segment down at least one more time to finish connecting some additional buildings on our segment.

We will continue to review the situation with Florida Power & Light and make a determination as to when we can return to Utility Power full-time. The "Power Patrol" would sure be happy if there was no longer a third shift.

The last few days have almost been business as usual. The key word is ALMOST. There are still several people at headquarters who have had many sleepless nights.

Our I.T. Department is still working 18 - 22 hour days building new servers and installing them at the NAP of the Americas in Miami. In fact, the crew of three that went down to the NAP last night at 9:00 PM, arrived back at the office at 5:00 AM.

Additionally, one lucky headquarters employee each night gets the task of "Power Patrol"; The "Power Patrol" is responsible for walking around the building once every ½ hour to check the property for suspicious individuals (remember there are still many in this area without power who would love to have our generator) and to check on the status of the generator. That person reviews the voltage output, amperage output and number of cycles (Hz - for computers to operate properly the generator must steadily put out 60Hz, and at least be within 2 Hz either way). The "Power Patrol", also, checks to see the level of the diesel remaining in the unit's tank. This may be the first day I've mentioned the nightly "Power Patrol", but there has been one every night since the generator was brought to the facility on Sunday.

Some nights, especially when it's hotter, the generator uses much more fuel than on other nights. Why, you may ask? The Air Conditioner. The data center is kept as cool as possible, and while the A/C is ALWAYS running, some nights (and days, for that matter) just require more A/C than others. The more electricity being pulled by the A/C, the harder the generator must work to keep up with the electricity requirements, the more fuel is used. Of course, that's not the only reason for the generator using more or less fuel, but it's a big reason.

Each night, the "Power Patrol", waits for the nightly Fuel delivery to arrive. And while waiting, prays for there to be enough fuel to last until they come. As was mentioned in a previous blog entry, we use about 200 gallons of Diesel Fuel per day to power the generator, but the generator's tank is only 100 gallons. So that means most days, we receive 2 to 3 deliveries of fuel. In reality, when we get down to 40 gallons, the delivery company is called, and within two hours they deliver.

Yesterday, there were many developments after the daily blog was written:

As we re-launched yesterday at 5:30 AM, our I.T. Staff along with our Director of Operations sat in our Ft Lauderdale Datacenter's Network Operations Center and watched every server. It was amazing to see how during the first hour after the site was brought online; over 100 unique visitors were visiting one model place each minute. By 6:30 AM, over 6,000 people were visiting the site at once. By 5:00 PM, over 17,000 people were using the site at once.

As we mentioned yesterday, we’re running on only ONE of our Four Internet connections due to storm related damage. This one Internet connection provides us with 45Mb per second of transfer; OMP normally operates with 3 times that, plus our connection at the NAP; Needless to say, we were starving from lack of bandwidth.

Today, we were informed that the second DS3 line serving the main datacenter would not be brought online for several weeks. As I mentioned yesterday, this was due to a fiber cut during the storm. Not to fret though, this evening, we WERE able to restore our 100Mb fiber connection from Data Center 2 in Ft Lauderdale to the NAP in Miami. As I type this blog entry, our I.T. staff is working diligently to route traffic over this connection.

Today we purchased the hardware required to build 20 new servers to be installed down at the NAP in Miami. We expect them to be in place within the next 7 days.

At about 8:00 PM yesterday, utility provided power returned to Data Center 2, OMP Studios and Our Print Shop. This has allowed our print department to begin working on print orders that were ordered before last Monday’s crash, and begin to accept new orders. Please bare with them as they play catch-up.

All Headquarters Staff reported to work today as if it was a normal business day; of course, our I.T. Staff and Director of Operations have been working almost 4 days straight with 1 or 2 hours of sleep a day.

This evening, at about 9:00 PM, our Director of I.T., a couple I.T. staff members, and our Director of Operations, drove down to the NAP in Miami to begin installing four new servers and that have been built over the last couple days. They will be brought online within the next day or two after some testing. Some additional hardware was installed for routing and switching.

As I walk around the NAP facility, I truly begin to feel confident that it is the right facility for us. Other documents we have published have noted that it is a Tier 1 Facility. I’m sure many do not understand what that really means. The largest internet backbone providers operate out of this facility, Walking down the rows and rows of servers and connectivity hardware, you can truly see a Who’s Who in the internet industry. Bellsouth, Centennial, Cogent, Expedient, FPL FiberNet, GlobalCrossing, Qwest, Spring, AT&T, WilTel Communications, Teleglobe, and MCI/Worldcom. That is just a list of some of the Telecom Companies, and really only the start of it.

It is looking at some of the other companies that truly build my confidence. I, really, only have to mention three: Google, Yahoo, and America Online (AOL). It’s quite funny to sit there and look at Google’s Cage and Yahoo’s Cage. They are direct competitors and their cages look directly at each other.

We are making steady progress towards recovery and expanding our infrastructure and in facilities that allow for almost unlimited bandwidth, space and expansion.

Our Telephone lines are still down at the Headquarters office, and BellSouth is telling us that it will be at least two weeks before they can repair. Today, we were able to forward the lines to a temporary Vonage IP Phone and setup an outbound message. Over the next week or so, should our lines not be restored, we are prepared to convert over 100% to IP Phones and restore our customer service via telephone. This conversion to utilizing the true strength of IP Phones was going to be made within the next two months anyway, we are just expediting this switch over should we be required.

We’re still operating on Emergency Generator at the Headquarters office. During the first 24 hours, the generator required over 250 gallons of diesel fuel.

On a final note, Thank you for your kind words of sympathy, praise and encouragement. Throughout the last week (an of course over the last 10 years), our one priority has been to provide quality service to our members. You are always on our minds and it is good to know that you are thinking of us too during this tough time both personally (as it related to our staff) and as it relates to our business.

The site was launched at 5:30 AM. Please understand that we are continuing with our plans to move to the NAP in Miami, there may be downtime within the next few weeks; we also cannot guarantee the quality of the generator power.

Due to the lack of connectivity via our 100mb fiber run to the Nap, we are running at less than 1/3 of our regular bandwidth. The site may be slow, however, please know that we are constantly working to bring all our services back up to spec, or better, as soon as possible.

Our Number one goal was to get the site up and running.

This blog will continue from this point forward as new information becomes available on any of our plans to upgrade our infrastructure, servers, and features.

Please stay tuned to this page for additional information as it becomes available.

The I.T. Team comes in early to again, power up all the servers one by one and make sure they are all in working order. They are required to build two new servers because they were damaged during the storm. (We believe a UPS - uninterruptible power supply shorted and then allowed the current to go directly to the systems and therefore frying the power supplies and other components.)

At around 7:00 PM, the generator arrives and is installed.

The Electrician disconnects OneModelPlace from the Main electric grid (which still isn't up), and connects us to the generator power.

The staff turns up one circuit breaker at a time to ensure that there are no faulty wiring and no voltage errors being created by the generator.

For the next 10 hours, the I.T. department works on bringing up each and every service necessary to launch OneModelPlace.com.

OneModelPlace's Director of I.T. arrives in Ft. Lauderdale to begin moving Datacenter 1 down to the NAP in Miami. Unfortunately, we are told that the additional space, power and bandwidth in our cage will not be available until Sunday.

Another team arrives in the neighborhood from Florida Power & Light. In reality, we could have been restored, but instead, due to the damage we had found on Tuesday, Florida Power & Light cuts the circuit right before OneModelPlace's Datacenter 1 & Office Building, as it was the only place they could cut before the damage. The Studios and Datacenter 2 were set to be restored.

A generator has become available in West Palm Beach from an amusement company who was just taking the generator off line from another company whose power had been restored. They, also, agree to provide diesel fill-ups every 12 hours. The generator will be available Sunday afternoon. The Management agrees that it would be best to get the site back up and running in Datacenter 1 on Generator Power, and then build new servers (as initially planned before the storm) to bring down to the NAP.

Florida Power & Light sends out a "Survey Team" to assess the damage to our Segment of the electric grid. They tell us nothing. 100,000 businesses and residences get their electricity restored, in the Ft Lauderdale area, however, OneModelPlace does not.

Diesel Fuel begins to become available in Ft Lauderdale, as more and more gas stations and Diesel distributors begin to regain power from the electric grid, or provided power by a generator. Port Everglades, the main port in Ft Lauderdale, also opens which allows diesel brought in by ship to market. Unfortunately, many generators that were available earlier in the week had been already rented to companies in Miami, who had been well ahead of Ft. Lauderdale in having gas stations and diesel distributors powered, or had been rented to the gas stations themselves.

All of OneModelPlace's Headquarters staff that is still in the Ft Lauderdale area is contacted via any available method (call, drive by, through others) and is asked to report to work. This is actually the first time that we were able to see if all of our staff members were ok; Many were truly out of contact for three plus days. While many of the non-I.T. personnel work outside to clean up the down trees and property, the I.T. staff began testing individual servers to ensure they are operational and can be moved to the NAP in Miami. They work in the dark using only Flash Lights and when available a loaned, low-power generator to power each computer separately.

Using low-powered generators, The I.T. staff tests the Internet connectivity and connectivity between datacenters. OneModelPlace's Primary Datacenter utilizes two 45mb DS3 lines from BellSouth with direct connectivity to the Internet. The Primary Datacenter is also connected to Datacenter 2 with 500 ft of Building-to-Building fiber optic cabling for both low-latency connectivity within our own network; and to be utilized as a backup should both DS3's be un-operational in the main datacenter.

Besides being connected back to the Primary Data Center, Datacenter 2 is also connected via a 100mb fiber-optic line running direct to the NAP of Americas in Miami.

The 100mb line between Datacenter 2 and the NAP were un-operational due to a fiber cut somewhere along the 60-mile stretch. Additionally, One of the two DS3 lines was un-operational. Should OneModelPlace be able to power the building with a stable generator, or begin to receive electricity from the public grid, the site could be brought online with at least limited bandwidth, assuming all servers were operational.

Tap water begins to flow in Ft Lauderdale; for the first time in 4 days our staff is able to flush a toilet, although the water is still isn't drinkable. Many staff members are able to take their first shower since before the storm - however, water heaters were not running since there was no natural gas lines and no electricity. So, needless to day, they suffered with a cold shower.

OneModelPlace's Director of Information Technology, who is stationed in Atlanta, finalizes emergency arrangements with NAP of Americas in Miami to provision additional space, bandwidth and power for all our servers to be moved to their facility beginning Saturday. It would still take a few days after the move-in to configure the servers to operate in the new facility.

OneModelPlace's Director of Marketing, who stayed behind in New York City after OneModelPlace's Modelween event due to the hurricane approaching Ft Lauderdale, calls Florida Power & Light and due to the international audience of our membership pleads for a priority to be given to our segment of the electrical grid.

Due to the time span between today and Saturday, OneModelPlace's Director of Operations attempts to find a generator from out-of-town that will power the entire Office and Datacenter. While generators are available as close as Orlando and as far away as Houston, the mass amounts of diesel fuel required (approximately 200 gallons per 24 hour period) to run them is not (because there is no electricity to operate the pumps at normal gas stations and most delivery companies are not operational yet).

A new Authoritative Domain Name Server is installed in our Cage at the NAP of the Americas, The OneModelPlace.com domain name is pointed to it - this will allow us to put up a temporary message explaining our current situation at OneModelPlace.com.

A State of Emergency is declared by Governor Jeb Bush in Palm Beach, Broward and Miami-Dade Counties. (Two of which affect OneModelPlace's operational ability.)

The IT Staff assesses damage to the offices, data center, studios and electric infrastructure in the area. No major damage was done to OneModelPlace.com assets, however, the strong winds of Wilma ripped off many copper tiles from the roofing and the side panels of the roof-top air conditioners. The roof-top air conditioners control the temperature in OMP's Primary Datacenter, it is a requirement that these are repaired before any servers could be started up.

Ninety-Five Percent of Broward County is in a total blackout. The staff found 3 electric poles that feed the main data center down. While it is hopeful that electricity could be restored quickly, the downed poles could add days or weeks to the restoration of electricity from the main electric grid. Florida Power & Light estimates all service will be restored to affected businesses and residences by November 22. Drinking water is also unavailable, the city utility shut down the tap water service after their emergency generators run out of gas causing the water in the system to be undrinkable.

As a result of the current state of the Ft Lauderdale area, I.T. staff begins planning for restoration of OneModelPlace.com and evaluating options to facilitate expediting the server migration and contingency plans which were initially being executed on a more aggressive schedule.