It’s ok we have a standby generator……
Many organisation protect themselves against power cuts by use of a standby generator. for their data centres they will use a UPS (uninterruptable power supply)combined with a standby generator. Often these can provide false comfort as they don’t actually work on the day for a variety of reasons, ranging from the generator being unable support the whole building ‘on load’ as the testing was done at the weekend, to the UPS and the generator not being configured properly and both failed simultaneously.
I have heard a statistic, which may or may not be true, that 40% of standby generators fail to work when required. The lesson is that if you rely on a standby generator or UPS you need to do lots of testing to make sure it will actually work on the day.
Sarah Armstrong-Smith sent me an excellent example of this…
“This happened about 7 years ago.There was a power cut affecting the local region where one of our primary Data Centres was situated. The UPS and generator kicked in as they should have. However, the generator started kicking out plumes of thick black smoke.
It wasn’t long before the Police & Environmental Health were at the door ‘ordering’ us to turn the generator off as the smoke was going over a motorway and causing danger to vehicles. We begged and pleaded and told them this was our Data Centre but this fell on deaf ears…so we agreed to start turning off non-essential services to try and reduce the load…off went all pre-production and test environments. It made a little bit of difference but not enough.
We thought about invoking DR for some of the systems, but this is back in the day of tape based recovery so it would have taken the best part of a day+ to recover services, so we closed down a few more services until we were left with the really critical ones!
Meanwhile we were frantically trying to contact the Electricity company to find out what had happened and when power would be restored, to which the response was ‘we have Engineers working on it’….not a very helpful response!
At the point where we had no choice but to shut the generator down and close the Data Centre…. the power came back up! Can you believe it! So we thought we had a lucky escape until lots of complaints started flooding through from local residents…the black smoke had covered their houses in soot, gone through the windows on to furniture etc. Not to mention the noise and smell. So that started a flood of compensation claims…
As you can imagine, the media picked up the story and were not very sympathetic!
The key lessons learned here are to really test your generators properly and on load. Most companies just switch it over for a few minutes to check it fires up and then turn it off again….which we were guilty of, but we thought we were smug in the knowledge that we had a generator and tested it!
Also, given the fact that we were in the middle of a residential area, I suspect that the noise alone coming from the generator would have been enough to force us to turn it off at night time…meaning that we probably would have no choice but to either invoke DR or stop running a 24*7 operation…. the cost of which would have been disastrous!
After the event, the strategy was reviewed including relocation of the Data Centre, but this proved too costly so our over reliance on generators had to be reviewed instead including improving DR plans and the ability to move services to other Data Centres. I think this is what kick-started our use of data replication to speed up recovery”!
