Too much fuel for trading floor

There's a sequel to the earlier blog entry where the generator failed due a poorly wired fuel supply.

Clearly some engineering works were necessary to ensure the fuel pump supplying the generator still had power during a mains outage. Some additional power cabling had to be run down to the basement to feed the fuel tank pump. At the same time the engineering company undertook some routine maintenance on the roof mounted generator. It was decided to run some tests to ensure the generator would start properly and that fuel would be properly supplied from the basement even if grid mains power was removed. Unfortunately it was decided to run the generator test during the trading day, presumably to save costs on engineering cover rather than paying overtime rates.

The traders were warned they might hear some slight noise from the generator, but there would be no interruption in power to the trading floor or other services. The generator test was scheduled for a quiet period in the trading day. The generator started flawlessly and ran well for at least 20 minutes. Stop and restart was tested. All appeared to be going well. I hadn't mentioned earlier, but the trading floor was on the top floor of the building and the generator was mounted on the roof above the trading floor.

People first started to notice problems when diesel fuel started to drip then pour through the ceiling of the trading floor on to the trading desks and traders below. Some urgent investigations discovered that diesel fuel was pooling in the containment bund/wall surrounding the roof mounted generator. During the maintenance work someone had left the end cap of the fuel pipe for the generator, unfortunately enough fuel was reaching the generator for it to continue to run. Small cracks in the roof below the generator was allowing fuel to leak through on to the trading floor below. The fire brigade were called. They had great difficulty in evacuating the traders who did not see why they should worry about a "little bit" of diesel. We had to close that part of the trading floor, but some urgent support staff intervention we were back in business within two hours.

The moral of this event? Don't run tests in a live environment unless it is really unavoidable. Potential saving on man hour costs can soon get wiped out by the consequences of an outage.