Data Centers, Skydiving & Downtime
There are situations when a mission critical system failure would be more horrible than we want to consider in our day-to-day thinking.
One of my favorite “downtime” activities is skydiving. The airplane climbs to an altitude of 12,000 ft, I dive out of the plane door, and the force of gravity accelerates me toward the predictably hard earth at speeds of up to 160 mph. And yet, during this anti-survival instinct sport, I have total confidence that my equipment will not fail me. How can this be?
My sense of security comes from the knowledge that I have not only my main parachute and a reserve chute, but also a small computer that would automatically deploy the reserve at a preset altitude if I passed out or simply forgot(!!) to pull the ripcord.
These key functions are interlocked and automated to preserve the parachute’s function and reduce the chance that human error will interfere. Despite the extra cost in materials and the room that this safety equipment takes up in the skydiving backpack, they are now mandated. For example, while the jumper can pack the main parachute, the US Parachuting Association requires that the automatic deployment device and reserve parachute be inspected and reset every six months by a licensed technician.
Henry Ford said, “Failure is only the opportunity to begin again more intelligently.”
That may be true, but what is the potential cost of that opportunity if the failure is the HVAC system in a mission critical facility?
Most businesses with mission critical facility infrastructures will experience financial losses when the network goes down due to power outages or other facility-related problems, regardless of how long the disruption lasts. According to the Disaster Recovery Journal, each year one in 500 data centers in the U.S. has a disaster resulting in network downtime. In this report, the cost of each shutdown ranged from $350,000 to $11 million with an average loss of $5 million. After the failure, 43% of the companies went out of business immediately and an additional 29% shut down within two years.
Hurricane Katrina taught us that when the facility relying on mission critical infrastructures contains people, the loss of human lives in a natural disaster can be unthinkable. In the 2005 tragedy of New Orleans, not all of the infrastructure losses were from wind and water destruction. Many IT circuitboards were destroyed from corrosion in high-humidity rooms as a consequence of air conditioning system failure. These interruptions might have been avoided had there been functioning redundant and automated backup systems.
As with skydiving equipment, the enormous consequences of a system shutdown are incomparable to the relatively minor cost of preventing the failure by careful design and preparedness protocols. What HVAC maintenance and operations procedures can be in place to ensure power redundancy and automation of backup systems without wasting energy or reducing system reliability?
Here are some mission critical facility automated essentials.
- Automatic transfer switches (ATS) to sense a utility power failure and start a backup generator for emergency power.
- Uninterruptible power supplies (UPS) with regularly tested batteries.
- Power distribution to the floor using power distribution units (PDUs) and remote power panels (RPPs).
- Load banks to ensure that the standby power system can accommodate the emergency load
- A fault tolerant and redundant control system for generators, switchgears, UPS systems, chillers, fire alarms, security, and other mechanical and electrical systems.
- Prior planning to connect temporary generators, fuel storage, UPS systems, batteries, load banks, chillers, and water to support your facility without a total building shutdown. ES