- Support hardware and software for large mainframe systems;
- Statewide telecommunications network;
- Specialized services such as supercomputer and geographic information systems; and
- LAN and network design.
In March of 1996, Teale issued an RFS to design and construct, per state specifications for a fixed price, a new data center. The center was designed to exclusively house computerized data processing operations with a large central plant, uninterruptible power systems (UPS), standby power systems, and administration and staff offices. The center consists of 150,000 sq ft of space, with 50,000 sq ft of access flooring supporting computer systems.
The goal of commissioning was to document the design intent, ensure equipment was installed correctly, verify equipment and system performance, train the operating staff, and institutionalize all documents. Our efforts at the Teale Data Center proved difficult at times, rewarding at others, but ultimately successful.
The business and financial impact of a data center outage is enormous. Teale supports many critical activities. One example is the California Highway Patrol; consequently, any disruption has the potential of placing an officer's life at risk. Hence, the term 'mission critical' has been applied to this type of facility.
Commissioning should be a requirement during the construction of facilities such as these. Here, the commissioning process added extraordinary benefit for the owner and will contribute to the long-term success of this facility.
Additionally, the lessons learned will allow us to refine our process.
The Design GoalsThe owner established interesting design goals for this project, such as:
- Failure of any single component of the mechanical or electrical system could not impact the computer loads;
- All equipment supporting the computer load must be able to be serviced and/or replaced without interruption to the computer operation; and
- The central plant must be expandable without interrupting the computer operations.
Establishing Design IntentThe state had issued its basic design intent in the RFS. Initially documenting the design intent was a matter of extracting information, clarifying it, and inserting it in the commissioning plan. However, many changes were made during the D-B process that had to be incorporated in the commissioning plan.
Additionally, even though the design intent was detailed in the RFS, the D-B team often determined the design basis. Practically speaking, these changes were made on a daily basis, but the D-B process did not lend itself to the rigor of detailed documentation in a timely fashion. Written substantiation of these changes proved difficult to obtain.
Several key design intent and basis of design examples are delineated in the RFS design intent statement:
"This facility is required to function 24 hours per day, seven days per week. All systems should be provided with redundant components such that the failure of a single component will not inhibit the capability of the mechanical system to support the operation. In addition, redundant components in the system must be designed to eliminate any single point of failure. All equipment must be able to be serviced and/or replaced without interruption of data operations. Central plant systems must be expandable without interruption of data operation."
During the course of commissioning, the bas and fuel oil delivery system interface failed the criteria established in the RFS. Testing disclosed that failure of power and/or data communications to the bms panel controlling the fuel oil delivery system caused all pumps to stop running.
The bms was reprogrammed and the basis of design was changed to read: "If there is a failure of electrical power or data communication to the bas controlling the fuel oil pumps, all fuel oil pumps will run."
Though this might sound innocuous, the end solution, with proper documentation (the letter from the mechanical engineer ensuring that running all pumps simultaneously would not be detrimental to the pumps or the system) took many months to obtain.
Also affecting the basis of design was the elimination of the requirement for input isolation transformers feeding the vfd's. Commissioning revealed that the required isolation transformers were not installed. The owner negotiated with the D-B team and accepted the vfd's as installed. Then appropriate documents supporting this agreement had to be obtained and added to the commissioning plan.
Reviewing Equipment InstallationThe next step in the process was to ensure that equipment was installed correctly. Our methodology was to develop equipment-specific, prefunctional test forms and complete those forms during site inspections. There was significant resistance to these inspections. Resistance manifested itself in two ways: refusal to collaborate in the development or review of the test forms, and scheduling of inspections for equipment and systems that were not complete.
Through persistence and the owner's pressure, we did however manage to break through this resistance and implement our requirements. The issues discovered during the process confirmed our belief in the need for these requirements. Following are a sampling of items that were identified during the prefunctional inspections:
- Supply fan vfd's were installed in the fan motor compartment of two heating units causing restrictions and turbulence at the fan inlets.
- A vav terminal box was installed upside down.
- Incorrect motor heaters were installed on two fancoil units.
- Input isolation transformers were not installed on any of the vfd's.
- Condenser water circuit-balancing devices were missing on each cooling tower circuit.
- Makeup water to cooling towers was installed through a single pressure-regulating valve, thereby creating a "single point of failure."
- The bms graphic user interface display did not agree with actual floor layout of computer room a/c units.
- One chilled water valve was installed backwards.
System/equipment TestingThe third and most important step was the testing of equipment, systems, and interactions between systems.
Virtually all of the items identified had the potential for catastrophic effects on continuous operation of the data center. Following are major issues discovered during the system tests:
- Two air handlers did not meet design airflow requirements.
- On these same air handlers, the hand/off/auto switches did not control unit dampers and chilled water valves; they only start and stop the unit fans.
- Four computer room air conditioning unit motor overload relays did not work.
- Two chilled water pumps that were wired to UPS power, failed during power transfer because local C panel was powered by non-UPS power.
- Several critical systems failed to operate when power or data communications from bms panels were lost.
Interaction Between SystemsWe have found that the most difficult concept to understand, and most important to test, is the interaction between the electrical and mechanical systems.
For example, a mechanical engineer typically does not understand the maintenance requirements of a circuit breaker. Industry and professional standards recommend that circuit breaker testing be performed every three years. Circuit breaker testing requires the circuit breaker to be removed from service. How does this impact the electrical system? If the circuit breaker provides control power to a bms, and the bms control panel's power is 'daisy chained,' this could have enormous consequences.
Since typically during the design process electrical and mechanical engineers don't have detailed discussions on their respective system's interactions, there is a risk element introduced into the process. Someone that is knowledgeable in the interactions of these systems needs to guide this process.
Lessons LearnedSo what were the lessons learned?
- At the onset, everyone must agree in principal that commissioning is beneficial and that they intend to cooperate fully.
- Reality eventually sets in. Everyone has a different vision about individual roles and obligations during commissioning.
- The engineers and contractors realize they have not budgeted sufficient funds to meet requirements.
- The commissioning process is viewed as an impediment to the schedule.
- Builders hope that commissioning engineers will go away, by benign neglect.
How To ImproveWe offer the following suggestions for improving the commissioning process:
- The owner should retain the commissioning agent directly.
- Thoroughly detail the commissioning requirements in the specifications.
- Discuss the commissioning requirements of all parties, as early in the process as possible.
- Specify and enforce damages that result from not meeting the commissioning specifications.
- The commissioning agent should budget enough money to spend a great deal of time at the jobsite a minimum of two days a week.
- The commissioning agent should differentiate between what he has the authority to enforce vs. what has value to his client.
- Be persistent.
And finally, near a project's end, there is great difficulty in retaining the interest of the design and construction team. Their interest now lies with the next project. The institutionalizing of accurate 'as built' documents and proper training is essential to the successful operation of any facility. Don't walk away without them. ES