The range of HVAC design options associated with the various types of data centers has expanded and evolved over the past 30 years in order to keep pace with the wide array of server transformations and deployment strategies. With the extensive variations of HVAC equipment, control strategies, and system applications, there are almost unlimited options for the design engineer, or the facility engineer, to consider when expanding, renovating, or building from the ground up. A common thread in our design strategies is the implementation of variable air volume components and sequences into the new or renovated systems. The VAV strategies are coupled with other variable water-flow designs and variable-temperature sequences for a comprehensive approach.
With the increasing emphasis on energy management in conjunction with the ever-present uptime requirements, designers should take advantage of the numerous opportunities that variable air volume systems can offer the building owner or operator.
SERVERS AND POWER DENSITY
Thirty years ago — when most data center designs seemed to have been based on approximately 50 W/sq ft and the server components (or data processing equipment) were more negatively sensitive to warm temperatures and low humidity levels — the precision-type constant-volume HVAC equipment developed explicitly for this market was very appropriate. At that time, most designers and building operators wanted to make sure that the room environment as a whole was maintained appropriately for any server, switch, or other processing machine that might be set into a rack or cabinet. The large rates of air turnover in these spaces ensured the operator and engineer that the overall space was properly conditioned. This type of environmental equipment required reheat and dehumidification components in order to more adequately regulate the space temperatures and humidity levels commensurate with the requirements of the server manufacturers.
Many times, the installed capacity of the HVAC equipment continually surpassed the actual server load, yet more often than not the unit fans were never de-energized. Operating the fans on a continuous basis throughout the year probably seemed like a minor expense when compared to the cost of downtime within the data center. Dedicated management of the perforated floor tiles, and all of the other openings in the raised floor assembly, was generally not a priority with the various operators. Even in recent years, customers of colocation businesses would judge the effectiveness of the HVAC system by the temperature and quality of the environment as they walked up the ramp into the data center, as opposed to the conditions within the cold aisles.
Today, the specific emphasis is now on the condition and quality of the air entering the server. The surrounding areas of the room — even within the raised-floor environment — are of a secondary concern. The data center of today is solely a process cooling application, and if the trend of increased power density within server racks continues, then the cooling systems will continue to evolve into micro-systems dedicated expressly to the process at hand.
MICROPROCESSOR SERVER EQUIPMENT AND INTERNAL VAV FUNCTIONS
The continuous improvements in microprocessor technology, and thus subsequently to server technology, directly impacts the HVAC design. Servers with larger processing capacity generate more heat. Servers that are more resilient to a wider range of temperature and humidity levels allow the data center operators to consolidate more machines into common racks thus significantly increasing the density of the heat gain. The other aspect of new server technology is that manufacturers are providing these machines with integral temperature regulation controls that include variable-airflow cooling fans. As the server utilization load rises, the cooling fans increase in speed to remove the heat generated by the microprocessor core, and conversely they reduce speed as the computation load declines.
VAV TECHNOLOGY – SIMPLE MANUAL OPERATION
Engineers enjoy a wide array of solutions to the problem of heat management and heat removal from the data center environment — specifically, dealing with the heat generated by the servers. The early transitions to cold aisle / hot aisle rack arrangements, in order to more dependably supply conditioned air to the respective aisle, provided a certain amount of thermal consistency within the environment. The numerous variations of cold aisle containment, hot aisle containment, in-row cooling, high-density cooling, and rack-mounted cooling have also significantly improved the way the process loads are served. The incorporation of variable volume airflow technology into the above HVAC systems, in order to parallel the internal server cooling operation, has provided significant improvements to the stability of the process environment.
In a project for a colocation company many years ago, we worked with the critical systems managers to provide a design that was deliberately more attentive to maintaining cold aisle conditions utilizing the original overhead ductwork systems from traditional watercooled computer room A/C units. The facility managers provided a partial cold aisle containment barrier (no overhead ceiling), and our duct design directed the supply diffusers to discharge directly into the cold aisle. No hot aisles were treated. The supply ducts were manually balanced depending upon the load, but this balancing process was not precise, nor was it accomplished in real time. We also found that the cold aisles were way too cold for the operators, but of equal importance, these aisles were way colder than necessary for the server equipment. When supply air temperatures were reset to more reasonable levels (between 65°F and 75°), the aisles were much more tolerable, the servers operated without issue, and the compressors in the CRAC units were operating less. The supplemental free cooling coils in the units were also able to be utilized for more hours in the year, since the supply air temperatures could be maintained with warmer condenser water temperatures. This manual VAV system was rudimentary, but it provided the owner with an economical solution to an existing situation, and it also helped to validate subsequent designs where the sequences and operations were fully automated.
TRADITIONAL VAV TERMINALS AND SUPPLY DUCTWORK
Various applications of traditional air distribution systems, utilizing variable airflow terminals, have been implemented across the country over the years. When server density loads are more moderate, this type of system affords the owner some flexibility in managing the environmental load while allowing for a certain amount of energy management. One renovation project that we participated in included a 40,000-sq-ft raised-floor environment with approximately 188 VAV terminals served from 14 rooftop A/C units. The low pressure ductwork from the VAV boxes served long sections of linear air devices and the associated supply air plenums. The original operator/builder had identified rack layout to the design engineer, who then proceeded to target the resultant cold aisles with airflow. The ceiling was a large, common return air plenum with traditional “egg crate” style return registers over the hot aisles, or directly over the server racks. The individual VAV terminals utilized ceiling-mounted temperature sensors above the return air registers in the respective areas to modulate the control damper.
The existing HVAC infrastructure was so extensive that we worked with the owner/operator to maintain the airside systems, replacing the rooftop air cooled A/C units with roof-mounted chilled water units and a central water cooled centrifugal chiller plant. This saved a significant amount of first-cost expenditure, reduced the construction period and provided for a substantial improvement in operating costs related to mechanical cooling. The rooftop air handling units were also provided with airside economizers for winter operation, and small humidifiers to maintain a minimal space humidity level.
Design and control issues included the acceptance of a wide ranges of humidity levels due to the influx of fresh air in mild or cold weather conditions and the limitations associated with large air devices (and the associated supply air plenums) that cannot be easily moved to accommodate changes in rack layouts or cage installation. In applications with higher server power densities, this design would become burdensome to the operator, as the overhead distribution could not be easily targeted to specific servers or racks.
However, in an environment where the density was limited to between 75 W/sq ft and 100 W/sq ft, and there were no rack loads in excess of 4 kW, this design was suitable for all of the noted reasons. Managing the use of the humidifiers, in order to retain the energy savings associated with the airside economizers, was a very important strategy that incorporated supply air reset, increased space temperatures, and the recognition by the owner that the old forms of the standard lease agreements needed to be modified to allow for wider fluctuations in space humidity levels.
HIGH DENSITY APPLICATIONS AND VAV COMPONENTS
The introduction of the in-row cooling concepts, utilizing cold aisle containment, has proven to be a great way to utilize existing chilled water capacity — such as piping services under a raised floor — in order to increase the white-space efficiency of server deployment. A recent project in the Midwest included the use of Emerson / Liebert in-row, chilled water CRV modules. The existing chilled water system utilized air-cooled chillers, which was a limiting factor in reducing PUE.
The owner had a desire to install a new application in their data center, but they had limited floor space. They agreed to increase the average density of the various racks within this application to approximately 12 kW, and install a partial cold-aisle containment barrier (again, with no ceiling). They had traditionally managed their power densities to around 5 kW per rack. The design included a series of in-row cooling modules, strategically placed within the server rack line-up so as to afford adequate air distribution to the left and right of the cooling units. The new system was set-up with an N+1 configuration. The engineer and owner were uneasy about this new application and included a number of perforated floor tiles in the cold aisle. These proved to be unnecessary and have subsequently been removed.
The in-row coolers could operate in many different modes, and in working with the manufacturer, a more refined sequence of operation was developed during startup and in the prefunctional testing. In the final setup, the cooling unit controllers modulated both fan speed and chilled water control valve position in unison, in order to maintain a 75° cold aisle and a minimal temperature gradient from the lowest server, to the highest server, in the racks. The use of integrated temperature sensors as provided by the manufacturer was instrumental in managing the environment at the inlet to the servers. This system does require the operators to install blank-off panels in the racks where no server exists, but most new racks have options for these panels.
We incorporated an extensive array of temperature sensors, pressure sensors, gauges, and other monitoring devices in order for the owner to track the effectiveness of the installation. The final commissioning included rack-mounted electrical load banks spread throughout the installation so that the owner could vary the location and intensity of the load. A series of unit failure scenarios were implemented. The in-row coolers adjoining the “failed” modules smoothly increase the fan speed to account for gradient changes and increases in discharge air temperatures.
The original design parameters included utilizing 45° chilled water supply with a 12° temperature differential, plus a discharge air temperature of approximately 55° (all of which was based on how the owner was operating the existing plant). This proved to be too conservative. As in the previous example, the cold aisle temperature was too cold for the operators and unnecessarily cold for the servers. Field adjustments to maintain a 70° discharge air temperature and a 16° chilled water temperature differential provided an acceptable operating condition. The controllers were varying the fan speeds as the local load increased or decreased. The incremental variations of the fan speeds provided a steady cold aisle environment.
TRADITIONAL CRAH UNIT APPLICATIONS WITH VAV INTEGRATION
Chilled water CRAH units are increasingly installed (or retrofitted) with VAV components. These applications include the use of either electronically commutated (EC) fans or variable frequency controllers on standard centrifugal fans. In traditional raised floor environments, these units can provide effective control of the environment while more efficiently managing the fan energy. When combined with underfloor differential pressure sensors and supply air temperature sensors, these new assemblies can modulate airflow to maintain only the most necessary static pressure levels in the raised floor plenum.
In a raised floor expansion project in the southeastern U.S., we worked with the owner to develop a VAV sequence of operation for the new CRAH units. The expansion space is approximately 5,000 sq ft within a much larger facility, and is constructed to be an independent zone, environmentally isolated from the other raised floor spaces. The construction requirements include diligently sealing the perimeter walls at the concrete floor, eliminating plenum losses at the PDU modules, sealing the edges around the CRAH unit floor stands, and minimizing floor tile penetrations due to the structured cabling services.
Each CRAH unit is installed with a remote differential pressure sensor which is used to generate an average differential floor pressure. The units then modulate to maintain the desired setpoint. These sensors are integrated into a central controller, through the communications loop, so the normal operation is for all the unit controllers to modulate the respective EC fans based on the averaged value. In the event that the central controller fails, or if a communications failure occurs, the individual units will operate to maintain the raised-floor differential pressure in the local area around the CRAH unit.
As the facility operators install new server racks, the floor grates with integral dampers are set into position and adjusted open. As with the first example above, the balancing of the actual airflow rate at the server rack is a manual operation. However, this system design provides for an automatic modulation of the main system airflow (and the chilled water control valve). The use of EC fans allows for an added control feature that spins the fans to create a virtual isolation damper in the event that the raised floor has not been fully populated and the actual heat gain load is low, where disabling a unit altogether is the best decision energy-wise. The EC fans can operate in this minimum speed mode to prevent air under the raised floor from moving back through the unit. This integrated control mode provided by the unit manufacturer utilizes far less energy than the standard backdraft damper with an electric actuator.
CONTINUING DEVELOPMENT OF NEW VAV SYSTEMS
The increasing cost of electrical energy, combined with the ultra-high power consumption of data centers, has generated a significant and varied number of HVAC solutions. It is safe to say that the development of more efficient systems to maintain the server environment is ongoing. Many of the original CRAC unit manufacturers (e.g, Liebert, APC) have added products to their offerings which significantly change the way conditioned air is delivered to the server racks. Overhead high-density cooling systems, VRF systems, and in-row cooling modules all incorporate VAV functions. New manufacturers such as Inertech/Skanska are offering innovative ideas and equipment for the higher-density rack applications that incorporate VAV technology.
It is important to note that the use of VAV systems and sequences, when properly applied and utilized, have been proven to reduce energy costs over the life of the HVAC system. However, as with all projects that include technically superior systems, the first-cost considerations can be oppressive. A well-researched life-cycle cost evaluation should be implemented in order to prove the worthiness of the particular application. Simplicity of maintenance is also a significant consideration.
As IT managers and co-location owners wrestle with the various decisions regarding effective deployment of server equipment, the mechanical engineer has a wide array of equipment selections to choose from. Each data center is distinct, and the one-size-fits-all strategy is not an effective, professional approach. Our experiences with hot aisle containment, cold aisle containment, in-row cooling, high-density cooling, and traditional CRAH/CRAC unit cooling methods have all been positive — when the VAV solution is properly applied and the control sequences are thoroughly developed. ES
