I’ve had a handful of jobs in this intriguing industry of building automation. The first one was actually working for a consulting engineering firm. In my office, specifically, a lot of our fees were earned as a third-party commissioning agent acting as an owner’s representative for local universities on new laboratory buildings. Generally speaking, plumbing, electrical, HVAC, and other systems in laboratory buildings are significantly more complicated, in many ways, than typical commercial office buildings. One of those ways is in the HVAC control systems. Here is a story of how an issue with a single $50 dollar temperature sensor’s reading caused a multimillion dollar laboratory building to behave very strangely.
Commissioning agent firms often include “seasonal testing” in their scope of work, during which they’ll work with building owners to see how the building is operating in the opposite season from which it was tested at the conclusion of construction. In this example, construction wrapped up over the summer, so our visit was in the wintertime, and I was watching the weather forecasts to get as close to a “design heating day” as I could get. I visited the building as part of this seasonal testing, and I worked with one of the university’s staff members, Jeff (who I’m about 99% sure has since retired).
This building includes wet labs with chemical containment fume hoods. Virtually all of the laboratory spaces are served supply air by three large variable air volume (VAV) air-handling units manifolded together, totaling around a maximum of 150,000 cfm of 100% outside air, which are scheduled to operate 24/7. Also serving the laboratories is an exhaust system including multiple fans whose speed varies to maintain exhaust duct header pressure and has an outside air bypass to maintain stack velocity (also operating 24/7). The air handlers feature chilled water coils for cooling and hot water coils for preheat with local recirculation pumps at each air-handling unit.
Jeff and I made it through a relatively uneventful morning of running groups of terminal units in full heating mode, so I could do my coil performance calculations. Returning to the site after lunch, we walked up to the building and tried to open the door to one of the stairwells and found that neither of us could pull the door open — it would move a little but only a fraction of an inch. After some head-scratching, we both pulled on the door at the same time and eventually managed to get it open. A gust of air rushed past us into the building, and it became perfectly clear — this building was under some incredibly negative pressure.
Jeff and I jumped into action and bounded the steps up four stories to the penthouse mechanical room. There, we quickly figured out that all of the laboratory air-handling units were off, and the exhaust fans were running full-tilt. We checked the graphics on the BAS front-end and noticed that all three of the units were shut down because of the freezestats and the exhaust system couldn't meet the stack velocity set point or the duct pressure set point.
So, what happened? One of the most important sensors in a building automation system is a dry bulb temperature sensor for the outside air, and, in this case, it turned out to be a single point of failure with a domino effect.
For the morning, the outside air temperature transmitter was reading correctly but then it malfunctioned, slowly changing from a reading of 17°F to more like 65°.
This malfunction caused the preheat recirculation pumps on each of the air-handling units to be disabled (they would only run if the outside air temperature was less than 50°).
The preheat coil control valves opened up to 100%, but, without the recirculation pumps, they weren’t able to keep up with the freezing air.
The freezestat devices worked correctly and interrupted the safety circuit in the air-handling units’ supply fan speed drives to shut down the units.
Because all systems were setup for 24/7 operation, there wasn’t any interlock between the exhaust system and the air-handling units, so they continued to run and increased speed because of the lack of makeup air.
After deducing all of this, Jeff and I took a few steps to return the building systems to normal operation. We checked if there were any damaged chilled water coils (fortunately there weren’t any). Then, we marked the outside air temperature as “unreliable” in the BAS and overrode it to a value closer to real conditions. This enabled all of the preheat pumps and let the preheat sections of the air handlers warm up. Next, we reset all the freezestat devices. I particularly enjoyed this step because the units were so tall that each unit had at least three freezestat devices, and I had to climb the coil piping to get to them! This enabled all of the supply fans, allowing us to manually increase the speeds with stepped overrides until static pressure was in line with set points on both the supply and exhaust ductwork. Then, we released them to normal control of the variable-speed drives.
Neither of us thought to call first responders during all of this, but someone else apparently did. By the time we got things back to normal, a fire marshal and police officer showed up in the mechanical room we were in. As it turned out, people evacuated the building while we were working on the systems, maybe even before. We all compared notes for a few minutes, and the fire marshal said he had taken some readings and found that pentane was being pulled out of the floor drains by all of the negative pressure. At the time, I didn’t understand why he said that, but looking back, I think he was pointing out an ignition risk and how this could have been a fire hazard. The police officer remarked that the pressurization seemed better in the building than it had been, and he asked me whether it was safe to let people back in. In a brief moment of knowing better, I told him it was a question for Jeff. I wrote up all of the findings from that day in our commissioning documentation, but I don’t recall any changes being made to mitigate this issue other than replacing the malfunctioning transmitter.
There is a local barbecue restaurant close to my office that I frequent, and I always have to give the door a good pull when I walk in. I don’t think they have their kitchen makeup fan turned on, and I’m always reminded of this story. I’m still amazed to think of the effects that one inexpensive sensor’s reading could have on such an expensive and complicated building. I’m even more amazed to think of the dumb luck I had to be around when it happened. Stories like this one are always fun to share, and each experience like this makes us handle the next project a little better than the last. I would love to hear your stories. Send me a line and share how you’ve carried those priceless learnings forward in your career.