Whenever I encounter a problem that involves processing massive amounts of data, I tend to lean on Python, a scripting language that allows us to use code to do just about anything you can think of.
Working in the building automation field brings many unexpected situations that need custom solutions. Let’s examine some tools that are at everyone's disposal that some might not be aware of. For those of you who share a tech mindset, I hope this brings some new ideas your way, and for those of you who do not, I hope this offers you insight into the solutions that are available. Data is the most important resource in today's modern world. We often have to pull this data out of our system to process for analysis, third-party commissioning, energy audits, etc. I wanted to share a recent event and my solution to the problem in case others have similar situations in the future.
A problem arose where an engineer needed to evaluate trends on a control system, but the data exported from the building automation system (BAS) was simply too large to open in any modern spreadsheet program, such as Excel or Apple Numbers. This data export included millions of rows of data and exceeded the limitations of these programs. Most modern spreadsheet applications tend to stop working after about 1 million rows of data. I was informed that the BAS was trending these particular gas sensors at an interval of one minute for the last few years. By the time this data was given to me, it was one large file with four to five years of one-minute-interval data.