Testing, Testing and more Testing

Lately, it seems like we are practically living in our test lab. As we thought about content for this month’s newsletter, an update on how we utilize the testing environment came to mind. Certainly a large part of our lab activities involve testing product enhancements, as well as confirming interoperability with other manufacturers’ equipment. We frequently receive compliments on the quality of our customer support and credit the investment in the test lab as a large component of this capability. Many situations that our partners run into in the field can be reproduced here in the test lab.

When The S4 Group was formed, we knew we needed to invest in a test environment. In fact, we wrote an article in October of 2004 for Automatedbuildings.com Extreme N2 Networking discussing the importance of establishing the test environment and including the flexibility to reconfigure it as needed.

That first test bed was one side of a single wire rack with the Metasys® N2 devices and a supervisory controller to provide us the ability to test our initial products. The primary focus was in being able to successfully communicate to the N2 bus and the N2 devices utilizing the documentation that we received as a result of becoming a member of the Metasys® Compatibility Program. This was no small feat since the N2 protocol is actually three sub-protocols that co-exist on the N2 bus. This meant that we needed to have sample N2 devices from each Metasys® N2 product family, along with a sample of N2 Vendor (VND) devices. Taking the sensitive timing characteristics of the N2 technology into account, we also needed to include enough devices to generate an adequate volume of transactions to exercise the N2 protocol, make sure that everything worked even when the bus was saturated, and ensure the integrity of all transactions.

One of the first enhancements to the S4 Open Appliances was the addition of our Upstream N2 Interface, enabling co-existence with the Metasys® supervisory controller. This meant an expansion in the test lab to include examples of each generation of Metasys® supervisory controller and the requisite tools to monitor both the upstream and downstream interfaces. We had to confirm the integrity of all N2 transactions independently of whether they originated from the Metasys® supervisory controller or from an OPC HMI or SCADA system, as would be the case with our first offering - The S4 Open: OPC-N2 Router. This product continues to be an important offering, especially in high availability environments, pharmaceuticals, and other applications where industrial or process control systems needed input from environmental control systems. The complexity of the product has increased dramatically with this enhancement: integration to three downstream N2 sub-protocols, the upstream N2 interface to the supervisory controller, and the upstream OPC interface. Testing has become correspondingly complex and adds to the demands on our test environment.

The S4 Open: BACnet-N2 Router would be introduced next. The introduction of this product meant that interoperability testing became a very important function of the test lab. After attending our first BACnet International PlugFest, we quickly learned that interoperability testing needed to be a standard part of our environment, not just a once a year event. Testing in a sterile environment was necessary but not adequate to deliver the quality that we were committed to. Even though BACnet is the most widely used communications standard in the BAS world, we still found it necessary to test each enhancement with a selection of different manufacturer global controllers and BACnet Operator Workstations. We found that once a year participation in PlugFest was just not enough to accomplish this task. So, another panel in the test bed is dedicated to hosting BACnet global controllers from various BAS manufacturers. A high end workstation in the test lab hosts virtual machines for every major BACnet OWS manufacturer in the industry. In a few cases, we found that the versions we were testing with did not perform well in a virtual machine environment, so we added dedicated machines to host these OWS environments.

We have put the necessary investments into the test environment to keep up with the ever increasing complexity of the product offerings. Along with this growth, our partners have grown with us. There has been a steady increase in the number if S4 Integration Partners and Distributors, as well as a number of strategic partnerships. This growth brought with it the requirement to have multiple tests running in parallel to make sure that everyone who needs support gets it as quickly as is practical, and so that we can keep product enhancements and new product development moving forward at the same time. To accommodate these needs, we split our available N2 devices into multiple color-coded N2 buses.

The need to support multiple S4 Open Appliance product families, and multiple generations of hardware, including a brand labeled version for one of the major BAS manufacturers, meant that we needed to have one panel dedicated to hosting an array of S4 Open Appliances. Again, flexibility is the key to successful utilization of the test environment. Any of the S4 Open appliances can be running the developmental firmware, or be rolled back to exactly match the firmware installed at a customer site we are providing support for. When necessary, we can have our partners export the configuration of their installed system so that we can load it onto one of the S4 Open appliances in the test lab and duplicate their installation to the extent possible in our test lab. Any of the appliances in our lab can be connected to any of the available N2 buses as needed for each test.

A flat Ethernet network was no longer adequate to test BACnet BBMD and Foreign Device Registration capabilities, so we expanded the test lab into multiple IP subnets. Support for testing cloud-based applications and applications hosted at remote locations where the BACnet-N2 Router is acting as an on-site agent for those applications is defining the next round of enhancements to the test environment. We already frequently utilize desktop sharing capabilities to share our lab facilities while working with partners. These new applications will require secure access from external locations.

Testing doesn’t start here at our S4 Test lab. Our development partner, Obermeier Software, of Verl, Germany puts every firmware build they produce through an automated test sequence before it is released to S4. This sequence includes tests to verify all the core system functionality. Then the automated tests go a step further to emulate the situation that exposed the root cause for all previous trouble tickets. This automated testing is a very thorough, but robotic, process. When we perform our Q/A testing here at S4, we attempt to simulate real world building conditions that the product will be working in when installed by our Integration Partners. We try to interject as much randomness as possible into the test process to simulate what might happen when different installation engineers approach the installation and integration process a bit differently. Of course, the final test is when the product is installed in a live project and performs as it is designed!

S4 Open Appliances are not just for Metasys® anymore. The lab has physically expanded to four wire racks with both sides of each rack able to support various test environments. From the outset, the architecture of the system anticipated many more integrations than just Metasys® N2. We’ve been chomping at the bit to get going on these projects so we have been building separate sections of the test lab in anticipation of the next two integrations to be released. What about those empty spaces on the racks? There is a long list of potential new integrations that we may offer in the future. Whenever the opportunity presents itself to acquire equipment that we might need for future testing, that’s where it ends up.

Is the test lab worth the investment in time and expense? Absolutely! It’s been proven that time and again with success stories. We’ve found an equation that works and we are going to keep evolving and improving it to make sure that the success continues into the future.