​​How Using a Test Executive Prevents Reactive Development​

A test engineer’s priority is ensuring a working product is delivered to the end customer. But as products become more complex, so does the testing. Let’s take a light bulb, for example. Smart light bulbs have built-in Wi-Fi and Bluetooth functionality—and sometimes even speakers. That means that over the last few years, a test team has added testing for power electronics, wireless connectivity, and acoustic test.

^{Figure 1: Advanced product capabilities have extended development times, shortening the time available for test development.}

Figure 1 illustrates the difference between the planned time spent on each phase and the reality. Because the deadlines stay the same, any delay in the earlier phases results in test development time getting cut short—test engineers are asked to test more functionality in less time. When we take time away from the test development phase, we force test engineers to be more reactive rather than proactive. They simply don't have time to consider how to approach this efficiently.

The Cost of Reactive Test Development

This trend of testing more in less time has been around for years but is often overlooked because the cost of this type of reactive development isn’t immediately evident. One significant side effect of reactive development is it leads to a lack of flexibility, which can lead to a loss of productivity for test engineers as they waste time finding ways to add features to a bloated, rigid tester. At a certain point, it becomes inefficient to continue building on the same tester, and engineers are forced to start over and waste time rebuilding the same features as before. 

Not only does this lead to a large amount of rework, but it also increases the number of testers that need to be maintained. The maintenance phase comes with its own set of challenges, especially if an engineer who built the tester leaves the company without adequately documenting it. Their departure can also lead to a rebuild of the entire system.

One obstacle to building test frameworks is finding someone with the time and qualifications necessary to do so. That’s why NI developed TestStand, an off-the-shelf and customizable test executive for all your test operations. TestStand provides a common framework for all your testers, streamlining the development of future testers. With all the standard components built, test engineers can focus on creating unique test functionality, reducing churn, and getting products to market faster.

Programming Language Flexibility

Test engineers are proficient in different languages, and it’s essential that they feel empowered to focus on building the steps that get the test done in the language they prefer. TestStand is language agnostic, meaning test organizations can program in various languages, including Python, C/C++, and .NET, and have it all plug into the same framework and work together.

Risk Reduction 

Developing your test management solution is only the beginning of the task. Software is not a one-time cost but an ongoing effort. Software must be maintained to meet the following goals:

• Add functionality as needed
• Fix bugs or defects that are found 
• Maintain compatibility with new software
• Keep up and take advantage of new technology advancements

This maintenance effort can often become significant and consume months of development time. On the other hand, ready-to-run test executive packages such as NI TestStand are regularly updated by NI to stay competitive and viable. This policy enables you to take advantage of the latest technologies (and get fixes to bugs and new functionality) at a fraction of the cost. Read our Test Executive Software – Build or Buy? guide to learn more.

Advanced Feature Implementation

The most obvious cost associated with building a custom test executive is the upfront development cost of implementing all the features of the test executive. The following table outlines some of the features associated with test frameworks. The table shows an estimate of the time required to implement each feature from scratch and compares it to the time it would take to implement the same features using NI TestStand.

^{Figure 4: NI worked with test professionals to ensure standard features were easy to implement with TestStand.}

Refer to the build or buy guide for more information.

Not all are relevant to every organization at any given time, but that doesn’t mean they won’t be relevant in the future.

In this section, we’ll discuss some valuable features you can add easily in TestStand—such as parallel execution, report generation, database connectivity, user interfaces, and debug/deployment tools. 

Parallel Execution

^{Figure 5: This diagram demonstrates TestStand’s ability to pipeline sequences and autoschedule resources to enable the testing of multiple UUTs (units under test) at once.}

One way to increase test throughput is simultaneously executing tests across different devices. When we look at how much time it takes to develop software that can perform parallel execution, it is clear why test organizations avoid creating this feature unless they must. The problem is that parallel execution requires considerable development effort, so when an organization realizes it’s needed, it is often too late to implement it. With TestStand, developing parallel systems is far more accessible, eliminating most of this effort and making parallel systems very cost-effective.

Report Generation and Database Connectivity

When building a test system, it is essential to consider how the test results will be handled. That’s why TestStand has built-in features that simplify database connectivity and report generation. Customizing reports in TestStand is a common task, and TestStand provides many features to customize the report content, functionality, and style. 

Although a test report does a great job of providing a snapshot of the passes and failures of a test run, it isn’t convenient when you want to poll historical test results from one or many test stations. Instead, NI TestStand can quickly log test results to almost any open database connectivity (ODBC) system, such as Oracle, SQL Server, or MySQL, out of the box.

Debug and Deployment 

Reduce the time you spend debugging your test code using TestStand’s built-in functionality. Take advantage of tools that allow you to monitor test values throughout a sequence and breakpoints to pause execution to further investigate a section of your test code.

^{Figure 6: Deployment involves packaging all necessary test system components using a deployment tool or server before distributing to the desired test stations.}

You can also simplify how you deploy your test code to production machines with utilities that automate building installers and distributions.

User Interfaces 

The operator interface is the display through which the operator interacts with the test system. Developing a custom operator interface can be a nontrivial time investment, especially when usability and consistency are considered. TestStand provides simple operator interfaces to configure and execute tests on deployed systems. You can also develop custom user interfaces in multiple programming environments for greater control over how operators interact with test systems.