Test Executive Software – Build or Buy? A Financial Comparison Using NI TestStand

Overview

Whether you are building a new automated test system, or upgrading an existing test system, it is important to fully understand the total cost of test. The capital cost of test equipment is just one of the costs involved in developing and maintaining an automated test system. In many cases, the biggest cost is not capital equipment, but the time spent building and automating the test system.
The test executive is a vital part of a modular automated test system architecture – the overarching software that manages, sequences, executes, and reports the results of the tests that you run on your products. When building your test system, one of the key questions you have to answer is whether to build your own test executive from scratch (using application development software such as NI LabVIEW, C++, or .NET) or to buy a commercial-off-the-shelf (COTS) test executive.

Contents

Role of a Test Executive

A test executive is the high-level software application for organizing, deploying, executing, and reporting on automated tests. Test executives typically have two modes of operation:

  • A development mode for defining testing processes or sequences
  • A run-time mode for executing sequences, displaying results, and generating reports on these test sequences

In most cases, the actual test programs that control the instrumentation hardware are separate software modules written in either a test programming language, such as LabVIEW or NI LabWindows™/CVI, or a general-purpose programming language, such as C/C++, scripting languages, or .NET. By following a modular software architecture that decouples the test code modules, which are often specific to the device being tested, from the test execution system, which is common to all the different devices being tested, you have a scalable and flexible architecture that is less expensive to develop, support, and maintain in the long term.

For more information on a modular software architecture, refer to:
Developing a Modular Test Software Architecture With NI TestStand and LabVIEW.

 

Typical Test Executive Functionality

The following list outlines some of the common functionality that test executives accomplish:

  • Test Sequence Development Environment
    • Editor to specify and organize individual tests
    • Branching/looping logic
    • Editor to specify test limits
    • Ability to call tests written in a variety of formats – DLLs, assemblies, VIs, scripts, EXEs, etc
    • Interface to test coding environments such as LabVIEW, LabWindows/CVI, Visual Studio, etc
    • Integration with Source Code Control providers
  • Custom Operator Interface
  • Sequence Execution Engine
    • Test result evaluation and display
    • Debugging capabilities
  • Results Reporting
    • Database logging
    • Report generation in a variety of formats – XML, text, HTML, ATML, etc
  • User Management
  • Parallel Testing Capabilities
  • Unit/Device Tracking and Serial Number Scanning
  • Test Deployment Tool
  • Localization Tools
  • Documentation

 

Back to top

Test Executive Cost Comparison Breakdown

Many developers undertake the task of building their own test executives before they fully realize the magnitude of the task. The following section outlines some of the specific costs incurred by developing and maintaining a test executive as well as training employees to use the test executive.

For a comparison, the costs are contrasted with the analogous cost incurred by choosing a COTS test executive. For the purpose of this cost breakdown, we have picked NI TestStand as the COTS solution because it is a ready-to-run test executive, but still allows for full customizability for your specific test applications

Note: The costs and man-hours used in this comparison are estimates based on our experience working with hundreds of automated test customers, however, they are just estimates of a common case. The important points here are the identification of the multitude of tasks required, and the significance of the investment required to complete these tasks.

 

Development Cost

The most obvious cost associated with building a custom test executive is the up-front development cost of actually implementing all of the features of the test executive. The following table outlines specific features required to build a test executive. The table includes an estimate of the time required to implement each of these features from scratch, as well as the time required to implement the same features using NI TestStand.

 

Feature Description

Development Time in Man-Days

Custom

NI TestStand

Test Sequencing Development Environment
Tools for specifying test programs; defining looping, branching, and conditional execution logic; specifying parameters to be passed to tests; specifying test limits and so on

100

Can vary significantly depending on the complexity of the sequence logic and variety of compatible test languages available

1

Custom Operator Interface 
Custom interface written in an application development language such as LabVIEW, C/C++, or .NET. Load, display, and run sequences; controls for executing and debugging sequences; pass/fail banners, operator prompts, and so on

40

8

Example operator interfaces in NI TestStand serve as a starting point for customizations

Sequence Execution Engine
Execute test sequences and call test code modules, while evaluating results and generating reports

15

0

Database Logging 
Store test data and results in various formats for Oracle, SQL Server Access, mySQL, and other ODBC and ADO databases

10

2

Report Generation 
Generate reports in various formats (ASCII, XML, HTML, ATML, Excel)

15

1

Built-in report generation that can be customized

User Management 
Password-protected access to test executive capabilities with different access levels. Environment for setting up/managing user information

5

0

Parallel Testing 
Architecture for loading and running test programs in parallel, while still providing full functionality of unit tracking, logging, etc. Should include constructs to prevent race conditions such as locks and semaphores

100

1

Unit (Serial Number) Tracking 
Track DUTs/UUTs by serial number for identification

5

1

Interface to Coding Languages 
Architecture that allows calling test code written in a programming language (such as LabVIEW, C/C++, or .NET) and getting results and measurements back. Assumes interfacing with two languages

15

Varies based on the number and variety of compatible test languages/file formats

0

NI TestStand calls LabVIEW VIs, LabWindows/CVI DLLs, ActiveX/COM modules, .NET Assemblies, scripting languages, and executables

Deployment Tool 
Create image/installer to replicate test system on production systems

20

3

Built-in deployment utility to create installers including NI TestStand Engine, test code, operator interfaces, configurations, and additional installers

Localization 
Ability to change languages for displayed text. Assuming one additional language.

15

Varies significantly based on the number of localized languages

2

Modular localization imports all strings from ini text files for easy localization

Documentation 
Both the development tools and the operator interface must be documented with hardcopy manuals and online help

20

1

Both NI TestStand and the shipping operator interfaces are fully documented

 

Cost Analysis

When evaluating the cost of developing custom software, it is important to realize all of the costs associated with an engineer. For instance, even though the cost of an engineer may be calculated by the hour or day, it is rare that this includes intangibles, such as benefits, healthcare, and pension, or the cost of office tools such as a computer, a desk, and office space. When all of these aspects are included, the average cost of a capable software engineer can easily reach $150,000 USD per year. However, in the analysis that follows, we will assume that, in USD, the cost of an engineer is $100,000 per year, or roughly $400 per working day.

  Custom NI TestStand
Development Time in Days

365

18

Development Cost Per Day

$400

$400

Number of Test Developers

10

10

Cost of Software Per Developer

$0

$4,223

Number of Test Systems

50

50

Cost of Software Per Deployment

$0

$561

Total Development Cost

$146,000

$ 81,703

Total savings by buying a COTS test executive: $64,297 (44%)

 

Maintenance and Support Cost

Developing your test management solution is only the beginning of the task. Software is not a one-time cost, but an ongoing effort. Software must be maintained over time in order to:

  • Add functionality as needed
  • Fix bugs or defects that are found
  • Maintain compatibility with new software
  • Keep up and take advantage of new technology advancements

Many times, this maintenance effort can become a major undertaking and consume months of development time. For instance, when new operating systems are released, simply porting the same application to a new OS can be a monumental task. Custom-developed software must be updated and maintained by the end-user or third-party developer on a contract basis and can be very expensive.

On the other hand, ready-to-run COTS test executive packages such as NI TestStand are updated by the vendor regularly, simply to stay competitive and viable. This policy enables users to take advantage of the latest technologies (and get fixes to bugs and new functionality) at a fraction of the cost.

Again, associating a definitive cost on maintenance and support is impossible because every case is different. However, in this example, we assume that the overhead of maintaining an in-house test executive requires the effort of 15 percent of the test development department (with these engineers often being called Test Executive Architects), and that the cost of maintaining the software increases by five percent every year. On the other hand, the overhead for NI TestStand would be substantially lower, at around five percent of the department, because fewer customizations are needed. Test Architects are still needed and we also included the cost associated with upgrading to a newer version of NI TestStand.

Keeping our earlier assumptions, and a test system lifetime of 10 years, we can develop the following cost analysis:

  Custom NI TestStand
Percentage of Time (Compared to Development Time) Spent in Maintenance  15%  5%
Maintenances Per Year  2  2
Annual Cost of Support and Upgrades  $0

$844

(Cost of SSP per developer)

First Year Total Maintenance Cost  $20,000  $13,440
Year-Over-Year Increase in Maintenance Cost  5%  5%
Total Maintenance Cost (10 Years)  $550,911  $281,619.01

Savings in first year by buying a COTS Test Executive: $21,410

Total savings by buying a COTS test executive: $269,293 (49%)

 

Training Costs

When standardizing on a standard test package, a company must provide detailed training to the appropriate users. For test developers to be productive at creating test code using a test executive, they must first be trained on how to use the test executive. In addition to basic training, developers can also greatly benefit from learning best practices and working through example problems.

Unfortunately, training is a major expense that many companies overlook when deciding on a software platform. When developing a custom test executive, keep in mind that in addition to developing the software, you will also have to provide standard training classes as well as local experts who can deliver the training material.

Many COTS software, including NI TestStand, offer on-site (as well as online) employee training that ranges from basic introductory topics, such as how to develop and deploy tests, to more advanced topics, like customizing the test executive and creating operator interfaces. In addition, these courses are taught in most major languages.

Although the actual cost of training varies between organizations, we can see the cost of developing and delivering training by making some assumptions. In this example, we continue with the assumptions from earlier.

  Custom NI TestStand
Course development time (course material and examples) in days

40

3

Course duration in days

5

5

Cost of course setup, computers, materials, etc

$10,000

0

Cost of NI TestStand course per developer (Part 1 – Test Development) -

$2,175

Cost of developing training material

$16,000

$0

Cost of delivering training – includes opportunity cost of developers

$28,000

$33,400

Total Cost of Initial Training

$63,000

$41,000

Initial training savings by buying a COTS test executive: $22,000 (35%)

 

Note: Although the savings in the training cost is dwarfed by savings in other categories, such as development and maintenance, keep in mind that

 

Back to top

Summarized Cost Comparison

 

Cost Over 5 Years


 

Total Cost Over Lifetime of Test System

 

Back to top

Build Versus Buy Trade-Off: Customizability Versus Off-the-Shelf Functionality

There are inherent trade-offs in deciding whether to build a custom test executive or buy a commercial off the shelf solution. A COTS test executive has the potential to reduce cost by taking away a lot of the development, maintenance, and training burden. However, every test application is different, and it is critical to evaluate whether the built-in functionality can adequately meet the wide variety of testing requirements for your specific application.

NI TestStand is a ready-to-run test executive that is fully customizable for any test application that is designed to deliver the cost benefits of a purchased product while maintaining the flexibility of a custom-developed solution. This customizability is made possible through a very modular software architecture that exposes all the functionality of the underlying execution engine via an easy-to-use API.

 

Back to top

Next Steps

Back to top