Sequence tracing provides immediate feedback and status of the current operation, such as Pass, Fail, Error, or Skipped. However, sequence tracing impacts performance by decreasing execution speed. The following approaches can help you improve execution speed without sacrificing the benefits of sequence tracing.
To improve the performance when sequence tracing is enabled, ensure that the tracing speed is set to the fastest speed possible. Select Configure» Station Options to launch the Station Options dialog box, as shown in Figure 1. On the Execution tab, slide the Speed control to Fast. This is the default setting.
Figure 1 – Setting the Sequence Tracing Speed
To further increase performance, you can also use the Station Options dialog box or the Execute menu to disable sequence tracing. However, if you disable sequence tracing, TestStand no longer provides direct feedback to the user about the progress and results of the test sequence.
Several sequence tracing options allow you to provide feedback to a user with less of an impact on performance. One option is to restructure your test sequence to hide some of the details inside subsequences and then use the Sequence Call Trace Setting to disable tracing when calling those subsequences. For example, on a motherboard test, tests for each component—such as RAM, Serial Ports, Video Out, CPU, and USB—might require several test steps. Tracing each individual step can be time-consuming and can impact the overall performance of your system. Create a top-level sequence and then a subsequence containing each of the component tests. Then, for each component’s SequenceCall step, select Properties»Run Options to set the Sequence Call Trace Setting property to Disable tracing in sequence, as shown in Figure 2. Nesting your test sequence in this way allows you to use sequence tracing for the top-level sequence without the performance overhead of tracing each substep. You can use this approach in both development environment and operator interfaces. However, additional overhead is required to process the sequence calls, which will reduce some of the performance gains.
Figure 2 - Sequence Call Trace Setting
Another way to improve performance without sacrificing valuable feedback is to disable sequence tracing and rely on UIMessage steps for feedback during testing. UIMessage steps provide the fastest response, but require more coding to post status updates throughout the test and requires an operator interface to monitor and display the UIMessage steps.
You can create a single UIMessage step by adding an ActiveX/COM action, as shown in Figure 3. This method is useful for posting UIMessages at certain points within the test sequence.
Figure 3 - UIMessage Step
Another way to use UIMessages, shown in Figure 4, is to call the AddPostStepCustomUIMessage method for executions. This approach applies to all steps in the execution, which can be a useful activity indicator. To decrease the rate of UIMessage posts, consider using an expression such as the one shown in Figure 4, which creates a UIMessage only on every fifth step.
Figure 4 - Post Step Custom UI Message
In TestStand 4.0 and later, you can call TestStand API methods directly from an expression instead of using an ActiveX code module. For more information about operator interfaces and UIMessages, refer to Best Practices for TestStand User Interface Development.
2. TestStand Configuration Options
TestStand has a number of configuration options that can improve performance. The following sections describe those options.
File format can affect speed and performance. Prior to TestStand 4.0, all sequences were saved in the INI format. TestStand 4.0 and later allows you to save sequences in three file formats: INI, XML, and binary. The binary format offers the fastest load and save times and also generates the smallest files. You can specify which format to use for new sequence files by clicking the File Format Options button on the Preferences tab of Station Options dialog box. To change the format of an existing sequence file, choose Edit»Sequence File Properties and select File Format on the General tab.
Figure 5 - Sequence File Properties Dialog Box
Another set of configuration items that can improve performance are the Load and Unload options, which control when sequence files and code modules load and unload. Loading and unloading files takes time, but if you load everything and leave it loaded, it may significantly increase the memory footprint of the test system. If you load too many files into memory, Windows will move code items into virtual memory, which also slows down the system. You can set the Load/Unload options at step level or at sequence-file level. In most test systems, you can combine the Preload when opening sequence file or Preload when execution begins options with the Unload when sequence file is closed option for the best performance. Table 1 and Table 2 describe the Load and Unload options.
|Load Option||Performance Effect|
|Preload when execution begins||This setting has high memory usage. While performance decreases when the execution is started, this setting offers the highest performance when running tests.|
|Preload when opening sequence file||This setting has high memory usage. Performance decreases when opening the sequence file instead of when the execution starts.|
|Load dynamically||This option has the lowest memory usage when combined with the Unload after step executes unload option. However, performance decreases because the step/sequence is loaded/unloaded every time a step executes. This setting has additional risks, such as the potential to lose global data within a code module when that code module unloads.|
Table 1 - Load Option Performance
|Unload Option||Performance Effect|
|Unload when precondition fails||This setting unloads a module if the precondition for a step is false. This frees memory during the test execution but decreases performance.|
|Unload after step executes||This setting unloads a module when the step execution is complete. This frees memory during the test execution but decreases performance.|
|Unload after sequence executes||This setting unloads a code module only when the sequence execution is complete. This option keeps the modules loaded longer than the Unload after step executes option but can require more memory. This option provides high performance during test execution.|
|Unload when sequence file is closed||This option has the highest memory usage. Performance decreases when closing the sequence file. However, this action is typically only taken when shutting down or switching to new tests, when timing is not as important. This will typically give the best overall performance, especially if you stop and restart executions in a repetitive manner, such as with the Single Pass option of the Sequential process model.|
Table 2 - Unload Option Performance
3. Search Directory Configuration
Another configuration option that affects performance is the Search Directories setting. The search directory configuration directly affects the time required to load sequence files and code modules when they are specified with relative paths. This setting is most important during the initial loading and execution of the tests, but it can also affect all subsequent iterations, depending on the Load and Unload option settings in your sequence files.
Use the search directory configuration utility to specify the order in which TestStand searches directories when loading code modules and sequence files. Select Configure»Search Directories to open the Edit Search Directories dialog box, as shown in Figure 6.
Figure 6 - Edit Search Directories Dialog Box
When TestStand loads a code module that has been specified by a relative path, it starts looking for the file based on the search directory listings in the Edit Search Directories dialog box. TestStand searches the directories in a top-to-bottom approach until it finds a code module with the specified name, such as MyTestStandVI.vi. Because TestStand searches the directories in order, that order directly affects the performance of module load times. To optimize load times, consider the directory structure of the tests, your test modules, and the following techniques for optimizing search performance.
- Include only those search directories that will be used by your tests.
- Keep all paths relative to the sequence file so that the first search path (the current sequence file directory) always returns the correct file and prevent the need for additional searches. This also allows you to move the entire folder to another location or computer without having to change the search directories.
Note: The Current Sequence File directory and Current Workspace directory are added automatically to the search path.
- If you expect modules to be found in subdirectories, use a partial relative path to the search directory. For example, instead of setting the search directory to search subdirectories when you expect that modules may be several layers deep, consider setting the module path to be relative to a higher-level directory, such as mysubdirectory1\mysubdirectory2\mycodemodule.dll. This technique can also help resolve ambiguities that can occur when more than one code module with the same name exists in multiple places.
- Place the directory in which items are most likely to be found at or near the top of the list.
- Place code modules in a central location (e.g. \test\bin) to minimize the number of search directories.
- Place the directories in which code modules are least likely to be found at the bottom of the list.
- Place directories for which you have enabled the Search Subdirectories option lower in the list. This setting is not recommended because it slows down the search process.
- Place directories located on network drives lower on the list, since network access is slower than local disk access.
Note: TestStand allows you to use the search directory path to easily switch the modules loaded from a debug version of a DLL to a release version of a DLL without any changes to your test sequence code. This is a powerful feature, but you must remember to return to the release version DLL when you are finished debugging in order to prevent a substantial decrease in performance.
4. Code Module Performance
TestStand calls code modules in various development environments to perform test steps. The configuration of these code modules and development environments can have a large impact on the performance.
General Code Module Optimization
In any code module environment, you can achieve better performance and ease of coding by passing only the necessary data into and out of a code module. Avoid passing large amounts of data that will not be accessed or modified by the code module.
Compiled Code Modules (DLLs)
Compiled code modules, such as .NET or standard DLLs, can compromise performance when you are using a debug version of the DLL rather than a release version. TestStand allows you to redirect search directories so you can easily switch between debug and release versions, but you must remember to switch back to the release version to prevent a decrease in your test speed. TestStand also allows you to build the debug DLL and the release DLL with the same name and location, but this technique introduces the risk of using the wrong version.
Instead of redirecting search directories, you can write a simple utility that copies either the debug or release DLL from different locations into the location where TestStand expects the DLL to be, making it safer to switch between debug and release DLLs.
National Instruments recommends using unique names for the debug and release DLLs and then using the Search/Replace functionality in TestStand to remap the DLL calls. This method is more time-intensive than switching between DLLs, but it eliminates confusion as to which DLL is being used and the risk of using the wrong DLL.
LabVIEW Code Modules
When you run LabVIEW VIs in the development environment, all of the debugging capabilities remain available, resulting in slower load and execution times. To improve LabVIEW code performance, configure your VIs to run in the LabVIEW Run-Time Engine by selecting Configure»Adapters»LabVIEW»Configure.
5. Test Sequence Optimization
Another way to improve performance is to optimize the sequence flow and hardware interaction required by the test sequence.
Sequence Flow Optimization
You can optimize test sequences to increase test speeds in a number of ways. For example, for some devices you can test different components in parallel to increase total speed. Note that this is parallel testing of components in a single UUT, rather than parallel testing of multiple UUTs. The Parallel and Batch process models do not automatically enable this sort of testing.
Parallel Testing of a Single UUT
When testing a single UUT, you may be able to test multiple portions of the system at the same time. Consider a test for a computer motherboard, where you can write a test for each of the SATA interfaces, serial port(s), parallel port(s), USB port(s), system clock, IDE interface(s), and memory bus. Many of these tests can be performed simultaneously on the system. Rather than writing a single test that works through each component individually, you can write a sequence that tests each component at the same time by launching new threads and/or executions.
To create parallel test sequences, first create subsequences for the different tests and use the Execution Options control on the Module tab for the SequenceCall steps which call the tests to direct the module to use a new thread or new execution, as shown in Figure 7.
Figure 7 - Sequence Calls Options
Both the new execution and new thread options cause TestStand to run the sequence in a new thread, but there are some subtle differences. By default, new executions have their own copy of any sequence file globals by default, though these copies are configurable on the sequence file Properties dialog box for each sequence file. You can display these new executions in a separate Execution window in the sequence editor, and you can run them using any of the TestStand process models. New threads, however, exist in the current execution and share the same sequence file globals as the calling sequence.
For this example, using a new thread is preferable to using a new execution because creating a thread requires fewer resources than creating a new execution.
You can also create and control TestStand threads and executions with the TestStand API. For more information about the available properties and methods related to threads and executions, refer to the NI TestStand API Reference Poster.
You must consider the available properties and methods when deciding whether to create a new thread or a new execution.
If you select the Use New Thread option, you can click the Advanced Settings icon to launch the Sequence Call Advanced Settings dialog box, as shown in Figure 8. Use this dialog box to specify various sequence call options, including whether the launching sequence waits for the asynchronously-launched subsequence to finish before completing. The default setting is Automatically Wait for the Thread to Complete at the End of the Current Sequence, which ensures that the top-level sequence waits until the subsequences executing in parallel complete before returning. This delay prevents your top-level sequence from returning before all of the parallel subsequences are complete, which could cause your results to prematurely show that testing is complete.
Figure 8 - Sequence Call Advanced Settings Dialog Box
In order to view the results of your asynchronous subsequences in the report and in the result list for database logging, use Wait steps at the end of the launching sequence to wait for the asynchronous sequence calls to complete. These Wait steps direct TestStand to attach the results of the asynchronous subsequence to the Wait step, making them available for report generation and database logging. Figure 9 illustrates the Wait Step Configuration dialog box. To wait on a sequence execution or thread, select Execution or Thread in the Wait for control and specify the execution or thread to wait for.
Figure 9 – Wait Step Configuration Dialog Box
You can optimize the test system hardware through the selection and quantity of the hardware used in the test system as well as through the hardware configuration and communication settings.
When you select the hardware for a test system, you may need to make tradeoffs between the channel count on an expensive piece of measurement equipment and the use of multiplexers or switches to enable your equipment to be used on more than a single test point. Using more measurement channels in parallel is typically more expensive, but also shortens the time required to test each UUT.
You should analyze your tests to determine the length of each test and the acceptable amount of hardware duplication. This helps you to predict which pieces of equipment may become bottlenecks during the tests and should therefore be duplicated. Also, benchmarking with various hardware configurations can help you determine the exact performance benefits of duplicating hardware.
When you share equipment between parallel tests or multiple UUTs, you must configure the software such that no two threads access the same resource at the same time. Refer to the Testing Multiple UUTs in Parallel section of this document for more information about this topic.
Hardware Capability Optimization
For any given hardware configuration, there are a number of common factors that can decrease test effectiveness. For example, you can use an oscilloscope to measure the rise time, fall time, RMS, and peak values of a signal. If you program the oscilloscope to capture the entire waveform, transfer the waveform to the test system, and then perform post-processing on the data to extract the desired measurements, you will experience decreased performance because of the large amount of data being transferred. Performance is also affected by the latency of the communication bus, so you should consider whether your instrument has a high-latency, such as a LAN or serial connection, or a low-latency bus like PCI or PXI.
If you configure the oscilloscope to measure the rise time, trigger an acquisition, read back the rise time from the instrument, and repeat for each measurement, you must then reconfigure and retrigger the oscilloscope for each measurement. This option also tends to be slow and inefficient.
Considering that many modern oscilloscopes have multiple measurement channels, use the following steps for faster test execution:
- Set the test to configure four measurement channels (one for each of the above measurements) in the oscilloscope.
- Trigger a single acquisition.
- Read each of the four measurement channels back.
The communications link between instruments and test software is another area where you can improve performance. Upon initializing communication with a device, some instrument drivers will transfer large amounts of data to verify communications and configurations. To minimize the decrease in performance caused by the re-initialization of instrument communications, use Process Model callbacks, which are special sequences in the process model that are intended to be overridden by sequences within a client sequence file, which is the sequence file containing the MainSequence sequence for a particular test sequence. If a client sequence file implements a Process Model callback, then, when the process model calls a callback sequence, the Process Model will call the implementation in of the callback sequence in the client sequence file.
The most common example of a Process Model callback is the MainSequence sequence. To create sequences in your client sequence file to override these Process Model callbacks, open the Sequence File Callbacks dialog box by selecting Edit»Seqence File Callbacks or by right-clicking in the Sequences pane and selectingSequence File Callbacks. From this dialog, select the callbacks from the list and click Add. Click OK when you have added all the callbacks you require. The sequences should now be available in your client sequence file.
For example, instrument initialization routines and calibration routines should be part of the ProcessSetup callback, while instrument communication closure should be part of the ProcessCleanup callback. By placing the initialization and cleanup into these callbacks, the instrument communications code executes once for all UUTs rather than once for each UUT.
6. Result Collections Optimization
There are a number of ways to optimize the impact of result collection on the performance of the system.
There are two general methods for logging data and results for a test system—on-the-fly logging and PostUUT logging. Each method has benefits and trade-offs.
With on-the-fly logging, TestStand logs each result or data item immediately after each step executes. The benefit to this method is that in the event of test system failure, the log retains all of the data up to the failure point. However, on-the-fly logging adds time to each step execution. You can mitigate the decrease in performance by decoupling the data-logging thread, as described in the Data Storage Mechanism section of this document.
In PostUUT logging, TestStand keeps all test data in memory until the entire test sequence is complete, at which point all of the results are committed to the data storage device at once. This method has better test speed compared to on-the-fly data logging method, but it has a larger memory footprint.
The following aspects of data storage can impact the speed of a test: the data storage mechanism, the quantity of data stored, and the storage location.
Data Storage Mechanism
You can store results data in a number of different mechanisms, such as files (e.g. tab-delimited text, comma-separated variable, HTML, XML) or databases. In general, logging results to a file is faster than logging to a database. Additionally, the way in which the data is handled in an intermediate storage location, such as a cache, affects system performance. One way to improve the speed of data logging is to use queues to transfer data to a separate logging thread, such as TestStand or Microsoft Message Queues (MSMQ). This decouples the data logging process from the data generation process.
However, decoupling the data logging process may require more memory, and if the test sequence is generating data faster than the system can log it, that memory requirement will grow unchecked until a limiting mechanism is placed on the data size. That limiting mechanism will slow the tests down again when too much data has been generated.
Another benefit of decoupling the data logging process from the data generation process is that the system can take advantage of spare processing power that is available when the system is waiting for I/O or hardware. On multi-core processors, you can place the data logging thread in a different processor than the main thread to improve performance.
Data Storage Quantity
Another factor to consider is the amount data you are logging to the system. Performance decreases as the amount of logged data increases. If you are using TestStand’s default process models, you can exclude recording results for individual steps by deselecting the Record Result option (Step»Properties»Run Options). You also can set an entire sequence to exclude result recording using the Disable Result Recording For All Steps setting on the Sequence Properties dialog box.
The following are some additional ways to minimize the performance impact of data logging.
- Log only UUTs that fail or error.
- Log passing UUTs only periodically.
- Log only failing steps.
- Log only step property results on failing steps.
Data Storage Location
As a general rule, data stored locally to a hard drive is logged faster than data stored to a network location, though some network data transfer mechanisms are faster than others. For example, using Microsoft Message Queues (MSMQ) to communicate with a database creates local intermediate data files that are then transferred over the network. This is generally faster and safer than writing directly to the remote database.
The report generation process can be time consuming and can decrease throughput, depending on when the report is created. For example, creating a detailed report immediately after each test takes some time, decreasing the throughput. If the report contains details that are needed immediately, such as calibration data, the reduced throughput may be a valid tradeoff. However, for reports that can be postponed, such as a certificate that ships with a device, you can offload the report-generation portions of the test to another system or process and only invoke those portions when needed. The key to post-test report generation is ensuring that you have logged all of the information required to create the report at a later date.
7. Early Test Termination
In a production environment, you may only need to know that a UUT failed any test, rather than knowing specifically which test failed. In those cases, you can terminate the test at the first failure in order to free system resources, and then obtain a more detailed failure analysis at a later time. Use the Sequence Properties dialog box to select the sequence you want to terminate at the first failure and set it to Immediately Goto Cleanup on Sequence Failure. You can also make this the default behavior for the entire test station by enabling the Immediately Goto Cleanup on Sequence Failure setting in the Station Options dialog box.
8. Testing Multiple UUTs in Parallel
TestStand installs three process models. The default process model is the Sequential model, which is well-suited for the repetitive testing of a class of UUTs, where one UUT after another is tested on the same test station. The other process models, Parallel and Batch, are better suited for testing multiple UUTs at the same time on the same test station.
Parallel Process Model
The Parallel process model allows you to test multiple devices simultaneously or in quick succession, such as connecting and starting Device 2 after the testing on Device 1 begins but while Device 1 is still running. The Parallel process model is designed for testing multiple UUTs using the same test sequence.
Batch Process Model
Like the Parallel process model, the Batch process model is designed for testing multiple UUTs at the same time. While the Parallel process model allows you to start and finish tests of UUTs at different times, the Batch process model is designed to start and finish the test sequence on all UUTs at the same time. This makes the Batch process model ideal for testing a group of UUTs that are tested as a group, such as a group placed in a temperature chamber.
Test sequences that use the Batch process model can use a special kind of synchronization called batch synchronization. Batch synchronization allows you to specify a multithreading behavior for a step or a group of steps by defining a synchronized section. You can define synchronized sections in one of the following ways:
- Synchronization step properties: Synchronize an individual step
- Batch Synchronization step types: Specify sections of a sequence in which you want to synchronize multiple threads that belong to one batch.
Both methods have the same three types of synchronization settings: Parallel, Serial, and One Thread Only.
Parallel synchronization sections require all UUTs to enter and exit the section at the same time in order to synchronize a specified portion of your sequence. The code within the synchronization section executes across all UUTs in the batch at the same time. You do not need to use this type of synchronization if you want the entire sequence to run in parallel because that is the default behavior of the batch model.
Serial synchronization sections are designed to execute tests on UUTs sequentially, one UUT at a time. This can be useful when you are sharing hardware across all of the UUTs and you need to start and finish a set of steps together. When the test execution of a UUT reaches the Serial synchronization section, the UUT waits until all UUTs arrive to that point within the sequence. Then, TestStand allows one UUT at a time to execute the tests within the section. As each UUT finishes, it waits at the end of the section while TestStand executes the tests on the next UUT. Once all of the UUTs have completed the tests within the synchronization section, TestStand allows all of them to continue executing in parallel.
An example of this type of synchronization section is a DMM measurement in which a single DMM is connected to all UUTs through a switch matrix. Code within the synchronization section can execute on only one UUT at a time, ensuring synchronized access to the shared hardware.
One-Thread-Only synchronization sections perform a test or action once, regardless of the number of UUTs being tested in the batch. Like Serial and Parallel synchronization, the system waits for all UUTs to reach the section and then allows one of the UUT threads to execute the steps within the section. While that UUT executes, the other UUTs remain paused. When the UUT finishes and reaches the end of the section, all of the other UUTs will jump to the end of the section, skipping the code within the section and then continuing to execute in parallel. This type of synchronization is useful for actions that need to be performed only once for the entire batch of UUTs, such as raising the temperature of a chamber that already contains all of the UUTs.
Hardware Sharing Considerations
When testing multiple devices at the same time, you have the choice of duplicating the necessary hardware, using a switching matrix to share resources, or a combination of these approaches. Sharing resources may not result in faster testing unless you design the tests appropriately.
In Figure 10, the Sequential Testing image, tests 1-3 must run on each UUT in a sequence, resulting in a total test execution time of 12 time units (assuming the tests require the same time). If you have the hardware available to test all the UUTs in parallel without sharing hardware, you can complete the tests in 3 time units, as shown in Figure 11. However, if there is not enough hardware for every UUT to have its own dedicated hardware, you can use multiplexing to share the hardware resources. Use the Parallel process model with resource locks for a total test execution time of 6 time units, as shown in Figure 12. You can further optimize your testing time by using Auto-Scheduling, which reorders tests based on available resources. In this example, you could use Auto-Scheduling to finish testing all four UUTs in 4 units of time, as shown in Figure 13.
Figure 10 – Sequential Testing
Figure 11 – Parallel Testing
Figure 12 – Parallel Testing With Shared Hardware
Figure 13 – Autoscheduling
There are a number of different strategies for optimizing the performance of test systems based on TestStand. Each method has its benefits and drawbacks. Ultimately, good test design and careful planning, combined with the techniques described in this document, can result in test systems that perform substantially better than expected, saving you time, money, and resources.
10. View Additional Sections of the NI TestStand Advanced Architecture Series
11. About the Authors
Aaron Gelfand, Senior Systems Engineer, has been with VI Technology in Austin, Texas, since 2004. Aaron graduated from the Florida Institute of Technology with BS degrees in Electrical Engineering and Ocean Engineering. While at VI Technology, Aaron has utilized LabVIEW and TestStand to design customer solutions for multiple industries needing enterprise test systems. He is a Certified LabVIEW Architect, Certified TestStand Developer, and Project Management Professional.
Daniel Elizalde, Product Development Manager, has been with VI Technology in Austin, Texas, since 2006. Daniel graduated from ITESM Mexico City with a BS degree in Electrical Engineering. Before joining VI Technology, he worked as a Senior Systems Engineer at National Instruments where he served as a TestStand consultant for many Fortune 500 companies. Daniel is a Certified TestStand Architect and a Project Management Professional.