TestStand is built on a modular architecture, with many decoupled components, including the TestStand engine, process models, test code, and user interface. This architecture is beneficial for V&V efforts, because each component can be more easily analyzed independently. Furthermore, changes to existing test systems require less revalidation, because the impact can often be limited to a single component.
The modular architecture of TestStand allows each component to be verified and validated independently and reduces the impact of changes on the test system as a whole.
When designing a test system, it is important to consider verification and validation, and how you can architect the system to make these efforts easier. The following sections provide best practices for designing components of your test system with V&V in mind.
Designing Sequence Files
When developing test sequences, keep the following goals in mind:
- Componentize functionality by utilizing subsequences for logical blocks of functionality.
- Reduce the interaction between steps, which lessens the impact of changes to individual steps on the test system.
Focusing on these goals will allow you to better track how requirements are covered in the test code and reduce the impact of changes you make to individual steps.
Using Separate Steps versus Step Settings
When defining a condition for a step, consider if the condition will ever apply to multiple steps. Using If/Else or Case steps to implement logic are more visible in the TestStand environment, are expandable, and can be modified to run different options and include more than one step per condition. However, they introduce relationships between the test steps and the flow control. For logic that will always apply to a single step, using a precondition allows you to contain the logic and the step in a single component.
Use a similar approach when implementing switching in your test code. For routes that apply to a single test, using the switching settings of a step allow you to componentize the switching functionality with the test code. For switching that impacts multiple steps, using switching steps are more visible in a sequence and make the sequence more self-documenting.
Modularize Using Subsequences
To combine the benefits of componentization of built-in step settings with the extensibility of using separate steps, you can create subsequences to encapsulate related sets of steps. By containing sets of such sequences in a separate sequence file, you can effectively create a sequence file which is a library of functions, that can be independently validated and shared between multiple test applications.
In addition, the test sequence should be composed almost entirely of Sequence Call steps. which each implement logical grouping of tests. The organization of subsequences should map to the test specifications, where high level requirements, such as “the system shall test the audio capabilities of the device”, should map to a sequence in the test, while lower level supporting requirements, such as “the maximum sound volume shall not exceed 80 dB”, map to steps within the sequence.
Reduce Dependencies Between Multiple Steps
You can also introduce dependencies between steps is the case when a step requires data obtained in the previous step. Avoid using the PreviousStep property to directly access data from another step, and instead use a local variable to store the data from the first step, then access the variable in the later step.
You should also ensure that each step independently sets or verifies that the required conditions are set before the test is executed. For example, if an Audio Volume Test step includes a volume test at Low, Medium, and High volume, and a subsequent step performs audio quality testing that must be done at Medium volume, you should ensure that the volume is set to medium before performing the test. This ensures that both tests are independent: making changes to the audio volume test will not impact the Audio quality test.
Documenting Test Sequences
Clear documentation of your sequence files is an important part of ensuring that all requirements in the specification are sufficiently covered. However, you should avoid repeating information in documentation, such as limit values or test parameters.
You can use the step name and description to document the purpose of a step. The step name should describe what the step does and why it performs an action, but should not contain parameter values that are defined in the step. For example, instead of naming a step ‘Wait’, name the step ‘Wait for system to boot’. If the name requires more information, use the Description property of the step to specify additional details.
You can also use TestStand integration with NI Requirements Gateway to effectively track where requirements are covered in actual test code. NI Requirements Gateway allows you to quickly see which requirements are covered and allows you to navigate to the step which covers the requirement, speeding the process of verification.
NI Requirements Gateway allows you to create linkages between your requirements documents, test sequences, and test reports to ensure all requirements are covered
You use the Requirements field, which is available for steps, sequences, and sequence files, to provide information on the requirements that they cover. You can use these fields with NI Requirements Gateway to create a mapping between your requirements document and your test code to quickly see where requirements are covered.
Use the requirements field to map steps, sequences, or sequence files to specific requirements in your specification documents
For more information on using NI Requirements Gateway with TestStand to track requirements, refer to the Coupling NI Requirements Gateway with NI TestStand Tutorial.
Developing Independent Code Modules
TestStand steps call code modules to communicate with instrumentation and automation hardware. Code modules can be implemented in a number of languages, including LabVIEW, C++, or C#. Since TestStand provides a natural boundary between steps and code modules, it is beneficial to write code modules which can be tested and validated independently of the TestStand sequence.
To ensure code modules can be tested outside of the test sequence, avoid using the SequenceContext or other TestStand references to access data directly, and instead pass data into the code module through parameters. For cases where using the SequenceContext is necessary, such as implementing a termination monitor, design the code module so that it can function without the TestStand-specific code. In a LabVIEW code module, you can use the “not a reference” function to check if the SequenceContext is valid before using it.
With independently executable code modules, you can design test fixtures which loop over the code module, and pass all permutations of input parameters. The test fixture can then compare the results against known correct results to validate that the code modules are behaving as expected.
Making Changes to Process Models Using Plug-ins
TestStand process models handle test functionality that is not specific to a unit under test, including UUT tracking, report generation, database logging, and batch or parallel testing. The process models that ship with TestStand are complex, so making changes to these models requires significant validation effort.
The process models use a plug-in architecture which implements the default report generation and database logging capabilities. You can utilize this architecture to extend the functionality of the existing process models without making changes to the process model sequence files themselves. To do so, you create a custom plug-in, which is implemented in a separate plug-in sequence file. You can validate the behavior of this plug-in independently.
Process models plug-ins are separate components from the process model files, which can be individually validated
If you need to make changes to functionality directly in the process models themselves, such as making changes to how the model collects the UUT serial number, consider disabling the steps in the process model which implement the existing functionality, then creating a separate plug-in for the new functionality. By using this approach, future changes you make to the custom behavior can be limited to only the plug-in, which will be easier to revalidate.
For more information on how to customize the process models and plug-ins, refer to the Best Practices for NI TestStand Process Model Development and Customization document.
Managing Test Settings & Configuration
When developing a test system, it is important to ensure that all TestStand settings are the same across all stations executing the test, and are never modified without proper validation of the change. However, since many TestStand components have individual settings, it can be difficult to ensure that no changes are present. In addition to TestStand settings, you must also ensure that instrument settings are consistent across test systems. Instrument settings can include NI-DAQmx settings made in NI Measurement & Automation Explorer (MAX), or GPIB and COM settings for a device. Such settings can be numerous and validation tests difficult to design.
One way to ensure that settings are correct is to set them programmatically in your test sequence, then query each setting to ensure that the setting was accepted by the instrument or program. If the setting cannot be queried, find where the setting is stored and read it from a text, INI, or XML file. The system can verify and record the state of items outside of TestStand and keep them under control.
Manage Files Using Source Code Control Tools
Another approach to managing settings is to strictly control the files that contain the settings. Configuration files which store all TestStand settings are stored in the <TestStand Application Data>\Cfg directory. For other settings, refer to the product specific documentation for information on the location of settings files
The industry-accepted method for monitoring, controlling, and storing test system files is Source Code Control (SCC) programs such Subversion, Perforce, and Microsoft Visual Source Safe. Many of these programs are designed to conform to the Microsoft SCC interface and you can use them from within TestStand or LabVIEW. In some cases, you cannot modify a file without taking temporary ownership of it and documenting your changes in order to save them. These programs can often tell you which files changed as well as analyze the old and new files to highlight the changes to help simplify verification.
You can also use file checksums to ensure that settings files have not changed from the validated state. To use this approach, you can add steps which compare the checksum against a checksum you calculate for the validated file value, and generate a test failure if they do not match.