Choosing the Appropriate NI TestStand Report Generation Strategy

NI TestStand 2012 supports new report generation features, including asynchronous report generation, offline report generation, and support for a new version of the Automatic Test Markup Language (ATML) report format. These features greatly increase the flexibility of NI TestStand report generation capabilities. This paper discusses the new report generation features and outlines how to choose the appropriate report generation strategy for your test system.

1. Introduction
2. New Report Generation Features in NI TestStand 2012
3. Report Generation Considerations
4. Use Cases
5. Custom Result Processing Plug-ins
6. Conclusion
7. Additional Resources

Introduction

Most automated test systems are required to generate a report that logs information about each test run against the unit under test (UUT), including the test status, measurement results, test parameters, and diagnostic information collected during the test. Test systems also have system-level requirements and constraints—such as hardware platform, test throughput, tester up-time, and linkage to Manufacturing Execution Systems (MES) or other enterprise management systems—that can influence the report generation strategy the test system uses. In most cases, the report generation strategy defined for one test system may not meet the requirements of another test system, even within the same organization. Test system architects must typically consider many or all of the following requirements when defining the appropriate report generation strategy for a particular test system:

• Maximize test system throughput
• Minimize report file size
• Include all necessary report content
• Interoperate with other processes and systems
• Support post-failure information recovery
• Generate reports immediately for the current UUT before testing the next one
• Generate multiple reports for each UUT
• Generate and view the report during test sequence execution

This paper describes how the new report generation features in NI TestStand 2012 can help meet these requirements and discusses the tradeoffs to consider when designing report generation for your test system. Use cases illustrate how common test system characteristics and requirements map to specific report generation strategies.

New Report Generation Features in NI TestStand 2012

NI TestStand 2012 features a new result processing architecture that includes many significant enhancements compared to earlier versions.  A key element of the new result processing architecture is a plug-in architecture for model sequence files. This plug-in architecture decouples the process model from report generation and significantly improves the consistency and performance of the report generators while adding flexibility and customization to report generation. The new reporting features in NI TestStand 2012 help you complete the following tasks:

• Generate reports asynchronously with respect to test execution
• Generate multiple reports in different formats
• Generate reports on-the-fly with significant performance improvement compared to NI TestStand 2010 SP1 or earlier
• Generate reports that validate against version 5.0 of the ATML Test Results and Session Information schema
• More easily customize report generation functionality without having to modify the process model files and vice versa
• Log result data in a fast and compact raw result format (.tsr) for offline processing
• Generate reports offline using the NI TestStand Offline Results Processing Utility

Generate Reports Asynchronously to Test Execution

In NI TestStand 2010 SP1 or earlier, the built-in process models call report generation callbacks synchronously in the UUT loop execution thread. NI TestStand generates the report for one UUT before testing the next UUT. This can negatively impact throughput of the test system because your test instrumentation is typically idle during report generation.

With NI TestStand 2012, you can choose to invoke the report generation callbacks in a separate thread that is asynchronous to the UUT loop execution thread. In this architecture, the UUT loop execution thread begins testing the next UUT without waiting for NI TestStand to generate the report for the previous UUT. In systems with multiple CPUs and sufficient system memory, this approach can improve instrument use and overall throughput of the test system.

Generate Multiple Reports in Different Formats

In NI TestStand 2010 SP1 or earlier, the built-in process models can generate only one report for each UUT. Modifying the built-in process models to generate multiple reports requires non-trivial changes. When upgrading to a new version of NI TestStand, you must reapply these changes.

In NI TestStand 2012, you can choose to generate multiple reports for each UUT. You can enable and configure each report independently of the other and generate each report in a different format. For example, you can generate an ASCII text report for readability and a detailed ATML report for post-processing.

Generate Reports On-the-Fly with Significant Performance Improvement Compared to NI TestStand 2010 SP1 or Earlier

NI TestStand includes a built-in, on-the-fly report generation framework to support generating reports during test execution. This framework consists of NI TestStand report generation sequence files and an on-the-fly server. During the testing of a UUT, the report generation sequence files execute to generate report snippets from step results and invoke the on-the-fly server to store the generated report snippets. At defined intervals, the report generation sequence files invoke the server to persist the partial report to disk.

You can enable the On-The-Fly Reporting option on the Contents tab of the Report Options dialog box. On-the-fly reporting is useful when you want to view partial reports during the testing of a UUT in order to monitor testing progress. It is also useful when you want to maximize the number of results persisted to disk in case a failure occurs during the testing of a UUT.  

In addition, you can enable the Only Display Latest Results option (named Conserve Memory and Only Display Latest Results in NI TestStand 2010 SP1 or earlier) in the Model Options dialog box to save memory during the testing of a UUT. When you enable this option, the on-the-fly report generation framework frees NI TestStand ResultList elements and report snippets for results that have already been persisted to the report. When using this option, NI TestStand displays only a partial report with results that have not yet been persisted to disk.

On-the-Fly Reporting in NI TestStand 2010 SP1 or Earlier

In NI TestStand 2010 SP1 or earlier, you can use on-the-fly reporting for XML, HTML, and ASCII reports but not for ATML reports. In addition, you can enable the Conserve Memory and Only Display Latest Results option for HTML and ASCII reports but not for XML or ATML reports.

Report Format	On-the-Fly Reporting	Conserve Memory option
HTML
ASCII
XML
ATML

Figure 1: On-the-Fly Report Options Supported in NI TestStand 2010 SP1 and Earlier

To avoid the inefficiency of writing results to disk after every step, the NI TestStand 2010 SP1 or earlier on-the-fly server persists results to disk at a fixed time interval (10 seconds).

On-the-Fly Reporting in NI TestStand 2012

In NI TestStand 2012, you can use on-the-fly reporting for all report formats, including ATML reports. The Conserve Memory and Only Display Latest Results option is called Only Display Latest Results, and it now supports all report formats.

Report Format	On-the-Fly Reporting	Conserve Memory option
HTML
ASCII
XML
ATML

Figure 2: On-the-Fly Report Options Supported in NI TestStand 2012

In NI TestStand 2012, the on-the-fly server was overhauled to increase performance and reduce memory usage. By default, the on-the-fly server uses several factors to determine the interval at which to persist results to disk. You can adjust these factors in the Advanced Result Processing Settings dialog box.

Generate Reports That Validate Against Version 5.0 of the ATML Test Results and Session Information Schema

NI TestStand 2012 can generate ATML reports that validate against the non-finalized version 5.0 of the ATML Test Results and Session Information schema. If necessary, an NI TestStand update will be made available after version 5.0 of the ATML Test Results and Session Information schema is finalized to support the final format.  

If you do not need to include information in your report other than the information the schema defines, you can disable the Include TestStand Extension Elements option on the Contents tab of the Report Options dialog box to reduce the size of the ATML 5.0 report.

More Easily Customize Report Generation Functionality Without Having to Modify the Process Model Files and Vice Versa

In NI TestStand 2010 SP1 or earlier, the default report generation framework is tightly coupled with the built-in process model. Most of the report generation code resides directly in the process model sequences. To customize report generation functionality, you need to thoroughly understand the design of the process models, the report generation framework, and the integration of the report generation framework with the process model. This coupling makes it difficult to change the report generation functionality to add support for a new report format or to generate multiple reports.

In NI TestStand 2012, the built-in process models use sequence files called process model plug-ins to process test results and create reports. Each plug-in contains sequences NI TestStand calls during process model execution to accomplish result processing tasks, such as report generation and database logging. These plug-ins are completely decoupled from the process model code. NI TestStand includes built-in plug-ins for report generation, database logging, and raw results logging. You can create custom process model plug-ins to extend or modify the functionality of the process models without changing the process model files.

The new architecture also supports creating multiple result processing configurations, which are named sets of plug-in instances you configure to accomplish a specific task, such as creating multiple reports with different formats or logging data to multiple databases. You can specify the active configuration to quickly change how NI TestStand processes results. For example, you can create one configuration used during production that logs to a database and generates a summary report, and you can create another that is used during debugging and generates a detailed report.

Log Result Data in a Fast and Compact Raw Result Format for Offline Processing

In NI TestStand 2012, you can log result data in a fast and compact raw result format (.tsr) for offline processing at a time other than when sequence execution occurs or on a different computer. This raw results file contains all the information NI TestStand requires to generate a report in any supported format offline. This approach can significantly improve throughput of your test system.

You can configure the raw result plug-in to automatically start the NI TestStand Offline Results Processing Utility to process a generated raw results file.

Generate Reports Offline Using the NI TestStand Offline Results Processing Utility

In NI TestStand 2012, you can use the new Offline Results Processing Utility to process raw results files and generate reports at a time other than when sequence execution occurs, such as overnight or on a different computer. Use this utility in situations in which test system performance is more important than immediate report generation.

Because the utility is a separate application, you can use command-line arguments and the Microsoft Windows Task Scheduler to automate the utility. You can also use an NI TestStand Call Executable step with command-line arguments in a sequence to launch the utility. If the utility is already running, the command-line arguments apply to the running instance of the utility.

The utility uses profiles to specify the set of raw results files to process and the set of result processing configurations to use as well as to remember the set of files the utility has already processed.

You can create result processing configurations to process results inline (at run time) or offline. For example, you can create one configuration that processes offline results differently than another configuration that processes inline results, or you can specify different configurations to use when processing offline result files, depending on where the offline results files are stored on disk.

Report Generation Considerations

Test system architects must typically consider many or all of the following requirements when defining the appropriate report generation strategy for a particular test system:

• Maximize test system throughput
• Minimize report file size
• Include all necessary report content
• Interoperate with other processes and systems
• Support post-failure information recovery
• Generate reports immediately for the current UUT before testing the next one
• Generate multiple reports for each UUT
• Generate and view the report during test sequence execution

The following sections examine each of these requirements, recommend NI TestStand report generation settings and configurations that best meet each requirement in typical scenarios, and identify other requirements you might have to sacrifice to optimize your system for the requirement. National Instruments recommends  you use this discussion as a starting point for your system and iteratively measure and modify your system to achieve the optimal balance among the requirements.

You should consider the different scenarios you will run your test system and which requirements are most important for each scenario. For example, when you first develop and debug a test system, throughput might not matter at all and support for post-failure information recovery might be critical. When you move to production, throughput might be the most important requirement. With NI TestStand 2012, you can use multiple results processing configurations to switch easily between settings based on your test system execution scenario.

Maximize Test System Throughput

Automated test systems are usually designed to maximize throughput by minimizing the time required to verify a UUT. Test system architects spend a considerable amount of time optimizing test system performance. In a typical test system, the amount of time required for report generation depends on a number of factors, including available resources (processor and memory), the format of the report being generated, the number of results, and additional data being logged.

Recommended Report Options:

• Single-CPU Computers: National Instruments benchmarks for logging 500 or fewer step results on single-CPU computers show that ASCII reports have the least negative impact to throughput and XML reports have the most negative impact to throughput. When logging more than 500 step results or large data sets, benchmarks show that TSR reports have the least negative impact to throughput and XML reports have the most negative impact to throughput. Benchmarks also show asynchronous report generation does not improve throughput on single-core computers, except in cases where a substantial amount of sequence execution time is spent waiting on operations that do not utilize the CPU, such as instrument I/O.
  
  500 or fewer step results:

  Report Option	Value
  Format	ASCII, TSR*, ATML, HTML, XML (Best to worst)
  Asynchronous	False
  On-the-Fly Report	False
  Only Display Latest Results	False

  * The TSR format generates a compact raw results file and, if used, requires the NI TestStand Offline Results Processing Utility to generate the final report.
  
  More than 500 step results:

  Report Option	Value
  Format	TSR, ASCII, ATML, HTML, XML (Best to worst)
  Asynchronous	False
  On-the-Fly Report	False
  Only Display Latest Results	False

• Multi-CPU Computers: National Instruments benchmarks for logging 500 or fewer step results on multi-CPU computers show  ASCII reports have the least negative impact to throughput and XML reports have the most negative impact to throughput. When logging more than 500 step results or large data sets, benchmarks show that TSR reports have the least negative impact to throughput and XML reports have the most negative impact to throughput. Benchmarks also show  asynchronous report generation improves throughput on multicore computers for all report formats.
  
  500 or fewer step results:

  Report Option	Value
  Format	ASCII, ATML, TSR, HTML, XML (Best to worst)
  Asynchronous	True
  On-the-Fly Report	False
  Only Display Latest Results	False

  
  More than 500 step results:

  Report Option	Value
  Format	TSR, ASCII, ATML, HTML, XML (Best to worst)
  Asynchronous	True
  On-the-Fly Report	False
  Only Display Latest Results	False

Tradeoffs:
Consider the impact to the following requirements when you optimize for test system throughput:

• Minimize report file size
• Include all necessary report content
• Interoperate with other processes and systems
• Support post-failure information recovery
• Generate reports immediately for the current UUT before testing the next one
• Generate multiple reports for each UUT
• Generate and view the report during test sequence execution

Minimize Report File Size

Automated test systems can generate a significant amount of data when testing a UUT. The amount of data generated depends on the complexity of the UUT being tested and the capabilities of the test equipment. The size of the report generated depends on the format of the report and the amount of data being logged. The data logged in the report is influenced primarily by system-level requirements of storage for traceability or analysis and by the testing and debugging processes used. In all cases, benchmarks the TSR report format is the most efficient at minimizing size on disk.

Recommended Report Options:

Tradeoffs:
Consider the impact to the following requirements when you optimize for report size:

• Maximize test system throughput
• Include all necessary report content
• Interoperate with other processes and systems
  

Include All Necessary Report Content

Test systems have different data logging requirements influenced by system-level requirements of storage for traceability or analysis and by the testing and debugging processes used. Each report format NI TestStand can generate contains different pieces of information. For example, the ASCII and HTML report formats contain only the step name, status, limits, limit comparison type, module time, and any additional results associated with the step. The TSR format contains all the information available in the ASCII and HTML formats and additional information such as the parent step IDs of steps and the execution thread IDs.

You can consider the set of data available in each report format according to the following categories:

• Minimal - Contains minimal context information such as the station ID, user name, execution start date and time, overall status, and minimal step-related information (such as the step name, module time, measurement value, and limits and module time).
• Controlled – Contains all the information in a minimal report and information associated with a step, such as step ID.
• Moderate – Contains all the information in a controlled report and information such as the step type, step group, total step execution time, and report options.
• Complete – Contains all the information in a moderate report and information such as the parent step IDs and execution thread IDs. Refer to the ResultLogRecordTypes Enumeration topic in the NI TestStand 2012 Help for the complete list of all data logged.

Report Content	ASCII	HTML	ATML 2	ATML 5	ATML 2 (w/ extensions)	ATML 5 (w/ extensions)	XML	TSR
Minimal
Controlled
Moderate
Complete

Recommended Report Options:

Tradeoffs:
Consider the impact to the following requirements when you optimize for report content:

• Maximize test system throughput
• Minimize report file size
• Interoperate with other processes and systems
  

Interoperate With Other Processes and Systems

Many test systems integrate with an Enterprise Resource Planning (ERP) system or a Manufacturing Execution System (MES) that processes test system reports to provide business-critical data to other systems. In some cases, an engineer or technician might process the report for statistical analysis on a separate machine. These post-processing requirements might dictate the report generation format for the test system.

The ATML and XML reports are saved in XML file formats that are structured and easily processed using SAX or DOM parsers. The ASCII and HTML reports are saved in ASCII file formats that are unstructured and not as easy to post-process as XML. The TSR file format is defined by National Instruments and requires the NI TestStand Engine and the NI TestStand Offline Results Processing Utility to generate another file in a format you can process directly.

Recommended Report Options:

* Although you cannot process TSR files directly, you can use the Offline Results Processing Utility to generate files you can process directly.

Tradeoffs:
Consider the impact to the following requirements when you optimize for interoperating with other processes and systems:

• Maximize test system throughput
• Minimize report file size
• Include all necessary report content
• Support post-failure information recovery
• Generate reports immediately for the current UUT before testing the next one
• Generate multiple reports for each UUT
  

Support Post-Failure Information Recovery

In the case of a catastrophic failure, it might be critical for step result information collected during the execution to be available for analysis. You can configure NI TestStand to generate reports on-the-fly while testing a UUT. By default, NI TestStand is configured to generate and persist the on-the-fly report after 500 step results have been collected or after every 1.5 seconds of execution time, whichever occurs first. Logging reports to disk can be slow and negatively impact throughput. In some cases, you can improve performance by changing the values of the Processing Interval or the Maximum Number of Results options in the Advanced Result Processing Settings dialog box. You can also improve performance by enabling the Only Display Latest Results option on the Contents tab of the Report Options dialog box. National Instruments benchmarks show TSR reports provide better throughput while generating an on-the-fly report.

Recommended Report Options:

Tradeoffs:
Consider the impact to the following requirements when you optimize for post-failure information recovery:

• • Maximize test system throughput
  • Interoperate with other processes and systems
  • Generate and view the report during test sequence execution
    

Generate the Report for the Current UUT Before Testing the Next UUT

At the end of the test sequence execution, some systems require a printout of a report associated with the UUT be moved to the next stage of the manufacturing process along with the UUT. For such reports, raw results generators and asynchronous report generation are not appropriate. This requirement does not impact other report formats. You can generate on-the-fly reports to satisfy this requirement because on-the-fly reports are generated synchronously.

Recommended Report Options:

Tradeoffs:
Consider the impact to the following requirements when you optimize for ensuring the report for a UUT is generated before the next UUT is tested:

• Maximize test system throughput
• Generate and view the report during test sequence execution
• Generate multiple reports for each UUT
  

Generate Multiple Reports for Each UUT

Some test systems might specify a requirement to generate more than one report for each UUT. For example, you might want to generate a fast and simple summary ASCII report and a detailed ATML report that can post-processed by an MES. You might also want to generate a summary report for the customer and a detailed report for internal use. You can configure NI TestStand to generate multiple reports for each UUT. National Instruments benchmarks show generating a raw results file and processing it offline to generate multiple reports yields the maximum throughput on multi-CPU computers.

Recommended Report Options:

Tradeoffs:
Consider the impact to the following requirements when you optimize for generating multiple reports:

• Maximize test system throughput
• Interoperate with other processes and systems
• Generate reports immediately for the current UUT before testing the next one
• Generate and view the report during test sequence execution
  

Generate and View the Report During Test Sequence Execution

Some test procedures can require a significant amount of time (several hours or more) to complete because the procedures contain a significant number of tests or individual tests take a long time to complete. To help diagnose issues early in such systems, it is necessary to generate and display a partial report while the test sequence is executing rather than wait until the end of execution to generate and display the report.

You can configure NI TestStand to generate a report on-the-fly to support such test sequences. National Instruments benchmarks show  if you want to view the entire report as the testing progresses, an on-the-fly ASCII report provides the best throughput. If you do not need to see the entire report, all reports generated on-the-fly when you enable the Only Display Latest Results option provide approximately equivalent throughput. Because you cannot generate the report associated with raw results until after the execution completes, you cannot use the raw results file format to generate and view the report during the test sequence execution.

Test sequences that specify a large number of steps generate very large reports. Generating and viewing a large report while executing a test sequence can cause significant performance issues. Therefore, when you generate reports on-the-fly, National Instruments recommends you enable the Only Display Latest Results option.

Recommended Report Options:

• Test sequence that contains a few steps and takes a long time to execute:

  Report Option	Value
  Format	ASCII, ATML, HTML, XML (Best to worst)
  Asynchronous	N/A
  On-the-Fly Report	True
  Only Display Latest Results	False

• Test sequence that contains a large number of steps:
  

  Report Option	Value
  Format	ASCII, ATML, HTML, XML (Best to worst)
  Asynchronous	N/A
  On-the-Fly Report	True
  Only Display Latest Results	True

Tradeoffs:
Consider the impact to the following requirements when you optimize for this requirement:

• Maximize test system throughput
• Interoperate with other processes and systems
• Support post-failure information recovery
• Generate reports immediately for the current UUT before testing the next one
  

Use Cases

The following use cases provide examples of different test system requirements and different report generation configurations to meet those requirements.

Use Case 1

Consider a test system in which the report generation functionality must maximize test system throughput; generate and display a summary report immediately after testing the UUT; and generate a detailed report an MES can post-process. You can create one report configuration to generate a high-level summary report and then different configurations for more detailed reports, as the following recommended report options describe:

Recommended Report Options:

Summary Report:

Detailed Report (Variant 1):

Detailed Report (Variant 2):
In some cases, the MES might not post-process the detailed report at the time the UUT is tested but instead at a later time or on a different computer. In this case, you can generate a TSR file and post-process results using the offline result processing utility to generate an ATML report that MES can post-process.

Use Case 2

Consider a test system that runs production tests and is located off-site at a contract manufacturer facility. The report generation functionality must have post-failure information recovery to help diagnose and isolate faults and  store the report for each UUT tested for traceability or analysis. You can generate a compact raw results file at the offsite test system, transfer the TSR file on-site, and generate the final report using the offline results processing utility.

Recommended Report Options:

Custom Result Processing Plug-ins

In some cases, your test system might specify a unique set of requirements that the built-in report generators cannot satisfy. National Instruments recommends  you choose a report format that most closely satisfies your set of requirements and modify it to fit your requirements. With NI TestStand 2012, you can create custom result processing plug-ins.

For example, your test system might require the performance throughput of an ASCII report format, the interoperability of an XML report format, and the ability to log a small amount of data to the report. Using NI TestStand 2012, you can create a custom result processing plug-in that generates an XML report containing only the information you require. Refer to the Process Model Plug-in Architecture topic in the NI TestStand 2012 Help for more information about creating a custom result processing plug-in.

Conclusion

Test systems often have complex and competing requirements or constraints. NI TestStand 2012 introduces several report generation features to help you address these requirements and constraints. The information discussed in this paper can help you choose the most appropriate NI TestStand report generation strategy for implementing the functionality and performance you require.

Additional Resources

Watch the What's New in NI TestStand 2012 webcast

Download a free evaluation of NI TestStand

0 Ratings | 0.00 out of 5

Read in  Chinese (Simplified) English Japanese |  Print |  PDF

Reader Comments | Submit a comment »