When investigating a test system which is exhibiting unwanted memory growth, you should characterize the issue to determine the rate and magnitude of the growth as well as the section of test code which is causing memory to grow. This document describes several approaches to narrowing down memory growth issues and provides information on tools that you can use to measure the memory utilization of your test system.
It is important to recognize that memory growth does not always indicate a problem with the test system. For example, if a code module is acquiring measurement data from hardware and storing the data in an array, the code module’s memory usage will increase as the data is stored. This type of memory growth is typically expected to occur. However, if memory allocated by code modules or other test system components is not released when it is no longer needed, memory usage will continue to increase as the test system executes and an error or crash can eventually occur once the test system uses all available memory. This uncontrolled memory growth is often referred to as a memory leak.
One of the most common contributors to memory growth in TestStand systems is the result collection and reporting process. If result processing and report settings are not configured to conserve memory, you may see memory growth as test results are collected and retained in memory. In test sequences which execute for a long period of time or generate a large number of results, this can eventually result in an out-of-memory error.
The report and database model plugins that ship with TestStand will process result information either during the test sequence (if On-The-Fly Reporting is enabled) or after a UUT is finished testing (if On-The-Fly Reporting is disabled). For this reason, report generation is unlikely to cause problematic memory growth unless a single UUT test generates a large number of results or executes for a long period of time.
To eliminate result collection and reporting as potential contributors to memory growth, you should disable result collection for all steps and disable all reporting and database logging model plugins.
Complete the following steps to disable result recording for all steps:
Complete the following steps to disable reporting and database logging plugins:
After configuring these settings, you should run your test sequence again to determine whether a memory leak is still present. If you no longer see a memory leak, this indicates that the result recording and report generation settings are not configured to optimize memory usage. You can follow the steps in the Addressing Memory Issues with Report Generation in TestStand whitepaper to configure these settings to minimize memory growth in your test system.
If you continue to observe a memory leak after disabling all result collection and reporting settings, you should isolate the problem down to a subset of your test code. This will allow you to more closely examine a single code module or set of code modules for potential memory leaks.
There are several approaches you can use to begin narrowing the set of test system code, including:
For each of these approaches, you can configure the test sequence to execute only the set of code you wish to test by setting the Run Mode of all other steps to Skip. After ensuring that the sequence is only configured to execute the steps you are interested in testing, you should run multiple iterations of the test sequence while monitoring memory usage using one of the tools described in the Tools to Use for Characterizing Memory Growth document. You can examine the output of your chosen tool to determine whether memory is leaking in the current test case.
As you execute test cases to narrow down the issue, you should maintain detailed records of each test case as well as the memory usage and growth observed in each case. This information will enable you to determine which components of the test code appears to be contributing to memory growth so that you can examine these sets of code in more detail to locate potential memory leaks. In many cases, multiple components in the test system may be contributing to memory growth. When this is the case, you can use the data you have collected to determine which of the potential memory leaks you wish to investigate first.
Once you have identified a smaller set of code that you wish to investigate, you should review the code within individual code modules to ensure that all references and allocated memory blocks are being handled and released properly. In many cases, you may be able to eliminate portions of code from the code module similarly to the process you followed previously to narrow down the test sequence memory growth. Depending on the code modules you are investigating, you may also be able to use memory profiling tools within the code module development environment to track memory usage and identify potential leaks. This process is described in greater detail in the Identifying and Resolving Memory Issues document.
In a large test system, it is often not apparent why memory growth is occurring during execution. This document describes a process you can follow to methodically eliminate sections of the test system and narrow down the issue to a small set of test code. By following this process, you can efficiently determine where the memory growth is occurring and investigate that set of code more closely to determine why memory is growing.