Using Memory Profiling Tools to Characterize Memory Growth Issues in Test Systems

Overview

When characterizing memory growth behavior, it is important to measure how the memory changes over time. There are several tools at your disposal that allow you to view memory usage and other metrics of a test system. Coupling these tools with unique test cases will enable you to isolate the component in your test system that is causing the memory growth. This document describes these tools.

Contents

General Purpose Tools

The tools described in this section allow you to view the memory usage of a process.  You can use this information to determine if a process is experiencing memory growth. These tools are a useful starting point when investigating a possible memory growth issue in a test system.

 

Task Manager

The Windows Task Manager monitors basic resource consumption of an application over time and is a good first step to determine whether a memory growth issue exists in a test system.

You can launch Task Manager by right-clicking the Windows Taskbar and selecting Start Task Manager or through the keyboard shortcut Ctrl+Shift+Esc. In Windows 8 or newer, you will also need to click the More Details button at the bottom of the Task Manager dialog to display the full set of options.

The Processes tab of the Task Manager dialog (or the Details tab in Windows 8 and newer) contains a list of all running processes on the system as well as a set of information about each process. You can configure the display settings for Task Manager by clicking View » Select Columns… (Windows 7 and previous) or right-clicking the header of the table and clicking Select Columns (Windows 8 and newer). For memory growth issues, it is often useful to enable the Memory (Private Working Set), Handles, Threads, USER Objects, and GDI Objects columns. All of these metrics are a snapshot of the resources currently used by the application. The following table explains the meaning of each column.

Column Purpose
Memory (Private Working Set) The set of virtual memory that is currently loaded into physical memory. Does not accurately represent the entire private memory space that a process may be using.
Handles The number of object handles in the process’s object table. Maintaining references to instrument driver sessions, loading LabVIEW VIs, and many other operations can cause the number of handles to increase.
Threads The number of threads running in the process.
USER Objects The number of USER objects currently being used by the process, which can include objects such as windows, menus, cursors, and other internal objects.
GDI Objects The number of objects from the Graphics Device Interface (GDI) library currently being used by the process.

For more information on the purpose of each of these columns, please refer to MSDN.

After enabling these columns, you can locate the TestStand process to view information about the running test system. The name of this process is SeqEdit.exe if you are executing your test system in the TestStand Sequence Editor, or the name of your user interface (such as TestExec.exe) if you are executing your test system in a user interface.

Note: If you are executing code modules in an external process, such as LabVIEW.exe, then the memory growth may occur in that process as well.

To use Task Manager to determine whether memory growth is occurring in the system, you should make note of the value of each column after running the test system in a loop for a few iterations, allow the system to run for a longer period of time, and compare the current values of each column to the values you observed at the beginning of execution to determine if significant growth has occurred in any of the column values. By executing the test code in a loop using the Test UUTs process model or by looping within a sequence, you can more easily detect memory growth that may be occurring during the execution of your test system. During this process, you may see memory growing and then resetting. Sometimes, this is a result of .NET garbage collection.

You may experience out-of-memory errors even if values in the Task Manager columns are below the memory limit of a process on a Windows system. This discrepancy is often due to differences between how much physical memory a process is using versus the virtual memory a process requires.

Physical memory is the physical set of RAM on the system, and is likely to range in size from 4GB to 16GB or more. This is the total physical memory available for use by all processes on the system. The operating system manages how processes use physical memory.

Virtual memory is the memory space available to an individual process, and can range from 2GB to 4GB for a 32-bit process, or up to 8TB for a 64-bit process. Virtual memory is the limiting factor of memory available to a process, so a 32-bit process can only access a maximum of 2-4GB of memory, even on a system with 16GB of physical memory installed. For more information on virtual memory limitations, refer to the Memory Limits of TestStand Systems document.

When you are examining the memory values in Task Manager or any of the other tools described in this document, it is important to understand whether the value is a count of virtual memory or physical memory. For example, the Memory (Private Working Set) column in Task Manager displays the amount of a process’ virtual memory that is currently loaded into physical memory. Since this value is not indicating the total amount of virtual memory being used by the process, it is not an accurate indication of how close the process is to its total memory limit.

For this reason, when examining memory counters in Task Manager or any of the tools in this document, you should consider how the counters are changing over time rather than focusing on the absolute value of the counters.

 

Performance Monitor

Performance Monitor is a more robust and feature-rich monitoring tool that is helpful for graphically visualizing resource consumption of an application over time. Unlike Task Manager, Performance Monitor will measure the memory usage of the process at regular intervals, display this information on a graph, and optionally save the data to a log file. This is useful if the test system must run for a long period of time before memory growth is apparent.

Performance Monitor is provided as part of the Windows operating system.  You can launch it by opening the Windows Start Menu and searching for perfmon.exe or by navigating to C:\Windows\System32 and launching the perfmon.exe application from that directory. After opening the Performance Monitor tool, you can view a graph of memory usage by clicking the Performance Monitor entry under the Monitoring Tools section on the left side of the window.

Performance Monitor displays information by graphing the values of counters. These counters represent various aspects of memory usage and system performance, such as the total number of memory bytes used by a process or the percentage of CPU utilization of the system. You can configure the set of counters displayed on the Performance Monitor graph.

Recommended Counters

The following table of counters may be useful for examining memory growth in a test system.

Counter Purpose
Process » Private Bytes The amount of private memory in use by the process. A process’ virtual memory space consists of private memory, which can only be accessed by the process, and shared memory, which can be accessed by other processes.
Process » Handle Count The number of object handles in the process’s object table. Maintaining references to instrument driver sessions, loading LabVIEW VIs, and many other operations can cause the number of handles to increase.
Process » Thread Count The number of threads running in the process.
.NET CLR Memory » # Bytes in all Heaps The total amount of memory being used by the process to store .NET objects.
.NET CLR Memory » Gen 2 heap size The amount of memory being used by the process to store .NET objects on the Gen 2 heap. The Gen 2 heap is an area of .NET memory where objects which have been in memory for a long period of time are stored. This value may indicate that .NET objects are not being released when they are no longer needed.

 

Adding Counters

Complete the following steps to display counters on the Performance Monitor graph:

  1. In the table at the bottom of the Performance Monitor window, select any existing counters and press the Delete key on the keyboard to remove these counters from the graph display.
  2. Right-click in the table at the bottom of the window and select Add Counters…
  3. In the Add Counters dialog, you will see a scrollable list box with all of the counters available to you, grouped into sections. Locate the section you are interested in, expand it, and select a counter.
  4. Below the list of counters, select the process of interest in the Instances of selected object list. This is typically SeqEdit or the name of your user interface executable, depending on whether you are executing your test system in the TestStand Sequence Editor or a user interface.
  5. Click the Add button to add this counter to the Added counters list.
  6. After configuring all counters, click OK on the Add Counters dialog.

Adjusting Counter Scale and Sampling

After adding counters, you will notice that the Performance Monitor graph begins updating with the values of the counters. You will likely need to adjust the settings of the graph to ensure that all counter values are scaled properly for the graph display and to ensure that the graph displays data from a long enough period of time for you to determine whether memory usage is growing as the system executes.

Complete the following steps to configure the Performance Monitor graph:

  1. Select all counters displayed in the table at the bottom of the window. Right-click and select Scale Selected Counters. This will scale the values of each counter so that the counters are visible in the graph window.
  2. Right-click in the graph and select Properties.
  3. On the General tab, you can set the sample rate and total number of samples displayed on the graph. You should configure these numbers so that the entire graph will display memory growth over a long period of time. You can set the Sample every value to 10 and the Duration value to 10000 to graph a new value every 10 seconds and display a total of approximately 2.5 hours of information in the graph window.

After configuring the counters and graph display, you can execute a test case and examine the Performance Monitor graph to determine whether any of the counter values are growing as the test system executes. Before running a test, you can clear the graph data by right-clicking in the graph area and selecting Clear.

Example Use Case

This is an example of what the Performance Monitor graph might look like after running a test case:

You can see from this image that the Private Bytes counter (blue line) grows at a constant rate, while the # Bytes in all Heaps and Gen 2 heap size counters do not. This indicates that the memory space of the Sequence Editor process is growing over time, which indicates that a memory leak could occur. Since the # Bytes in all Heaps and Gen 2 heap size counters are not growing over time, the memory growth does not appear to be caused by .NET memory. If these counters were growing, this would indicate that the process is continuing to open .NET references and not closing the references that are no longer needed.

For more information about the significance of Gen 2 heap size and .NET garbage collection, please refer to MSDN.

 

Specialized Tools

Once you have determined that memory growth is occurring in a test sequence, you may wish to use a more specialized tool to gather more detailed information about the type of memory growth that is occurring. This section describes some specialized memory debugging tools that you may use to gather this information.

.NET Memory Profiler

The .NET Memory Profiler is a third-party tool that allows you to take a snapshot of all .NET objects in the memory of a process. The tool also allows you to compare multiple snapshots, which you can use to determine which .NET objects are remaining in memory throughout multiple executions of the test sequence. If the Performance Monitor graph indicates that .NET memory is growing as your test sequence executes, you can use the .NET Memory Profiler to determine which .NET objects are contributing to the memory growth.

The .NET Memory Profiler tool is not a National Instruments product and does have a paid license. However, you can download a free trial of the tool from www.memprofiler.com. The trial is fully functional and will allow you to use the tool to investigate .NET memory growth issues for a limited period of time.

 

Conclusion

When investigating a test system which is exhibiting memory growth, you should track memory usage in the system. This allows you to make changes to the test system—such as modifying report settings and reducing the size of the test sequence—and understand how these changes impact memory growth in the system.  You can use this information to determine which settings should change or which steps you should investigate more closely.