Use the Test Program Performance Analyzer to visualize performance log data. Graph types include step times statistics, single step times, scatter distribution, and parallel test efficiency (PTE). Use the Graph Type control to select the type of graph to display. Use the Timing Method section of the analyzer to specify timing options for the graph.

Graph Type Task
Step Times Statistics

Select Step Times Statistics in the Graph Type control to display a graph you can use to analyze performance bottlenecks by visualizing statistical information on a step’s test times. Use the Sorting Method section of the analyzer to specify sorting options for the graph. By default, TSM enables the following options:

  • Show Average Step Times—Displays a colored bar indicating the average time it takes for a step to run. To hide this plot, remove the checkmark from the Show Average Step Times checkbox.
  • Show PTE%—The top x-axis of the graph displays the PTE percentage of the step. The bottom x-axis of the graph displays the step time in milliseconds. A diamond point over the bars in the graph displays the calculated PTE for each step and number of sites. To hide this plot, remove the checkmark from the Show PTE% checkbox.
  • Show Distribution Box Plot—Displays the statistical distribution of step times as a box plot. To hide this plot, remove the checkmark from the Show Distribution Box Plot checkbox.

Hover over a bar or box plot in the graph to display a tooltip with the following information:

  • Step name
  • Number of Sites (in the multisite configuration)
  • Average (if Show Average Step Times is enabled)

When you select Compare Data Sets in the Mode section of the analyzer, the value in the tooltip includes the value of the bar the mouse is hovering over, the matching value in the other log file, and the difference between the two values.

The Show Distribution Box Plot option displays the following additional statistical distribution information at the bottom of the tooltip:

  • Minimum
  • Lower Quartile
  • Median
  • Upper Quartile
  • Maximum

Click a bar in the graph to display a graph of the Single Step Times for that step name and the number of sites. To return to the Step Times Statistics graph, click the back button next to the title in the upper left corner of the graph.

Click a plot name to the left of the graph area to display a scatter graph of the step times for that step name. To return to the Step Times Statistics graph, click the back button next to the title in the upper left corner of the graph.

When you right-click the step name, a context menu opens. Click Go to Step in Sequence Editor to navigate to the step's location in the sequence editor. If the sequence file cannot be found, or if the step or sequence have been deleted, the sequence editor displays an error message.
Single Step Times

Select Single Step Times in the Graph Type control to display a graph you can use to view the time it takes for one step to run. Use the corresponding options in the Graph Type section of the analyzer to specify the step name and number of sites to display in the graph for the step. You can also click a step in the Step Times Statistics graph to display the graph for the step.

The upper x-axis of the graph displays the site. It also displays the data set log data file when you select Compare Data Sets in the Mode section of the analyzer, using the letter B to indicate the base data set log file and M to indicate the modified data set log file. The lower x-axis of the graph displays the batch index number.

The graph groups sites by subsystem and overlays a code module bar on sites that run a code module on that subsystem.

Hover over a bar in the graph to display a tooltip with the following information:

  • Plot Name
  • Step Name
  • Number of Sites
  • Batch Index
  • Site
  • Subsystem
  • Code Module Time or Value (When you select Compare Data Sets in the Mode section of the analyzer, the value in the tooltip includes the value of the bar the mouse is hovering over, the matching value in the other log file, and the difference between the two values.)

Click the back button next to the title in the upper left corner of the graph to view the Step Times Statistics graph.
Scatter Distribution

Select Scatter Distribution in the Graph Type control to display a graph you can use to view the statistical distribution of test times. Use this view to identify outliers in test time distribution. Select Socket Time to view the distribution of your entire test program. Alternatively, select Code Module Time in the Timing Method section, and then select MainSequence Time to view the distribution of the entire test program. Select Single Step Time and specify a step name to view the distribution of a single step's test times.

Each plot represents a given number of sites. Each point represents the average test time over all sites in the configuration on the given batch index. Click a point on the Scatter Distribution graph to go to that batch index in the Single Step graph to view more detail on individual sites. Click the back button next to the title in the upper left corner of the graph to return to the Scatter Distribution graph.

Hover over a point in the graph to display a tooltip with the following information:

  • Plot Name
  • Batch Index
  • Number of Sites
  • Value

Click the back button next to the title in the upper left corner of the graph to return to the Step Times Statistics graph.
Socket Times/PTE (Parallel Test Efficiency)

Select Socket Times/PTE in the Graph Type control to display a graph you can use to view the overall socket time or MainSequence time of the test program for different multisite configurations.

The bars on the graph display the test time, and the diamonds on the graph display the calculated PTE of the test time relative to the single site configuration test time. TSM uses the following formula to calculate the PTE:

PTE ( n )   =   [ 1 − T n − T 1 T 1 × ( n − 1 ) ]   ×   100 %

where

  • n = number of sites
  • T1 = single site test time
  • Tn = n site test time

Use the following options in the Target Socket or MainSequence Time section of the analyzer to configure an optional plot that displays the target test time plot for each multisite configuration. The graph displays a horizontal line on each bar to indicate the time under which the target PTE percentage has been met.

  • Target PTE(%)—Specify the target PTE value.
  • Benchmark Single Site Socket or MainSequence Time (ms)—Specify the single site socket or MainSequence time in milliseconds to use to calculate the target test time.

Hover over a bar in the graph to display a tooltip with the following information:

  • Plot Name
  • Value (When you select Compare Data Sets in the Mode section of the analyzer, the value in the tooltip includes the value of the bar the mouse is hovering over, the matching value in the other log file, and the difference between the two values.)

Timing Method

The Timing Method section of the analyzer includes the following options:

  • Total Time—Total time to run the step, including any TestStand overhead and the time it takes to run any code modules.
  • Code Module Time—Only the time it takes to run the code module, if any, for the step.

When you select Socket Times/PTE in the Graph Type control of the analyzer, the Timing Method section includes the following options:

  • Socket Time—Total time to run the MainSequence sequence, including any TestStand overhead. Socket time is different from cycle time, which includes the socket time and the time required to perform other tasks, such as binning, placing DUTs, and generating reports.
  • MainSequence Time—Only the time it takes to run the MainSequence sequence.

Sorting Method

The Sorting Method section of the analyzer for the Step Times Statistics Graph includes the following options based on whether you select Single Data Set or Compare Data Sets in the Mode section:

  • Execution Order—Sorts the steps in execution order. This is the default setting for the Sorting Method option.
    Note If you are comparing data sets and the execution order differs between the two sets, the analyzer uses the execution order of the base log file data. If the execution order changes after the first time you take a test program performance measurement, the analyzer reflects the execution order used in the first run.
  • Step Name—Sorts the steps alphabetically by step name.
  • Test Time (Base, Modified, Difference)—Sorts steps by the selected test time statistic, highest to lowest. You can sort steps by the average, minimum, lower quartile, median, upper quartile, or maximum of their test times. For multiple sites, the analyzer sorts by the time of the highest number of sites. If you are comparing data sets, you can sort by the test times in the base log file data, the modified log file data, or the difference between the two.
    • Show Top X Test Times—Specifies the number of steps with the highest test times to display on the graph.
    • Show All Test Times—Displays all the steps with the highest test times.
  • Sort by PTE (Base, Modified)—Sorts steps by the worst PTE value. If you are comparing data sets, you can sort by the PTE values in the base log file data or the modified log file data.

Show Single Site Data

The analyzer displays this option only when performance data contains only data for single site and one other multisite configuration. In this situation, the analyzer hides the single site data by default because it typically is being included only for the purpose of calculating PTE. Enabling this option overrides this behavior and displays the single site data in the graph.