To reduce the need to design and maintain your own data file format, National Instruments has created a flexible data model called technical data management (TDM), which is natively accessible through NI Measurement Studio, LabWindows/CVI, LabVIEW and DIAdem and is portable to other applications such as Microsoft Excel. A TDM DLL is also available for accessing the TDM data model with third-party software applications. The TDM data model offers several unique benefits, including the abilities to scale to your specific project requirements and to easily attach descriptive information to your measurements while streaming your data to disk. The TDM data model supports two file formats, TDM and TDM Streaming (TDMS). The new TDMS .NET Class Library is optimized for streaming well-documented data to disk. The other file format of the TDM data model, written by the TDM DLL, offers the same flexibility with regard to documenting data, but is not optimized for streaming data to disk. The new TDMS API provides the widest coverage of different use cases, and as a result it is the recommended way to interface with the TDM data model. The TDM white paper provides additional information including a comprehensive overview of the NI TDM platform.
This white paper explains the TDM data model and demonstrates how the Measurement Studio TDMS API reduces your programming time while increasing the value of the data you are collecting.
2. The TDMS File Format of the TDM Data Model
The TDM data model offers three levels of hierarchy – root, group, and channel (see Figure 1). Each level accepts an unlimited number of user-defined attributes, making the data “search ready.”
Figure 1. Each TDMS file written in Measurement Studio contains descriptive information on the root, group, and channel levels.
Unlike TDM files, which have a strictly required XML-based header file, TDMS files have a binary index file with the extension *.TDMS_Index. The TDMS_Index file provides consolidated information on all the attributes and pointers in the bulk data file, and accelerates access to the data while reading. The TDMS_Index file is automatically regenerated if deleted or removed.
The TDMS file can contain several groups, and each group can contain several channels. You can insert your own custom properties at each of the three levels. Creating and using channel groups and custom properties are discussed in greater depth later in this white paper.
The descriptive information located in the TDMS file, a key benefit of this format, provides an easy way to document the data without having to design your own header structure. As your documentation requirements increase, you do not have to redesign your application; you simply extend the TDMS data model to meet your specific needs.
3. Measurement Studio TDM Streaming API for Writing and Reading TDMS Files
Like other file I/O functions in Measurement Studio and .NET, you have to open, write to, and close the file with the TDMS API. You can also use the native TDMS functions from the API to save the properties you choose rather than worrying about file formatting, data types, and other tasks normally associated with writing additional information to a measurement file. Review the following sections for the basic steps of writing, reading, creating properties, and sharing your TDMS files.
TDMS Writing Data Basics
The simplest form of writing measurement data with the Measurement Studio TDMS functions is illustrated in Figure 2. In this example, a sine wave is generated, saved as an array of measurement values, and passed to the
Figure 2. Write data to a TDMS file at the channel level.
Notice the calls to the
TdmsFile.AddChannelGroup function and the
TdmsChannelGroup.AddChannel function. In the example above, the channel associated with the sine wave data is described by a specific group and channel name. The acquired signal is then appended to the channel identified in the
TdmsChannelGroup.AddChannel function. Figure 3 demonstrates the structure of the TDMS file written in the example code above.
Figure 3. TDMS files store descriptive information at the file, group, and channel levels.
TDMS Reading Data Basics
Figure 4 illustrates the simplest form of reading data using the TDMS API functions. Here, the
TdmsFile object obtains a file reference to the TDMS file and calls the
TdmsFile.GetChannelGroups function. After getting the appropriate channel from the channel group and the number of values stored in that channel, the
TdmsChannel.GetData function reads all data in “Main Group” into memory.
Figure 4. Read data back from a TDMS file.
TDMS Write Custom Properties
The TDMS data model automatically creates certain properties when you write some data types. However, in many cases, you may want to create a property specific to your particular needs, such as UUT, serial number, or test temperature. You can accomplish this task using the
AddProperty function, which you can use to write properties at the file, group, or channel levels, as shown in Figure 5.
The file level of a property determines which set of property functions need to be called. For example, to write a group property, only the
TdmsChannelGroup.AddProperty function needs to be called. This is also true when setting file and channel properties. These cases are illustrated in Figure 5.
Figure 5. Write custom properties at three different levels before writing data to the file.
With the TDMS
AddProperty functions, you can specify an individual property by itself or specify many properties by using arrays. Figure 5 specifies two properties at the file level (
Plant). You can expand this to specify many more properties by just increasing the size of the array. Arrays are not necessarily needed if only a single property is written, such as a serial number.
Property values can also be different data types. In Figure 5, string properties are written at the file and group levels. But at the channel level, two numeric properties are written to specify the minimum and maximum temperature.
Figure 6 illustrates how you can use the TDMS
AddProperty function to save additional descriptive information at the file level. In this example, the date and the user of the test are built into an array and then written as custom properties.
Figure 6. Use the AddProperty function to save additional descriptive information at the file level.
TDMS Read Custom Properties
Once you have written a TDMS file, you can read the properties back into Measurement Studio using the TDMS
GetProperties functions. For example, Figure 7 shows how to get properties for the file level by using the
TdmsFile.GetProperties function. This process is very similar to writing the properties.
Figure 7. Read TDMS properties from three levels in the data storage model.
In this configuration, the property values are returned as a string array because we specify the data type in the
TdmsProperty.GetValue function. The data could be a string, double, boolean, or some other data type and the function would return an object array if you did not specify a data type. You can display the data on the user interface as an object or typecast it in .NET code to the appropriate data type.
TDMS Defragment File and TDMS Save File
Included with the TDMS API functions are the defragment and save functions for optimizing read and write operations.
Each call of the TDMS write functions records a block of data to file, which may contain several channels. This prevents the data for a single channel from being written contiguously across multiple writes. The
TdmsFile.Defragment function reorganizes the file by channel to optimize read operations. In this way, the user can take the performance penalty only once by defragmenting the TDMS file “offline” rather than with every read. For example, the user is able to acquire and store data very rapidly and then defragment the file before performing any read operations.
TdmsFile.Save function forces all cached data to disk, a useful feature to ensure that no data is lost in the event of a system crash. Using the
TdmsFile.Save function costs some system performance. If
AutoSave is true, the
TdmsFile is automatically saved to disk after every change. If
AutoSave is false, you must call
Save explicitly to write the cached data to disk.
AutoSave is set to true on the write functions by default so there is typically no need to manually call the
View TDMS Files
TDMS File Viewer Shipping Example
Measurement Studio is shipped with two well-documented examples demonstrating how to write and read TDMS files. The TDM Streaming example sets up a TDMS file and then streams data to disk.
Figure 8. The TDM Streaming example documents the streaming capabilities of the TDMS file format.
The TDMS File Reader example loads the file generated by the TDM Streaming example, displays the different levels of data in the file such as groups and channels, and displays the properties and data inside each level that is selected from the tree structure. Figure 8 shows an example of the File Reader.
Figure 9. The TDMS File Reader provides easy access to your TDMS data file.
Load TDMS Files into Excel
Using the Microsoft Excel add-in, which is included with Measurement Studio 8.6 and later, you can load and work with TDMS files in Excel. Excel limits the amount of data that you can load per data channel to 65,535 values (1,048,576 values in Excel 2007); however, with the Excel add-in, you can specify which starting point to read from if your measurements exceed this limit.
Load TDMS Files into DIAdem
TDMS files can also be loaded and viewed in National Instrument DIAdem data management software. With the National Instruments DataFinder included in DIAdem, not only can you locate the data you need to work with faster, you also have the ability to mine your test data to search for trends and correlations.
In addition to being able to manage and mine your data in DIAdem, you can also perform perform advanced offline analysis on your data as well as generate automated reports.
4. Additional TDMS Information
TDMS files are native in National Instruments Measurement Studio, LabVIEW, LabWindows/CVI and DIAdem software.
Measurement Studio and .NET offer a variety of file I/O options for reading and writing data. The TDMS .NET Class Library provides benefits that go beyond traditional file operations in Measurement Studio. With the TDMS API, you can:
Quickly stream data to disk in a structured format.
Easily save descriptive information with test data, making it search-ready.
Categorize data under groups to add structure (for example, Group 1 = temperature sensors, Group 2 = pressure sensors, etc).
Use the API to abstract the low-level file I/O normally associated with saving data.
Include descriptive properties to help analyze data at a later time.