From 6:00 PM CST Friday, Feb 15th - 2:00 AM CST Sunday, Feb 17th, ni.com will be undergoing system upgrades that may result in temporary service interruption.

We appreciate your patience as we improve our online experience.

Introduction to the LabWindows™/CVI™ TDMS Library

Publish Date: May 19, 2014 | 7 Ratings | 4.14 out of 5 | Print

Overview

This technical white paper provides an overview of the features and use cases of the National Instruments LabWindows/CVI Technical Data Management Streaming (TDMS) Library.

Table of Contents

  1. Introduction
  2. The TDMS File Format of the TDM Data Model
  3. LabWindows/CVI TDM Streaming API for Writing and Reading TDMS Files
  4. Additional TDMS Information
  5. Conclusion

1. Introduction

To reduce the need to design and maintain your own data file format, National Instruments has created a flexible data model called technical data management (TDM), which is natively accessible through NI LabWindows/CVI and NI DIAdem and is portable to other applications such as Excel. 

The TDM data model offers several unique benefits such as the ability to scale to your specific project requirements and easily attach descriptive information to your measurements while streaming your data to disk. The TDM data model supports two file formats, TDM and TDM Streaming (TDMS). The new TDMS Library is optimized for streaming well-documented data to disk. The DIAdem Connectivity API writes TDM files and offers the same flexibility with regard to documenting data. The DIAdem Connectivity Library functions read and write TDM and TDMS files; however, these functions do not have the speed that the TDMS Library functions offer for writing data to disk. The TDMS Library functions provide the widest coverage of different use cases, so they are the recommended way to interface with the TDM data model. The TDM white paper provides additional information including a comprehensive overview of the NI TDM Platform.

This white paper explains the TDM data model and demonstrates how the LabWindows/CVI TDMS Library reduces your programming time while increasing the value of the data you are collecting.

Back to Top

2. The TDMS File Format of the TDM Data Model

The TDM data model offers three levels of hierarchy – root, group, and channel (see Figure 1). Each level accepts an unlimited number of user-defined attributes, making the data “search ready.”


Figure 1. Each TDMS file written in LabWindows/CVI contains descriptive information on the root, group, and channel levels.

Unlike TDM files, which have a strictly required XML-based header file, TDMS files have a binary index file with the extension *.TDMS_Index. The TDMS_Index file provides consolidated information on all the attributes and pointers in the bulk data file, and speeds up access to the data while reading. The TDMS_Index file is automatically regenerated if deleted or removed.

 

The TDM data model file can contain several groups, and each group can contain several channels. You can insert your own custom properties at each of the three levels. Learn more about creating and using channel groups and custom properties later in this white paper.

 

The descriptive information located in the TDMS file, a key benefit of this format, provides an easy way to document the data without having to design your own header structure. As your documentation requirements increase, you do not have to redesign your application; you simply extend the TDMS data model to meet your specific needs.

Back to Top

3. LabWindows/CVI TDM Streaming API for Writing and Reading TDMS Files

Like other file I/O functions in LabWindows/CVI, you have to open, write to, and close the file with the TDMS API. You can also use the native TDMS functions from the library to save the properties you choose rather then worrying about file formatting, data types, and other tasks normally associated with writing additional information into a measurement file. Review the following sections for the basic steps of writing and reading, creating properties, and sharing your TDMS files.

TDMS Writing Data Basics

The simplest form of writing measurement data with the LabWindows/CVI TDMS functions is illustrated in Figure 2. In this example, a sine wave is generated, saved as an array of measurement values, and passed to the TDMS_AppendDataValues function.

 

#include <cvitdms.h>
#include <analysis.h>
#include <userint.h>
#include <cvirte.h>

int main (int argc, char *argv[])
{

 double measuredData[200];
 TDMSChannelHandle mainChannel;
 TDMSChannelGroupHandle mainChanGroup;
 TDMSFileHandle tdmsFileHandle;
 
 if (InitCVIRTE (0, argv, 0) == 0)
  return -1;    /* out of memory */
 
 // Create and set up TDMS file
 TDMS_CreateFile ("c:\\signals.tdms", TDMS_Streaming, "DAQ Test",
    "", "", "", &tdmsFileHandle);
 TDMS_AddChannelGroup (tdmsFileHandle, "Main Group", "Dev1",
    &mainChanGroup);
 TDMS_AddChannel (mainChanGroup, TDMS_Double, "Main Channel", 
    "ai0", "", &mainChannel);
   
 // Write data
 SinePattern (200, 1.0, 0.0, 1.0, measuredData);
 TDMS_AppendDataValues (mainChannel, measuredData, 200, 1); 
   
 // Close file
 TDMS_CloseFile(tdmsFileHandle);  
 
 return 0;

}

 
Figure 2. Write data to a TDMS file at the channel level.

Notice the calls to the TDMS_AddChannelGroup function and the TDMS_AddChannel function. In the example above, the channel associated with the sine wave data is described by a specific group and channel name. The acquired signal is then appended to the channel identified in the TDMS_AddChannel function. Figure 3 demonstrates the structure of the TDMS file written in the example code above.

Figure 3. TDMS files store descriptive information at the file, group, and channel levels.

TDMS Reading Data Basics

Figure 4 illustrates the simplest form of reading data using the TDMS Library functions. Here, the TDMS_OpenFile function passes a file reference to the TDMS_GetChannelGroups function. After getting the appropriate channel from the channel group and the number of values stored in that channel, the TDMS_GetDataValues function reads all data in the “Main Group” group into memory and displays the waveform on a graph.

 

#include <cvitdms.h>
#include <analysis.h>
#include <userint.h>
#include <cvirte.h>

int main (int argc, char *argv[])
{

 unsigned __int64 numChanValues;
 TDMSChannelHandle channels[2];
 TDMSChannelGroupHandle channelGroups[2];
 double measuredData[200];
 TDMSFileHandle tdmsFileHandle;
 
 if (InitCVIRTE (0, argv, 0) == 0)
  return -1;    /* out of memory */
 
 // Open TDMS file
 TDMS_OpenFile ("c:\\signals.tdms", 1, &tdmsFileHandle);
   
 // Read group data
 TDMS_GetChannelGroups (tdmsFileHandle, channelGroups, 1);
 TDMS_GetChannels (channelGroups[0], channels, 1);
 TDMS_GetNumDataValues (channels[0], &numChanValues);
 TDMS_GetDataValues (channels[0], 0, numChanValues,
                  measuredData);

 // Close file
 TDMS_CloseFile(tdmsFileHandle);
 
 return 0;

}


Figure 4. Read data back from a TDMS file.

TDMS Write Custom Properties


The TDMS data model automatically creates certain properties when you write some data types. However, in many cases, you may want to create a property specific to your particular needs, such as UUT, serial number, and test temperature. You can accomplish this task using the TDMS_SetFileProperties function, with which you can write properties at the file, group, or channel levels, as shown in Figure 5.

 

The file level of a property determines which set of property functions need to be called. For example, to write a group property, only the TDMS_SetChannelGroupProperty needs to be called. This is also the case when setting file and channel properties. These cases are illustrated in Figure 5.

 

#include <cvitdms.h>
#include <analysis.h>
#include <userint.h>
#include <cvirte.h>

int main (int argc, char *argv[])
{

 int i;
 TDMSChannelHandle mainChannel;
 TDMSChannelGroupHandle mainChanGroup;
 double measuredData[200];
 TDMSFileHandle tdmsFileHandle;
 
 char * filePropertyNames[2] = {"TestID", "Plant"};
 char * filePropertyValues[2] = {"Test1", "Site B"};
 char * groupPropertyName = "Serial #";
 char * groupPropertyValue = "012345";
 char * chanPropertyNames[2] = {"Max Temp", "Min Temp"};
 int chanPropertyValues[2] = {100, 0};
 
 if (InitCVIRTE (0, argv, 0) == 0)
  return -1;    /* out of memory */
 
 // Create TDMS file
 TDMS_CreateFile ("c:\\signals.tdms", TDMS_Streaming, "DAQ Test",
    "", "", "", &tdmsFileHandle);
 TDMS_AddChannelGroup (tdmsFileHandle, "Main Group", "Dev1",
     &mainChanGroup);
 TDMS_AddChannel (mainChanGroup, TDMS_Double, "Main Channel",
     "ai0", "", &mainChannel);
   
 // Write file properties
 for (i = 0; i < 2; i++)
  TDMS_SetFileProperty(tdmsFileHandle, filePropertyNames[i],
     TDMS_String, filePropertyValues[i]);
   
 // Write group properties
 TDMS_SetChannelGroupProperty(mainChanGroup,
     groupPropertyName, TDMS_String, groupPropertyValue);
   
 // Write channel properties
 for (i = 0; i < 2; i++)
  TDMS_SetChannelProperty(mainChannel, chanPropertyNames[i],
     TDMS_Int32, chanPropertyValues[i]);
   
 // Write data
 SinePattern (200, 1.0, 0.0, 1.0, measuredData);
 TDMS_AppendDataValues (mainChannel, measuredData, 200, 1); 
   
 // Close file
 TDMS_CloseFile(tdmsFileHandle);
 
 return 0;

}

Figure 5. Write custom properties at three different levels before writing data to the file.

With the TDMS Set Properties functions, you can specify an individual property by itself or specify many properties by using arrays. Figure 5 shows two properties at the file level (TestID and Plant) being specified. You could expand this to specify many more properties by just increasing the size of the array. Arrays are not necessarily needed if only a single property is written, such as a serial number.

 

Property values can also be different data types. In Figure 5, string properties are written at the file and group levels. But at the channel level, two numeric properties are written to specify the minimum and maximum temperature.

 

Figure 6 illustrates how you can use the TDMS Set Properties functions to save additional descriptive information at the file level. In this example, the date and the user of the test are built into an array and then written as custom properties.

 

#include <analysis.h>
#include <userint.h>
#include <cvirte.h>

int main (int argc, char *argv[])
{

 int i;
 unsigned int nameSize;
 TDMSChannelHandle mainChannel;
 TDMSChannelGroupHandle mainChanGroup;
 double measuredData[200];
 TDMSFileHandle tdmsFileHandle;
 char * customFilePropertyNames[2] = {"Date", "Author"};
 char * customFilePropertyValues[2] = {"", ""};
 
 if (InitCVIRTE (0, argv, 0) == 0)
  return -1;    /* out of memory */
 
 customFilePropertyValues[1] = malloc(100 * sizeof(char));
   
 // Create and set up TDMS file
 TDMS_CreateFile ("c:\\signals.tdms", TDMS_Streaming, "DAQ Test",
     "", "", "", &tdmsFileHandle);
 TDMS_AddChannelGroup (tdmsFileHandle, "Main Group", "Dev1",
     &mainChanGroup);
 TDMS_AddChannel (mainChanGroup, TDMS_Double, "Main Channel",
     "ai0", "", &mainChannel);
   
 customFilePropertyValues[0] = DateStr ();
 GetCurrentUser (customFilePropertyValues[1], 100, &nameSize);
   
 // Write file properties
 for (i = 0; i < 2; i++)
  TDMS_SetFileProperty(tdmsFileHandle,
      customFilePropertyNames[i], TDMS_String,
      customFilePropertyValues[i]);
   
 // Write data
 TDMS_AppendDataValues (mainChannel, measuredData, 200, 1); 
   
 // Close file
 TDMS_CloseFile(tdmsFileHandle);
   
 free (customFilePropertyValues[1]);
 
 return 0;

}

Figure 6. Use the Set Properties functions to save additional descriptive information at the file level.

TDMS Read Custom Properties


Once you have written a TDMS file, you can read the properties back into LabWindows/CVI using the TDMS Get Properties functions. For example, Figure 7 shows how to get properties for the file level by using the TDMS_GetFileProperty function. This process is very similar to writing the properties.

 

#include <ansi_c.h>
#include <utility.h>
#include <cvitdms.h>
#include <cvirte.h>

int main (int argc, char *argv[])
{

 int i;   
 unsigned int numProperties;
 TDMSChannelHandle channels[2];
 TDMSChannelGroupHandle channelGroups[2];
 TDMSFileHandle tdmsFileHandle;
 char * allFilePropertyNames[10];
 char * allFilePropertyValues[10];
 
 if (InitCVIRTE (0, argv, 0) == 0)
  return -1;    /* out of memory */
 
 // Open TDMS file
 TDMS_OpenFile ("c:\\signals.tdms", 1, &tdmsFileHandle);
   
 // Read file properties
 TDMS_GetChannelGroups (tdmsFileHandle, channelGroups, 1);
 TDMS_GetChannels (channelGroups[0], channels, 1);
 TDMS_GetNumFileProperties(tdmsFileHandle, &numProperties);
 TDMS_GetFilePropertyNames(tdmsFileHandle,
    allFilePropertyNames, numProperties);
 for (i = 0 ; i < numProperties; i++) {
  allFilePropertyValues[i] = malloc (100* sizeof(char)); 
  TDMS_GetFileProperty(tdmsFileHandle,
     allFilePropertyNames[i], allFilePropertyValues[i],
     100);
 }
   
 // Close file
 TDMS_CloseFile(tdmsFileHandle);
 
 return 0;

}

Figure 7. Read TDMS Properties from three levels in the data storage model.

In this configuration, the property values are returned as a variant array because the data could be a string, double, Boolean, and so on. You can display the data on the user interface as a variant or convert it in LabWindows/CVI to the appropriate data type.

View TDMS Files

TDMS File Viewer Shipping Example

LabWindows/CVI is shipped with two well-documented examples demonstrating how to write and read TDMS files. The TDM Streaming Write Benchmark example streams data to disk and shows the average streaming rate. 

Figure 8. The TDM Streaming Write Benchmark example documents the streaming capabilities of the TDMS file format.

The TDM Streaming File Reader example loads the file generated by the TDM Streaming Write Benchmark example, displays the different levels of data in the file such as groups and channels, and displays the properties and data inside each level that is selected from the tree structure. Figure 8 shows an example of the File Reader.

   

Figure 9. The TDM Streaming File Reader provides easy access to your TDMS data file.

Load TDMS Files into Excel

Using the FREE Microsoft Excel add-in, you can load and work with TDM and TDMS files in Excel. Excel limits the amount of data that you can load per data channel to 65,535 values; however, with the FREE Excel add-in, you can specify which starting point to read from if your measurements exceed this limit.

Download the FREE Excel Add-In for Loading TDM Files into Excel

TDMS Defragment File and TDMS Save File

Included with the TDMS Library functions are the defragment and save functions for optimizing read and write operations.

Each call of the TDMS_AppendDataValues function records a block of data to file, which may contain several channels. This prevents the data for a single channel from being written contiguously across multiple writes. The TDMS_DefragmentFile function reorganizes the file by channel to optimize read operations. In this way, the user can take the performance penalty only once by defragmenting the TDMS file “offline” rather than with every read. For example, the user is able to acquire and store data very rapidly and then defragment the file before performing any read operations.

The TDMS_SaveFile function forces all cached data to disk, a useful feature to ensure that no data is lost in the event of a system crash. Using the TDMS_SaveFile function costs some system performance.

Back to Top

4. Additional TDMS Information

The LabWindows/CVI TDMS Library functions are designed to write data quickly and efficiently to the disk. They were also designed to work on real-time systems so you can quickly store data on a deterministic system. Therefore, with the TDMS Library functions, you can keep track of your data in a structured format on a real-time system instead of writing data in a binary format with no predefined framework.

TDMS files are native in National Instruments LabWindows/CVI and DIAdem software. With DIAdem, you can mine your test data based on the attributes stored with your TDMS files to find trends and correlations.

Read More about the TDMS File Format Internal Structure 

Back to Top

5. Conclusion

LabWindows/CVI offers a variety of file I/O options for reading and writing data. The TDMS Library functions provide several benefits that go beyond traditional file operations in LabWindows/CVI. With the TDMS Library, you can:

  1. Stream data to disk quickly in a structured format
  2. Easily save descriptive information with test data, which makes it search ready
  3. Categorize data under groups to add structure (for example, Group 1 = temperature sensors, Group 2 = pressure sensors)
  4. Easily use the API to abstract the low-level file I/O normally associated with saving data
  5. Include properties to help describe data when analyzed later

Next Steps


To view a comparison of file formats supported in LabWindows/CVI include ASCII, XML, binary, INI, TDM, and TDMS, view the Best Practices for Storing Data with LabWindows/CVI webcast.

Run the Example Code in the LabWindows/CVI 7-Day Evaluation  
View the LabWindows/CVI Resource Page

Back to Top

Bookmark & Share


Ratings

Rate this document

Answered Your Question?
Yes No

Submit