PDF1. Introduction
In the past, LabVIEW programmers who needed to capture information such as serial numbers, test operators, etc, usually resorted to defining and maintaining their own file format. This usually looks simple at the outset but results in ongoing maintenance headaches. The file format inevitably changes over time to store new information, causing you to have to revise the VI that reads your custom files. Each time you update your file reader VI, it has to be compatible with all previous file versions as well, and you have to distribute the new reader VI to all clients potentially interested in reading those files.
TDM aims to solve these file I/O versioning issues, by providing a standard data model and LabVIEW APIs that interface to it directly. When you add new information to your TDM format file, you are guaranteed that this information will be automatically read in by the TDM reader VI that is shipped with LabVIEW. Offloading ownership of your data file format to LabVIEW frees you up to spend more time on other, more profitable and/or more interesting aspects of application development.
TDM also makes it easy for you to save well-documented data, because adding this information involves only a few extra VIs from the LabVIEW palette and incurs no extra file format maintenance hassles. Saving well documented data can have enormous benefits for you and your company. For example, when data is well documented, and when you have a data management system set up to mine that data, it becomes easy and automatic to answer questions such as, “why did this test fail” or “who ran this test”. Often the answers to these questions are a large part of why you took the measurements in the first place.
The NI TDM Data Management Platform makes it easy both to save well documented data and to mine those stored data sets – without every user calling you each time they want to find a particular data set. Saving “search ready” data files in the TDM format is the best way in LabVIEW to save well documented data, but it is only the first step in managing your test data. The full spectrum of NI TDM components spans several NI products, primarily LabVIEW, DIAdem, and LabWindows/CVI. Additional components extend the NI TDM platform to cover legacy data files and connect to third-party C-based applications and even Excel.
The remainder of this paper outlines the NI TDM technologies and provides insight on how you might best apply them in your application development.
Figure 1. The NI TDM Components
2. NI TDM Components - Overview
Figure 1 above shows a simple view of the NI TDM components, the foundation of which is the TDM Data Model. Data files are either written with this data model using the Native TDM File APIs, or they are mapped to the TDM Data Model with a DataPlugin. The NI DataFinder then automatically creates and maintains a file index of the descriptive information for each file, as exposed in the TDM Data Model. Finally, you use NI DIAdem and its DataFinder interface to mine test data. DIAdem provides two methods for searching data. The first, is an Internet-like search where you type in key words and DIAdem returns all instances of channels or files that contain the search string. There also is a more advanced search that allows you to created parameterized searches based on specific attributes states. NI DIAdem combines data searching functionality with a general purpose interactive data visualization, analysis, and reporting environment.
3. TDM Data Model
At the center of the NI data management offering is the TDM Data Model, which has three hierarchical levels – File (Root), Groups, and Channels – each of which can have an unlimited number of user-defined, scalar properties. Each Channel additionally contains a 1D array of data values. Both the properties and the data arrays are stored in your choice of a variety of standard data types. NI provides Native TDM File APIs to create and read data files with the TDM Data Model or just write a TDM header file. In all cases you can import these files into Excel with the TDM Excel Add-in.
Figure 2. The TDM Data Model provides three levels to structure your measurement information. 4. Native LabVIEW TDM File APIs
In LabVIEW there are two different APIs for interacting with the TDM Data Model. The fastest and most flexible is the TDM Streaming API, introduced in LabVIEW 8.20, which writes TDMS files, (the S in TDMS stands for streaming). The other TDM interface is through the LabVIEW Data Storage VIs, introduced in LabVIEW 7.1, which write a flavor of TDM files, using an XML-based header file. The Data Storage VIs also include additional functions for querying data from a file and loading data from non-TDM files via DataPlugins. The Data Storage VIs will read and write TDM and TDMS files; however, they do not have the speed that the TDM Streaming API offers for writing data to disk. Because the TDM Streaming API accommodates the widest range of applications, and there are VIs for converting between TDM and TDMS, NI recommends using the TDM Streaming API for writing data to the TDM Data Model.
Tip! Poll all the end users of the application you are developing on which information they would like to capture in the test files. Then using the TDM Streaming API, create attributes at the most appropriate level, either the file, group, or channel. You can add or take away attributes at will. Not being locked into to your initial choices is a key advantage of adopting the TDM data model and the associated LabVIEW APIs.
Additional Interfaces to the TDM Data Model
Occasionally you might have to work with TDM files outside of LabVIEW, or you have existing binary files that you would like to read using the Data Storage VIs. For these cases, there is a C DLL for reading and writing TDM files, and a LabVIEW API to write XML header files for arbitrary binary files.
See Also:
Application Note: Using the TDM Streaming API
Application Note: Using the Data Storage VIs
Learn more about TDM Interfaces
5. DataPlugins
A common challenge facing test system developers, is working with legacy data files. NI DataPlugins fill this gap by providing a unified way to interface to and access data, regardless of file format. For example, ni.com/dataplugins lists more than 70 DataPlugins available for download; one of them happens to be for Nicolet scope files. With the Nicolet DataPlugin, you can use the LabVIEW Data Storage API to access data from Nicolet data files. You can also create your own DataPlugins for files not listed on the Web site. There is a programmer’s reference guide, complete with examples to aid you in creating your own DataPlugins.
Another advantage of using DataPlugins, is not only do they unify access of data files in LabVIEW, but also by virtue of mapping data onto the TDM data model, they open all files up to data mining using the DIAdem DataFinder.
Figure 4. DataPlugins map the contents of non-TDM files onto the TDM Data Model, unifying file access in LabVIEW and enabling searching with the DIAdem DataFinder.
See Also:
Learn more about DataPlugins
6. DIAdem DataFinder
DIAdem DataFinder, integrated into NI DIAdem, is an automatic file-indexing and search-providing service that works exclusively with data files exhibiting the TDM Data Model. This includes both the TDM and TDMS data files created by the Native TDM File APIs as well as any data files for which there is an associated DataPlugin. DIAdem DataFinder is automatically installed locally with DIAdem 10.0 or later versions.
You configure DataFinder first by declaring a series of search areas. Each search area starts with a certain parent directory and implicitly includes all files and directories inside the parent directory, though you may choose to exclude certain subdirectories. This causes DataFinder to consider indexing only files contained in a directory included beneath a declared Search Area. The second way you configure DIAdem DataFinder is to associate each file extension you want to index with the appropriate DataPlugin to use in reading that file. This second step is not needed if all you have are TDM and/or TDMS files, but it is very important if you have one or more DataPlugins for legacy file types. This feature causes DataFinder to consider indexing only data files that have a valid file extension (*.TDM, *.TDMS, etc.), and DataFinder will automatically know which DataPlugin to use for each file.
Once DataFinder is installed and configured, it automatically builds and maintains an index of all data Files, Groups, and Channels (see TDM Data Model) that satisfy the file type and location criteria in the DataFinder configuration. Properties on all three hierarchy levels are automatically stored in the DataFinder database and can be used in query conditions. When a valid data file is created, deleted, or edited, DataFinder automatically notices and reindexes the hierarchy and properties of this file. When properties not yet in DataFinder are encountered in a newly created file, these properties are automatically added to the table structure. DIAdem DataFinder dynamically manages its own data tables and updates them based on file events and the TDM Data Model contents of each file.
Figure 5: The DIAdem Data Index correlates the meta information from all data files that have an associated DataPlugin.
7. More on DIAdem
NI DIAdem provides a built-in user interface to the DataFinder to search easily for all indexed data files with a simple Google-like text search or a powerful SQL-like advanced search. DIAdem combines this data searching functionality with general-purpose interactive data visualization, analysis, and reporting environment. DIAdem natively uses the TDM Data Model and can read and write both TDM and TDMS data files and can read any files for which there is a registered DataPlugin. In addition to being an interactive data handling environment, DIAdem can also be fully automated with VBScript and with custom dialogs and is often used as the basis for a fully customized data management and evaluation application.
See Also:
More information on DIAdem
8. TDM Excel Add-in
The TDM Excel Add-in expands the toolbar of an existing Excel installation with a new “Import a TDM File” icon, which will import any data file written with the Native TDM File APIs. The TDM Excel Add-in can be configured to load only the desired properties from each TDM Data Model level, and it will automatically load only a subset of rows if the Excel version has fewer available rows than are stored in the data file.
See Also:
Download the TDM Excel Add-in
9. Conclusion
The NI TDM Platform covers all the bases of a search-ready data management system. You no longer need to worry about the details of file parsing, data model construction, data table structure, database management, and design of search GUIs. All you have to do is plug into the NI TDM Platform by creating data files with the Native TDM File APIs and/or using DataPlugins for your legacy files, and then use the easy-to-configure NI DataFinder to automatically create and maintain a data index. Finally, you can take advantage of the DIAdem built-in search GUIs in DIAdem DataFinder and its easy-to-use interactive data analysis and reporting environment to quickly turn your saved data into the reports and result information you have needed all along.
