Test Data Management for the Enterprise

This paper is designed to help managers, engineers, and scientists understand the different options available for managing large amounts of test and simulation data. It explores the different approaches to test data management, their advantages and drawbacks, and introduces a new National Instruments solution to the increasingly costly and time-consuming challenge of data management.

Online Presentation of NI DataFinder Server Edition and NI DIAdem

Pricing and Purchasing Information

Lesen Sie dieses Dokument in deutscher Sprache

1. The Pain Points of Data Management
2. Traditional Data Management Approaches
3. NI Technical Data Management Solution
4. NI DataFinder
5. DIAdem DataFinder and NI DataFinder Server Edition
6. Analyzing and Reporting Your Test Data
7. Conclusion
8. Additional Resources

The Pain Points of Data Management

Today’s complex products require data acquisition throughout the complete design and development process. Companies make a significant investment in the data they collect, from the cost of simulation systems, data acquisition hardware, and automation systems to the associated personnel required to perform and analyze the tests. Increasing microprocessor speed and storage capacity together with decreasing costs for hardware and software have resulted in an explosion of data stored in files and databases. While technology is enabling faster and richer data retention, managing and making good use of this data remains the real challenge. Yet more than ever in today’s fiercely competitive business environment, companies need to rapidly turn test and simulation data into usable information to efficiently drive product development.

Many simulation and test systems have grown over several years, often independently of each other and with equipment from different suppliers. As a result, data is stored in different file formats, with little or no descriptive information and in different locations. All of these factors present roadblocks that impede the optimal exchange of information, and make it extremely difficult to locate a particular data set and derive decisions from it. As a result, many companies are seeing a loss in efficiency and increasing development costs as the amount of data they store increases.

Even if you have a history of consistent data formats containing descriptive information, the challenge still remains to make the data available for others to both easily access and effectively use. How do you ensure that all data consumers in your organization can find the data they need? Once they do find the desired data, how do you guarantee that they have the right tools to quickly turn that data into usable information? Common approaches to these challenges range from primitive file and folder naming conventions to complex and costly database solutions. Your ability to turn raw data into interpretable results and easily share these results throughout your organization is critical to the efficiency of your overall development process.

Traditional Data Management Approaches

The challenges involved in building and maintaining scalable data management systems have caused many companies to continue managing their critical data assets by using simple file and folder naming conventions. Engineers are left to manually search through deep directory structures to find the data they need. Many times this data is stored on different machines and in different file formats, adding even more complexity to the situation. Correlation of data and test results across multiple machines and formats under these circumstances can be challenging and time consuming. File- and folder-naming data management systems are highly ineffective, resulting in lost productivity, and are easily corrupted when files are inadvertently moved out of their designated folders or renamed.

Figure 1. Traditional File and Folder Approach across Multiple Test Machines

Many engineers tolerate these challenges because they can load much of this data into readily available spreadsheet tools such as Microsoft Excel. However, the cost saved by not having to purchase dedicated engineering software is soon lost when time and resources are spent recreating common engineering analysis and reporting routines unavailable in business-centric spreadsheet packages and manually searching for data.

At the opposite end of the spectrum is the approach of using dedicated databases for managing technical data. Because of their reputation for organizing data for easy searching and retrieval, databases have garnered the attention of many test engineers as one way to manage their test data. There are some obvious and some not-so-obvious drawbacks to adopting a standard database such as Oracle or Access for your data management system. First, these databases are not set up to work out of the box with your test data; there is a significant amount of data model and database design work that must take place up front. Designing a data model and the database can take months of preparation and planning. For larger solutions, the initial cost of setting up a database solution can easily reach six-digit dollar figures. But more important is the often overlooked ongoing maintenance required to keep databases up and running. You also need to invest extra time and effort to expand the database design and custom client software every time your test needs change. This maintenance and scaling over time not only costs more and more money but usually requires you to engage IT experts who know how to keep these systems up and running. And, oftentimes, the IT group does not have the bandwidth to help test groups.

Figure 2. Database Approach of Storing Data in a Central Location

In the end, neither the file and folder approach nor the database approach meets the needs of most test groups. While they may seem like viable solutions at design time, you are still left with maintenance headaches and inefficiencies as your test needs change over time and you continue to store more data.

NI Technical Data Management Solution

National Instruments recognizes that engineers and scientists face these data management challenges every day when designing their test systems. Whether it is an engineer who spends hours a week tediously searching through directories and files to find just one specific test or an engineer who has to redesign an expensive database and client application to accommodate additional information, they both want the same thing – engineering intelligence from their test data. Engineers spend a large amount of time designing and performing tests so they can move quickly from test data to test results. All too often today, engineers are expected to make the crucial transition from test data to test results on their own. Therefore, National Instruments has developed a data management solution to solve these data management challenges and make it easier and faster for engineers to move from data collection to usable results.

The National Instruments technical data management (TDM) solution consists of three components – the TDM data model for storing descriptive information with your test files, NI DataFinder for searching and mining your test data regardless of file format, and NI DIAdem for analysis and reporting. Many engineers already have components in place for raw data file writing as well as analysis and reporting, the lowest common denominator being saving ASCII files and performing analysis in Excel. However, most engineers are lacking the middle component, NI DataFinder, to complete their data management approaches. You can use DIAdem DataFinder or NI DataFinder Server Edition to quickly query and mine your test data based on the descriptive information stored. For example, within seconds, you can search through all of your test files for all tests run on a certain day by a specific person that failed without ever having to open any files or knowing where they are located. Using NI DataFinder, you can quickly find trends and correlations in data sets across multiple files and formats.

The foundation of the National Instruments solution for data management is storing rich meta information with your data, which you can then search on using NI DataFinder. With the NI technical data management (TDM) data model, you can easily store properties and organize your data hierarchically within your test file. TDM and TDM Streaming files, written natively using NI DIAdem, LabVIEW, or LabWindows^™/CVI, provide three levels of hierarchy to structure your test data – file, group, and channel levels. The TDM data model removes the burden of designing and maintaining your own custom file format. Designed to scale with your needs, the TDM data model provides the most efficient way to properly document your test and simulation data in LabVIEW and LabWindows/CVI. Having a rich set of properties increases the range of possible search conditions.

Figure 3. The TDM Data Model for Saving Well-Documented Test Data

Once you have properly documented your data with properties, NI DataFinder, introduced in DIAdem 10.0, gives you the ability to search and mine your test data without the headache and expense of setting up and maintaining a large database. With NI DataFinder, you can perform Internet-like searches across all your data files, regardless of format and location within your company intranet. Simply install NI DataFinder with DIAdem, point it to the location of your data files, and, seconds later, you can search for your files just as you would search for information on the Internet.

NI DataFinder does this by automatically building and maintaining an index of all files that satisfy the file type and location criteria in the NI DataFinder configuration. You can use properties which are automatically stored in the NI DataFinder index in query conditions. When a valid data file is created, deleted, or edited, NI DataFinder automatically notices and reindexes the hierarchy and properties of this file. When you save properties not yet in NI DataFinder in a newly created file, these properties are automatically added to the index. NI DataFinder dynamically manages its own data tables and updates them based on file events and the contents of each file. Therefore, unlike many expensive database solutions, you can change and add information as your needs change, without ever redesigning your data management solution.

Figure 4. The NI TDM Solution Consists of Three Components

The final component of the National Instruments TDM solution for moving quickly between raw data and engineering results is NI DIAdem for post-test analysis and reporting. Using DIAdem, you can take advantage of hundreds of built-in engineering analysis functions to extract meaningful results from your raw data, and then share these results with easy-to-build, reusable report templates. DIAdem also provides a built-in interface to both DIAdem DataFinder and NI DataFinder Server Edition. Therefore, you can run interactive searches within DIAdem as well as save and automate your queries. Within one environment, you can mine your data for trends, analyze these trends, and then save these results to reports. Based on the size of your test group and the amount of data you are collecting, there is an NI DataFinder solution to meet your needs.

NI DataFinder

This pivotal middle component of the NI TDM solution gives you the ability to customize your data management and mining needs. Once you have installed NI DataFinder, configure it to search through particular folders and directories for specific file types. NI DataFinder then automatically builds an index of the metadata of these files, which you can query. You can add and remove search areas with one click as well as configure NI DataFinder to index your own custom file formats.

Once you have set up your search areas, you can query your data from two interfaces in DIAdem – the simple and advanced search. In the simple search, you type in words or values to run a quick search and return files that contain all of these values somewhere within the file. Using the advanced search, you can search for properties on specific hierarchy levels (file, group, and channel) and return your results as channel, group, or file listings. For example, you could search on “Group.Test_Status = fail” and return all the groups that contain this property and failed. In this case, you can search for all channels across multiple files and formats that meet your search criteria and then load just the returned channels from each of the files. You now have more control of your data because you only load the channels or properties of interest from each file.

Figure 5. Using the advanced search, you can quickly find trends and correlations within your test data.

Out of the box, NI DataFinder automatically indexes TDM and TDM Streaming files within the search areas you specify. However, you may have a number of existing files in different formats that you also want to search using NI DataFinder. For these, you can use a DataPlugin to load your files in DIAdem or LabVIEW. Once you have a DataPlugin for the file, you can index, search, and load it just like a TDM file. There are more than 75 existing FREE DataPlugins available for download at ni.com/dataplugins. You also can use the DataPlugin API, which includes examples to help you write your own DataPlugin for your custom formats.

Figure 6. The index stores all the descriptive information included with a file, so you can mine and search on these values.

As mentioned previously, the NI DataFinder index is completely self-configuring and maintaining. Once you point NI DataFinder to the folders and directories you want to index, it builds the index of all the metadata stored within the files. As you collect more test data and store the files within the search areas, NI DataFinder receives a Windows File Event notification and automatically indexes the new data files. NI DataFinder also monitors file events that are triggered when a file is changed or deleted, so you always have the most up-to-date index of your data. Also, NI DataFinder has a number of native properties, such as name and description, which are stored with all files. However, in many tests that you perform, you may want to store your own custom properties such as Test Status or Test Procedure that are also indexed and searchable. Because NI DataFinder is a completely self-scaling index of all your metadata, the index automatically adapts to these new properties and indexes them along with the native properties. With a traditional database solution, adding new properties can be a huge undertaking if the database is not designed properly from the beginning.

DIAdem DataFinder and NI DataFinder Server Edition

National Instruments offers two data management options to meet your needs – DIAdem DataFinder and NI DataFinder Server Edition. DIAdem DataFinder, included in all packages of DIAdem, helps you search data stored locally on your machine or on a network. Larger groups accessing data across multiple machines with limited network bandwidth and numerous file formats require a more unified and robust approach.

NI DataFinder Server Edition is software installed on a server machine, but it is designed for easy installation and configuration without the need for IT involvement. Just like the local DIAdem DataFinder, you configure particular search areas and file formats for indexing. Once you have configured these search areas, multiple client computers using DIAdem have simultaneous visibility to collected test data using the one central shared index on the server machine. Your entire group or department can find and analyze data without the hassle of searching through files across several machines. Just select to connect to the NI DataFinder Server Edition, and you are immediately connected to the server index just as if you were connected to your local DIAdem DataFinder index.

Figure 7. NI DataFinder Server Edition centralizes data from multiple test stations so you can easily access and mine it with multiple clients simultaneously.

Because NI DataFinder Server Edition is running on a high-bandwidth server machine running Windows Server 2003 OS, you also benefit from its performance and connection flexibility, which allows up to 25 users to simultaneously connect to each index. Also, your network traffic is greatly decreased because you are only passing a query over the network to the central server as opposed to continually crawling data files over the network as you would if you were using the local NI DataFinder.

NI DataFinder Server Edition also gives you more control over who is able to search through particular files and folders. Just as in DIAdem DataFinder, you can select which search areas you want to index with NI DataFinder Server Edition. However, you can also set up multiple indexes with NI DataFinder Server Edition. For example, say you work at a cell phone company. You have two groups – the software group and the electronics group – that work closely together to design new phones. Both groups are collecting data as they perform tests on their components. They can then store this data on a server machine running NI DataFinder Server Edition to search and mine through the raw data. However, each group only wants the query to return results from their data sets. In this case, you can configure multiple indexes with NI DataFinder Server Edition so that each group only searches within the index containing relevant information for that group’s particular search area.

Once you configure NI DataFinder Server Edition on the server machine, you can export configurations to the appropriate DIAdem clients in a few easy steps. Just select to export the current configuration from the settings menu. This creates a *.urf file, which you can give to all the users who want to connect to that NI DataFinder Server Edition index. The users just need to double-click the *.urf file to register the NI DataFinder Server Edition profile and any associated DataPlugins for the files on the server. Now users can open their local copies of DIAdem and select the remote NI DataFinder from the menu. These two simple client steps are all you need for everyone in your test group to have easy, intuitive access to the test results.

Figure 8. The NI DataFinder Server Edition server manager makes it easy to configure new indexes and deploy client configurations.

Using NI DataFinder Server Edition, you now have faster and easier access to not only to your data but your neighbors’ data. Having faster and easier access to more data helps you focus on the important tasks of design and analysis as opposed to wasting time searching for data. Besides just having the ability to find the data you are looking for, you have visibility into multiple files at once to find important trends or anomalies.

Managers also benefit from this centralized approach to data management. Managers can access test data without having to request data from each engineer. Though this benefit is also possible with a database solution, the initial cost of NI DataFinder Server Edition is considerably lower than most database solutions, without the subsequent costs for maintenance or scaling. Also, with a self-configuring and maintaining index, managers do not have to worry about IT bandwidth with NI DataFinder Server Edition. Bottom line, NI DataFinder Server Edition gives a wide range of people access to the important information and data collected so you can achieve the highest return on your investments in test equipment and test data.

Analyzing and Reporting Your Test Data

The last element in the NI TDM solution is the query and reporting client, DIAdem. DIAdem provides a built-in user interface to NI DataFinder to help you search easily through all indexed data files with a simple Internet-like keyword search or a powerful advanced search.  DIAdem combines this data-searching functionality with general-purpose interactive data visualization, analysis, and reporting environments. Once you have found the data you need – either through DIAdem DataFinder or NI DataFinder Server Edition – you can analyze the data using hundreds of built-in engineering analysis functions and create consistent reports to share your results with others. In addition to being an interactive data-handling environment, DIAdem and NI DataFinder also can be fully automated with VBScript and custom dialogs. Engineers often use this solution as the basis for a fully customized data management and evaluation application.

Figure 9. DIAdem for Post Analysis, Reporting, and Automation of Data

Conclusion

The NI data management solution covers all the bases of a scalable, cost-effective, search-ready data management system. You no longer need to worry about the details of file parsing, data model and data table structure, database management, search GUI design, or support from outside groups such as the IT department. All you have to do is use the easy-to-configure NI DataFinder or NI DataFinder Server Edition to automatically create and maintain a data index. Finally, you can take advantage of the DIAdem built-in search GUIs and easy-to-use interactive data analysis and reporting environment to quickly turn your saved data into the reports and result information you need. 

NI DataFinder and NI DataFinder Server Edition benefits include:

• Low cost of initial purchase and ongoing maintenance
• No need to install, design, and maintain expensive and complex databases
• Simple setup – simply point at the directories you wish to index and install the correct DataPlugins for all your data formats
• Automatic indexing of data files added to NI DataFinder search areas
• Automatic scaling from single users to complete engineering groups and enterprises
• Easy DataPlugin creation for your custom file formats
• Interactive or automatic data analysis and reporting with DIAdem

Additional Resources

• Online Presentation of NI DataFinder Server Edition and NI DIAdem 
• Pricing and Purchasing Information
• DIAdem
• TDM

13 Ratings | 3.69 out of 5

Read in  English German Korean |  Print |  PDF

Reader Comments | Submit a comment »