Automate Measurement Data Standardization, Enrichment, and Validation with DataFinder Server


Data from various sources may use different units, might use different nomenclatures, or may be corrupt. Automatically standardize metadata, verify data quality, and convert to TDMS to ensure data is consistent regardless of origin with DataFinder Server. This whitepaper discusses the benefits of using DataFinder Server, a component of the SystemLink product family, to standardize and enrich your raw data files.


The Problem with Diverse Data Sources

If we start with a raw data set that includes data from multiple sources we typically run into 4 problems:


Figure 1: The Inconsistent Data Problem. 

  • Multiple Sources and Formats 
  • Different Metadata Names and Values
  • Different Engineering Units
  • Data Might Contain Errors


If these issues exist in the raw data, they ultimately result in inconsistent analysis for the following reasons.

  • Consolidating File Formats Manually Leads to Errors

Multiple file formats and sources are used in a test setup for a variety of reasons, but is most often attributed to different groups within the company testing subsystems or components. These different groups could use different hardware vendors or different processes. Unless an automated process is used, consolidating data from multiple file formats is a tedious and manual process that leaves room for possible human error and wasted time. 

  • Can’t Find the Data 

Data querying tools, like DataFinder Server, can reduce the time and effort it takes to find the data you need to analyze. Creating a query requires metadata names and values to be used. If these metadata names and values don't match and extra queries aren't used to expand the search results, the data won't be found and analyzed.

  • Analysis Results are Incorrect

Incorrect engineering units and erroneous raw data can lead to drastic changes in the results of analysis procedures resulting in invalid results. Different engineering units can be a serious issue for teams located in different sites where on standardized unit hasn't been agreed upon. The problem also tends to happen when testing is completed by other groups outside of the company. Sensor failures, data corruption, or human error can also have the same effect. These problems are typically only realized after a critical error occurs. 




The DataFinder Server Data Preprocessor

In addition to the DataFinder Server’s ability to automatically index files for group access and searching, DataFinder Server Advanced and Professional include the Data Preprocessor. The Data Preprocessor automatically standardizes, enriches, and validates data before using it for other purposes. By using the Data Preprocessor, you eliminate time wasted upfront in preparing your data files and save yourself from making costlier mistakes further downstream.

The Data Preprocessor has six configurable steps:


Figure 2: The Data Preprocessor has six configurable steps. 


  • Metadata Names—Map metadata, or property, names that have slight variations to a single, standardized name. For example, channel names “Engine,” “Engine Type,” and “Fuel” can be mapped to “Engine_Fuel_Type.”
  • Metadata Values—Similar to the step above, map property values with slight differences in representation to a single, standardized representation. For example, values “Diesel,” “D,” and “Fuel_Diesel” can be mapped to “Diesel.”
  • Engineering Units—Automatically convert measurements to standardized units using a comprehensive, built-in unit library. For example, convert displacement measurements in feet and inches to meters.
  • Signal Statistics—Calculate characteristic values for the data and store as properties. For example, calculate mean for one channel of measurement data.
  • Final Validation—Check that actual data is within an expected range. For example, if 1,000 data values are expected but only 250 are present in the file, exclude this data and flag for error condition.
  • Store Data—Store the standardized, enriched, and validated data as a “master” copy in another disk location on the server machine or over the network. This “master” copy is saved into a high-performance, low-footprint file format for optimized loading and storage. The original data files are never deleted or modified.


The 64-bit versions of DIAdem Advanced and Professional provide the interface to configure the above Data Preprocessor steps. At the end of the configuration, DIAdem saves a configuration file (*.DPP), which you upload to the DataFinder Server using the Server Manager web client. Afterward, the Server Manager becomes your interface for starting and stopping the preprocessor, as well as pointing it to the location of raw data to process.

Each Data Preprocessor uses a new DataFinder Server component called a compute node. A Data Preprocessor requires at least one compute node license to execute. More compute node licenses can be purchased to speed up preprocessing tasks by working in parallel.



DataFinder Server: The Perfect Companion to Analysis Server

The Analysis Server makes it possible for you to extract the maximum value from large amounts of data in minimal time by harnessing the power of server technology to analyze large amounts of data and generate reports. DataFinder Server complements the Analysis Server by ensuring data is standardized and verified with a Data Preprocessor before analysis, in addition to enabling the Analysis Server to perform complex search queries on data distributed throughout the organization.


Figure 3: DataFinder Server and Analysis Server work together to create a fully automated data management workflow.. 


SystemLink delivers measurable improvements in operational efficiency and productivity by providing you with a centralized web-based management interface for connected devices, software, and data. Although aligned with NI products such as LabVIEW, TestStand, and hardware systems, SystemLink also offers an open architecture for incorporating a wide range of third-party software and hardware technologies. DataFinder Server and Analysis Server are components of the SystemLink product family. To learn more about the other capabilities of SystemLink, visit


Additional Resources

  •  Contact Sales to learn more about DataFinder Server, Analysis Server, and SystemLink.