How to Set Up a Preprocessing Routine for DataFinder Server

Publish Date: May 19, 2017 | 0 Ratings | 0.00 out of 5 | Print | Submit your review

Overview

In this tutorial, you will learn with the help of examples how to configure Data Preprocessor to standardize metadata and prepare data for automated analysis. This is an important part of the Technical Data Management process chain to ensure consistent and automated analysis for every data point that is collected. There are other tutorials that cover the other steps in the process.

Table of Contents

  1. Data Preprocessor Configuration for TDMS Files
  2. Using DIAdem to Define a Data Preprocessor Configuration
  3. Deploy Data Preprocessor Package via Web Server
  4. Next Steps
  5. Additional Data Management Software Suite Tutorials

 

 

The Data Preprocessor is designed to convert raw data from different sources to homogenous “smart data” by standardizing property identifiers and values, applying unit conversion, adding additional statistical calculations from the individual channel data, and verifying the resulting data. The last step is to convert the data into a standard file format such as TDM or TDMS so that the modifications made during the preprocessing routine do not affect the original data file

 

 

1. Data Preprocessor Configuration for TDMS Files

The files you will see in the following example were created in DIAdem, where you specify how your data should be cleansed and unified.

This example shows you how to first change property identifiers and content to standardized and more suitable names and values and add some additional signal characteristics using the Data Preprocessor. The example data is stored in your National Instruments public example folder (for instance: C:\Users\Public\Documents\National Instruments and the example starts with the TDMS files located in …\National Instruments\DataFinder\Data.

 

Data Preprocessor Operation

The example TDM files are written with 3 levels of information and metadata saved at each level. For an overview of the TDM file format, read this whitepaper.  These example files contain some properties that describe a common set of information, such as test information, on file level, common channel information on group level, and sensor-specific information on channel level. As of DIAdem 2017, you can group properties in the Data Portal using the tilde character (~) for additional organizational structure.

In our data cleansing routine, we will replace ‘_’ with a ‘~’ to show how you can further organize your data.

 

File Level Properties

Group Level Properties

Channel Level Properties

Original Name

Processed Name

Original Name

Processed Name

Original Name

Processed Name

Test_Module

Test~Module

Test_NoOfSensors

Test~NoOfSensors

Limit_High

Sensor~Limit~High

Test_Name

Test~Name

Test_Status

Test~Status

Limit_Low

Sensor~ Limit~Low

Test_Operator

Test~Operator

 

 

Sensor_ID

Sensor~ID

Test_Procedure

Test~Procedure

 

 

Sensor_Type

Sensor~Type

UUT

Test~UUT

 

 

Sensor_Calibration_Date

Sensor~Calibration_Date

 

The screenshot below shows the effect of grouping properties in the DIAdem Data Portal using the file level property replacement as an example:

 

 

Use the “Replace Property Identifiers” function in the Data Preprocessor to build property groups by inserting the tilde character (~) into the property identifier as a separator.

 

Additionally, the individual channels have abbreviated channel names, which you are going to replace with more meaningful names using the “Replace Property Values” function. Because the names follow a certain pattern, you can apply a replacement pattern to convert “Temp” to “Temperature” using a single mapping rule.

 

Channel Name

Original

Standardized

Temp_A

Temperature_A

Temp_J

Temperature_J

 

 

We will then add some channel properties containing signal characteristics, such as:

 

Channel Data Statistics

Characteristic Value

Property Name

Minimum

Statistics~Minimum

Maximum

Statistics~Maximum

Arithmetic mean

Statistics~ArithmeticMean

Standard deviation

Statistics~StandardDeviation

 

 

 

Finally, you define the file format of the cleansed and enriched data. You can choose between the file formats TDM or TDMS. Our example uses the TDM format.

 

Back to Top

2. Using DIAdem to Define a Data Preprocessor Configuration

Use the Data Preprocessor Configuration dialogs to define the Data Preprocessor processing steps which will configure a Data Preprocessor in the TDM Server Manager.

For convenience, it is easier to load one example file into the DIAdem Data Portal and use drag&drop for some configuration steps:

1. Select the DIAdem NAVIGATOR panel.

2. Click the Delete Internal Data symbol.

 

 

3. Select File>>Open.

4. Change the file type to TDMS (National Instruments TDM Streaming).

 

 

5. Navigate to the TDM Server Demo Data folder (“C:\Users\Public\Documents\National Instruments\DataFinder\Data”) and select a file.

6. Click Load to load a TDMS example file into the DIAdem Data Portal.

 

 

Now specify the DataFinder Preprocessor configuration:

7. Select the DIAdem Script panel.

8. Select Settings>>TDM Server>>Data Preprocessor.

 

 

9. Click the New Data Preprocessor Package symbol.

 

 

10. Enter the Name of the new Data Preprocessor and an optional Description and Author.

 

 

11. Click OK.

Configure the Property Identifier Mapping

12. Select the Replace identifiers tab

13. Enable Replace property identifiers

 

 

14. Click Edit.

15. Click the Add DataPlugin Mapping symbol and select TDMS as DataPlugin name.

 

 

 

16. Click OK.

17. Select the TDMS column by clicking into the column header.

18. Click Edit DataPlugin Mapping to open the dialog box.

 

 

19. Select the root element in the Data Portal (the root of the tree structure).

20. Select all Custom Properties and drag&drop them into the still open Edit DataPlugin Mapping dialog.

 

 

21. Specify the <vacant> Standardized Identifier as defined in the table above. Where applicable replace the separators like “_” with “~” using the Find and Replace dialog box along with the Transfer Property Names tool to copy the Original Identifier text over. 

 

 

22. Repeat this procedure for ChannelGroup and Channel on the respective tabs.

 

 

 

23. Click OK.

24. Click Close.

Configure the Property Value Mapping

25. Switch to the Replace Values tab

26. Enable Replace property values

 

 

27. Click Edit

28. Select Channel/Name in the tree

 

 

29. Select three channels (Temp_A, Temp_B and Temp_C) in the Data Portal

30. Drag&Drop the selected channels into the Mapping rules area of the Replace Property Values dialog box.

 

 

31. For Temp_A specify Temperature_A as Output Value.

32. For Temp_B specify Temperature_B as Output Value.

33. For Temp_C specify Temperature_C as Output Value.

 

 

34. The Search pattern wizard has detected a pattern.

 

 

35. Click Yes to use this pattern and map all Temp_? channels to Temperature_?

36. Confirm the Edit rule dialog box by clicking OK.

 

 

37. Close the Replace Property Values dialog box.

Defining Additional Channel Statistics

38. Switch to the Statistics tab.

39. Enable Calculate statistics.

 

 

40. Click Edit.

41. Enable Minimum.

42. Use Statistics~Minimum as Custom Property Name.

43. Enable Maximum.

44. Use Statistics~Maximum as Custom Property Name.

45. Enable Arithmetic mean.

46. Use Statistics~ArithmeticMean as Custom Property Name.

47. Enable Standard deviation.

 

 

48. Use Statistics~StandardDeviation as Custom Property Name.

 

 

49. Click Close.

Define the Standard File Format

50. Switch to the Convert File tab.

51. Define Save with DataPlugin as TDM.

 

 

52. Click Save Data Preprocessor Configuration As and save the DPP file to a folder you specify.

 

 

Verify the Data Preprocessor Routine

53. Select Data Preprocessor Chain Test with Debugger

 

 

54. Browse to C:\Users\Public\Documents\National Instruments\DataFinder\Data

55. Select TDMS from the File name drop down menu

 

 

56. Select a TDMS file and click Load

57. A pop up dialog will appear letting you know the data has been processed and loaded into the Data Portal.

 

 

58. Explore the Data Portal and verify the channel names changed from Temp_X to Temperature_X and the Custom Properties show the new organization we set up.

 

 

Back to Top

3. Deploy Data Preprocessor Package via Web Server

Now we can deploy the Data Preprocessor to the server via the TDM Server Manager.

The Data Preprocessor uses a configuration file (*.DPP), which you created in DIAdem, to convert raw file data from one or more raw data folders to homogenous data, which you save in the respective processed data folder.

To setup the Data Preprocessor, complete the following steps in the wizard:

1. Open TDM Server Manager in your web browser and login using your credentials.

 

 

2. Select New>>Data Preprocessor to start the configuration wizard.

 

3. Enter ConvertExampleData as the Name of the new Data Preprocessor server, and click Next.

 

4. Enter the Name of the raw data area.

5. Specify the Raw data folder. Click the Choose a Folder symbol and select the TDM Server Demo Data.

 

 

6. Specify the Processed data folder. Click the Choose a Folder symbol and navigate to “C:\Users\Public\Documents\National Instruments\TDM Server Demo\Processed_Data”

7. Click Next to continue the configuration wizard

8. Select Use custom configuration and specify your Data Preprocessor that you saved in step 52.

 

 

In case you want to use a preconfigured Data Preprocessor file, you can find the Configuration file at “C:\Users\Public\Documents \National Instruments\Data Preprocessor\Configuration\Process_Example_Data_TDMS.DPP”.

9. Click Next to continue the configuration wizard.

10. Select the ADVANCED license

 

 

11. Click Finish to complete the initial configuration of the Data Preprocessor

12. Select the newly created Data Preprocessor in the list and click Start to initiate processing.

 

 

 

Back to Top

4. Next Steps

After configuring the preprocessor routine, you may want to test the functionality. To do this, open up three folders and arrange on your desktop:

1. C:\Users\Public\Documents\National Instruments\DataFinder\Data (Contains Data)

2. C:\Users\Public\Documents\National Instruments\TDM Server Demo\Raw_Data (Empty)

3. C:\Users\Public\Documents\National Instruments\TDM Server Demo\Processed_Data (Empty)

Now copy the data files into the Raw_Data folder and see how the Processed_Data folder updates.

 

Back to Top

5. Additional Data Management Software Suite Tutorials

Follow the remaining tutorials for the Data Management Software Suite.

 

 

 

 

Back to Top

Bookmark & Share


Ratings

Rate this document

Answered Your Question?
Yes No

Submit