Table Of Contents

Defining Data Integrity Rules

Version:
    Last Modified: November 20, 2020

    Define data integrity rules to ensure that all data entering the SystemLink file service complies with the metadata requirements for your data analyses.

    Define data integrity rules in a JSON file that you upload to a DataFinder dedicated to indexing the file service.
    1. In a text editor, create a JSON file that defines the required metadata at the root, group, and channel level for all TDM data entering the file service.
      spd-note-tip
      Tip  

      Update the file version when updating a definition file. DataFinder adds the "FileVersion" property to the checked files to indicate which version of the definition file they comply with. This enables you to search for data that complies with a specific version of the definition file.

      Refer to the code example below for a metadata definition file at the bottom of this page.
    2. Upload the JSON file to the DataFinder that indexes the file service.
      1. Select the Data Integrity Rules tab.
      2. Upload the metadata definition from your disk and click OK. The DataFinder instance checks all new files entering the file service for compliance with the metadata definition.
      3. To remove the file, click Remove required metadata definition.
    3. Optional: Reset the entire index to execute a compliance check on already indexed files in the file service.
      1. Select the Index tab.
      2. Click the arrow next to Reset Index and choose Entire Index. The DataFinder instance reindexes all files in the file service and checks them for compliance. This process may take a while. During the check, DataFinder adds and optimizes two properties:
        Added Properties Values
        RequiredMetadata~Compliant (searchable in Data Navigation) 0 = no

        1.0 = version of the metadata definition file with which the metadata complies

        RequiredMetadata~ErrorText (not searchable in Data Navigation) Missing keys: metadata1, metadata2, ...

    Example for a metadata definition:

    {
    "FileVersion": 1,
    "Description": "Required metadata keys for SystemLink (this line is ignored)",
    "RequiredMetadataVersion": 1.0,
    "RequiredMetadataKeys": {
    "root": ["product_identifier", "product_type", "product_design_step"],
    "group": ["package", "dut_id"],
    "channel": ["ResultStatArithMean", "ResultStatMax", "ResultStatMin"]
    }
    }
    Search for the files that do not comply with the data integrity rules in the advanced search in Data Navigation and add the missing metadata.

    Recently Viewed Topics