Normalizing Incoming Data Automatically

Create a routine to automatically convert incoming data files into a data table.

  1. In the SystemLink web application, navigate to Analysis » Scripts.
  2. Click Python 3 (ipykernel).
  3. Use the linked example as a template to create ETL pipelines.
    The example includes steps to complete the following actions.
    • Extract the file contents.
    • Transform the data into a data table format.
    • Load the data into storage with the DataFrame Service API.
    Note When you create a data table, you must do the following.
    • Define the column type.
    • Define the data type for each column.
    • Define the total number of columns. The total number of columns cannot exceed 2500.
    You cannot change these settings after creating the table. However, you can define and update additional metadata for existing tables or columns.
    Column Type Description
    NORMAL The column has no special properties. This behavior is the default behavior.
    INDEX The column provides a unique value per row. Each table must provide one INDEX column with one of the following data types.
    • INT32
    • INT64
    • TIMESTAMP
    NULLABLE The column allows rows to contain null values. You can exclude NULLABLE columns when appending rows. Appended rows will use null values for that column.
    Note After you finish appending data to a table, free up resources associated with the table. Use the route POST /nidataframe/v1/tables/{id}/data where {id} is the table ID. Set endOfData to true in the JSON body of the request.
  4. Create a routine to execute your notebook and specify File uploaded for the event. Refer to Automating Actions with Routines to learn how to create a routine.
    When files are created, your notebook will execute to convert the file data into a data table.
You can view data tables in the SystemLink web application under test results or in dashboards. To download a data table, use one of the following methods.
From a dashboard Click the top of the data table panel in your dashboard and select Inspect. Click Download CSV.
Note Data tables downloaded from a dashboard are decimated based on the query settings for the dashboard.
From a test result Select a data table in the Data Tables tab and click Download CSV.
Note Data tables downloaded from a test result are undecimated.