The Data Storage Directory in NI InsightCM™

Publish Date: May 16, 2019 | 0 Ratings | 0.00 out of 5 | Print

Overview

NI InsightCM collects, stores and manages data from distributed monitoring devices that send the data to the InsightCM Server. During initial installation or upgrade, users must configure data directory locations to be used by the server. Additionally, InsightCM 3.2 has introduced data storage improvements, which are explained below. Refer to the NI InsightCM Readme for more information on installation, upgrades and server hardware requirements.

Table of Contents

  1. Data Storage Directory
  2. Changing Data File Directory from Database Directory (Optional)
  3. Data Set Migration in InsightCM 3.2
  4. Run Data Set Migration
  5. Support
  6. Next Steps

1. Data Storage Directory

With InsightCM 3.0, all data was stored in a central database. Users can configure the data directory location during installation with the default location given below. In certain scenarios, InsightCM servers with large amounts of collected data could experience disk thrashing due to large amounts of incoming data sets and server hardware limitations.

  • Database Directory: C:\ProgramData\National Instruments\InsightCM 3.0\

With InsightCM 3.2, data is stored in both a central database and as files on disk. This change results in better performance by decreasing memory used by the InsightCM database and more uniform load time for viewing data in the Data Viewer. This change is not apparent to the end user and does not change user interaction with Data Viewer. Users can configure the data storage locations during installation, and the data files are stored in this location by default.

  • Database Directory: C:\ProgramData\National Instruments\InsightCM 3.0\
  • Data File Directory: C:\ProgramData\National Instruments\InsightCM 3.0\Files

Users can configure the Data File directory to be on a different drive than the the Database directory to improve data viewing performance. Changing the Data File directory should be done immediately after installation or upgrading to InsightCM 3.2. For InsightCM 3.2 and later installations, a recommended storage configuration for optimal perfomance would be a Windows Server machine with three solid state drives with the InsightCM Server 3.2 installed on the C:\ drive, the Database directory installed on the D:\ drive, and the Data File directory installed on E:\ drive.

Figure 1. NI InsightCM Data Directory selection during Installation

 

Back to Top

2. Changing Data File Directory from Database Directory (Optional)

Changing the Data File directory should be done immediately after installation or upgrading to InsightCM 3.2. These steps are only applicable to InsightCM 3.2 and they are not required if storing data files and the database on the same drive is acceptable. For systems with large numbers of devices, storing data files and the database on separate drives can improve data loading performance. To change the data set file storage location, use the InsightCMConsole to export the DataManagementConfig and edit the BaseDir location. Once the data set file location is set, you can then run movedatafiles command and data sets will be migrated to the set folder location. See below for steps to move the file storage location.

  1. Open Windows Command Prompt and run the below commands. These commands use the default installation directory and will need to be updated if InsightCM was not installed to the default location.
  2. cd C:\Program Files\National Instruments\InsightCM Server 3.0\
  3. InsightCMConsole exportconfig -k DataManagementConfig -o C:\Users\Public\c.json
  4. Open c.json and update the BaseDir location. For example, to change from the default location to F:\InsightCM\Files, you make the below change:
    \"BaseDir\":\"C:\\\\ProgramData\\\\National Instruments\\\\InsightCM 3.0\\\\\\\\Files\"
    \"BaseDir\":\"F:\\\\InsightCM\\\\\\\\Files\"
  5. Open Windows Services and stop the main NI InsightCM service.
  6. Run the following command in the Windows Command Prompt used earlier:
    InsightCMConsole importconfig -f C:\Users\Public\c.json
  7. Start the main NI InsightCM services in Windows Services.

Any data events that are received by the InsightCM server will be saved in the Data File and Database directories. If the user changes the Data File directory after files have been received, the user must manually copy files to the new Data File directory.

 

Back to Top

3. Data Set Migration in InsightCM 3.2

When upgrading to InsightCM 3.2 from a previous version, users must manually run the data set migration command from the InsightCMConsole to move data from the database to files in InsightCM 3.2. Trend and waveform data is still available in the Data Viewer while the data set migration is in process. However, users may experience slowdowns when loading in the Data Viewer and using the InsightCM web application. It is recommended to minimize web application usages during the migration. The migration runs as a background process and can take on the order of hours to days to complete, depending on the amount of data to be moved. It is not recommended to stop the migration process once it has began as this may cause the data event currently being processed to be lost. 

 

Back to Top

4. Run Data Set Migration

These steps are only applicable to upgrading InsightCM 3.0 or 3.1 to InsightCM 3.2. The general process for migrating Data Sets is to run the movedatafiles command, run the archivedataevents command, compact the database (optional) and then restart the main NI InsightCM service and the NI InsightCM MongoDB service. Once the data set migration is complete, the data set files are now located on disk in the Data directory configured during installation. The data set files are located inside the Files folder in the Data directory.

To start migrating data to the file system, use the InsightCMConsole.exe with the movedatefile option. The movedatafiles command takes the waveform portion of data sets that are stored in the InsightCM database and moves them to files. One consideration when doing the data set migration is that the storage drive must have at least double the space on the data drive to complete the data set migration. This is because the InsightCM database does not automatically release disk space back to the drive after a data set is migrated. For example, if the size of the Data folder stored on the D:\ is 40 GB, then the user would need 80 GB to complete the migration.

Once the movedatafiles command has complete, run the archivedataevents command. The archivedataevents command moves all remaining data set components (data events, data set metadata, and trend points) older than 30 days out of the database to files. This further reduces the size of the InsightCM database and improves data viewing perfomance. An optional step is to compact database to release unused space to the storage drive. If your InsightCM data directory is greater than 100 GB, consider compacting the database to reclaim storage space. Contact NI Technical Support and reference 'InsightCM Database Compaction' for more information. The final step is to restart the NI InsightCM MongoDB service. To safely do so, first stop the main NI InsightCM 3.2 service, restart the NI InsightCM MongoDB service, and then start the NI InsightCM service.

  1. Open Windows Command Prompt and run the below commands.
  2. cd C:\Program Files\National Instruments\InsightCM Server 3.0\
  3. InsightCMConsole movedatafiles
  4. Allow the movedatafiles command to complete. When completed, run the below command.
  5. InsightCMConsole archivedataevents
  6. Allow the archivedataevents command to complete.
  7. (Optional) Compact database to release drive. Contact NI Technical Support and reference 'InsightCM Database Compaction' for more information.
  8. In Windows Services, stop the main NI InsightCM 3.2 service, restart the NI InsightCM MongoDB service, and then start the NI InsightCM service.

 

Figure 2. Running the InsightCMConsole movedatafiles Command

 

Back to Top

5. Support

A team of NI engineers worldwide can provide technical support. In addition, NI’s network of partners offer installation and configuration support.

Post a question on the NI InsightCM Forum.

Contact NI Technical Support.

Find a NI condition monitoring partner.

 

Back to Top

6. Next Steps

Back to Top

Bookmark & Share


Ratings

Rate this document

Answered Your Question?
Yes No

Submit