The Data Storage Directory in InsightCM

With InsightCM 3.0, all data was stored in a central database. Users can configure the data directory location during installation with the default location given below. In certain scenarios, InsightCM servers with large amounts of collected data could experience disk thrashing due to large amounts of incoming data sets and server hardware limitations.

• Database Directory: C:\ProgramData\National Instruments\InsightCM 3.0\

Starting in InsightCM 3.2, data is stored in both a central database and as files on disk. This change results in better performance by decreasing memory used by the InsightCM database and more uniform load time for viewing data in the Data Viewer. This change is not apparent to the end user and does not change user interaction with Data Viewer. By default, the following directories are used:

• Database Directory: C:\ProgramData\National Instruments\InsightCM 3.0\

• Data File Directory: C:\ProgramData\National Instruments\InsightCM 3.0\Files

Users can configure the Data File directory to be on a different drive than the the Database directory to improve data viewing performance. Changing the Data File directory should be done immediately after installation or upgrading to InsightCM 3.2 or later. For InsightCM 3.2 or later installations, a recommended storage configuration for optimal perfomance would be a Windows Server machine with two solid state drives with the InsightCM Server and the Database directory installed on the  C:\ drive, and the Data File directory installed on D:\ drive.

These steps are only applicable to InsightCM 3.2 or later and they are not required if storing data files and the database on the same drive is acceptable. For systems with large numbers of devices, storing data files and the database on separate drives can improve data loading performance. To change the data set file storage location, use the InsightCM Console. See below for steps to move the file storage location.

1. Open Windows Command Prompt and run the below commands. These commands use the default installation directory and will need to be updated if InsightCM was not installed to the default location.
2. cd C:\Program Files\National Instruments\InsightCM Server 3.0\
3. InsightCMConsole setdatadir -dir "D:\InsightCM Data"
4. Start the NI InsightCM service.
    

Any directory can be specified in step 3. "D:\InsightCM Data" is just an example.

Any data events that are received by the InsightCM server will be saved in the Data File and Database directories. If the user changes the Data File directory after files have been received, the user must manually copy files to the new Data File directory.

If desired, the location of the database directory can be changed as well. See below for steps to move the database directory:

1. Stop the NI InsightCM and NI InsightCM MongoDB services.
2. Backup the original database directory folder: C:\ProgramData\National Instruments\InsightCM 3.0\mongodb
3. Move the following folder to the desired directory, such as D:\Data: C:\ProgramData\National Instruments\InsightCM 3.0\mongodb\db
4. Modify the fifth line of C:\ProgramData\National Instruments\InsightCM 3.0\mongodb\mongod.cfg to match the new directory:
   
   "dbPath: <insert new directory here>"
   
   
5. Run the following command from the command prompt:
   
   "C:\Program Files\MongoDB\Server\3.2\bin\mongod.exe" --config "C:\ProgramData\National Instruments\InsightCM 3.0\mongodb\mongod.cfg" --install
   
   
6. Start the NI InsightCM and NI InsightCM MongoDB services.
   

When upgrading from InsightCM 3.0 or 3.1, users must manually run the data set migration command from the InsightCMConsole to move data from the database to files in InsightCM 3.2 or higher. Trend and waveform data is still available in the Data Viewer while the data set migration is in process. However, users may experience slowdowns when loading in the Data Viewer and using the InsightCM web application. It is recommended to minimize web application usages during the migration. The migration runs as a background process and can take on the order of hours to days to complete, depending on the amount of data to be moved. It is not recommended to stop the migration process once it has began as this may cause the data event currently being processed to be lost. 

These steps are only applicable when upgrading from InsightCM 3.0 or 3.1. The general process for migrating Data Sets is to run the movedatafiles command, run the archivedataevents command, compact the database (optional) and then restart the main NI InsightCM service and the NI InsightCM MongoDB service. Once the data set migration is complete, the data set files are now located on disk in the Data directory configured during installation. The data set files are located inside the Files folder in the Data directory.

To start migrating data to the file system, use the InsightCMConsole.exe with the movedatefile option. The movedatafiles command takes the waveform portion of data sets that are stored in the InsightCM database and moves them to files. One consideration when doing the data set migration is that the storage drive must have at least double the space on the data drive to complete the data set migration. This is because the InsightCM database does not automatically release disk space back to the drive after a data set is migrated. For example, if the size of the Data folder stored on the D:\ is 40 GB, then the user would need 80 GB to complete the migration. Alternatively, the user could set the Data File directory to another drive with at least 40 GB of available space.

Once the movedatafiles command has complete, run the archivedataevents command. The archivedataevents command moves all remaining data set components (data events, data set metadata, and trend points) older than 30 days out of the database to files. This further reduces the size of the InsightCM database and improves data viewing perfomance. An optional step is to compact database to release unused space to the storage drive. If your InsightCM data directory is greater than 100 GB, consider compacting the database to reclaim storage space. Contact NI Technical Support and reference 'InsightCM Database Compaction' for more information. The final step is to restart the NI InsightCM MongoDB service. To safely do so, first stop the main NI InsightCM service, restart the NI InsightCM MongoDB service, and then start the NI InsightCM service.

1. Open Windows Command Prompt and run the below commands.
2. cd C:\Program Files\National Instruments\InsightCM Server 3.0\
3. InsightCMConsole movedatafiles
4. Allow the movedatafiles command to complete. When completed, run the below command.
5. InsightCMConsole archivedataevents
6. Allow the archivedataevents command to complete.
7. (Optional) Compact database to release drive. Contact NI Technical Support and reference 'InsightCM Database Compaction' for more information.
8. In Windows Services, stop the main NI InsightCM service, restart the NI InsightCM MongoDB service, and then start the NI InsightCM service.

Figure 2. Running the InsightCMConsole movedatafiles Command

A team of NI engineers worldwide can provide technical support. In addition, NI’s network of partners offer installation and configuration support.

Post a question on the InsightCM Forum.

Contact NI Technical Support.

Find a NI condition monitoring partner.

• InsightCM Known Issues
• Access the InsightCM Help Documentation
• Browse Purchase Options
• Request a Follow-Up From a Local Sales Engineer