Consider techniques to reduce the amount of data that must be archived. The amount of reduction possible depends on the system requirements.
Code that runs faster than necessary reduces the resources available for future expansion and may generate superfluous data.
- Run loops only as fast as necessary to fulfill the system requirements.
- Provide the appropriate execution timing to allow other processes to execute.
- Consider allowing additional CPU time for future expansion.
- See LabVIEW 2009 Real-Time Module Help: Timing Deterministic Applications (RT Module) for more information.
- Consider the rates in the entire system.
- For example, a cRIO 9074 reading one channel of thermocouple (TC) data from a 9211 module through the default scan engine. The RT code can execute in a few microseconds, the scan engine updates every 10ms, but the 9211 has a scan rate of 14 scans/second. Archiving TC samples to disk faster than 14Hz results in duplicate data.
Data Size and Type
When a small amount of data is used to represent a data point, less data must be written to disk to archive the point.
- Consider archiving RAW data instead of calibrated and scaled data.
- Raw data is typically 12, 16, 20 or 32-bits per sample.
- Scaled and calibrated data is typically 64 bits.
- Properly packing and writing raw data may require 2-4 times less disk access (see Write Less Often in the Optimization section of this document).
- Consider the actual data resolution.
- All digitizers have a finite resolution.
- Use the smallest numeric representation possible.
You can choose to write to several types of files from LabVIEW.
Items marked with an asterisk write data in binary format. This data is not human readable but often consumes less space on disk. The other formats are human readable, but often require more space on disk.
For example, a DBL uses 64 bits in a Binary file. The same DBL uses 8 bits per digit in a Text file (up to 120 bits depending on the number). See LabVIEW 2009 Help: Numeric Data Types Table for more information.
Use a binary file format to reduce read/write time on the real-time system.
Reduce the number of data points archived to disk. There are built-in LabVIEW functions to decimate data.
- Sample Compression
- Decimate 1D Array
- Decimate (single shot)
- Decimate (continuous)
You can also build your own decimation algorithms. The decimation factors in the following examples are arbitrary. The following considerations are made.
- Data is always processed.
- Data is not always archived to disk.
- Data history – how do we know how much data was not logged?
- In these examples, there is a constant loop rate and a file timestamp.
- A sample counter could have been used.
- How will the next file-write be determined?
- We apply the algorithm to the last point written to file.
- The algorithm could be applied to a constant set point.
Decimate by Sample
Log every N’th sample (Figure 1). A simple rollover counter can be used to log every N’th sample. None of the samples between N, 2N, 3N… are logged to file.
Averaging N Samples
Log the average of the last N points to file (Figure 2). An additional Shift Register or Feedback Node is used to store the running sum, and the average is calculated and written to disk.
Decimate by Time
Log one point of data every M units of time (Figure 3). When configured to reset, the Elapsed Time Express VI is suited for this task. None of the samples between t = M, 2M, 3M… are logged to file.
Average Samples over Time
Average samples over time (Figure 4). One Shift Register/Feedback Node is used to store the running sum and another is used to store the number of iterations since the last file write. The average is calculated and written to disk.
A strict delta filter logs data only if the value changes (Figure 5).
Extend delta filtering to absolute deadbanding. An offset from the current data point must be exceeded to qualify for a disk write (Figure 6). The In Range and Coerce function is suited for the comparison.
Absolute deadbanding can be modified to implement alarming and bandpass filtering. The min and max values are provided to the In Range and Coerce function to identify the window.
Rather than use an offset to define the deadband, it may be more meaningful to define the deadband as a percent change (Figure 7). For example, if the new value is within 5% of the last value written, do not write the value to disk.
The nature of the data and system requirements will determine what statistical parameters are of interest. There are built-in LabVIEW functions to calculate statistics.
- Min & Max Function
- Statistics Express VI
A simple data set may include the following.
- Number of samples
Using these simple building blocks, a robust algorithm may be implemented. Consider the following combination.
- Use a Deadbanding algorithm to determine if a threshold has been exceeded.
- Use Statistical Profiling to build up a picture of the data over time.
- If the deadband has not been exceeded, write and reset the statistical profile every hour.
- If the deadband has been exceeded, write and reset the statistical profile as well as the offending data point.
Reduce the size of the data archived to disk.
Write an initial value and delta (either as a scalar value or percentage) to disk. A smaller data type may be used to represent the delta than the data point itself. This technique is mostly applicable to continuous phenomena (temperature in an incubator for example).
Data Specific Compression
There are a variety of compression algorithms that have been written for specific data sets (audio, video, image, etc.). These algorithms may be used to reduce the amount of data that gets written to disk. A decompression algorithm is used to extract the data when it’s needed. These algorithms are beyond the scope of this document, but an Internet search for “Compression Algorithms” will get you started.
There is also a 3rd party API called OpenG Zip Tools that can be used to compress files on LabVIEW RT targets. This API is not supported by National Instruments.