Configuring MongoDB for Optimal Data Table Metadata Query Performance

To obtain optimal performance and increase system stability, connect the DataFrame Service to a dedicated MongoDB instance.

Note NI recommends using MongoDB Atlas. This edition contains monitoring and optimization tools that help improve performance. Use these tools to monitor CPU, memory, and disk usage of the MongoDB instance.

If your performance does not meet expectations, adjust the resources. You can also configure indexes to accelerate common queries.

Configuring MongoDB Indexes

Accelerate queries by using the MongoDB Atlas Performance Advisor to create indexes.

Note Performance Advisor is only applicable for deployments that use MongoDB Atlas or MongoDB Enterprise Advanced.

Performance Advisor monitors slow queries and suggests indexes that can accelerate the query. For more information, refer to the MongoDB Performance Advisor Indexing in Atlas documentation and the MongoDB Performance Advisor Indexing in Enterprise Advanced documentation.

If you have several SystemLink Enterprise deployments, you can configure those deployments identically through the infrastructure as code (IaC) development method. To implement IaC, NI recommends creating additional indexes through the Helm chart of the DataFrame Service.

For example, a data table may contain a high-cardinality custom property called TesterNumber. You would then want to efficiently query all data tables associated with a given tester number. To conduct an efficient query, create an index on the TesterNumber custom property.

  1. Locate and open the Helm values file for your deployment.
    Note This file usually shares a file name similar to the systemlink-values YAML file.
  2. Add the name of the custom data table property to the dataframeservice.database.propertyIndexes configuration.
  3. Ensure the property name casing in the Helm values file matches the custom data table property casing.

    The following example configuration adds a custom index on the TesterNumber custom data table property:

    dataframeservice:
  database:
    propertyIndexes:
      - TesterNumber
  4. Deploy SystemLink Enterprise.
  5. Wait for the new DataFrame Service Pods to start up and create the requested indexes.

Deployments with millions of data tables may experience a delay and consume significant MongoDB resources. Monitor MongoDB performance while the Pods are starting up and consider adding resources if necessary. When the Pods start again, queries that include an equality filter should execute faster and consume fewer resources.

The Helm chart also supports configuring indexes on first-class fields through the dataframeservice.database.additionalFieldIndexes configuration. An example field is the SupportsAppend field.

For a complete list of supported fields, refer to comments next to the configuration in the systemlink-values YAML file.

Monitoring MongoDB Performance

For optimal query performance, size MongoDB appropriately for your organizational workload.

To size a MongoDB deployment, monitor the following system performance information:
  • CPU usage
  • Memory usage
  • Disk usage

If any of these indicators are too high, increase the resources until performance meets expectations.

For more information on how to monitor MongoDB resource usage, refer to How to Monitor MongoDB.

Adding Analytics Nodes

For MongoDB Atlas deployments, use read-only Analytics nodes to horizontally scale data table metadata queries.

For more information on how to add Analytics nodes to a MongoDB Atlas instance, refer to How to Add Analytics Nodes to a MongoDB Deployment. The DataFrame Service automatically targets data table metadata queries to any present Analytics nodes.

Adjusting Timeout Values

Modify the timeout length for metadata queries in the DataFrame Service by adjusting a Helm configuration file.

The DataFrame Service imposes a timeout on queries for data table metadata from a SystemLink UI client. If queries exceed these timeouts, the service returns a MetadataStoreTimeout error. You can adjust this timeout by modifying the dataframeservice.interactiveMetadataQueryTimeout configuration in the Helm values file of the deployment.
Note You should carefully monitor the MongoDB resources after making an adjustment. Setting the timeout too high can exhaust the IOPS of the MongoDB instance.

The following configuration sets the timeout at 15 seconds:

dataframeservice:
  interactiveMetadataQueryTimeout: "00:00:15"