SystemLink Enterprise secrets may need to be modified or rotated based on changes to your environment, your organization's security policies, or other operational requirements. Different secrets require different procedures to ensure proper operation after modification.

For detailed information about secret types, formats, and management approaches, refer to Required Secrets.
Note This document provides guidance for manually rotating Helm-managed secrets. If you are using external secret management tools (such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault), refer to your secret management system's documentation for rotation procedures. External secret managers may handle automatic rotation and synchronization differently than the manual procedures described here.

Common Procedures for Secret Rotation

  • Applying secret changes:

    After modifying secrets in the Helm values files, apply the changes using the Helm upgrade command:

    helm upgrade <release> oci://downloads.artifacts.ni.com/ni-docker/ni/helm-charts/systemlink --install --version <version> --namespace <namespace> --values systemlink-values.yaml --values systemlink-secrets.yaml
    Table 21. Upgrade Parameters
    Parameter Description
    release The Helm release name from the initial install.
    namespace The name space of the SystemLink Enterprise deployment.
    version The SystemLink Enterprise version.
  • Restarting Deployments:

    Some secrets require manual deployment restarts to ensure services pick up the new values. Use the kubectl rollout restart command:

    kubectl rollout restart deployment/<release>-<service-name> -n <namespace>

    To restart all deployments at once:

    kubectl rollout restart deployment -n <namespace>
  • Restarting StatefulSets:

    To restart stateful services, use the command:

    kubectl rollout restart statefulset/<release>-<service-name> -n <namespace>

    To restart all statefulsets at once, use the command:

    kubectl rollout restart statefulset -n <namespace>
  • Monitor Pod Health

    After rotating secrets and restarting services, verify all pods return to a healthy state:

    kubectl rollout status deployment/<release>-<service-name> -n <namespace> --timeout=10m
kubectl rollout status statefulset/<release>-<service-name> -n <namespace> --timeout=20m
kubectl get pods -n <namespace>
kubectl logs -l app.kubernetes.io/name=<service-name> -n <namespace> --tail=50

General Rotation Process

Most secret rotations follow this pattern:

  1. Modify the secret values in the appropriate Helm values file.
  2. Apply changes using the Helm upgrade command. See Common Procedures above.
  3. Restart the affected deployments or statefulsets if required. See individual secret procedures below.
  4. Monitor pods to ensure they return to a healthy state.

Some secrets have specific requirements or additional steps as documented in the procedures below. Secrets that use environment variables typically require manual pod restarts, while secrets mounted as volumes may be picked up automatically depending on the application's configuration.

Secret Rotation Procedures

  • Image Pull Secrets:

    Modify the image pull secret in systemlink-secrets.yaml and apply the changes by running the Helm upgrade command.

    Restart all deployments and statefulsets. Refer to Restarting Deployments and Restarting StatefulSets for instructions. Refer to Monitoring Pod Health to verify all pods return to a healthy state.

  • Authentication Secrets:

    Modify the authentication secret in systemlink-secrets.yaml and apply the changes by running the Helm upgrade command. Restart the web server deployment. Refer to Restarting Deployments for instructions.

  • Proxy Server Credentials:

    Modify the proxy server secret in systemlink-secrets.yaml and apply the changes by running the Helm upgrade command. Restart the web server deployment. Refer to Restarting Deployments for instructions.

  • Whitelisted API Keys and Hashes:

    Modify the whitelisted API keys and hashes secret in systemlink-secrets.yaml and apply the changes by running the Helm upgrade command. Restart the service deployment associated with the rotated API key. Refer to Restarting Deployments for instructions.

    Refer to Required Secrets for information about API keys and their associated services.

  • Encryption Keys:
    Warning Encryption keys used to encrypt data at rest cannot be rotated without migrating or re-encrypting existing data. Changing these keys may result in permanent loss of access to encrypted data. Contact NI support before rotating encryption keys to determine which keys can be safely rotated.
    For encryption keys that can be safely rotated, modify the encryption keys secret in systemlink-secrets.yaml and apply changes using the Helm upgrade command. Restart the deployment associated with the specific encryption key. Refer to Restarting Deployments for instructions.
  • Dremio Credentials:

    Modify the Dremio credentials secret in systemlink-secrets.yaml and apply changes using the Helm upgrade command. This includes both Dremio authentication credentials and the Nessie bearer token. Restart the dataframeservice and dataframeservice-nessie deployments. Refer to Restarting Deployments for instructions.

    Reset Dremio after changing the credentials. Refer to Resetting Dremio for instructions.

  • Elasticsearch Credentials:
    Note This procedure applies only if Elasticsearch is deployed as part of your SystemLink Enterprise installation.

    Modify the Elasticsearch credentials in both elasticsearch-secrets.yaml and systemlink-secrets.yaml. Upgrade the Elasticsearch chart first to update the user credentials, then upgrade the main SystemLink chart.

    Restart the fileingestioncdc and fileingestion deployments. Refer to Restarting Deployments for instructions.

  • PostgreSQL Credentials:

    Modify the PostgreSQL credentials secret in systemlink-secrets.yaml and apply changes using the Helm upgrade command. Restart the testmonitorservice, dynamicformfields, and dashboardhost deployments. Refer to Restarting Deployments for instructions.

  • MongoDB Credentials:

    Modify the MongoDB credentials secret in systemlink-secrets.yaml and apply changes using the Helm upgrade command. Restart all deployments that use MongoDB. Refer to Restarting Deployments for instructions.

    Refer to Required Secrets for information about services that use MongoDB.

  • RabbitMQ Credentials:

    Modify the RabbitMQ credentials secret in systemlink-secrets.yaml. Before applying changes, delete the RabbitMQ stateful set to stop all RabbitMQ pods:

    kubectl delete statefulset <release>-rabbitmq -n <namespace>

    Verify all RabbitMQ pods have stopped:

    kubectl get pods -n <namespace> --selector=app.kubernetes.io/name=rabbitmq

    After you stop all pods, list and delete all RabbitMQ persistent volume claims:

    kubectl get pvc -n <namespace> --selector=app.kubernetes.io/name=rabbitmq
kubectl delete pvc -n <namespace> --selector=app.kubernetes.io/name=rabbitmq

    Apply changes using the Helm upgrade command. See Common Procedures to redeploy the RabbitMQ stateful set with the new credentials.

    Note The system typically deletes persistent volumes automatically when their claims are removed. If you do not configure the cluster for automatic deletion, you can manually delete orphaned persistent volumes to recover resources.
  • Redis Credentials:

    Modify the Redis credentials secret in systemlink-secrets.yaml. Delete the Redis stateful set:

    kubectl delete statefulset <release>-webserver-redis -n <namespace>

    List and delete all Redis persistent volume claims:

    kubectl get pvc -n <namespace> --selector=app.kubernetes.io/name=webserver-redis
kubectl delete pvc -n <namespace> --selector=app.kubernetes.io/name=webserver-redis

    Apply changes using the Helm upgrade command. Ssee Common Procedures to redeploy the Redis stateful set with the new credentials.

  • Object Storage Credentials:
    • Feeds, File Ingestion and Notebook Execution:

      Modify the feeds, file ingestion, and notebook execution secrets in systemlink-secrets.yaml. Apply changes using the Helm upgrade command. Restart the feedservice, file, nbexec-execution-service, and nbexec-notebook-service deployments. refer to Restarting Deployments for instructions.

      Note This procedure applies only if you store credentials as an access token on the cluster. Refer to Object Storage for details on credential storage options.
    • DFS and Dremio:

      Modify the DFS secrets and Dremio secrets in systemlink-secrets.yaml. Apply changes using the Helm upgrade command. Reset Dremio. Refer to Resetting Dremio for more details.

      Note If not using the SystemLink Enterprise secret template, additionally modify the Dremio distributed storage configuration. Refer to aws-secrets.yaml or azure-secrets.yaml for the storage configuration.
  • SMTP Credentials:

    Modify the SMTP credentials secret in systemlink-secrets.yaml and apply changes using the Helm upgrade command. Restart the SMTP deployment. Refer to Restarting Deployments for instructions.