Notebook Execution Service Configuration

Set up the Notebook Execution Service to manage notebook execution environments, resource allocation, Argo Workflows orchestration, and lifecycle management execution.

Note For SystemLink Enterprise April 2024 and earlier, configure AWS S3 lifecycle parameters to clean up artifacts from Argo Workflows. NI recommends setting the lifecycle to a minimum of 2 days.

Execution Parallelism Configuration

Configure the maximum number of notebook executions that can run simultaneously.

Note By default, each execution requires a minimum of 0.2 CPU and 2,176 MiB of memory.
Table 65. Execution Parallelism Configuration
Parameter Details Required
parallelism The maximum number of notebook executions that SystemLink can run in parallel. When setting this value, you must consider cluster capacity. Yes
maxNumberOfWorkflowsToSchedule The maximum number of workflows that SystemLink can schedule for execution. Yes
Note For 150 parallel executions, the SystemLink Kubernetes cluster requires approximately 30 vCPU and 319 GiB of memory. For shorter executions, consider using fewer parallel executions to maintain a smaller cluster size.

Resource Profiles Configuration

Configure resource profiles to maximize the use of system resources.

Note This configuration is available in SystemLink Enterprise October 2024 and later.
Table 66. Resource Profiles Configuration
Profile Parameter Path Default CPU Request Default Memory Request Default Memory Limit
Low resourceProfiles.low.requests/limits 0.1 4Gi 4Gi
Medium resourceProfiles.medium.requests/limits Configurable Configurable Configurable
High resourceProfiles.high.requests/limits Configurable Configurable Configurable

Execution Persistence Configuration

Configure how long notebook executions persist in the database.

Table 67. Execution Persistence Configuration
Parameter Details Recommendation
daysToPersistExecutions The number of days an execution persists in the database. This parameter is based on the started_at value. To avoid deleting executions before completion, set this parameter to a minimum of 2 days.

Node Selection Configuration

Limit notebook execution pod scheduling to specific nodes using node selectors and tolerations.

Table 68. Node Selection Configuration
Configuration Type Parameter/Command Details
Node Selector notebookExecutionNodeSelector Adjust in node-selectors.yaml to target specific nodes.
Tolerations notebookExecutionTolerations Adjust in node-selectors.yaml to tolerate node taints.
Node Taint kubectl taint nodes <node> notebook_execution=true:NoSchedule Prevents other pods from scheduling on an execution node.
Node Label kubectl label nodes <node> notebook.executor=true Identifies the nodes for notebook execution placement.

Argo Workflows Integration

Configure Argo Workflows integration when an existing Argo Workflows deployment exists on your cluster.

Table 69. systemlink-values.yaml Configuration
Parameter Details Required
argoworkflows.argo-workflows.controller.instanceID.enabled Set to true to enable instance ID configuration. Yes
argoworkflows.argo-workflows.controller.instanceID.explicitID Set to sl-notebook-execution-0 for SystemLink integration. Yes
Note SystemLink only requires the previous parameters when an existing Argo Workflows deployment is present.
Table 70. systemlink-admin-values.yaml Configuration
Parameter Details Required
argoworkflowscrds.crds.install Set to false to disable installation of Argo Workflows Custom Resource Definitions when using existing deployment. Yes
Note SystemLink only requires the previous parameter when an existing Argo Workflows deployment is present.

Viewing Notebook Workflow Details

Connect to the Argo server to view notebook workflow details.
Note To view the details of a notebook workflow, you must be an admin.

  1. In the Command prompt, run the following command to set port forwarding to the service.

    kubectl port-forward services/<helm-release-name>-argo-workflows-server 2746:2746 -n <namespace>

    In this command, substitute the following variables.

    • <helm-release-name> is the release of the SystemLink that contains the chart.
    • <namespace> is the namespace of the new SystemLink chart.

    After running this command, you can access the server by connecting to localhost:2746.

  2. Run the following command to get the name of the service account token.

    SECRET=$(kubectl get sa helm-release-name-argo-workflows-server -o=jsonpath='{.secrets[0].name}' -n namespace)

  3. Run the following command to retrieve the token value.

    echo $ARGO_TOKEN="Bearer $(kubectl get secret $SECRET -o=jsonpath='{.data.token}' -n namespace | base64 --decode)"
  4. Navigate to the Argo server.
  5. Under Client Authentication, enter the token value.