Process Isolation on the NI Linux Real-Time OS

Commonly required in time-sensitive, real-time systems, process isolation is the segregation of software processes or threads to protect them from interrupting each others' execution. Implementation of process isolation can range from limiting memory usage of certain threads to specifying which CPU cores a thread is able to run on. By restricting additional thread execution parameters, the overall system becomes more secure and reliable.

In its simplest state, thread isolation consists of ensuring that thread "A" does not use the memory space allocated for thread "B" through implementing virtual address spaces. Different software packages provide detailed forms of thread isolation by restricting the following parameters based on the thread ID (PID):

• Memory usage
• CPU usage
• I/O Access
• CPU affinity
• Network access
• Device access
• Thread suspension
• Memory and CPU accounting

One of the main APIs that NI Linux Real-Time uses to isolate threads is known as cgroups or control groups. These groups form a Linux kernel API that a developer can use to isolate tasks from each other with several different parameters. Cgroups is one of the many APIs Linux users can take advantage of to accomplish process isolation. Some others include Linux Containers (LXC), chroot, and systemd. Each cgroup, or control group, is a group of processes and threads that is assigned specific execution limitations such as maximum group CPU usage, memory usage, network usage, and so on. The cgroups API works by assigning subsystems to each cgroup that give different parameters to a group of threads. The Linux OS subsystems are:

• cpuset: Sets which core of the CPU can be used for executing the cgroup's tasks.
• memory: Limits the memory usage of the tasks in each cgroup and accounts for the memory usage of each cgroup.
• blkio: Implements the block I/O controller that controls the respective cgroup's communication to and from peripheral devices.
• cpuacct: Groups tasks using cgroups and accounts for their collective CPU usage.
• devices: Allows or denies access to devices by tasks in a cgroup.
• net_cls: Tags network packets so the Linux traffic controller can identify which packets came from a particular cgroup task.
• net_prio: Assigns the cgroup tasks' network traffic priority on each network interface.
• freezer: Freezes and unfreezes the tasks in a cgroup.

By combining these subsystems, the user can give the threads in each cgroup very specific operating limits. Use cases for cgroups include thread isolation, execution logging, safety protocol, and the overall reliable operation of complex systems.

Currently, NI Linux Real-Time takes advantage of only two of the eight subsystems available: cpuset and cpuacct. With ssh, users can locate these hierarchies under the /dev/cgroup/ directory in Linux RT.

cpuset

The cpuset hierarchy contains four control groups:

• Root (.): Holds tasks that cannot be assigned to the system and LabVIEW Timed Loop control groups. It is the top-level control group.
• System (system_set): Contains all tasks not related to LabVIEW timed structures or Scan Engine.
• LabVIEW Timed Loops (LabVIEW_tl_set): Contains all of the tasks corresponding to LabVIEW timed structures (Timed Loops and timed sequences).
• LabVIEW Scan Engine (LabVIEW_ScanEngine_set): Contains Scan Engine threads such as the IO thread.

For each of these control groups, only a few files should concern the user. The tasks file is the most important for any control group because it lists which threads are executing in the associated control group. The cpus file lists the cpus that the threads in the associated cgroup have access to execute on. The other files under each cpuset control group usually do not require inspection.

cpuacct

The cpuacct hierarchy contains five control groups, the last of which is temporary:

• Root (.): Serves as the top-level control group.
• LabVIEW_Timed-Structures_group: Contains tasks related to the Timed Loop structures generated from LabVIEW that do not have critical priority.
• LabVIEW_Time-Critical_group: Contains tasks related to the Timed Loop structures generated from LabVIEW that do have critical priority.
• ISR_group: Includes tasks related to interrupt service routines (ISRs).
• other_group: Features tasks in the process of being assigned to other groups.

Cpuacct control groups maintain the same tasks file that lists which tasks are associated with the control group. Each group also contains cpuacct.usage_percpu, cpuacct.usage, and cpuacct.stat files that record CPU usage information for each control group.

LabVIEW Real-Time SMP Utility VIs

Most users interact with cgroups using LabVIEW Real-Time symmetric multiprocessing (SMP) utility VIs. These VIs assign which CPUs can participate in automatic load balancing and indicate the distribution of load across the CPUs of the real-time device. Find more information on these VIs in the LabVIEW Real-Time help documentation:

LabVIEW Real-Time SMP Utility VI Help

These VIs directly affect the control groups in the cpuset subsystem. Specifically, the RT Set CPU Pool and the RT Set CPU Pool Assignments VIs affect the cpus file of the specified control group. If a cpu is available for automatic load balancing, the cgroups include that specific cpu core in their cpus files.

Note: The LabVIEW Scan Engine cgroup cannot be controlled with these VIs.

Assigning Threads to Existing Control Groups

To assign tasks to currently running control groups, a developer can use the echo command to write the thread ID to the tasks file of the associated cgroup:

# /bin/echo PID > tasks

Note that the /bin/echo should be used instead of the bash echo command because the /bin/echo command provides errors for every write() command unlike bash echo. Also, only one thread can be attached to a control group with each command.

Mounting Additional Subsystems

More often than not, a developer wants to use a subsystem other than the two provided by default on Linux Real-Time, cpuset, and cpuacct. In this case, the developer needs to make some modifications to the Linux kernel to enable the desired subsystems for mounting.

Recompiling the NI Linux Kernel

First, the user needs to download the NI Linux kernel from the GitHub repository

NI Linux Kernel GitHub© Repository

After downloading the repository, the user can edit the .config file with a kernel configuration tool such as menuconfig.

After editing the .config file to include cgroups, the user should be able to proceed with the compilation of the Linux kernel as listed in the guide.

Mounting Enabled Subsystems

After the subsystems have been enabled in the .config file and the kernel has been recompiled, mounting and using the subsytems should follow the standard cgroups functionality published by Kernel.org:

Kernel.org cgroups Documentation

The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis.