README.ISOLCPUS

# Copyright (C) 2005-2017 The RTAI project
# This [file] is free software; the RTAI project
# gives unlimited permission to copy and/or distribute it,
# with or without modifications, as long as this notice is preserved.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.

                      *** EXPLOITING CPUs ISOLATION ***

Written by:
Bernhard Pfund <bernhard@chapter7.ch>
and
Paolo Mantegazza <mantegazza@aero.polimi.it>

RTAI can take advantage of the possibility Linux affords to isolate CPUs
from any of its scheduling activity on multi processor (MP) machines.

Contents:
=========
 1. Isolcpus
 2. IsolCpuMask in RTAI
 3. Cpusets & CPU hotplug
 4. Example init script


1. Isolcpus
-----------

There should be no better explanation of what it is than the following
excerpts from Linux 'Documentation/kernel-parameters.txt':

isolcpus=       [KNL,SMP] Isolate CPUs from the general scheduler.
                Format:
                <cpu number>,...,<cpu number>
                or
                <cpu number>-<cpu number>
                (must be a positive range in ascending order) or a mixture
                <cpu number>,...,<cpu number>-<cpu number>
                This option can be used to specify one or more CPUs
                to isolate from the general SMP balancing and scheduling
                algorithms. The only way to move a process onto or off
                an "isolated" CPU is via the CPU affinity syscalls.
                <cpu number> begins at 0 and the maximum value is
                "number of CPUs in system - 1".

                This option is the preferred way to isolate CPUs. The
                alternative -- manually setting the CPU mask of all
                tasks in the system -- can cause problems and
                suboptimal load balancer performance.

Add the following to the above, from the very same source:
acpi_irq_nobalance [HW,ACPI]
		  ACPI will not move active IRQs (default)
		  default in PIC mode

Irq balancing can be disabled directly at kernel configuration and has little
effect on what will follow. Nonetheless you should prefer to be sure that Linux
does not manipulate hardware related stuff on its own. So if you forgot to
disable irq balancing when you made your kernel there is no need to remake it,
just set it at boot time.

From what above it should be easy to infer that by using "isolcpus" you can
be sure that Linux will have none of its tasks running on the isolated cpus.
That is not entirely true since a few kernel threads and kworkers will have 
a copy of each of them repeated and assigned to all of the available CPUs.
It is assumed that they will have very little to do if no interrupt arrives 
on the isolated CPUs. Thus if RTAI is able to care avoiding any hard interrupt 
on the isolated CPUs they will be fairly well isolated from Linux. 

Nevertheless, to further help isolation, RTAI removes any kernel thread and 
kworker at the insmoding of its HAL. Even if such an action does not impede
their further dynamic creations there is some evidence that it is useful
anyhow. It would also be possible to block such dynamic creations, an action
that is postponed till more experience is gathered with the use of what above.
To easily check if any following dynamic insertion of something within the
isolated cpus has happened the following might be of help:
cat /proc/*/task/*/status |grep "allowed:" |grep <mask>,
where mask is the mask of what you want either to see not or see. 

With what above only Linux rescheduling interrupts should appear on the
isolated CPUs. They are not difficult to block too, but it does not seem
that much more has to be gained by doing so. Therefore, as for what above,
it is momentarily set aside as a possible future TODO action.

Finally, combining all of the above with the possibility of forcing RTAI 
enabled Linux process/thread/kthreads to stay on the isolated CPUs the 
latter will find themselves processing just RTAI real time stuff, with 
a significant reduction of latencies/jitters. In the case of many RTAI 
tasks running on the isolated CPUs all issues producing latency/jitter 
will still have an effect, but it will be reduced to the least possible, 
without any added bus/pipe/cache interference from Linux.

It should be noticed however that there will remain a single interrupt still
managed by Linux, i.e. the one to the local APIC timer. Such an interrupt
will not come from the hardware but from a software inter processor interrupt
(IPI) generated by RTAI to keep Linux happy. It is likely that such an IPI
could be sent only to non isolated CPUs but no testing of such a solution
has been done up to now. The possible change requires just the substitution
of a single line of code but I'm afraid it could damage Linux somehow.


2. IsolCpuMask in RTAI
----------------------

Then what you have to do on the RTAI side is to load the core RTAI HAL module
using something like: "insmod rtai_hal.ko IsolCpusMask=<xxx>", where <xxx>
is the mask of isolated CPUs. Please notice that Linux uses a list if isolated
CPUs while RTAI requires the corresponding mask (I'm lazy and let you do it).

Suppose you have a quadcore system with cores #0 - #3 and want everything but
core #0 to be used by RTAI. In that case, just like in the cpuset example below,
your IsolCpusMask would be 0xE (1110).

What will happen next is that RTAI will divert all of Linux interrupts away
from the isolated CPUs, keeping those requested for RTAI (rt_request_irq)
on the isolated CPUs, without you having to care for it. In such a way there
will be no Linux activity on the isolated CPUs whatsoever and they will remain
within complete RTAI ownership.

Finally it is up to you to exploit such a feature by assigning all of your
tasks to the isolated CPUs, according to your needs, by using:
"rt_task_init_cpuid", "rt_thread_init_cpuid", "rt_task_init_schmod".

Notice that you have also the possibility of further specialising your CPUs
isolation scheme by diverting any real time interrupt you use to a single
specific CPU, or CPU cluster within the isolated CPUs, by using the RTAI
function: rt_assign_irq_to_cpu.

Naturally you can set RTAI "IsolCpusMask" even without setting the Linux
"isolcpus" list, still with some beneficial effects though, for sure, they
will not be as good as with the complete isolation setting described
previously.

Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation mask from
the kernel. So there is nomore the need to set IsolCpuMask at the insmoding of 
rtai_hal.ko. Nonetheless if IsolCpuMask is not null it will supersede the one
assigned by Linux at boot time.

3. Cpusets & CPU hotplug
------------------------

If you want to make sure nothing unwanted is _ever_ scheduled on a specific
CPU or core the hotplug system is your friend. (Logically) offlined CPUs are
removed from the Linux scheduler and thus no longer get tasks assigned. Online
again and assigned to a cpuset, RTAI tasks can be scheduled on these CPUs. See
'Documentation/cpusets.txt' and 'Documentation/cgroup.txt'.

Consequently you should inform RTAI about the allowed CPUs (1-3 in the example)
and initialise your tasks accordingly (see section 2).

The example below shows a quadcore CPU partitioned into three sections.

+-Quadcore CPU---------------------+
|                                  |
|                  +-Production-+  |
|    +--------+    | +--------+ |  |
|    |        |    | |        | |  |
|    | Core 0 |    | | Core 2 | |  |
|    |        |    | |        | |  |
|    +--------+    | +--------+ |  |
|                  |            |  |
|                  |            |  |
|  +-RTnet------+  |            |  |
|  | +--------+ |  | +--------+ |  |
|  | |        | |  | |        | |  |
|  | | Core 1 | |  | | Core 3 | |  |
|  | |        | |  | |        | |  |
|  | +--------+ |  | +--------+ |  |
|  +------------+  +------------+  |
+----------------------------------+

 - The root set where only core #0 remains. The kernel and all non real-time
   tasks run on that single core.

 - The 'RTnet' cpuset on core #1 where all RTDM related tasks are scheduled on.

 - The 'Production' cpuset where the real-time controller tasks are executed.
   This cpuset has load-balancing enabled, hence tasks assigned to that set are
   scheduled on either core #3 or #4 and can be moved within the scheduler domain
   if necessary.

Memory isolation is not used in this example and no overlapping sets exist, it's
even actively prevented by using the cpu_exclusive attribute.

By appropriately exploiting both isolation methods illustrated above it is mostly
possible to reduce latencies to single microsec digit figures, especially if the
hard timer periodic mode is adopted.


4. Example init script
----------------------

############# START OF SCRIPT #############

# Environment
MKDIR=/bin/mkdir
MOUNT=/bin/mount
ECHO=/bin/echo
CPUSET=/dev/cpuset
DEVICES=/sys/devices/system/cpu
SLEEP=/bin/sleep
NAME=cpuset

reset_cpus()
{
         # Shutdown CPU cores 1 - 3
         $ECHO 0 > $DEVICES/cpu1/online
         $ECHO 0 > $DEVICES/cpu2/online
         $ECHO 0 > $DEVICES/cpu3/online

         # Noop
         $SLEEP 1

         # Enable CPU cores 1 - 3
         $ECHO 1 > $DEVICES/cpu1/online
         $ECHO 1 > $DEVICES/cpu2/online
         $ECHO 1 > $DEVICES/cpu3/online
}

mount_cpuset()
{
         $MKDIR $CPUSET

         # Configure the root set
         if [ -e $CPUSET ]; then
                 $MOUNT -t cgroup -ocpuset cpuset $CPUSET
                 # Disable load balancing in the root set
                 $ECHO 0 > $CPUSET/cpuset.sched_load_balance
         fi
}


create_rtnet_set()
{
         # Configure the RTnet cpuset
         if [ -e $CPUSET ]; then
                 $MKDIR $CPUSET/rtnet
                 $ECHO 1 > $CPUSET/rtnet/cpuset.cpus
                 $ECHO 1 > $CPUSET/rtnet/cpuset.cpu_exclusive

                 # Disable load balancing in the rtnet set
                 $ECHO 0 > $CPUSET/rtnet/cpuset.sched_load_balance
         fi
}

create_production_set()
{
         # Configure the production cpuset
         if [ -e $CPUSET ]; then
                 $MKDIR $CPUSET/production
                 $ECHO 2,3 > $CPUSET/production/cpuset.cpus
                 $ECHO 1 > $CPUSET/production/cpuset.cpu_exclusive

                 # Enable load balancing in the production set
                 $ECHO 1 > $CPUSET/production/cpuset.sched_load_balance
         fi
}

case "$1" in
   start)
         reset_cpus
         mount_cpuset
         create_rtnet_set
         create_production_set
   ;;
   stop)
         reset_cpus
   ;;
   *)
     echo "Usage: /etc/init.d/$NAME {start|stop}" >&2
     exit 2
   ;;
esac

exit 0
############# END OF SCRIPT #############