Discussion: Real Time Cores (Preempt-RT, RTAI, Xenomai and others)

[hdiethelm]

After the long discussions in #3873 and #3875, I investigated the real time options a bit more in depth and created a summary.

LinuxCNC has quite a number of available real time cores. Some of them are broken, others work fine. What I found so far:

All userspace variants are configured by --with-realtime=uspace and built if the needed library is installed. During runtime, a variant is detected using the kernel name, version and other hints and selected automatically or also not if your kernel name doesn't match expectations.

Format: {Text shown at start of LinuxCNC}:{My short description}

• User Space: --with-realtime=uspace
  • POSIX non-realtime: Any old kernel or you forgot make setuid
    • Works, but big latency
    • Basically what starts when you don't have any other realtime or it is not properly detected
  • POSIX realtime: PREEMPT_RT
    • Works
    • Probably the most used option
    • Debian stock RT kernel works fine
  • XENOMAI (posix-skin) realtime: Xenomai2: Might be / Xenomai3: Works / Xenomai4: Not compatible
    • Works: I managed to get it running with Xenomai3
    • Probably seldom used due to no up to date compiled kernel package is available
  • LXRT realtime: RTAI Userspace
    • Seams to be broken after 2.8.4. I had no success, 2.8.4 is buster where I don't have a VM. In Bookwork/Trixie I was not able to compile 2.8.4. 2.9.* and master builds but has an error starting.
• Kernel Space: --with-realtime=/usr/realtime-$VERSION
  • Nothing: RTAI Kernelspace
    • Works in 2.9
    • Broken in Master: Ruckig broke RTAI #3875
    • @NTULINUX maintains a kernel which is available in the LinuxCNC Debian repo: https://github.com/NTULINUX/RTAI

RTAI is not maintained any more, last commit ~4 years ago.

To help others wanting to try Xenomai3, I created a repo with all you need to get it running:
https://github.com/hdiethelm/xenomai3-linuxcnc

Xenomai4 has a new API: EVL. I did not found a posix skin. I managed to get the kernel and tools built for anybody interested in creating a Xenomai4 module for LinuxCNC:
https://github.com/hdiethelm/xenomai4-linuxcnc

If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.

BTW: Is it fine starting this discussion here or should I move it to the forum?

[hdiethelm]

Benchmarks so far on my CNC PC:
Hardware:

• H110M-A Mainboard
• i5-6500
• AMD Radeon RX 460 2GB (IGPU or Nvidia was crap, so I bought it second hand, way better)

Kernel command line: isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1

Results with latency-histogram, load is 2x glx gears and watching video on youtube with sound:

• PREEMPT_RT: Idle: 1.9us / Load: 42us
• Xenomai3: Idle: 1.1us / Load: 21us

[rene-dev]

In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.

[NTULINUX]

If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.

Sweet! Good luck!

[hdiethelm]

I had to try it anyway... ;-) It doesn't look that bad. configure, automake and that WithRoot() was the biggest issue so far. Now I have proper EVL threads:

linuxcnc:

Note: Using XENOMAI4 EVL realtime
linuxcnc-thread:88560
linuxcnc-thread:88561

evl ps -l

CPU   PID   SCHED   PRIO  ISW     CTXSW     SYS       RWA       STAT     TIMEOUT      %CPU      CPUTIME       WCHAN                 NAME
 15   88560  fifo    98   1       1         1         0          X          -           0.0      0:000.003    (null)                linuxcnc-thread:88560
 15   88561  fifo    97   1       1         1         0          X          -           0.0      0:000.001    (null)                linuxcnc-thread:88561

The harder part is finding anything which uses functions not suited for real time and port that to EVL. And I wont think about HW drivers so far... ;-)

[andypugh]

In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.

RTAI and Ethernet is probably not impossible, it just wasn't implemented.
Xenomai? Who knows.

LXRT RTAI should work with Ethernet too, and does have slightly better latency on my test system than preempt-rt.

(But my test system is a freak, less than 1k latency on preempt-rt and on RTAI so who knows what the typical LXRT/RTAI latency is?)

[hdiethelm]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master? For me it fails. Any tricks?

In Xenomai3, network runs out of the box, however, probably not as good as it could. My machine has a Ethernet Mesa card and just works with 2.9.8 / Xenomai3 kernel. However, I did not yet do extensive testing.

Xenomai3 and Xenomai4 are different and bot a bit special concerning syscalls / device drivers / network.
Both: As soon as you call a non-Xenomai part, it switches from the Xenomai to the normal kernel, does the work and switches back. Due to you can have a PREEMT_RT kernel with Xenomai, this should still be fast but not optimal. There is a way of tracing this, I have to look into it.

Xenomai3: It has a set of ported network drivers: https://v3.xenomai.org/rtnet/config/index.html
Xenomai4: A driver has to be "enabled" to work, what ever this is: https://v4.xenomai.org/core/net/index.html#evl-net-port

This presentation explains it: https://www.youtube.com/watch?v=WXKsLTfrOiE 23:40 for Xenomai3 / 27:33 for Xenomai4

[hdiethelm]

Xenomai3 network drivers: https://gitlab.com/Xenomai/xenomai3/xenomai/-/tree/master/kernel/drivers/net/drivers?ref_type=heads
Xenomai4: https://v4.xenomai.org/core/drivers/net/index.html :-D Looking in Git and grep for NET_OOB probably only Intel e1000e and 82575/82576: https://gitlab.com/Xenomai/xenomai4/linux-evl/-/blob/v6.12.y-cip-evl-rebase/drivers/net/ethernet/intel/Kconfig?ref_type=heads

It seams that Xenomai4 is still under heavy construction.

[andypugh]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master?

No, I haven't tried LXRT RTAI since 2022.

However LXRT RTAI has run at least once on 2.9:
https://sourceforge.net/p/emc/mailman/message/37718519/
(note the link to the .png)

[hdiethelm]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master?

No, I haven't tried LXRT RTAI since 2022.

However LXRT RTAI has run at least once on 2.9: https://sourceforge.net/p/emc/mailman/message/37718519/ (note the link to the .png)

I found it: #3879 Seams no one used this for a pretty long time...