Version 19 (modified by nettings, 14 years ago)
--

IRQ Priorities How-To

FIXME: find out which other tasklets should receive high priority, investigate consequences of demoting non-essential tasklets to SCHED_OTHER, verify that rtirq greps correctly for numbered handlers if the number is omitted (rtcX, usbX:...

FIXME: extract ffado-specific settings and feed the rest upstream into the RT Wiki at kernel.org

FIXME: sampo suggests defining tasklet first. will do. jörn

Even with an rt-patched kernel, setting the scheduling priority of the IRQ handler on its own is not enough to ensure proper operation. Since most of the actual work that has to be done to respond to an IRQ is done in a tasklet, one should also ensure that the priority of the softirq-tasklet processes is high enough. Otherwise tasklets are not executed in time. This has been experimentally verified.

All softirq-tasklets run SCHED_FIFO with a priority of 50 by default. To ensure glitch-free audio, it is important that those tasklets responsible for audio operation have a higher priority than the others.


Manual priority setting

You can adjust IRQ tasklet priorities manually (which is a good exercise to understand what is going on), and to be able to fix things quickly if your hardware changes during a session.

Here's a typical laptop example. To work out the hierarchy of priorities, try to follow the flow of the data in your mind, from the bottom up.

NOTE: This walkthrough assumes kernels <= 2.6.31-rt with per-device interrupt tasklets (as opposed to per-IRQ as before). These allow more fine-grained tuning on shared interrupt lines, as is common and often unavoidable on notebooks.

1. Look up the relevant interrupts

To find out which IRQ lines handle which hardware, just ask the kernel:

nettings@hoppetosse:/local/ffado-svn/libffado> cat /proc/interrupts
           CPU0
  0:      89132   IO-APIC-edge      timer
  1:       1538   IO-APIC-edge      i8042
  8:          1   IO-APIC-edge      rtc0

The realtime clock (on IRQ 8) is used to generate "alarms" (timed interrupts) and should have the highest priority. We are using 90 here.

FIXME: Does ffado make use of alarm timers? If so, add a brief explanation.

  9:        274   IO-APIC-fasteoi   acpi
 12:      28208   IO-APIC-edge      i8042
 14:      20841   IO-APIC-edge      ata_piix
 15:          0   IO-APIC-edge      ata_piix
 16:      27747   IO-APIC-fasteoi   uhci_hcd:usb2, yenta, ohci1394, i915@pci:0000:00:02.0, eth0

Here's a tricky one. This example laptop uses a ffado device via a pc-card IEEE1934 adaptor. The module responsible for the cardbus slot is called "yenta" on IRQ 16. It must have the next highest priority, because if it's not handled in time, the actual firewire chip on it can't do anything. Let's make it 88.

Next thing is the firewire module itself, called ohci1394. As you can see, it's sharing an interrupt with the yenta module (naturally). We will set its priority to 86.

You can also see that a USB hub sits on the same interrupt. So it's a bad idea to use the corresponding USB port for your external audio disk. Moreover, the wired network interface is sharing this busy interrupt line, too. Hence, if possible, bring down or at least don't use the network interface during important recording sessions on this particular machine.

 17:          0   IO-APIC-fasteoi   uhci_hcd:usb3, mmc0
 18:          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:       6501   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5

Here's a nice USB 2.0 hub on a little-used interrupt. An ideal place for our external recording disk. Let's up this to 84.

 21:         32   IO-APIC-fasteoi   ipw2200
...

The rest of the interrupts are not related to our audio chain.

2. Find out which processes handle the IRQs

To get a list of all the kernel IRQ tasklets, you can (as root) do

hoppetosse:~ # ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
    3  FF     49  89   - [sirq-high/0]
    4  FF     49  89   - [sirq-timer/0]
    5  FF     49  89   - [sirq-net-tx/0]
    6  FF     49  89   - [sirq-net-rx/0]
    7  FF     49  89   - [sirq-block/0]
    8  FF     49  89   - [sirq-tasklet/0]
    9  FF     49  89   - [sirq-sched/0]
   10  FF     49  89   - [sirq-hrtimer/0]
   11  FF     49  89   - [sirq-rcu/0]
   24  FF     50  90   - [irq/9-acpi]
   33  FF     50  90   - [irq/12-i8042]
   34  FF     50  90   - [irq/1-i8042]
   68  FF     50  90   - [irq/14-ata_piix]
   69  FF     50  90   - [irq/15-ata_piix]
  293  FF     50  90   - [irq/19-ehci_hcd]

This is the USB 2.0 tasklet for our audio disk. Its process ID is 293.

  429  FF     50  90   - [irq/16-uhci_hcd]
  436  FF     50  90   - [irq/17-uhci_hcd]
  441  FF     50  90   - [irq/18-uhci_hcd]
  442  FF     50  90   - [irq/19-uhci_hcd]
 1114  FF     50  90   - [irq/8-rtc0]

The real-time clock tasklet is running as PID 1114.

 1117  FF     50  90   - [irq/16-yenta]

Our pc-card module has the process ID 1117.

 1147  FF     50  90   - [irq/21-ipw2200]
 1169  FF     50  90   - [irq/17-mmc0]
 1220  FF     50  90   - [irq/23-Intel IC]
 1225  FF     50  90   - [irq/16-ohci1394]

The firewire card handler has PID 1225. Again, the last few tasklets are not relevant for low latency audio.

3. Set the tasklet priorities

To tune realtime scheduling classes and priorities, you can use the chrt command which is part of the schedtools package. Obviously, it requires root privileges. Now we use the PIDs we have determined in the previous step.

hoppetosse:~ # chrt -f -p 90 1114
hoppetosse:~ # chrt -f -p 88 1117
hoppetosse:~ # chrt -f -p 86 1225
hoppetosse:~ # chrt -f -p 84 293

Just like any well-behaved UNIX command, chrt remains silent when it has accomplished its job.

To verify that all is well, take a look at the IRQ tasklets again:

hoppetosse:~ # ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
...
  293  FF     84 126   - [irq/19-ehci_hcd]
...
 1114  FF     90 130   - [irq/8-rtc0]
 1117  FF     88 128   - [irq/16-yenta]
...
 1225  FF     86 126   - [irq/16-ohci1394]
...

4. (optional) Find a good slot for your external USB disk

Plug your USB drive into your machine. Then use the lsusb command to find out which bus it's sitting on. The number of the bus should correspond to the number of the "good" USB port as listed in /proc/interrupts. Swap USB ports until you've hit it. (But don't be frustrated if you don't - some mainboards provide "optional" USB ports that are not connected to any physical socket on the outside and can't be used easily.)

nettings@hoppetosse:/usr/local/lib> lsusb
...
Bus 001 Device 007: ID 0fce:d075 Your pluggable USB Hard disk
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
...

Automate priorities setting with rtirq

The fun thing about IRQ tasklets is:

  • thanks to APIC, interrupt numbers can (and will change) between reboots
  • PIDs of tasklets will change between reboots and on hardware hotplug events
  • USB bus numbers will change between reboots

That means there is no simple way to script those adjustments you've just made. rtirq to the rescue. It's a straightforward bash utility written by Rui Nuno Capela that employs some grep and awk magic to automate what you just did by hand.

(The USB port you found above will not change, though.)

Here's an example configuration (/etc/sysconfig/rtirq) that corresponds to the manual setting described earlier:

NOTE: Current low latency kernels enable per-device handlers (as opposed to per-interrupt as before). That means if two devices are sharing an interrupt (very common on notebooks), you can apply more fine-grained tweaks. rtirq has been updated to deal with this, so be sure to get the very latest version (scroll down a bit on that page). Still, you may encounter problems. Holler on the lists if you do.

# IRQ thread service names
# (space separated list, from higher to lower priority).
RTIRQ_NAME_LIST="rtc yenta ohci1394 ehci_hcd:usb"

# Highest priority.
RTIRQ_PRIO_HIGH=90

# Priority decrease step.
RTIRQ_PRIO_DECR=2

# Whether to reset all IRQ threads to SCHED_OTHER.
RTIRQ_RESET_ALL=0

# On kernel configurations that support it,
# which services should be NOT threaded
# (space separated list).
RTIRQ_NON_THREADED="rtc"

# Process names which will be forced to the
# highest realtime priority range (99-91)
# (space separated list, from highest to lower priority).
# RTIRQ_HIGH_LIST="timer"

And here is the result:

hoppetosse:~ # /etc/init.d/rtirq.sh start
Setting IRQ priorities: start [rtc] irq=8 pid=1128 prio=90: OK.
Setting IRQ priorities: start [yenta] irq=16 pid=1124 prio=88: OK.
Setting IRQ priorities: start [ohci1394] irq=16 pid=1240 prio=86: OK.
Setting IRQ priorities: start [ehci_hcd:usb] irq=19 pid=426 prio=84: OK.

To add this script to the current runlevel, do (as root)

hoppetosse:~ # insserv rtirq.sh