IRQ Priorities How-To

Some concepts explained

IRQs (short for "interrupt requests") allow peripheral devices to get the immediate attention of the CPU and the kernel that runs on it. Some IRQ lines correspond to actual wires on your mainboard, some are implemented in software.

An interrupt handler is a piece of kernel code that deals with interrupts. For efficiency reasons, interrupt handlers in Linux are divided into top halves, which acknowledge that an IRQ was received (so that the device that issued it is at peace), and bottom halves, where the actual work gets done. Bottom halves can be delayed if the kernel has more pressing things to do.

The mechanism that provides this delayed execution is called the tasklet API. Hence, bottom halves of interrupt handlers are frequently called "tasklets".

SCHED_FIFO is a scheduling class of real-time enabled kernels. SCHED_FIFO processes can use all available CPU until another SCHED_FIFO process with higher priority needs it, and they take precedence over all other processes on a system. Consequently, a rogue SCHED_FIFO process has the potential to lock up the system.

On a real-time kernel (one that includes the RT patches and has the CONFIG_PREEMPT_RT option enabled), all interrupt handler top halves run SCHED_FIFO, with a priority of 50 by default. To ensure glitch-free audio, it is important that those responsible for audio operation have a higher priority than the others. Moreover, the jackd realtime threads should run at a priority higher than the standard IRQs but lower than the audio-related ones (use the -P option to set it accordingly, or the "Priority" value in qjackctl).


Manual priority setting

You can adjust interrupt priorities manually (which is a good exercise to understand what is going on), and to be able to fix things quickly if your hardware changes during a session.

Here's a typical laptop example. To work out the hierarchy of priorities, try to follow the flow of the data in your mind, from the bottom up.

NOTE: This walkthrough assumes kernels >= 2.6.31-rt with per-device interrupt handlers (as opposed to per-IRQ as before). These allow more fine-grained tuning on shared interrupt lines, as is common and often unavoidable on notebooks.

1. Look up the relevant interrupts

To find out which IRQ lines handle which hardware, just ask the kernel:

nettings@hoppetosse:/local/ffado-svn/libffado> cat /proc/interrupts
           CPU0
  0:      89132   IO-APIC-edge      timer
  1:       1538   IO-APIC-edge      i8042
  8:          1   IO-APIC-edge      rtc0

The realtime clock (on IRQ 8) is used to generate "alarms" (timed interrupts) and should have the highest priority. We are using 90 here.

  9:        274   IO-APIC-fasteoi   acpi
 12:      28208   IO-APIC-edge      i8042
 14:      20841   IO-APIC-edge      ata_piix
 15:          0   IO-APIC-edge      ata_piix
 16:      27747   IO-APIC-fasteoi   uhci_hcd:usb2, yenta, ohci1394, i915@pci:0000:00:02.0, eth0

Here's a tricky one. This example laptop uses a ffado device via a pc-card IEEE1934 adaptor. The module responsible for the cardbus slot is called "yenta" on IRQ 16. It must have the next highest priority, because if it's not handled in time, the actual firewire chip on it can't do anything. Let's make it 88.

Next thing is the firewire module itself, called ohci1394. As you can see, it's sharing an interrupt with the yenta module (naturally). We will set its priority to 86.

You can also see that a USB hub sits on the same interrupt. So it's a bad idea to use the corresponding USB port for your external audio disk. Moreover, the wired network interface is sharing this busy interrupt line, too. Hence, if possible, bring down or at least don't use the network interface during important recording sessions on this particular machine.

 17:          0   IO-APIC-fasteoi   uhci_hcd:usb3, mmc0
 18:          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:       6501   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5

Here's a nice USB 2.0 hub on a little-used interrupt. An ideal place for our external recording disk. Let's up this to 84.

 21:         32   IO-APIC-fasteoi   ipw2200
...

The rest of the interrupts are not related to our audio chain.

2. Find out which processes handle the IRQs

To get a list of all the kernel threads that deal with IRQs, you can (as root) do

hoppetosse:~ # ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
    3  FF     49  89   - [sirq-high/0]
    4  FF     49  89   - [sirq-timer/0]
    5  FF     49  89   - [sirq-net-tx/0]
    6  FF     49  89   - [sirq-net-rx/0]
    7  FF     49  89   - [sirq-block/0]
    8  FF     49  89   - [sirq-tasklet/0]

This is the tasklet thread where all "bottom halves" of all interrupt handlers end up. Its PID is 8.

Even with an rt-patched kernel, setting the scheduling priority of the IRQ handler on its own is not enough to ensure proper operation. Since most of the actual work that has to be done to respond to an IRQ is done in a tasklet, one should also ensure that the priority of the softirq-tasklet processes is high enough. Otherwise tasklets are not executed in time. This has been experimentally verified. We will set the priority of this thread higher than any "normal" IRQ handlers, but lower than any of our audio-related ones.

This example laptop is a single-core system. On multiprocessor systems, you will see several of these tasklet threads, one per CPU. You should given each of them the same, higher, priority.

    9  FF     49  89   - [sirq-sched/0]
   10  FF     49  89   - [sirq-hrtimer/0]
   11  FF     49  89   - [sirq-rcu/0]
   24  FF     50  90   - [irq/9-acpi]
   33  FF     50  90   - [irq/12-i8042]
   34  FF     50  90   - [irq/1-i8042]
   68  FF     50  90   - [irq/14-ata_piix]
   69  FF     50  90   - [irq/15-ata_piix]
  293  FF     50  90   - [irq/19-ehci_hcd]

This is the USB 2.0 handler for our audio disk. Its process ID is 293.

  429  FF     50  90   - [irq/16-uhci_hcd]
  436  FF     50  90   - [irq/17-uhci_hcd]
  441  FF     50  90   - [irq/18-uhci_hcd]
  442  FF     50  90   - [irq/19-uhci_hcd]
 1114  FF     50  90   - [irq/8-rtc0]

The real-time clock handler is running as PID 1114.

 1117  FF     50  90   - [irq/16-yenta]

Our pc-card module has the process ID 1117.

 1147  FF     50  90   - [irq/21-ipw2200]
 1169  FF     50  90   - [irq/17-mmc0]
 1220  FF     50  90   - [irq/23-Intel IC]
 1225  FF     50  90   - [irq/16-ohci1394]

The firewire card handler has PID 1225. Again, the last few interrupts are not relevant for low latency audio.

3. Set the IRQ handler priorities

To tune realtime scheduling classes and priorities, you can use the chrt command which is part of the schedtools package. Obviously, it requires root privileges. Now we use the PIDs we have determined in the previous step.

hoppetosse:~ # chrt -f -p 90 1114
hoppetosse:~ # chrt -f -p 88 1117
hoppetosse:~ # chrt -f -p 86 1225
hoppetosse:~ # chrt -f -p 84 293
hoppetosse:~ # chrt -f -p 82 8

Just like any well-behaved UNIX command, chrt remains silent when it has accomplished its job.

To verify that all is well, take a look at the IRQ kernel threads again:

hoppetosse:~ # ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
...
    8  FF     82 122   - [sirq-tasklet/0]
  293  FF     84 124   - [irq/19-ehci_hcd]
...
 1114  FF     90 130   - [irq/8-rtc0]
 1117  FF     88 128   - [irq/16-yenta]
...
 1225  FF     86 126   - [irq/16-ohci1394]
...

4. Get the jackd priority Just Right(tm)

Now that all hardware-related priorities are settled, you can increase your jackd priority. Obviously, it does not make sense to raise it above the IRQ handlers we need for audio. But it makes a lot of sense to set it higher than the standard tasklet value, which is FF/50 on most systems. 70 would be a good value in most cases. Note that jackd will create some threads with higher and some with lower priorities for itself, and finally bestow RT privileges on its clients. It's generally helpful if these are still above the non-audio IRQ handlers, i.e. above 50.

Use jackd's -P parameter, or set the priority value in qjackctl accordingly. The following command will show you all your SCHED_FIFO threads, sorted in decreasing priority:

hoppetosse:~ # ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "FF" | sort -r
    99  FF    15 139   - [watchdog/0]
    99  FF    14 139   - [posixcputmr/0]
    99  FF     3 139   - [migration/0]
    95  FF    12 135   - [sirq-hrtimer/0]
    93  FF     5 133   - [sirq-timer/0]
    91  FF    10 131   - [sirq-tasklet/0]
    89  FF    11 129   - [sirq-sched/0]
    87  FF    57 127   - [irq/8-rtc0]
    85  FF   681 125   - [irq/16-yenta]
    83  FF   713 123   - [irq/16-ohci1394]
    81  FF    49 121   - [irq/19-ehci_hcd]
    80  FF 16446 120   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
    71  FF 16446 111   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
    70  FF 16446 110   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
    70  FF 16446 110   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
    69  FF 16446 109   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
    65  FF 16579 105   - /usr/local/lib/ardour2/ardour-2.8.11
    65  FF  4161 105   - qjackctl
    50  FF   710  90   - [irq/22-Intel IC]
    50  FF   701  90   - [irq/23-Intel IC]
    50  FF   698  90   - [irq/17-mmc0]
    50  FF   678  90   - [irq/21-ipw2200]
    50  FF   247  90   - [irq/16-i915]
    50  FF    55  90   - [irq/1-i8042]
    50  FF    54  90   - [irq/12-i8042]
    50  FF    53  90   - [irq/19-uhci_hcd]
    50  FF    52  90   - [irq/18-uhci_hcd]
    50  FF    51  90   - [irq/17-uhci_hcd]
    50  FF    50  90   - [irq/16-uhci_hcd]
    50  FF    44  90   - [irq/15-ata_piix]
    50  FF    43  90   - [irq/14-ata_piix]
    50  FF    29  90   - [irq/9-acpi]
    49  FF    13  89   - [sirq-rcu/0]
    49  FF     9  89   - [sirq-block-iopo]
    49  FF     8  89   - [sirq-block/0]
    49  FF     7  89   - [sirq-net-rx/0]
    49  FF     6  89   - [sirq-net-tx/0]
    49  FF     4  89   - [sirq-high/0]
     9  FF 16579  49   - /usr/local/lib/ardour2/ardour-2.8.11
     1  FF 16446  41   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
     1  FF 16446  41   - /usr/bin/jackd -P70 -p1024 -t9999 -dfirewire -dhw:0 -r48000 -p2048 -n3
     1  FF    17  41   - [events/0]
     -  TS 17282  19   0 grep FF

Note how the jack clients (ardour and qjackctl in this case) rank above the mundane IRQ handlers. At the bottom, you will see low-priority RT threads of jackd and ardour. Leave these alone - we should assume that the programmers know what they are doing.

5. (optional) Find a good slot for your external USB disk

Plug your USB drive into your machine. Then use the lsusb command to find out which bus it's sitting on. The number of the bus should correspond to the number of the "good" USB port as listed in /proc/interrupts. Swap USB ports until you've hit it. (But don't be frustrated if you don't - some mainboards provide "optional" USB ports that are not connected to any physical socket on the outside and can't be used easily.)

nettings@hoppetosse:/usr/local/lib> lsusb
...
Bus 001 Device 007: ID 0fce:d075 Your pluggable USB Hard disk
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
...

Automated priorities setting with rtirq

The fun thing about IRQ handlers is:

  • thanks to APIC, interrupt numbers can (and will change) between reboots
  • PIDs of top halves will change between reboots and on hardware hotplug events
  • USB bus numbers will change between reboots

(The USB port you found above will not change, though.)

That means there is no simple way to script those adjustments you've just made. rtirq to the rescue. It's a straightforward bash utility written by Rui Nuno Capela that employs some grep and awk magic to automate what you just did by hand.

Here's an example configuration (/etc/sysconfig/rtirq) that corresponds to the manual setting described earlier:

NOTE: Current low latency kernels enable per-device handlers (as opposed to per-interrupt as before). That means if two devices are sharing an interrupt (very common on notebooks), you can apply more fine-grained tweaks. rtirq has been updated to deal with this, so be sure to get the very latest version (scroll down a bit on that page). Still, you may encounter problems. Holler on the lists if you do.

# IRQ thread service names
# (space separated list, from higher to lower priority).
RTIRQ_NAME_LIST="rtc yenta ohci1394 ehci_hcd:usb"

# Highest priority.
RTIRQ_PRIO_HIGH=90

# Priority decrease step.
RTIRQ_PRIO_DECR=2

# Whether to reset all IRQ threads to SCHED_OTHER.
RTIRQ_RESET_ALL=0

# On kernel configurations that support it,
# which services should be NOT threaded
# (space separated list).
RTIRQ_NON_THREADED="rtc"

# Process names which will be forced to the
# highest realtime priority range (99-91)
# (space separated list, from highest to lower priority).
# RTIRQ_HIGH_LIST="timer"

And here is the result:

hoppetosse:~ # /etc/init.d/rtirq.sh start
Setting IRQ priorities: start [rtc] irq=8 pid=1128 prio=90: OK.
Setting IRQ priorities: start [yenta] irq=16 pid=1124 prio=88: OK.
Setting IRQ priorities: start [ohci1394] irq=16 pid=1240 prio=86: OK.
Setting IRQ priorities: start [ehci_hcd:usb] irq=19 pid=426 prio=84: OK.

You can add this script to the current runlevel, so that it will be executed automatically when the system starts up. As root, do

hoppetosse:~ # insserv rtirq.sh

Note that this won't be sufficient if you later plug in audio-relevant hardware. In that case, manually execute the script after all devices have been connected.

System-V style startup scripts (/etc/init.d/$foo {start|stop} and the insserv command) may not be present on all systems (notably the latest Ubuntu) - check your distro documentation about how to get the job done.

FIXME: find out which other tasklets should receive high priority, investigate consequences of demoting non-essential tasklets to SCHED_OTHER, verify that rtirq greps correctly for numbered handlers if the number is omitted (rtcX, usbX:)...

FIXME: extract ffado-specific settings and feed the rest upstream into the RT Wiki at kernel.org

FIXME: find out how to set sirq-tasklet to a sane value with rtirq (thanks to edogawa on #ffado)