RPi 3 and the real time kernel

As a beta tester for MOD I thought it would be cool to play around with netJACK which is supported on the MOD Duo. The MOD Duo can run as a JACK master and you can connect any JACK slave to it as long as it runs a recent version of JACK2. This opens a plethora of possibilities of course. I’m thinking about building a kind of sidecar device to offload some stuff to using netJACK, think of synths like ZynAddSubFX or other CPU greedy plugins like fat1.lv2. But more on that in a later blog post.

So first I need to set up a sidecar device and I sacrificed one of my RPi’s for that, an RPi 3. Flashed an SD card with Raspbian Jessie Lite and started to do some research on the status of real time kernels and the Raspberry Pi because I’d like to use a real time kernel to get sub 5ms system latency. I compiled real time kernels for the RPi before but you had to jump through some hoops to get those running so I hoped things would have improved somewhat. Well, that’s not the case so after having compiled a first real time kernel the RPi froze as soon as I tried to runapt-get install rt-tests. After having applied a patch to fix how the RPi folks implemented the FIQ system the kernel compiled without issues:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

And the RPi seems to run stable with acceptable latencies:

Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops
Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops

So that’s a maximum latency of 75 µs, not bad. I also spotted some higher values around 100 but that’s still okay for this project. The histogram was created with mklatencyplot.bash. I used a different invocation of cyclictest though:

cyclictest -Sm -p 80 -n -i 500 -l 300000

And I ran hackbench in the background to create some load on the RPi:

(while true; do hackbench > /dev/null; done) &

Compiling a real time kernel for the RPi is still not a trivial thing to do and it doesn’t help that the few howto’s on the interwebs are mostly copy-paste work, incomplete and contain routines that are unclear or even unnecessary. One thing that struck me too is that the howto’s about building kernels for RPi’s running Raspbian don’t mention the make deb-pkg routine to build a real time kernel. This will create deb packages that are just so much easier to transfer and install then rsync’ing the kernel image and modules. Let’s break down how I built a real time kernel for the RPi 3.

First you’ll need to git clone the Raspberry Pi kernel repository:

git clone -b 'rpi-4.9.y' --depth 1 https://github.com/raspberrypi/linux.git

This will only clone the rpi-4.9.y branch into a directory called linux without any history so you’re not pulling in hundreds of megs of data. You will also need to clone the tools repository which contains the compiler we need to build a kernel for the Raspberry Pi:

git clone https://github.com/raspberrypi/tools.git

This will end up in the tools directory. Next step is setting some environment variables so subsequent make commands pick those up:

export KERNEL=kernel7
export ARCH=arm
export CROSS_COMPILE=/path/to/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
export CONCURRENCY_LEVEL=$(nproc)

The KERNEL variable is needed to create the initial kernel config. The ARCH variable is to indicate which architecture should be used. The CROSS_COMPILE variable indicates where the compiler can be found. The CONCURRENCY_LEVEL variable is set to the number of cores to speed up certain make routines like cleaning up or installing the modules (not the number of jobs, that is done with the -j option of make).

Now that the environment variables are set we can create the initial kernel config:

cd linux
make bcm2709_defconfig

This will create a .config inside the linux directory that holds the initial kernel configuration. Now download the real time patch set and apply it:

cd ..
wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/patch-4.9.33-rt23.patch.xz
cd linux
xzcat ../patch-4.9.33-rt23.patch.xz | patch -p1

Most howto’s now continue with building the kernel but that will result in a kernel that will freeze your RPi because of the FIQ system implementation that causes lock ups of the RPi when using threaded interrupts which is the case with real time kernels. That part needs to be patched so download the patch and dry-run it:

cd ..
wget https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
cd linux
patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run

You will notice one hunk will fail, you will have to add that stanza manually so note which hunk it is for which file and at which line it should be added. Now apply the patch:

patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

And add the failed hunk manually with your favorite editor. With the FIQ patch in place we’re almost set for compiling the kernel but before we can move on to that step we need to modify the kernel configuration to enable the real time patch set. I prefer doing that with make menuconfig. You will need the libncurses5-dev package to run this commando so install that with apt-get install libncurses5-dev. Then select Kernel Features - Preemption Model - Fully Preemptible Kernel (RT) and select Exit twice. If you’re asked if you want to save your config then confirm. In the Kernel features menu you could also set the the timer frequency to 1000 Hz if you wish, apparently this could improve USB throughput on the RPi (unconfirmed, needs reference). For real time audio and MIDI this setting is irrelevant nowadays though as almost all audio and MIDI applications use the hr-timer module which has a way higher resolution.

With our configuration saved we can start compiling. Clean up first, then disable some debugging options which could cause some overhead, compile the kernel and finally create ready to install deb packages:

make clean
scripts/config --disable DEBUG_INFO
make -j$(nproc) deb-pkg

Sit back, enjoy a cuppa and when building has finished without errors deb packages should be created in the directory above the linux one. Copy the deb packages to your RPi and install them on the RPi with dpkg -i. Open up /boot/config.txt and add the following line to it:

kernel=vmlinuz-4.9.33-rt23-v7+

Now reboot your RPi and it should boot with the realtime kernel. You can check with uname -a:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

Since Rasbian uses almost the same kernel source as the one we just built it is not necessary to copy any dtb files. Also running mkknlimg is not necessary anymore, the RPi boot process can handle vmlinuz files just fine.

The basis of the sidecar unit is now done. Next up is tweaking the OS and setting up netJACK.

Edit: there’s a thread on LinuxMusicians referring to this article which already contains some very useful additional information.

RPi 3 and the real time kernel

More ARM goodies II

Received the BeagleBone Black (BBB) and the MK808 with a RK3066 SoC. My first impressions are really positive. Especially the BBB is quite an awesome device that I’m probably going to use a lot in favor of the Raspberry Pi. At first glance I had something like, the BBB blows the RPi away, but as soon as I started looking for documentation on how to put Debian on it for instance it became clear that the RPi is still the device to beat. The RPi community is huge, documentation for it is well laid out and working with the RPi is just so easy. The BBB on the other hand lacks a vivid community, is $10 more expensive and a lot more difficult to work with. Take the Debian install for example, seems quite some work to get that going.

The MK808 is surely an improvement over the UG80X I already own. It comes with a HDMI port instead of a HDMI plug, has an extra USB OTG port, a heatsink, hardware serial console access, a reset button and a power indicator LED. The pre-installed Android version looks better too. I flashed my RT kernel recovery image on it, inserted the Micro SD from my UG80X and it booted without any issues. So I’m going to pursue my goal to get a real-time, low-latency environment running on a RK3066 based device on the MK808 and find another purpose for the UG80X.

Edit: Getting Debian to work on the BBB is actually quite easy: http://elinux.org/BeagleBoardDebian#Demo_Image
Next time I’ll promise to make better use of my Google skills.

More ARM goodies II

Hacking an Android TV stick, the sequel

jeremy@rk3066:~$ uname -a
Linux rk3066 3.0.36-rt58 #1 SMP PREEMPT RT Thu Jul 4 13:18:23 CEST
2013 armv7l GNU/Linux

Managed to compile and run a real-time kernel on the Android TV stick with the RK3066 SoC. Packaged the latest version of amSynth (1.4.0 which has been released recently), installed it, fired up JACK and amSynth and so far no xruns, nothing. And this is with -p64!

jackd -P84 -p32 -t2000 -dalsa -dhw:Device -n3 -p64 -r44100 -s -P

I should measure the latency of the $2 USB audio interface I’m using to find out what the total latency of this set-up is. Well, at least I got the system latency for usage with softsynths like amSynth down to 64/44100*2=3ms. Now that’s a usable situation.

jeremy@rk3066:~$ lsusb | grep -i c-media
Bus 002 Device 006: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter (Planet
UP-100, Genius G-Talk)
jeremy@rk3066:~$ cat /proc/asound/cards 0 [RK29RK1000 ]: RK29_RK1000 - RK29_RK1000 RK29_RK1000 1 [HDMI ]: ROCKCHIP_HDMI - ROCKCHIP HDMI ROCKCHIP HDMI 2 [Device ]: USB-Audio - Generic USB Audio Device Generic USB Audio Device at usb-usb20_host-1.1, full speed

A big pro of this stick is that it suffers less from SD card corruption than my RPi. Yesterday evening I wrecked up yet another SD card when testing my RPi with a real-time kernel, it’s getting a bit cumbersome. Speaking of real-time kernels, it was quite some work to apply the RT patchset to the RockChip kernel source. Had to add stuff by hand and when I finally got everything in place it wouldn’t compile. But I managed to solve all the build errors. After flashing the kernel image the TV stick wouldn’t boot of course, it hung at some point. But I quickly saw that the issue was with the SD card reader and that it was similar to the SD card reader issue on the RPi for which I found a workaround. So I added an #ifdef clause to the RockChip SD card reader driver, recompiled, reflashed and wham, it continued booting. Now I have to clean up my build directory and get a usable diff of it against the pristine RK3066 kernel sources.

Hacking an Android TV stick, the sequel