RPi 3 and the real time kernel

As a beta tester for MOD I thought it would be cool to play around with netJACK which is supported on the MOD Duo. The MOD Duo can run as a JACK master and you can connect any JACK slave to it as long as it runs a recent version of JACK2. This opens a plethora of possibilities of course. I’m thinking about building a kind of sidecar device to offload some stuff to using netJACK, think of synths like ZynAddSubFX or other CPU greedy plugins like fat1.lv2. But more on that in a later blog post.

So first I need to set up a sidecar device and I sacrificed one of my RPi’s for that, an RPi 3. Flashed an SD card with Raspbian Jessie Lite and started to do some research on the status of real time kernels and the Raspberry Pi because I’d like to use a real time kernel to get sub 5ms system latency. I compiled real time kernels for the RPi before but you had to jump through some hoops to get those running so I hoped things would have improved somewhat. Well, that’s not the case so after having compiled a first real time kernel the RPi froze as soon as I tried to runapt-get install rt-tests. After having applied a patch to fix how the RPi folks implemented the FIQ system the kernel compiled without issues:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

And the RPi seems to run stable with acceptable latencies:

Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops
Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops

So that’s a maximum latency of 75 µs, not bad. I also spotted some higher values around 100 but that’s still okay for this project. The histogram was created with mklatencyplot.bash. I used a different invocation of cyclictest though:

cyclictest -Sm -p 80 -n -i 500 -l 300000

And I ran hackbench in the background to create some load on the RPi:

(while true; do hackbench > /dev/null; done) &

Compiling a real time kernel for the RPi is still not a trivial thing to do and it doesn’t help that the few howto’s on the interwebs are mostly copy-paste work, incomplete and contain routines that are unclear or even unnecessary. One thing that struck me too is that the howto’s about building kernels for RPi’s running Raspbian don’t mention the make deb-pkg routine to build a real time kernel. This will create deb packages that are just so much easier to transfer and install then rsync’ing the kernel image and modules. Let’s break down how I built a real time kernel for the RPi 3.

First you’ll need to git clone the Raspberry Pi kernel repository:

git clone -b 'rpi-4.9.y' --depth 1 https://github.com/raspberrypi/linux.git

This will only clone the rpi-4.9.y branch into a directory called linux without any history so you’re not pulling in hundreds of megs of data. You will also need to clone the tools repository which contains the compiler we need to build a kernel for the Raspberry Pi:

git clone https://github.com/raspberrypi/tools.git

This will end up in the tools directory. Next step is setting some environment variables so subsequent make commands pick those up:

export KERNEL=kernel7
export ARCH=arm
export CROSS_COMPILE=/path/to/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
export CONCURRENCY_LEVEL=$(nproc)

The KERNEL variable is needed to create the initial kernel config. The ARCH variable is to indicate which architecture should be used. The CROSS_COMPILE variable indicates where the compiler can be found. The CONCURRENCY_LEVEL variable is set to the number of cores to speed up certain make routines like cleaning up or installing the modules (not the number of jobs, that is done with the -j option of make).

Now that the environment variables are set we can create the initial kernel config:

cd linux
make bcm2709_defconfig

This will create a .config inside the linux directory that holds the initial kernel configuration. Now download the real time patch set and apply it:

cd ..
wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/patch-4.9.33-rt23.patch.xz
cd linux
xzcat ../patch-4.9.33-rt23.patch.xz | patch -p1

Most howto’s now continue with building the kernel but that will result in a kernel that will freeze your RPi because of the FIQ system implementation that causes lock ups of the RPi when using threaded interrupts which is the case with real time kernels. That part needs to be patched so download the patch and dry-run it:

cd ..
wget https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
cd linux
patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run

You will notice one hunk will fail, you will have to add that stanza manually so note which hunk it is for which file and at which line it should be added. Now apply the patch:

patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

And add the failed hunk manually with your favorite editor. With the FIQ patch in place we’re almost set for compiling the kernel but before we can move on to that step we need to modify the kernel configuration to enable the real time patch set. I prefer doing that with make menuconfig. You will need the libncurses5-dev package to run this commando so install that with apt-get install libncurses5-dev. Then select Kernel Features - Preemption Model - Fully Preemptible Kernel (RT) and select Exit twice. If you’re asked if you want to save your config then confirm. In the Kernel features menu you could also set the the timer frequency to 1000 Hz if you wish, apparently this could improve USB throughput on the RPi (unconfirmed, needs reference). For real time audio and MIDI this setting is irrelevant nowadays though as almost all audio and MIDI applications use the hr-timer module which has a way higher resolution.

With our configuration saved we can start compiling. Clean up first, then disable some debugging options which could cause some overhead, compile the kernel and finally create ready to install deb packages:

make clean
scripts/config --disable DEBUG_INFO
make -j$(nproc) deb-pkg

Sit back, enjoy a cuppa and when building has finished without errors deb packages should be created in the directory above the linux one. Copy the deb packages to your RPi and install them on the RPi with dpkg -i. Open up /boot/config.txt and add the following line to it:

kernel=vmlinuz-4.9.33-rt23-v7+

Now reboot your RPi and it should boot with the realtime kernel. You can check with uname -a:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

Since Rasbian uses almost the same kernel source as the one we just built it is not necessary to copy any dtb files. Also running mkknlimg is not necessary anymore, the RPi boot process can handle vmlinuz files just fine.

The basis of the sidecar unit is now done. Next up is tweaking the OS and setting up netJACK.

Edit: there’s a thread on LinuxMusicians referring to this article which already contains some very useful additional information.

RPi 3 and the real time kernel

Moving to OpenStack

In the coming days I’m going to move the VPS on which this blog resides from VMware to the Fuga OpenStack cloud. Not because I have to but hey, if I can host my stuff on a fully open source based cloud instead of a proprietary solution the decision is simple. And Fuga has been around for a while now, it’s rock solid and as I have a lot of freedom within my OpenStack tenant I can do with my VPS whatever I want when it comes to resources.

Moving the VM will cause some downtime. I’ve opted for the solution to shut down the VM, copy it from the ESXi host on which it lives to a server with enough storage and the libguestfs-tools package so that I can do some customization and the python-openstackclient package so that I can easily upload the customized image to OpenStack. Then I need to deploy an OpenStack instance from that uploaded image, switch the DNS and my server should be back online.

Moving to OpenStack

MOD sysadmin

A sysadmin is also jumping in to help with the infrastructure – mailing lists and wiki – so we can make sure there are usable tools available to us, after all, a brand new community is to be born!!!

From the MOD Kickstarter page, update #29.

So yes, I’m an official member of the MOD team now! I’m going to help out maintaining the current infrastructure and with the upcoming release of the MOD Duo some parts will have to be rebuilt or built up from scratch.

I will also be beta testing the MOD Duo, I’m eagerly awaiting for it to arrive in the mail. The MOD team has made some real good progress in getting the most out of the Allwinner A20 board they’re using. Also the amount of plugins that will be available for it will be staggering. This won’t be just an ordinary modelling FX unit but a complete all-in-one musical box that can also be used as a synthesizer or a loop station. And it’s going to be a rock-solid unit, the team working on making this possible contain some big names from the Linux audio community. The MOD Duo beta testing unit I will be receiving has been with Fons before for example, from what I understood he has done an in-depth analysis of the audio codec being used on the MOD Duo board.

MOD Duo final revision
MOD Duo final revision
MOD sysadmin

The Zynthian project

Recently I found out that I was not the only one trying to build a synth module out of a Raspberry Pi with ZynAddSubFX. The Zynthian project is trying to achieve the exact same goal and so far it looks very promising. I contacted the project owner to ask if he would be interested in collaborating. I got a reply promptly and we both agreed it would be a good idea to join forces. The Zynthian project has all the things that I still had to set up already in place but I think I can still help out. The Zynthian set-up might benefit from some optimizations like a real-time kernel and things like boot time can be improved. Also I could help out testing, maybe do some packaging. And if there’s a need for things like a repository, web server or other hosting related stuff I could provide those.

Protoype of the Zynthian project
Zynthian prototype

I’m very happy with these developments of our projects converging. Check out the Zynthian blog for more information on the current state of the project.

The Zynthian project

The hardest part

At least, for me. Now I have to code something that enables me to select banks and instruments on my synth module. I’ve settled for pyliblo to talk to ZynAddSubFX but I’m just no coder. My elbow now rests on “Learning Python” from O’Reilly, I started reading it like a year ago but never got past chapter 3 or something. Time to persist, I’ve been wanting to be able to code a little bit for years now.

As a note to self, and maybe it’s helpful for others too, what follows are some of the relevant OSC messages to change banks and instruments in ZynAddSubFX.

Changing banks can be done with /loadbank. pyliblo comes with an example script send_osc.py and loading a bank with send_osc.py works as follows:

send_osc.py 7777 /loadbank ,i 3

7777 is the port ZynAddSubFX runs on. The /loadbank message wants an integer (,i) and ,i 3 loads the fourth bank (Choir and Voice) as ZynAddSubFX starts counting from 0. Bear in mind that this will only load the bank, it won’t change the instrument that is loaded. To load an instrument from a loaded bank the following send_osc.py incantation does the trick:

send_osc.py 7777 /setprogram ,c $'\x03'

So /setprogram loads an instrument from an active bank. It takes a character (,c) as an argument because the program numbers are in hex. But hex are multiple characters so you have to add some escape sequences to make it work (and I lost it there so I could be completely wrong). The above command should load the fourth instrument from the Choir and Voice bank (Voice OOH) as that bank should have been loaded by the previous /loadbank message.

It is also possible to load instrument (.xiz) files. This can be done with the /load_xiz message:

send_osc.py 7777 /load_xiz 0 "/usr/share/zynaddsubfx/banks/Bass/0001-Bass 1.xiz"

This will load the Bass 1 instrument fom the Bass bank into part 1 (remember that ZynAddSubFX starts counting from 0). So to load the Bass 2 instrument into part 2 you’d do the following:

send_osc.py 7777 /load_xiz 1 "/usr/share/zynaddsubfx/banks/Bass/0002-Bass 2.xiz"

So now I have to incorporate this stuff into Python code that gets called when I press buttons on my LCD display. These are the mappings I’d like to accomplish:

  • Up: toggle next bank and display first instrument from that bank
  • Down: toggle previous bank and display first instrument from that bank
  • Left: toggle previous instrument and display instrument
  • Right: toggle next instrument and display instrument
  • Select: select displayed instrument

That shouldn’t be too hard right? Well, first hurdle, pyliblo can only send or dump OSC messages, it doesn’t seem to be able to handle return messages from the OSC server (ZynAddSubFX in this case). To be continued…

Edit: of course sending just messages with pyliblo won’t handle any return messages, you will need a receiving part. But maybe I should take a look at using MIDI with mididings for instance. Thanks Georg Mill for the tip!

The hardest part

Two steps further

For my little synth module project I created the following systemd unit file /etc/systemd/system/zynaddsubfx.service that starts up a ZynAddSubFX proces at boot time:

[Unit]
Description=ZynAddSubFX

[Service]
Type=forking
User=zynaddsubfx
Group=zynaddsubfx
ExecStart=/usr/local/bin/zynaddsubfx-mpk
ExecStop=/usr/bin/killall zynaddsubfx

[Install]
WantedBy=local-fs.target

/usr/local/bin/zynaddsubfx-mpk is a simple script that starts ZynAddSubFX and connects my Akai MPK:

#!/bin/bash

zynaddsubfx -r 48000 -b 64 -I alsa -O alsa -P 7777 -L /usr/share/zynaddsubfx/banks/SynthPiano/0040-BinaryPiano2.xiz &

while ! aconnect 'MPK mini' 'ZynAddSubFX'
do
  sleep 0.1
done

/usr/local/bin/zynpi.py

/usr/local/bin/zynpi.py in its turn is a small Python script that shows a message and a red LED on a 16×2 LCD display so that I know the synth module is ready to use:

#!/usr/bin/python
# Example using a character LCD plate.
import math
import time

import Adafruit_CharLCD as LCD

# Initialize the LCD using the pins
lcd = LCD.Adafruit_CharLCDPlate()

# Show some basic colors.
lcd.set_color(1.0, 0.0, 0.0)
lcd.clear()
lcd.message('Raspberry Pi 2\nZynAddSubFX')

The LCD is not an Adafruit one though but a cheaper version I found on Dealextreme. It works fine though with the Adafruit LCD Python library. Next step is to figure out if I can use the buttons on the LCD board to change banks and presets.

synth_module_lcd_startup
Raspberry Pi synth module with 16×2 LCD display
synth_module_test_setup
The synth module test environment
Two steps further