RPi 3 and the real time kernel

As a beta tester for MOD I thought it would be cool to play around with netJACK which is supported on the MOD Duo. The MOD Duo can run as a JACK master and you can connect any JACK slave to it as long as it runs a recent version of JACK2. This opens a plethora of possibilities of course. I’m thinking about building a kind of sidecar device to offload some stuff to using netJACK, think of synths like ZynAddSubFX or other CPU greedy plugins like fat1.lv2. But more on that in a later blog post.

So first I need to set up a sidecar device and I sacrificed one of my RPi’s for that, an RPi 3. Flashed an SD card with Raspbian Jessie Lite and started to do some research on the status of real time kernels and the Raspberry Pi because I’d like to use a real time kernel to get sub 5ms system latency. I compiled real time kernels for the RPi before but you had to jump through some hoops to get those running so I hoped things would have improved somewhat. Well, that’s not the case so after having compiled a first real time kernel the RPi froze as soon as I tried to runapt-get install rt-tests. After having applied a patch to fix how the RPi folks implemented the FIQ system the kernel compiled without issues:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

And the RPi seems to run stable with acceptable latencies:

Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops
Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops

So that’s a maximum latency of 75 µs, not bad. I also spotted some higher values around 100 but that’s still okay for this project. The histogram was created with mklatencyplot.bash. I used a different invocation of cyclictest though:

cyclictest -Sm -p 80 -n -i 500 -l 300000

And I ran hackbench in the background to create some load on the RPi:

(while true; do hackbench > /dev/null; done) &

Compiling a real time kernel for the RPi is still not a trivial thing to do and it doesn’t help that the few howto’s on the interwebs are mostly copy-paste work, incomplete and contain routines that are unclear or even unnecessary. One thing that struck me too is that the howto’s about building kernels for RPi’s running Raspbian don’t mention the make deb-pkg routine to build a real time kernel. This will create deb packages that are just so much easier to transfer and install then rsync’ing the kernel image and modules. Let’s break down how I built a real time kernel for the RPi 3.

First you’ll need to git clone the Raspberry Pi kernel repository:

git clone -b 'rpi-4.9.y' --depth 1 https://github.com/raspberrypi/linux.git

This will only clone the rpi-4.9.y branch into a directory called linux without any history so you’re not pulling in hundreds of megs of data. You will also need to clone the tools repository which contains the compiler we need to build a kernel for the Raspberry Pi:

git clone https://github.com/raspberrypi/tools.git

This will end up in the tools directory. Next step is setting some environment variables so subsequent make commands pick those up:

export KERNEL=kernel7
export ARCH=arm
export CROSS_COMPILE=/path/to/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
export CONCURRENCY_LEVEL=$(nproc)

The KERNEL variable is needed to create the initial kernel config. The ARCH variable is to indicate which architecture should be used. The CROSS_COMPILE variable indicates where the compiler can be found. The CONCURRENCY_LEVEL variable is set to the number of cores to speed up certain make routines like cleaning up or installing the modules (not the number of jobs, that is done with the -j option of make).

Now that the environment variables are set we can create the initial kernel config:

cd linux
make bcm2709_defconfig

This will create a .config inside the linux directory that holds the initial kernel configuration. Now download the real time patch set and apply it:

cd ..
wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/patch-4.9.33-rt23.patch.xz
cd linux
xzcat ../patch-4.9.33-rt23.patch.xz | patch -p1

Most howto’s now continue with building the kernel but that will result in a kernel that will freeze your RPi because of the FIQ system implementation that causes lock ups of the RPi when using threaded interrupts which is the case with real time kernels. That part needs to be patched so download the patch and dry-run it:

cd ..
wget https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
cd linux
patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run

You will notice one hunk will fail, you will have to add that stanza manually so note which hunk it is for which file and at which line it should be added. Now apply the patch:

patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

And add the failed hunk manually with your favorite editor. With the FIQ patch in place we’re almost set for compiling the kernel but before we can move on to that step we need to modify the kernel configuration to enable the real time patch set. I prefer doing that with make menuconfig. You will need the libncurses5-dev package to run this commando so install that with apt-get install libncurses5-dev. Then select Kernel Features - Preemption Model - Fully Preemptible Kernel (RT) and select Exit twice. If you’re asked if you want to save your config then confirm. In the Kernel features menu you could also set the the timer frequency to 1000 Hz if you wish, apparently this could improve USB throughput on the RPi (unconfirmed, needs reference). For real time audio and MIDI this setting is irrelevant nowadays though as almost all audio and MIDI applications use the hr-timer module which has a way higher resolution.

With our configuration saved we can start compiling. Clean up first, then disable some debugging options which could cause some overhead, compile the kernel and finally create ready to install deb packages:

make clean
scripts/config --disable DEBUG_INFO
make -j$(nproc) deb-pkg

Sit back, enjoy a cuppa and when building has finished without errors deb packages should be created in the directory above the linux one. Copy the deb packages to your RPi and install them on the RPi with dpkg -i. Open up /boot/config.txt and add the following line to it:


Now reboot your RPi and it should boot with the realtime kernel. You can check with uname -a:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

Since Rasbian uses almost the same kernel source as the one we just built it is not necessary to copy any dtb files. Also running mkknlimg is not necessary anymore, the RPi boot process can handle vmlinuz files just fine.

The basis of the sidecar unit is now done. Next up is tweaking the OS and setting up netJACK.

Edit: there’s a thread on LinuxMusicians referring to this article which already contains some very useful additional information.

RPi 3 and the real time kernel

MOD sysadmin

A sysadmin is also jumping in to help with the infrastructure – mailing lists and wiki – so we can make sure there are usable tools available to us, after all, a brand new community is to be born!!!

From the MOD Kickstarter page, update #29.

So yes, I’m an official member of the MOD team now! I’m going to help out maintaining the current infrastructure and with the upcoming release of the MOD Duo some parts will have to be rebuilt or built up from scratch.

I will also be beta testing the MOD Duo, I’m eagerly awaiting for it to arrive in the mail. The MOD team has made some real good progress in getting the most out of the Allwinner A20 board they’re using. Also the amount of plugins that will be available for it will be staggering. This won’t be just an ordinary modelling FX unit but a complete all-in-one musical box that can also be used as a synthesizer or a loop station. And it’s going to be a rock-solid unit, the team working on making this possible contain some big names from the Linux audio community. The MOD Duo beta testing unit I will be receiving has been with Fons before for example, from what I understood he has done an in-depth analysis of the audio codec being used on the MOD Duo board.

MOD Duo final revision
MOD Duo final revision
MOD sysadmin

LAC2013: Saturday 11th of May

Somehow I managed to arrive just in time to set everything up for my workshop, like always. I’m very good at stumbling in at the very last moment. The Nvidia binary blob didn’t like a beamer attached to it so I couldn’t properly display my slides. The well-filled room didn’t seem to care, they were more interested in what I had to say and the equipment I brought with me. I talked and talked and only briefly demonstrated the Raspberry Pi I brought with me. Guess I could’ve done a whole workshop about the RPi because apparently that little device caught most of the attention.

Right after my workshop I rushed to Rui’s intercalated workshop about the software he develops. I missed his first workshop which took place on Thursday. But since Rui had so much more to show after that initial workshop the LAC2013 organization decided to allot him more time for an extra workshop. But I also didn’t want to miss Jörn’s workshop so I decided I also wanted to see part of that workshop too. Even though I’ve been using Rui’s software for years (I’m a QA, Qtractor Afficionado) he showed things I had never seen before. And I probably attended one of the best parts of Jörn’s workshop where he showed and made us listen to what he did with a live recording. An ear opener, really, amazing what you can do with a good pair of ears, years of experience and the right tools.

Then it was lunchtime already. Missed out on the warm lunch but when that was all cleared the alternative food stand was set up again which had probably even better food. They had great salads, fruit juice, bread and other tasty things, all for free. Many, many thanks to the organization for setting this up, it really added up to the overall positive vibe of the conference.

Now I had a bit of a problem. I needed to go to the Forum Stadtpark to do a sound check for the Linux Sound Night. But walking was not an option with all my stuff and public transport would take too long as there was no direct connection. Luckily I could tag along with the guys from SuperDirt² so I hopped into their car and off we went. SuperDirt² had to play last so they did their sound check first. I watched their sound check in awe, these guys were good! I was up next. Everything went smooth, monitors were good, the sound guy was a really cool guy and so was the stage manager for the event. All omens were very positive!

We got back right on time for Albert Graef’s talk on creating LV2 plugins with Faust. We witnessed a glimpse of the future. If I got it right it will be extremely easy to create your own LV2 plugins in the near future. Just throw some Faust code against it and upload it with your browser and within moments you can download your own LV2 plugin! Time to learn some Faust I guess. One lightning talk later we witnessed another glimpse of the future. No, not my acrylic guitar in the hands of Bruno Gola but the world premiere of the MOD Quadra digital pedal board. The excitement was tangible. But the MOD guys quite easily redeemed the high expectations. The MOD Quadra is simply an amazing device, the web GUI looks stunning and it’s all so easy. And it runs on Linux people, using the LV2 framework. If this doesn’t propel LV2 into mass adoption then what?

For dinner we ended up in a nice Italian restaurant. The Gösser tasted good, same for the dish I ordered (Calamari alla griglia). After dinner we walked to the Forum Stadtpark where we were welcomed by the pleasant chaos of Android drummers. I installed the app and joined the concert. It was fun. Then the beamer got switched on showing us two terminal windows with vim on the right side and something compile-ish on the left. On stage a person in front of a notebook, coding live. The result? Really cool stuff if you ask me, just watch for yourself.

Algorave to the max all y’all!

Then it was time for something remotely dance related, namely me. Really enjoyed the gig even though my voice let me down after the third song. The guy that came after me unfortunately was a bit the odd one out so the contrast with the last act, SuperDirt², couldn’t have been bigger. As soon as Käpt’n Dirt hit the strings of his cello we knew this was going to be a blast. And when Ras Tilo kicked in the party was complete. What a great show!

After the encore of SuperDirt² it was time for the Open Jack Session. Not Jack as in JACK but Jack as in 3.5″ mini-jack. It was lying there on the table on stage and after ClaudiusMaximus Marije Baalman plugged in. Live coding in SuperCollider with every once in a while Marije stretching her arms because of the anti-RSI alarms that kept popping up. Great stuff, very enjoyable to listen to and the added humorous note of the anti-RSI alarms perfectly summed up what this conference was all about: having a great time. Despite the submerging inebriation I was enthralled.

We stayed until we got kicked out. That’s how it should be. Sole minor blemish: the beer. That Murauer stuff was close to undrinkable.

LAC2013: Saturday 11th of May

LAC2013: Friday 10th of May

“Hello, I’d like to know if I can take my electric guitar with me as hand luggage, would that be possible?”
“Well that depends on the whims of the cabin crew.”
“So if I can’t take it with me in the cabin would that cost me extra?”
“Yes, that will be €200.”

So I decided to take my guitar apart, wrap it in a big towel and put in my suitcase so that I could take it with me as standard baggage. I didn’t want to run the risk of having to pay €200 extra. When I arrived at the airport I was sitting next to a young lady with, yes you guessed it right, a guitar as hand luggage. She didn’t have to pay anything extra. Grmbl. You’ll see that when I unpack my stuff in Graz the neck of the guitar will be broken or something. Let’s hope not.

I had a transfer in Munich which went smoothly. The whole journey went smoothly actually. Thanks to 3G internet, QR codes of my tickets sent to my mobile phone and Google maps. Especially those QR codes are very practical, they save time and paper. Every counter and gate has QR scanners and the personnel just puts your mobile phone in the scanner and you’re done. Maximum efficiency. Google maps helped me out on quite some occasions too although the Grazians were very helpful too.

So on Friday around 2 PM I walked into the main conference building. I couldn’t be there on Thursday because of my son‘s birthday. So I already missed quite some interesting workshops, presentations and lightning talks. I was kind of bummed about that at first but as soon as I entered the building I forgot about all that. It was still lunch time and everyone got together in the main building to eat something. It was a warm welcome and great to see all those people again. I immediately spotted the MOD guys (my goal for this LAC was to get a MOD endorsement) and got acquainted with the one and only Kirill Alferov which was quite a nice surpise. After having something to eat I decided to check on the status of my guitar and to put it back together again. Luckily the guitar had survived the journey unharmed.

As soon as I started screwing the neck back on the MOD guys approached me to ask if they could borrow my guitar for their lightning talk on Saturday. Sure, but only if I could play around a bit with the MOD Quadra they brought with them. This was no problem so a few minutes later I found myself happily noodling and tapping footswitches on this amazing device. Even though it still had some rough edges it all felt, looked and sounded very professional to me. Yes, the MOD is a fine example of my view on Linux audio development: world domination. And I mean it. Within 10 years Windows and Mac OS will only exist for tablets or other small portable devices with touchscreens and Linux will be the only viable alternative on both other ends of the scale: embedded devices and fully fledged audio workstations in professional studios.

Right after that I headed off with Marc Groenewegen to the Linux/Ardour in a Recording Studio workshop where I soon found myself going through the mixing console manual together with Frank Neumann to find any references to GPL clauses as the console was running on Linux (see how quickly world domination is approaching already?). Of course we found nothing. The workshop was a bit too specialized for me so I decided to check in at the hotel and get rid of my baggage. The plan was to eat Schnitzels for dinner but when it was time to go to the restaurant it was pouring so we (me, Funs & girlfriend) arrived at the restaurant completely soaked. But I couldn’t care less when I got served my Schnitzel XL with a big pint of Puntigamer.

After dinner we went to the Mumuth. Amazing building. But me and electroacoustic music don’t go together that well. It’s not that I don’t like it. I’ve seen Sachiko M once and she blew me away with just a sine wave out of a sampler. Not just with the sine wave but more with her performance, her being there and getting totally absorbed by the sounds she was making. They had to carry her off the stage after she was done. Well, that didn’t happen at the Mumuth. Or I should’ve stayed until the end because now I’m basing my judgement on a mere three perfomances because I left earlier as I didn’t really dig the vibe.

LAC2013: Friday 10th of May

MOD – your next digital pedalboard

Tja wat moet ik hier nou over zeggen? Check het zelf uit, fantastisch idee, een digitale gitaar FX processor die geheel op Linux draait en gebruik maakt van het LV2 plug-in framework.

text-align: center;

MOD Quadra prototype

En ja het is de bedoeling dat dit apparaat echt in productie gaat. Er worden er binnenkort 35 gemaakt, zou er graag een willen hebben uiteraard, maar zal een aardig prijzige aangelegenheid worden. Heb me ingeschreven op de mailinglist dus we merken het wel. I’ll keep you posted.

Site: http://www.portalmod.com/en/index.html

MOD – your next digital pedalboard