RPi 3 and the real time kernel

As a beta tester for MOD I thought it would be cool to play around with netJACK which is supported on the MOD Duo. The MOD Duo can run as a JACK master and you can connect any JACK slave to it as long as it runs a recent version of JACK2. This opens a plethora of possibilities of course. I’m thinking about building a kind of sidecar device to offload some stuff to using netJACK, think of synths like ZynAddSubFX or other CPU greedy plugins like fat1.lv2. But more on that in a later blog post.

So first I need to set up a sidecar device and I sacrificed one of my RPi’s for that, an RPi 3. Flashed an SD card with Raspbian Jessie Lite and started to do some research on the status of real time kernels and the Raspberry Pi because I’d like to use a real time kernel to get sub 5ms system latency. I compiled real time kernels for the RPi before but you had to jump through some hoops to get those running so I hoped things would have improved somewhat. Well, that’s not the case so after having compiled a first real time kernel the RPi froze as soon as I tried to runapt-get install rt-tests. After having applied a patch to fix how the RPi folks implemented the FIQ system the kernel compiled without issues:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

And the RPi seems to run stable with acceptable latencies:

Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops
Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops

So that’s a maximum latency of 75 µs, not bad. I also spotted some higher values around 100 but that’s still okay for this project. The histogram was created with mklatencyplot.bash. I used a different invocation of cyclictest though:

cyclictest -Sm -p 80 -n -i 500 -l 300000

And I ran hackbench in the background to create some load on the RPi:

(while true; do hackbench > /dev/null; done) &

Compiling a real time kernel for the RPi is still not a trivial thing to do and it doesn’t help that the few howto’s on the interwebs are mostly copy-paste work, incomplete and contain routines that are unclear or even unnecessary. One thing that struck me too is that the howto’s about building kernels for RPi’s running Raspbian don’t mention the make deb-pkg routine to build a real time kernel. This will create deb packages that are just so much easier to transfer and install then rsync’ing the kernel image and modules. Let’s break down how I built a real time kernel for the RPi 3.

First you’ll need to git clone the Raspberry Pi kernel repository:

git clone -b 'rpi-4.9.y' --depth 1 https://github.com/raspberrypi/linux.git

This will only clone the rpi-4.9.y branch into a directory called linux without any history so you’re not pulling in hundreds of megs of data. You will also need to clone the tools repository which contains the compiler we need to build a kernel for the Raspberry Pi:

git clone https://github.com/raspberrypi/tools.git

This will end up in the tools directory. Next step is setting some environment variables so subsequent make commands pick those up:

export KERNEL=kernel7
export ARCH=arm
export CROSS_COMPILE=/path/to/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
export CONCURRENCY_LEVEL=$(nproc)

The KERNEL variable is needed to create the initial kernel config. The ARCH variable is to indicate which architecture should be used. The CROSS_COMPILE variable indicates where the compiler can be found. The CONCURRENCY_LEVEL variable is set to the number of cores to speed up certain make routines like cleaning up or installing the modules (not the number of jobs, that is done with the -j option of make).

Now that the environment variables are set we can create the initial kernel config:

cd linux
make bcm2709_defconfig

This will create a .config inside the linux directory that holds the initial kernel configuration. Now download the real time patch set and apply it:

cd ..
wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/patch-4.9.33-rt23.patch.xz
cd linux
xzcat ../patch-4.9.33-rt23.patch.xz | patch -p1

Most howto’s now continue with building the kernel but that will result in a kernel that will freeze your RPi because of the FIQ system implementation that causes lock ups of the RPi when using threaded interrupts which is the case with real time kernels. That part needs to be patched so download the patch and dry-run it:

cd ..
wget https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
cd linux
patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run

You will notice one hunk will fail, you will have to add that stanza manually so note which hunk it is for which file and at which line it should be added. Now apply the patch:

patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

And add the failed hunk manually with your favorite editor. With the FIQ patch in place we’re almost set for compiling the kernel but before we can move on to that step we need to modify the kernel configuration to enable the real time patch set. I prefer doing that with make menuconfig. You will need the libncurses5-dev package to run this commando so install that with apt-get install libncurses5-dev. Then select Kernel Features - Preemption Model - Fully Preemptible Kernel (RT) and select Exit twice. If you’re asked if you want to save your config then confirm. In the Kernel features menu you could also set the the timer frequency to 1000 Hz if you wish, apparently this could improve USB throughput on the RPi (unconfirmed, needs reference). For real time audio and MIDI this setting is irrelevant nowadays though as almost all audio and MIDI applications use the hr-timer module which has a way higher resolution.

With our configuration saved we can start compiling. Clean up first, then disable some debugging options which could cause some overhead, compile the kernel and finally create ready to install deb packages:

make clean
scripts/config --disable DEBUG_INFO
make -j$(nproc) deb-pkg

Sit back, enjoy a cuppa and when building has finished without errors deb packages should be created in the directory above the linux one. Copy the deb packages to your RPi and install them on the RPi with dpkg -i. Open up /boot/config.txt and add the following line to it:

kernel=vmlinuz-4.9.33-rt23-v7+

Now reboot your RPi and it should boot with the realtime kernel. You can check with uname -a:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

Since Rasbian uses almost the same kernel source as the one we just built it is not necessary to copy any dtb files. Also running mkknlimg is not necessary anymore, the RPi boot process can handle vmlinuz files just fine.

The basis of the sidecar unit is now done. Next up is tweaking the OS and setting up netJACK.

Edit: there’s a thread on LinuxMusicians referring to this article which already contains some very useful additional information.

RPi 3 and the real time kernel

More ARM goodies II

Received the BeagleBone Black (BBB) and the MK808 with a RK3066 SoC. My first impressions are really positive. Especially the BBB is quite an awesome device that I’m probably going to use a lot in favor of the Raspberry Pi. At first glance I had something like, the BBB blows the RPi away, but as soon as I started looking for documentation on how to put Debian on it for instance it became clear that the RPi is still the device to beat. The RPi community is huge, documentation for it is well laid out and working with the RPi is just so easy. The BBB on the other hand lacks a vivid community, is $10 more expensive and a lot more difficult to work with. Take the Debian install for example, seems quite some work to get that going.

The MK808 is surely an improvement over the UG80X I already own. It comes with a HDMI port instead of a HDMI plug, has an extra USB OTG port, a heatsink, hardware serial console access, a reset button and a power indicator LED. The pre-installed Android version looks better too. I flashed my RT kernel recovery image on it, inserted the Micro SD from my UG80X and it booted without any issues. So I’m going to pursue my goal to get a real-time, low-latency environment running on a RK3066 based device on the MK808 and find another purpose for the UG80X.

Edit: Getting Debian to work on the BBB is actually quite easy: http://elinux.org/BeagleBoardDebian#Demo_Image
Next time I’ll promise to make better use of my Google skills.

More ARM goodies II

Hacking an Android TV stick, the sequel

jeremy@rk3066:~$ uname -a
Linux rk3066 3.0.36-rt58 #1 SMP PREEMPT RT Thu Jul 4 13:18:23 CEST
2013 armv7l GNU/Linux

Managed to compile and run a real-time kernel on the Android TV stick with the RK3066 SoC. Packaged the latest version of amSynth (1.4.0 which has been released recently), installed it, fired up JACK and amSynth and so far no xruns, nothing. And this is with -p64!

jackd -P84 -p32 -t2000 -dalsa -dhw:Device -n3 -p64 -r44100 -s -P

I should measure the latency of the $2 USB audio interface I’m using to find out what the total latency of this set-up is. Well, at least I got the system latency for usage with softsynths like amSynth down to 64/44100*2=3ms. Now that’s a usable situation.

jeremy@rk3066:~$ lsusb | grep -i c-media
Bus 002 Device 006: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter (Planet
UP-100, Genius G-Talk)
jeremy@rk3066:~$ cat /proc/asound/cards 0 [RK29RK1000 ]: RK29_RK1000 - RK29_RK1000 RK29_RK1000 1 [HDMI ]: ROCKCHIP_HDMI - ROCKCHIP HDMI ROCKCHIP HDMI 2 [Device ]: USB-Audio - Generic USB Audio Device Generic USB Audio Device at usb-usb20_host-1.1, full speed

A big pro of this stick is that it suffers less from SD card corruption than my RPi. Yesterday evening I wrecked up yet another SD card when testing my RPi with a real-time kernel, it’s getting a bit cumbersome. Speaking of real-time kernels, it was quite some work to apply the RT patchset to the RockChip kernel source. Had to add stuff by hand and when I finally got everything in place it wouldn’t compile. But I managed to solve all the build errors. After flashing the kernel image the TV stick wouldn’t boot of course, it hung at some point. But I quickly saw that the issue was with the SD card reader and that it was similar to the SD card reader issue on the RPi for which I found a workaround. So I added an #ifdef clause to the RockChip SD card reader driver, recompiled, reflashed and wham, it continued booting. Now I have to clean up my build directory and get a usable diff of it against the pristine RK3066 kernel sources.

Hacking an Android TV stick, the sequel

YubiKeys, SSH, PAM en LDAP

Hoe tweeweg authenticatie op te zetten voor SSH logins met behulp van YubiKeys, PAM en LDAP (via SSL) op een Ubuntu 12.04 installatie.

Benodigde pakketjes

sudo apt-get install ldap-auth-client libpam-yubico

Conflicterende pakketjes

Mocht het pakketje nscd geïnstalleerd zijn, deïnstalleer deze. Dit pakket verhindert een goede werking van de YubiKey-LDAP authenticatiemethode.

ldap-auth-config

Tijdens het installeren van het pakketje ldap-auth-client wordt er een aantal gegevens gevraagd. Doorloop de set-up als volgt:

  • LDAP server Uniform Resource Identifier: ldaps://fqdn.van.ldap.server:poortnummer
  • Distinguished name of the search base: dc=voorbeeld,dc=com
  • LDAP version to use: 3
  • Make local root Database admin: No
  • Does the LDAP database require login? No

/etc/ldap.conf

Dit bestand regelt de verbinding met je LDAP server. De setup van ldap-auth-client heeft de boel nu ingesteld maar eigenlijk heb je deze hele setup niet nodig want met een /etc/ldap.conf met de volgende drie regels ben je er ook:

base dc=voorbeeld,dc=com
uri ldaps://fqdn.ban.ldap.server:poortnummer
tls_cacertfile /etc/ssl/certs/cacert.org.pem

Je kunt dus een deel van wat de setup heeft aangemaakt laten staan, de rest kun je uithekken. Zoals je ziet heb ik voor de SSL kant van het verhaal voor CAcert.org gekozen als CA. Je moet dan ook expliciet verwijzen naar de root certificaten van CAcert.org anders kan er niet gebind worden met de LDAP server. De LDAP server, in mijn geval een OpenDJ directory server, maakt dus gebruik van door CAcert.org getekende certifcaten.

/etc/nsswitch.conf

Dit bestand regelt op welke manier en in welke volgorde de verschillende nameservices worden aangesproken. In ons geval gaat het om de entries password, shadow en groups in dit bestand. Standaard staan die in Ubuntu 12.04 op compat. Om Ubuntu eerst de fysieke bestanden en daarna LDAP te laten checken moet compat veranderd worden in files ldap. Dit gaat het snelst met een sed commando:

sudo sed -i 's/compat/files ldap/g' /etc/nsswitch.conf

Nu /etc/nsswitch.conf is ingesteld kun je gaan testen of je LDAP connectie goed is:

id ldapgebruiker
getent passwd
getent group

PAM

Nu is het zaak PAM zo in te stellen dat je zowel tegenover de Yubico servers authenticeert met je YubiKey als tegenover LDAP wanneer je SSH’t naar je machine. Dit regel je in /etc/pam.d/sshd:

# PAM configuration for the Secure Shell service

# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
auth       required     pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
auth       required     pam_env.so envfile=/etc/default/locale

# Yubikey authentication
#
# Allow local accounts for the moment.
auth sufficient pam_unix.so

# Fetch LDAP password + OTP, verify OTP and move on ('required' control value).
# LDAP password is stripped and passed on.
auth required pam_yubico.so id=1234 ldap_uri=ldaps://fqdn.van.ldap.server:poortnummer [ldapdn=ou=People,DC=voorbeeld,DC=com] user_attr=uid yubi_attr=yubikeyid

# LDAP authentication
#
# Verify the LDAP password that has been passed on ('use_first_password' argument)
# and grant access when password gets accepted ('sufficient' control value).
auth sufficient pam_ldap.so use_first_pass

# Standard Un*x authentication.
@include common-auth

# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so

# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account  required     pam_access.so

# Standard Un*x authorization.
@include common-account

# LDAP session
#
# Create homedir for LDAP user if it doesn't exist yet.
session required pam_mkhomedir.so skel=/etc/skel/ umask=0022

# Standard Un*x session setup and teardown.
@include common-session

# Print the message of the day upon successful login.
session    optional     pam_motd.so # [1]

# Print the status of the user's mailbox upon successful login.
session    optional     pam_mail.so standard noenv # [1]

# Set up user limits from /etc/security/limits.conf.
session    required     pam_limits.so

# Set up SELinux capabilities (need modified pam)
# session  required     pam_selinux.so multiple

# Standard Un*x password updating.
@include common-password

De dikgedrukte gedeeltes zijn toegevoegd. Het id wat is toegevoegd bij de pam_yubico.so module kun je aanvragen via Yubico. Dit id is de zgn. API Key. Ook heb ik een extra opties toegevoegd:

yubi_attr=yubikeyid

Wat dit doet is de mapping gebruiker – YubiKey Public ID checken op de LDAP server, dus niet op de machine zelf. Het gros van de handleidingen gaat ervan uit dat je de mappings op de machine zelf maakt in platte bestandjes. Maar dat wil je helemaal niet, daar heb je nou juist je LDAP server voor, zodat je dit soort dingen als mappings centraal kan regelen. Wat ik heb gedaan is een objectclass yubikey toegevoegd aan de OpenDJ configuratie en deze een attribute yubikeyid meegegeven. Hoe ik dit heb gedaan komt nog wel aan bod in een latere entry waarin ik ook zal uitleggen hoe je users die in moeten kunnen loggen op je machine moet aanmaken in OpenDJ. De objectclass yubikey heb ik op zijn beurt weer toegevoegd als auxiliary objectclass bij de gebruikers die een YubiKey hebben. Vervolgens kan ik dan in het veld yubikeyid het Public ID ingeven van de YubiKey die bij een bepaalde gebruiker hoort. Het Public ID bestaat uit de eerste 12 tekens van de OTP die een YubiKey genereert. Je kan ook een bestaande attribute gebruiken, Zarafa gebruikt bijvoorbeeld het carLicense attribute. Maar dit gaat er bij mij niet in, een YubiKey Public ID is geen rijbewijs, dus krijgt het een eigen attribute.

Als je nu via SSH inlogt met je LDAP wachtwoord meteen gevolgd door je YubiKey OTP op dezelfde regel zou je keurig geauthenticeerd moeten worden. Eerst tegenover de Yubico Cloud, vervolgens stript de pam_yubico.so module het wachtwoord en haalt het YubiKey OTP eraf en geeft de rest door aan pam_ldap.so die je nu met het overgebleven wachtwoord authenticeert tegenover je LDAP server. En mocht je nog geen home directory hebben dan wordt deze aangemaakt door pam_mkhomedir.so.

YubiKeys, SSH, PAM en LDAP