Compiling a Custom Kernel on Fedora on Raspberry Pi

In the previous piece I showed how a current 64-bit Fedora distribution can be made to run on a Raspberry Pi 4. There was a catch though — we used the boot partition and the kernel from the Raspbian OS.

In this writeup we will replace this with our own kernel, and optionally add additional kernel features.

It is common to use an “initial ramdisk” — using either the initrd or initramfs scheme — an archive file containing a small filesystem and in it some drivers and tools — to equip a kernel at boot-time with extra capabilities (such as loading RAID or SAN drivers) before mounting the primary operating system disk. Adding this is possible, but is outside the scope of this article.

We will not use a mainline kernel from kernel.org, as some of the Raspberry hardware support is still only available in the downstream repository maintained by the Raspberry Pi people. We will instead use their latest release.

We will compile this kernel locally right on our arm64 Fedora, running on the Raspberry Pi 4 build in the previous writeup. This is an order of magnitude faster than using an arm64 emulator… and tidier than using a cross-compiler on a PC.

To keep this really simple, we will do this as root. And we’ll go old school and use menuconfig, because it’s just as awesome today as it was 25 years ago.

So, first things first:
Step 1: Grab the sources

Browse https://github.com/raspberrypi/linux/releases, drill into the latest release (raspberrypi-kernel_1.20200212–1 as of the writing of this piece), and at the bottom of its release page, grab the link of the tar.gz package. Then, on your raspberry:

# cd /usr/src
# wget https://github.com/raspberrypi/linux/archive/raspberrypi-kernel_1.20200212-1.tar.gz
# tar zxvf raspberrypi-kernel_1.20200212-1.tar.gz
...
# ln -nfs linux-raspberrypi-kernel_1.20200212-1 linux
# cd /usr/src/linux

Step 2: Install some tools

# dnf -y install gcc flex make bison openssl-devel elfutils-libelf-devel ncurses-devel

Step 3: Prepare a configuration file.

If you followed the process from the first writeup, you should have Raspbian kernel’s driver modules under /lib/modules/4.19.97-v8+

We will load a nice little module from there called configs

# modprobe configs

As a result, /proc/config.gz will be available. This is compressed text containing your running kernel configuration (e.g. which bits to compile in, which to compile as modules on the side, and which to leave out). We will extract it straight into our kernel source tree.

zcat /proc/config.gz > /usr/src/linux/.config

Run the menuconfig make target:

# cd /usr/src/linux
# make menuconfig

You can just go ahead and save the configuration as is. If it’s your first time doing this, I highly recommend you browse around. This gives an idea as to how expansive the Linux kernel is — including the many bits that don’t end up built into popular distributions like Fedora or Debian, and how much diverse stuff one can enable and get out of it, from drivers to packet filter capabilities, exotic filesystems, and upcoming technologies.

There are indeed many things you can do here — remove things you know you will never use — or add things you think you need. After you save your configuration, you will be ready to build.

For my nefarious purposes, I came here to add a few specific things — SELinux and CGroups. However, for these options to appear, their prerequisites also need to be ticked in.

To add these particular ones via menuconfig, I went to Security Options, and enabled “Enable different security models”, and in there enable “Socket and Networking Security Hooks. In “General Setup” I enabled “Auditing Support”. Under “Security” again, I enabled “NSA SELinux Support”.

Of course, we can just add the lines you want to .config directly— namely, in the case of my stuff, this:

CONFIG_AUDIT=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_GENERIC=y
CONFIG_AUDIT_COMPAT_GENERIC=y
CONFIG_INTEGRITY_AUDIT=y
CONFIG_INTEGRITY=y
CONFIG_LSM_MMAP_MIN_ADDR=32768
CONFIG_NETWORK_SECMARK=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=0
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_HOOKS=y

I got the above list by running a diff between the original .config and the one I saved in menuconfig right after ticking in the SELinux stuff.

Of course, you can do a million other things in here, adding drivers, changing things like the XFS filesystem driver from being a loadable module to being compiled into the kernel itself, or removing dead weight to make your kernel ever so lean.

Once done, save your .config file.

Step 4: Build the kernel and its modules

All we have to do now is build the kernel image (the uncompressed kernel Image make target — to keep in line with how the Raspberry people did it) and the modules (the modules and modules_install make targets).

# make Image 
# make modules 
# make modules_install

Note that the Pi 4 has four CPU cores, so running several jobs in parallel can speed things up four-fold (optimum jobs being core count +1, so 5). I got what appears as a race condition on my first time doing this, (An cp: cannot stat ‘./modules.order’: No such file or directory error)… maybe because I ran multiple targets at once. This didn’t happen from the second run onward and sped things up substantially:

# make -j5 Image 
# make -j5 modules 
# make -j5 modules_install

Another issue with this particular release is this error:

[root@localhost linux]# make Image modules modules_install
...
modules_install/usr/bin/ld: scripts/dtc/dtc-parser.tab.o:(.bss+0x10): multiple definition of `yylloc'; scripts/dtc/dtc-lexer.lex.o:(.bss+0x38): first defined here

That’s really not a biggie. If you encounter it, fix it so:

sed -i 's/^YYLTYPE yylloc;$/extern YYLTYPE yylloc;/'  /usr/src/linux/scripts/dtc/dtc-lexer.l

… and re-run the make command to keep going.

This will run for a while and produce a new uncompressed kernel in /usr/src/linux/arch/arm64/boot/Image as well as modules in /lib/modules/<kernel-name>, which, in the case of the kernel used for this writeup, is in /lib/modules/4.19.97-v9. Note that since the original raspbian kernel calls itself “4.19.97-v9+”, its modules live in /lib/modules/4.19.97-v9+ next door to where we’re installing ours.

Step 5: Install the kernel

The common way to “Install” this kernel — e.g. ask the active boot loader to offer it as a boot option at next boot — (make install — which runs the arch/arm64/boot/install.sh script) — we will not use here.

This is because the install script doesn’t know how to handle the Raspbian vfat boot partition (which doesn’t support symlinks) and stuff will break if you run this. Keep in mind neither grub nor uboot is being used here as a boot loader. We will take the low-tech approach instead — we will simply copy the old kernel to a backup, and the new one on top of the old one’s filename.

# cp -vrfp /boot/kernel8.img /boot/kernel8.img.bak
# cp -vrfp /usr/src/linux/arch/arm64/boot/Image /boot/kernel8.img

We’ll also copy our new kernel’s System.map file to /boot to enable resolving kernel symbol names and addresses:

# cp -vrfp /usr/src/linux/System.map /boot/System.map

That’s it.

Reboot the pi and you should be done.

To check it all works, make sure your pi boots and you can get back on.

Once you do, run

# uname -r 

To make sure the version of kernel you booted is the right one (in our case this will say “4.19.97-v9” if we succeeded, and bear the original Raspbian kernel name — “4.19.97-v9+” — if we got something wrong.

Then run

# lsmod

If you get a list of modules that are already loaded, you’re golden. If you get an empty list, the kernel can’t load modules.

Congrats. Within the limitations of the Raspberry Pi kernel source tree, you are now in control of your kernel.