In August this year, I ran a series of quick benchmarks using the Phoronix Test Suite, on a mission to find out: Do Custom Kernels Matter?

*Note: I kept getting distracted by other things, so I’m only writing this article a few months later. Also, this is more of a quick test than an in-depth benchmark.

There’s a lot of buzz around custom kernels on the internet, and about how some oh so dramatically improve performance. I wanted to find out if this was true, and, more importantly, if the gain was worth the hassle of compiling/installing the respective kernels.

I ran the benchmarks on EndeavourOS, since Arch Linux has support for the most custom kernels, due to the AUR. The PC they were run on was a custom-built AMD desktop.

I tested 7 different kernels: Linux Zen, Xanmod, Liquorix, Clear Linux’s, the CachyOS kernel, the TKG kernel, and, of course, the generic kernel.

Of course, there are more than just these, but I believe they are the most popular ones. Although some of them are officially only supported on Debian or similar distros, they are still in the AUR, so I tested them anyways.

The CachyOS and TKG kernels also support compiling with 6 different CPU schedulers, so, for the sake of completeness, I tested all those as well.

And yes, this means I compiled 20 kernels from source withing 5 days. The Ryzen 7 3800X hates me.

I did go through the setup process of each kernel (which is important for the Clear Linux and TKG kernels, since they provide the option to optimize the kernel for a specific CPU architecture), but I didn’t customize the kernels in any way after installation (e.g. setting the CPU scaling governor).

I tried to keep the conditions of every benchmark run the same, which meant not updating packages for a week.

Keep in mind I picked the benchmarks based on how long they would take to run (because I was impatient), and none of the benchmarks featured are actually properly designed to put the kernel under stress. Also, I wanted to see more how the kernel would impact day-to-day use, rather than “professional” database performance. So take my results with a small grain of salt.

I couldn’t include the result graphs in this article, but feel free to take a look via the hyperlinks below.

Kernel vs. Kernel

At first, I compared all the kernels against each other. You can view the results here.

Three things struck me:

1. The CachyOS kernel has more than 50% of the first place finishes, if only by a small margin.
2. The Liquorix kernel is leading the AI benchmark (TensorFlow Lite) by a noticeable margin.
3. Linux-TKG performed quite badly, even though it was optimized at compile time for my CPU.

I guess if you’re in the AI space, the Liquorix kernel might be a consideration for you, although it performs pretty badly otherwise (sharing the crown for “Most Last Place Finishes” with the Clear Linux kernel). Also, it draws more power and heats up the CPU more than the other kernels.

Overall, none of the kernels seemed so overwhelmingly better than the generic one that I would recommend them to anyone for their performance gains, the gains aren’t big enough in my opinion to justify installing an extra kernel on your system.

However, CachyOS, which ships the CachyOS kernel by default (surprise!) did a good job of optimizing it to make it just a little better.

CPU Schedulers

I then went on to compare the various CPU schedulers I had selected. I did this with the CachyOS kernel and the TKG kernel, which are the only ones to support customizing the CPU scheduler during compilation.

Luckily, the results were mostly consistent between the two, which makes it easy for me to interpret the results.

Basically, no big differences between the schedulers here either.

However, the PDS scheduler finished first place most of the time, and had the highest overall score (Geometric Mean). It also won by a respectable margin in the TensorFlow Lite workload.

This would explain why Liquorix performed so well there, since it uses the PDS scheduler.

I also noticed the BMQ scheduler performed similarly to PDS in the TensorFlow Lite benchmark, which leads me to believe they use a similar tuning or optimization.

However, BMQ didn’t really shine in the other tests, sitting more inbetween all the other schedulers.

Another thing that struck me was the Xonotic benchmark in the TKG schedulers comparison specifically: The results of this benchmark are more scattered than the other ones. The winner here, the BORE scheduler, performed about 14% better than last place (the PDS scheduler). This sort of difference was in none of the other benchmarks or comparisons.

First of all, this contradicts PDS winning everything else (I can’t really explain why). But second, it shows how the scheduler does impact gaming performance of a system.

One thing I was interested in was how the EEVDF scheduler would perform, because it replaces the CFS scheduler in the generic kernel since version 6.6.

EEDVF didn’t perform much better than CFS, but had higher CPU usage and temperatures, albeit by just a tiny margin.

I must admit I had higher hopes for it than that, but it was accepted into the generic kernel for the small performance improvement it brings, so I assume the tests I ran aren’t really representative. After all, the kernel devs probably know better than I how to properly test a CPU scheduler.

Conclusion

I ran these benchmarks to find out if custom kernels improve performance enough to justify installing and maintaining a kernel not officially supported by the Linux distro you’re running.

The answer for me is: No. The performance gains were mostly minimal, and hardly consistent across kernels.

The most interesting things I noticed, though:

1. I was quite disappointed by the TKG kernel. It performed so badly even though it allows for the most optimization during compilation.
2. The CachyOS kernel seems pretty well optimized, it came out on top most of the time. Since CachyOS is also a distro, not just a kernel, I’m interested to see where this project will go in the future.
3. Liquorix improves performance on AI workloads quite a bit, but at the cost of overall higher CPU usage and temperatures.

In the end, you have to decide if you want to try out a custom kernel after seeing these results. Maybe you’re an avid AI dev, and Liquorix would speed up your workflow nicely, or you want to squeeze out all the frames you computer can give you, then the CachyOS kernel might be something for you.

And don’t forget: Fun, experience and trying out new things are all still valid reasons to try out a custom kernel.