Linux vs FreeBSD Disk I/O: Why Is FreeBSD Faster?

General Discussions
, ,
1

I manage Linux for a living, I do Linux side jobs, and maintain a home lab to test the sorts of things I might want to propose at work, or do for clients.

Recently, I’ve begun doing testing on FreeBSD in relation to the Mailguard anti-spam system I’ve been maintaining.

Linux and FreeBSD are comparable in network performance, both very good, but
I was blown away at what a beast FreeBSD is at disk I/O. Every test I’ve run has FreeBSD with 50% higher throughput no matter what sysctl parameters or scheduler is used. I’ve tried all the tuned modes, and there are small differences, but nowhere near the amount needed to catch up to FreeBSD. On the other hand, Linux seems to have better latency.

Any hints on maximizing Linux disk throughput?

4 Likes
2

Great topic! Hm. It’s likely because FreeBSD often runs with a leaner default stack, so I/O passes through fewer layers.

Linux, on the other hand, offers more flexibility and features, which adds overhead: The I/O scheduler, blk-mq, cgroups, dm layers, LVM, md, and filesystem journaling, and you can end up with queues and locking a lot easier.

It’s still very fast! But yes, Linux can be slower than FreeBSD out‑of‑the‑box in some storage benchmarks and configurations.

As far as closing the gap, it’s a bit outdated now and only loosely related, but some of the tips may help:

In addition, I would say make sure to set the CPU governor to a max performance mode. If the CPU is sitting at 800 MHz because the governor downclocked it, storage I/O operations may be slower. It really depends on the exact hardware, but low‑frequency/powersave modes can absolutely hurt I/O latency and throughput compared to FreeBSD.

FreeBSD will tend to sit at or near a high non‑turbo frequency unless you explicitly enable powerd[xx] or tune power settings.

For NVMe SSDs, the none I/O scheduler is usually the best choice, and it is the default on most distributions.

Also check if your tests are passing through dm-crypt or any extra layer that might be running single threaded. FreeBSD handles a lot of this work in parallel, so Linux needs a bit of manual tuning to keep up.

You could say FreeBSD out of the box is like a sports car already in sport mode. It gives you raw power with nothing extra stepping in.

Linux out of the box is the same sports car, but it starts in comfort mode with more systems active. More flexibility, options/features, but you also feel a bit less of that straight power until you tune it.

4 Likes
3

:nerd_face: :backhand_index_pointing_up: actually
freeBSD would eat linux alive in performance comparison
but linux has bigger community and more mainstream
I won’t say it’s ULE scheduler or zfs but it’s the kernel optimization overall.

1 Like
4

It doesn’t appear to be not ZFS, as FreeBSD and Linux now use the same openzfs codebase.

It could be the scheduler for sure, and no doubt kernel optimization plays a part.

But in my testing I don’t see any real difference between Linux and FreeBSD in terms of network performance.

So it has to be something to do specifically with I/O scheduling/algorithms.

2 Likes
5

Now I understand why TrueNAS was built with FreeBSD. I wonder why they choose to move over to Debian? Thank you all for this nice tidbit.

2 Likes
6

One distro that keeps coming up in conversations like this is CachyOS. It kind of makes sense why it has blown up over the last year: Their whole goal is squeezing out more speed everywhere they can, and that includes disk IO.

They ship with the right scheduler defaults for NVMe, they have IO scheduler rules that automatically pick the best option for each device type, and they even have their own scheduler called ADIOS that’s tuned for modern multi-queue setups. So they’re clearly thinking about performance at the block layer, not just desktop responsiveness.

I haven’t tested CachyOS for raw disk IO myself yet. I’ve only clicked around the desktop to get a feel for it.

It does look like a solid pick for anyone who’s trying to get better throughput or wants a distro that focuses heavily on tuning and tweaking for faster IO.

1 Like