FreeBSD 7.0 Bests Linux In SMP Performance

cecom writes "After major improvements in SMP support in FreeBSD 7.0, benchmarks show it performing 15% better than the latest Linux kernels (PDF, see slides 17 to 19) on 8 CPUs under PostgreSQL and MySQL. While a couple of benchmarks are not conclusive evidence, it can be assumed that FreeBSD will once again be a serious performance contender. Some posters on LWN have noted that the level of Linux performance could be related to the Completely Fair Scheduler, which was merged into the 2.6.23 Linux kernel." Update: 03/06 21:32 GMT by KD : An anonymous reader sent in word that Linux kernel developer Nick Piggin reran the benchmark today and came to a different conclusion: In his benchmark Linux was faster than FreeBSD.

Well

I'd be interested to see results from pre-CFS kernels.

Not that FreeBSD hasn't made major performance improvements.

Also, I think that a database test isn't a complete picture. For example, some OSes like IRIX or Mac OS X perform very well on streaming of local video and audio, but I wouldn't benchmark Oracle or PostgreSQL on either.

Re:Well

And I like the article summary stating that FreeBSD may now be considered "a serious performance contender". Like FreeBSD was 1000% slower than Linux? Most servers spend their time spinning their wheels anyway, generally I'd rather look at security, how it handles under load and other metrics than whatever "performance" is considered in this instance. Linux is good for some things, BSD for others.

About the only really good news here is that MySQL performance is actually adequate. As MySQL has always been a dog (usable, but a dog) on FreeBSD, the general rule of thumb was that if you needed MySQL you should stick to Linux; all other factors being equal. So now at least we can get down to other factors that are important instead of one database that performs poorly on one system.

Re:Well

Most servers spend their time spinning their wheels anyway

And then along comes a /. article...

Re:Well

The linked PDF contains pre-CFS kernel benchmarks. Conclusion:

The new CFS scheduler in 2.6.23 is "Completely Fair" ...to FreeBSD

Re:Seriously: who cares?

But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!

So if you are google, and your software will all 100% run with Linux or BSD, you don't see the idea that 15% better performance means the same work with 15% less machines means something? In certain cases, 15% can mean thousands or millions of dollars, all for changing to an operating system that will basically run on the exact same hardware and run the exact same software, after a recompile.

No for most it isn't a big deal and may not make people CHANGE operating systems on existing hardware. We use both Linux and BSD, so it *might* make me consider BSD instead of Linux on the next new box. I'm likely not alone in this.

It's that year again!

Maybe now we can finally declare year of the linux desktop!

Wait, what?

Re:BSD Desktops

FreeBSD is not fragmented like the 100 and 1 Linux distros

I'm a FreeBSD fan, but what kind of logic is that? You pick one example out of a fragmented set, and compare it to an entire other set of operating systems.

You act as if NetBSD, OpenBSD, DragonflyBSD, Darwin, etc, do not exist. Of course and item cannot be fragmented if if you define it's containing set as "itself". Makes about as much sense as:

Ubuntu is not fragmented like the 100 and 1 BSD distros

Re:BSD Desktops

No, those are forks. FreeBSD is not just a kernel like Linux. FreeBSD is the entire package. You don't have other distros, you have forks of the original.

Well, both PC-BSD and DesktopBSD claim to not be forks. DesktopBSD explicitly identifies itself as a distribution of FreeBSD and PC-BSD says "PIF" (PC-BSD is FreeBSD). Here are the references: [pcbsd.org]http://faqs.pcbsd.org/index.php?action=artikel&cat=14&id=304&artlang=en [pcbsd.org]

http://desktopbsd.net/wiki/doku.php?id=doc:faqs [desktopbsd.net]

Of course, I appreciate the fact that FreeBSD's base is an integrated system which is maintained as a unit as opposed to Linux distributions which are sourced from multiple projects.

I look forward

to the enlightening and respectful conversation this article will provoke.

Re:I look forward

Screw you!

Wait, you forgot to mention ... you use BSD or Linux ?

I have Vista

but I heard it wasn't compatible with Windows or labtops.

Can you help? Sorry, I am not good with computers. :) I want to download the internet onto my labtop.

In short, wow

Very, very nice scaling performance under PGSQL is evident in the PDF, and I've no reason to assume the benches aren't legit. I think part fo the reason that PG was traditionally slower than MySQL was that it did lots of complicated locking to provide better scalability across processors, whereas we see MySQL performance dropping off after we go to more than eight cores. I think this was the same philosophy Sun took with "Slowaris", which was also far more scalabe than Linux at the time the moniker was in widespread use.

Still, I hope Linux can at least match this sort of superb scalability. CFS is fairly new, and I know there's optimisation work been done to it in .24 and .25, although it was a little sad to see the first iteration of CFS performing more poorly than its predecessor (and, if this is the case, I can see why Linus stonewalled CK's patches for so long, since they were mainly tested on desktop workloads). Are there any apples-to-apples comparisons out there that test various flavours and versions of Linux and BSD with a wide range of benchmarks? At the best review sites do a few benches with MySQL, and six months later everything has changed so it's incredibly difficult to do good performance comparisons.

Even so, it's refreshing to see precious little of the "BSD fudged their benchmarks!" trollspeak in the LKML thread, and plenty of talk about how to make Linux better. Open source is hippy capitalism - it also needs healthy competition to keep it in check :)

Offtopic: bug linked to in the LKML pointed me at this http://www.latencytop.org/ [latencytop.org] Sounds quite nifty

Some clarifications on my benchmarks

Hi, I am the one who performed these benchmarks and I'd like to clarify a couple of things:

* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels :-)

* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.

* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png [freebsd.org]. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.

* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png [freebsd.org]. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.

* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.

Hope this helps to clarify things.

Re:You don't have to be Kreskin

Nothing against Linux, but it's hardly an apples to apples comparison.

Exactly. I mean, who cares about performance on a dying OS? ;)

Re:You don't have to be Kreskin

Why on earth do you think FreeBSD is a dying OS?

'
Because netcraft confirms it.

Re:You don't have to be Kreskin

It probably has a lot to do with FreeBSD having a much more focused niche. FreeBSD is really tuned primarily for servers. You can use it on your desktop, but that's not really it's main purpose. Linux on the other hand, has really branched out. It has desktop distros, server distros, embedded distros, and probably a couple other areas I haven't thought of.

Re:You don't have to be Kreskin

You think so? I dunno, it seems to me that FreeBSD suits the desktop role really well; I use it for preference. Especially when you consider that the only OS with more packages is Debian, it makes sense that it can fit a desktop role extremely nicely.

Re:You don't have to be Kreskin

I dunno, it seems to me that FreeBSD suits the desktop role really well

It does (I use it too) BUT only in specific environments. FreeBSD hardware support is not bad, but it is nowhere near as complete as that found in the various Linux distro's. My wireless keyboard + mouse is supported under any recent Linux distro, on FreeBSD, only the keyboard works (fixable with a unofficial ums.ko though). No support under FreeBSD for my DVB-C PCI card either.

Kreskin sez

Linux is actually better than BSD because you can roast marshmallows over the schedular flamewars.

Re:finally

Your comment would have been funny if we were talking about NetBSD. NetBSD is the one that runs on everything, including a toaster.

Re:finally

Hello. I don't believe you know me, but you might be able to help. I use Ubuntu Linux, and I am helping the daughter of a friend to install Linux onto her PC this evening. I have never really spoken to a girl before, and was thinking that maybe if I made a joke, it might break the ice. You are obviously excellent at humour, and I was wondering if you could give me some tips? For instance, it is inevitable that something will go wrong during the process, and I was thinking that maybe if I said something like 'this is almost as unreliable as my beard trimmers!' then it would demonstrate that I am a funny guy. Can you offer any kind of critique of this line, or offer any other advice?

Re:Bad news for Linux?

Benchmarks are almost but not completely useless. In this particular setup, FBSD 7.0 runs postgres doing some specific set of queries faster than Linux.

Its a safe bet Linux will do some other set of things faster than FreeBSD does them, possibly even another specific set of PostgreSQL queries for that matter. Linux is definately more concerned with desktop app performance. I can say this safely because Linux actually cares about it, FreeBSD does not. Its there to serve, not run X. It will run X, and if they see a way to make performance better for the desktop apps AND the server apps, then it may go in the source tree. If its going to hurt the server side, don't bet on it.

While I use FreeBSD for my servers because its got a clean filesystem layout and is designed to be a server OS, I'd be willing to bet that someone with deep knowledge of PostgreSQL on Linux could give it a run for its money by tweaking the kernel for server performance.

Re:Bad news for Linux?

The article is probably misleading (surprise, surprise), as the tuning documentation for PostgreSQL *states* that good IO performance has more of an impact than good CPU performance. Additionally, some other information I've read (search for postgres tuning/optimi(z|s)ation online) recommends FreeBSD because of its strong IO performance. I'll go out on a limb and assume that MySQL's performance attributes are similar.

In my opinion, the article summary is a pretty big red herring because the SMP performance may not have a huge impact on the result.

Re:Bad news for Linux?

For one, CFQ is not supposed to be an optimized I/O scheduler for database loads. That's where the Deadline scheduler comes in. You wouldn't want a "Fair" scheduler on your database server, as you would end up putting the database in I/O wait to handle lower priority processes.

Re:Possibly

The beastie is not evil! Just because he has horns, a pitchfork and looks like the devil doesn't make him evil!

You can tell by his smile. It doesn't look freakish or anything ... does it ...