I'd be interested to see results from pre-CFS kernels.
Not that FreeBSD hasn't made major performance improvements.
Also, I think that a database test isn't a complete picture. For example, some OSes like IRIX or Mac OS X perform very well on streaming of local video and audio, but I wouldn't benchmark Oracle or PostgreSQL on either.
And I like the article summary stating that FreeBSD may now be considered "a serious performance contender". Like FreeBSD was 1000% slower than Linux? Most servers spend their time spinning their wheels anyway, generally I'd rather look at security, how it handles under load and other metrics than whatever "performance" is considered in this instance. Linux is good for some things, BSD for others.
About the only really good news here is that MySQL performance is actually adequate. As MySQL has always been a dog (usable, but a dog) on FreeBSD, the general rule of thumb was that if you needed MySQL you should stick to Linux; all other factors being equal. So now at least we can get down to other factors that are important instead of one database that performs poorly on one system.
Acoording TFA, FreeBSD didn't even have proper SMP support until relatively recently. Until 5.0, released in 2003, only one CPU could run kernel code at once. (To be fair, I think Linux still hasn't fully got rid of its own Big Kernel Lock, but...)
Having a threaded kernel or having fine-grained locking is hardly a reasonable barrier for "proper SMP support." If I have two processors and they can be both running code at the same time, it's SMP support - everything else is just icing.
On virtually every CPU bound application, the time is spent not in the kernel, but in userland.
The linked PDF contains pre-CFS kernel benchmarks.
Prior to 2.6.23, the 2.6 kernel used the Completely Fair Queuing (CFQ) scheduler, which was also bad for database workloads. Please see this note by Werner Puschitz [puschitz.com]:
The Completely Fair Queuing (CFQ) scheduler is the default algorithm in RHEL4 which is suitable for a wide variety of applications and provides a good compromise between throughput and latency. In comparison to the CFQ algorithm, the Deadline scheduler caps maximum latency per request and mainta
CFS is a thread scheduler, CFQ is a disk IO scheduler. They are independent but can coexist (and in many modern distro releases, do so by default). Both seem to be focused on desktop user experience, although they're not exactly "bad" for servers either.
You also wouldn't run Oracle or PostgreSQL on them unless you wanted to lose your job. The test is a great test for linux vs. freebds, because it's the type of environment where they'd compete.
That benchmark shows some sound methods and shows Linux ahead in performance. As TFS states, a benchmark or two doesn't mean too much. What one can take away from this is that under some tests a prerelease of FreeBSD beats a released Linux kernel and under some others a prerelease Linux kernel beats a prerelease FreeBSD kernel. That means there's probably not enough difference by the time both kernels are released to declare a clear winner based on performance, and if so then not by much. Other factors are
Just to clarify, the benchmark document says October 2007, and the release was February 2008. Where's the changes file to show what changed in the meantime? Plus, the tested scheduler is not yet the default scheduler in the released version.
The article also describes a FreeBSD 7.0 pre-release from October last year. This still had debugging code turned on in the builds, as mentioned on the NetBSD lists when Andrew Doran was comparing NetBSD -current SMP performance.
Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
My rule of thumb for upgrading equipment has been to not bother until we hit a full order of magnitude improvement. In other words, if 1) we can 10X the performance of a system AND 2) there have been complaints about performance, then the upgrade is probably worth it. Even then, the value is dubious. For example, in Postgres, (or any other database application) it's very typical to see 100x improvement simply by creating an index!
Maybe this is good for frail BSD egos, who have been long bruised by the mindshare success of Linux over the more historic and "more free" BSD. So be it. But it's not performance that's kept me from using BSD, it's familiarity and the pain of switching. And that's also what kept me running it yesterday, will today, and tomorrow too.
Don't get me wrong - I would hate to see BSD "die" in any meaningful way. The different cultures between Linux and BSD create a very rich, diverse environment where ideas can be tested, and the cross-feed of proven concepts and technologies (EG: Open SSH) benefits all involved!
But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!
But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!
So if you are google, and your software will all 100% run with Linux or BSD, you don't see the idea that 15% better performance means the same work with 15% less machines means something? In certain cases, 15% can mean thousands or millions of dollars, all for changing to an operating system that will basically run on the exact same hardware and run the exact same software, after a recompile.
No for most it isn't a big deal and may not make people CHANGE operating systems on existing hardware. We use both Linux and BSD, so it *might* make me consider BSD instead of Linux on the next new box. I'm likely not alone in this.
I would think the cost of changing the OS in an environment large enough such that a 15% performance increase could result in millions of savings might be an expensive proposition on its own. Despite the impression many home users or small shops have, changing an OS in a large environment is far from "free".
Compare 15% against Moore's law, and you find that it equates to a few weeks delay in the price-performance curve.
If it takes more than a few weeks to make the switch, you've already lost your benefit, as well as the potential of destabilizing your administration of those systems. Backups have be revisited, since the file tree will have changed. Network monitors will have to be updated, and tested for compatibility changes. Little one-off scripts to solve problem X or Y in a hurry will break. Admins will have to be trained, and will make more mistakes for a while until they find out what not to do. Unforeseen wrinkles will inevitably appear, Etc... Etc... Etc...
But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!
Unless it was Linux that recently edged ahead with a 15% increase, then every product manager with a Linux fanboy in his company would be indundated with demands to switch. I agree with your basic premise that one shouldn't blindly switch operating systems based on a few benchmarks. But I do have to disagree with your implication that only BSD user have ego problems.
Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
Obviously, it depends on cost:benefit, which is exactly why there is no rule of thumb. Blanket generalisations and "rules of thumb" are a bad way of making a decision for everyone that
Wrong! Commersial Database vendors will give their eye teeth to get 15% higher TPC-C scores. 15% isn't something to sneeze at when it comes to benchmarks. Now if you can get the 15% for "free" by switching to a different OS to submit your scores on.... They 100% will.
Yes, I meant that: who cares?
Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
I only use Linux as a desktop OS. I don't know much about the implications of this particular benchmark, but I care a lot about the current schedule work, for the simple fact that they have been causing me problems for my desktop use.
To be honest, when using Linux for database applications, you shouldn't use the Completely Fair Scheduler. I build and tune Oracle database servers for a living, and I've researched this very thoroughly. Take it from the master himself, Mr. Werner Puschitz [puschitz.com]:
The Completely Fair Queuing (CFQ) scheduler is the default algorithm in RHEL4 which is suitable for a wide variety of applications and provides a good compromise between throughput and latency. In comparison to the CFQ algorithm, the Deadline scheduler
I'd be interested to see results from pre-CFS kernels.
And I'd love to see how the last -ck patchset compares. CFS began as a politics-driven copy of RSDL, so it would be nice to see which actually performs better. Just to conclude that particular flamefest...
The closest to perfection a person ever comes is when he fills out a job
application form.
-- Stanley J. Randall
Well (Score:5, Interesting)
Not that FreeBSD hasn't made major performance improvements.
Also, I think that a database test isn't a complete picture. For example, some OSes like IRIX or Mac OS X perform very well on streaming of local video and audio, but I wouldn't benchmark Oracle or PostgreSQL on either.
Re:Well (Score:4, Interesting)
About the only really good news here is that MySQL performance is actually adequate. As MySQL has always been a dog (usable, but a dog) on FreeBSD, the general rule of thumb was that if you needed MySQL you should stick to Linux; all other factors being equal. So now at least we can get down to other factors that are important instead of one database that performs poorly on one system.
Re:Well (Score:5, Funny)
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Having a threaded kernel or having fine-grained locking is hardly a reasonable barrier for "proper SMP support." If I have two processors and they can be both running code at the same time, it's SMP support - everything else is just icing.
On virtually every CPU bound application, the time is spent not in the kernel, but in userland.
Re:Well (Score:5, Informative)
Re: (Score:3, Funny)
Re: (Score:2)
Prior to 2.6.23, the 2.6 kernel used the Completely Fair Queuing (CFQ) scheduler, which was also bad for database workloads. Please see this note by Werner Puschitz [puschitz.com]:
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
Re: (Score:1, Informative)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
From TFA... (Score:2)
Short version:
Linux pre-CFS is faster than post-CFS, but FreeBSD still comes out ahead, by maybe 5%.
Re:From TFA... (Score:4, Informative)
The article also describes a FreeBSD 7.0 pre-release from October last year. This still had debugging code turned on in the builds, as mentioned on the NetBSD lists when Andrew Doran was comparing NetBSD -current SMP performance.
Seriously: who cares? (Score:4, Interesting)
Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
My rule of thumb for upgrading equipment has been to not bother until we hit a full order of magnitude improvement. In other words, if 1) we can 10X the performance of a system AND 2) there have been complaints about performance, then the upgrade is probably worth it. Even then, the value is dubious. For example, in Postgres, (or any other database application) it's very typical to see 100x improvement simply by creating an index!
Maybe this is good for frail BSD egos, who have been long bruised by the mindshare success of Linux over the more historic and "more free" BSD. So be it. But it's not performance that's kept me from using BSD, it's familiarity and the pain of switching. And that's also what kept me running it yesterday, will today, and tomorrow too.
Don't get me wrong - I would hate to see BSD "die" in any meaningful way. The different cultures between Linux and BSD create a very rich, diverse environment where ideas can be tested, and the cross-feed of proven concepts and technologies (EG: Open SSH) benefits all involved!
But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!
Re:Seriously: who cares? (Score:5, Interesting)
So if you are google, and your software will all 100% run with Linux or BSD, you don't see the idea that 15% better performance means the same work with 15% less machines means something? In certain cases, 15% can mean thousands or millions of dollars, all for changing to an operating system that will basically run on the exact same hardware and run the exact same software, after a recompile.
No for most it isn't a big deal and may not make people CHANGE operating systems on existing hardware. We use both Linux and BSD, so it *might* make me consider BSD instead of Linux on the next new box. I'm likely not alone in this.
Re: (Score:2)
Re: (Score:2)
but migrating new machines that are constantly be added weekly is much less expensive, and ROI *might* be reasonable in the right circumstances.
Re:Seriously: who cares? (Score:4, Insightful)
If it takes more than a few weeks to make the switch, you've already lost your benefit, as well as the potential of destabilizing your administration of those systems. Backups have be revisited, since the file tree will have changed. Network monitors will have to be updated, and tested for compatibility changes. Little one-off scripts to solve problem X or Y in a hurry will break. Admins will have to be trained, and will make more mistakes for a while until they find out what not to do. Unforeseen wrinkles will inevitably appear, Etc... Etc... Etc...
Worth it for Google? Not a chance!
Re: (Score:2)
Unless it was Linux that recently edged ahead with a 15% increase, then every product manager with a Linux fanboy in his company would be indundated with demands to switch. I agree with your basic premise that one shouldn't blindly switch operating systems based on a few benchmarks. But I do have to disagree with your implication that only BSD user have ego problems.
Re: (Score:3, Insightful)
Obviously, it depends on cost:benefit, which is exactly why there is no rule of thumb. Blanket generalisations and "rules of thumb" are a bad way of making a decision for everyone that
Re: (Score:1)
Re: (Score:1)
Yes, I meant that: who cares? Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
I only use Linux as a desktop OS. I don't know much about the implications of this particular benchmark, but I care a lot about the current schedule work, for the simple fact that they have been causing me problems for my desktop use.
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094/ [launchpad.net]
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
And I'd love to see how the last -ck patchset compares. CFS began as a politics-driven copy of RSDL, so it would be nice to see which actually performs better. Just to conclude that particular flamefest...