Hi, I am the one who performed these benchmarks and I'd like to clarify a couple of things:
* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels:-)
* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.
* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png [freebsd.org]. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.
* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png [freebsd.org]. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.
* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
FYI. Your observations of mysql performance are right in line with what others including google have noticed. The famous google patches significantly improve the scalability regressions introduced. They should eventually make there way into the official Mysql in a couple months or so.
I absolutely agree. The good news is Mysql is going to open up its development process a little more to streamline the integration of changes / features from outside the company into official releases.
Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
The interesting part for me is really performance across the board for lots of other different workloads, on a variety of different systems.
If the numbers are really 15% different, that seems essentially equivalent to me. I never understood why 15% better really
Yes, I agree. We in fact track a spectrum of workloads, and try to optimize for all of them simultaneously. Sometimes there are trade-offs but this is rare. The main point you should take away from this is that FreeBSD 7.0 does not run into scalability bottlenecks on this configuration, so it can make efficient use of 8 CPUs where FreeBSD 6.x (and older Linux kernels) could not.
The scalability of the architecture is more important than a few % variation in absolute numbers, which as you say will be wiped o
The CFS may not scale well (and this needs fixing) but this is still primarily a server benchmark
I expect (but don't know) that desktop performance and the performance of other applications would be better with CFS than the FreeBSD Scheduler?...but saying that it is nice to see that FreeBSD (which I always considered the Linux that wasn't) is a contender again if nothing else it will keep the competition on it's toes...
Why do you "expect" this? The ULE scheduler is also designed with interactive (e.g. desktop) performance in mind, so your expectation seems unmotivated by prior probability and more by a prejudice that Linux must be better:-)
I expect it because the scheduler is supposed to be completely fair (it's in the name?) rather than biased towards processes that are highly active (which I would expect on a server)
Linux is a kernel, it is not a scheduler, there is nothing stopping anyone retrofitting the same algorithm into FreeBSD, or vice versa
Linux is not better just different same as OSX, Solaris, etc are different....
OK, so basically you're just going on marketing ("it says it's completely fair, that has GOT to be better than a BIASED scheduler!") and your own prior beliefs, instead of technical insight. Thanks for clarifying:)
Some clarifications on my benchmarks (Score:5, Informative)
* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels
* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.
* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png [freebsd.org]. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.
* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png [freebsd.org]. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.
* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
Hope this helps to clarify things.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
The interesting part for me is really performance across the board for lots of other different workloads, on a variety of different systems.
If the numbers are really 15% different, that seems essentially equivalent to me. I never understood why 15% better really
Re: (Score:2)
The main point you should take away from this is that FreeBSD 7.0 does not run into scalability bottlenecks on this configuration, so it can make efficient use of 8 CPUs where FreeBSD 6.x (and older Linux kernels) could not.
The scalability of the architecture is more important than a few % variation in absolute numbers, which as you say will be wiped o
Re: (Score:2)
I expect (but don't know) that desktop performance and the performance of other applications would be better with CFS than the FreeBSD Scheduler?
Re: (Score:3, Interesting)
Re: (Score:2)
Linux is a kernel, it is not a scheduler, there is nothing stopping anyone retrofitting the same algorithm into FreeBSD, or vice versa
Linux is not better just different same as OSX, Solaris, etc are different
Re: (Score:2)
Re: (Score:2)