Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Operating Systems Software BSD

NetBSD 2.0 vs FreeBSD 5.3 Benchmarks 110

Posted by CowboyNeal from the default-install-shootout dept.
diegocgteleline.es writes "According to OSnews, Gregory McGarry benchmarked NetBSD 2.0 against FreeBSD 5.3 and found that NetBSD 2.0 surpasses FreeBSD 5.3 in most of benchmarks. The machine used for benchmarks is a 3 Ghz P4 so it doesn't reflect the improvements of FreeBSD 5.3 in the SMP arena, which is where their developers have put their efforts in the last years and where NetBSD is still using a "big-lock" model. Newsforge is also carrying a interview with some NetBSD developers about the technology behind NetBSD 2.0."
This discussion has been archived. No new comments can be posted.

NetBSD 2.0 vs FreeBSD 5.3 Benchmarks More

NetBSD 2.0 vs FreeBSD 5.3 Benchmarks

Comments Filter:

  • Benchmarks valid? (Score:4, Interesting)

    by marcovje ( 205102 ) on Friday January 07, 2005 @04:30AM (#11285921)

    FreeBSD's focus is typically on overall throughput of massive server, with responsetime as a tradeoff, while this benchmark with all of its timings of single OS operations is more something realtimish?

    If NetBSD aspires to be a RT focussed distro/OS, they'd better have benchmarked it against some Linux with RT patches?

    • Re:Benchmarks valid? (Score:5, Informative)

      by molnarcs ( 675885 ) <csabamolnarNO@SPAMgmail.com> on Friday January 07, 2005 @07:29AM (#11286395) Homepage Journal
      Also, it must be emphasised that these are microbenchmarks. Robert Watson explains [freebsd.org] nicely the situation:

      Since the performance optimization on FreeBSD for the last few years, with features like SMPng, libpthread, and UMA, has been focussed on macro performance not micro performance, it's not surprising the micro performance requires tuning. However, there is lots of on-going work on this front so I'd expect to see continued improvement in the immediate (5.4) life time. There are a number of optimizations in 6.x that are on the merge path for 5.x that will directly impact the results in these measurements -- in particular, what is clearly a bug in the way mutexes are released on UP kernels that adds almost a hundred cycles to every mutex release operation. This was identified in my micro-benchmarking shortly after the release of 5.3, and may play a substantial part in the posted results, especially for the very micro benchmarks that involve kernel memory allocation.

      It is also unfortunate, that the otherwise excellent benchmark, when mentioning the barrage of criticism leveled at freebsd (of which 90% is posted by random and anonymous trolls on both ./ and osnews I might add) he refers to the "article" of this guy. [slashdot.org] This was my reply [slashdot.org] - and his adequate asnwers seems to be putting me on his foe list (lol.)

      Anyway, I don't want' to downplay the importance of this benchmark (nor does rwatson, if you read the whole of his post) - in fact, I'd like to see more of these coming. And overall, it was a good reading :)

      • Re:Benchmarks valid? (Score:5, Interesting)

        by marcovje ( 205102 ) on Saturday January 08, 2005 @06:45AM (#11296005)

        Correct. And this is how it went with 4.x either. 3.6 was significantly better than the first 4.xRC.

        It's not even something of FreeBSD in particular. In the project we are RC'ing for 2.0, and all kinds of detail benchmarks are popping up.

        However overall, for significant programs 2.0.x is generally faster than 1.0.x even when 2.0.x doesn't even use optimizations (the project is a compiler).

        The failure at microbenchmarks is just that the 1.0.x branch had 6 revisions in 3-4 years, some a full year tuning apart even.

  • I use them (Score:4, Interesting)

    by brilinux ( 255400 ) on Friday January 07, 2005 @04:31AM (#11285923) Journal
    As an OS junky, I have used both of these BSDs, and am very impressed with them. I use FreeBSD on my P4 3GHz laptop, as Project Evil lets me use the WiFi card nicely, and ACPI kind of works. I use NetBSD on my Sparc Station 5 and my Athlon desktop, and I have found both to be wonderful for desktop use and as servers. I salute both of the groups of people who make these OSes, and wish that I had more time to contribute.
  • I would assume that FreeBSD has a "default install" as tested that supports at least 2-way SMP, given the focus that has been made in FreeBSD. This compared with NetBSD could make a huge difference, is NetBSD's default install also setup for SMP. It is a known fact that using an SMP compiled kernel on pretty much ANY OS will result in slower performance on a single processor, as the added overhead of locking data structrues will have to occur.

    • Re:One flaw with tests (Score:4, Interesting)

      by TheGratefulNet ( 143330 ) on Friday January 07, 2005 @07:21AM (#11286368)
      no.

      in freebsd (I know this for a fact, I've gotton burned by it before) - if you compile a kernel for SMP, its NOT like linux in that it will fallback to UNIprocessing. it does NOT fallback! it hangs!

      when moving an SMP custom compiled kernel to a non-smp box, you MUST rebuild the kernel or boot from an alternate non-smp kernel.

      that sucks - but that's how it is. at least it is when I move a p6-built kernel from my dual xeon box to my single k8 box. both DO accept p6 as the ARCH type but the SMP code just causes a lockup upon boot.

      (been running freebsd at home and at work for about 3 yrs now; and linux for about 10)

      fyi
      • I know on the older Linux (not sure if it applies today), if you had an SMP kernel, and moved it to a uniprocessor, you had to have an APIC compatible with the SMP configurations for it to work. I accept that I could very well be wrong (as other comments have suggested), although optimization and coding for SMP can and probably will cause performance decreases for uniprocessor machines now matter how you slice it. An algorithem that works very well in a uniprocessor or "monolithic lock" situation could tu
        • on all linux kernels I have built (start at 1.2), enabling smp was ALWAYS safe. I can say that to linux's favor.

          on freebsd 4.x (at least) I can say this isn't true. you enable these 2 for smp:

          # To make an SMP kernel, the next two are needed
          options SMP # Symmetric MultiProcessor Kernel
          options APIC_IO # Symmetric (APIC) I/O

          but it might be the apic option that causes the lockup. doesn't matter since both are required and both=on means an single cpu system will crash on bootup with t

          • I think you may be right. The APIC_IO code in Linux I think would automatically downgrade even if selected, but in BSD it didn't. If you happen to have had a dual processor motherboard with a single processor loaded, BSD would work fine, or one with a chipset that could support SMP, but didn't have the socket for it. It's been a while since I had to mess with servers to this level, having moved on to networking.

            • I think you may be right. The APIC_IO code in Linux I think would automatically downgrade even if selected, but in BSD it didn't. If you happen to have had a dual processor motherboard with a single processor loaded, BSD would work fine, or one with a chipset that could support SMP, but didn't have the socket for it. It's been a while since I had to mess with servers to this level, having moved on to networking.

              Most modern K7/P4 motherboards have an integrated local APIC in any event.

              I also seem to reme

      • This was fixed a long time ago (by dillon before he left - that long ago).

      • Re:One flaw with tests (Score:4, Informative)

        by molnarcs ( 675885 ) <csabamolnarNO@SPAMgmail.com> on Friday January 07, 2005 @10:21AM (#11287418) Homepage Journal
        SMP kernel does not cause lockups on UNI machines. You may be confusing ACPI lockups that troubled pre 5.3 releases (and in some cases 5.3 as well)? FreeBSD from 5.2 to 5.3 came with "options SMP" in GENERIC enabled. You say:

        but that's how it is. at least it is when I move a p6-built kernel from my dual xeon box to my single k8 box. both DO accept p6 as the ARCH type but the SMP code just causes a lockup upon boot.

        Are you sure it's the SMP code itself? As I said, 5.2/5.2.1 shipped with an SMP kernel by default, and I didn't see hang problems because of SMP (ACPI was a different issue). Also, someone mentioned APIC - which might be the case on some boards, but right now I run an APIC enabled non-smp kernel on my UP box (athlon xp 2400+, crappy asrock mb). I think it is more likely that code compiled on your dual xeon box won't run smoothly on your athlon64, smp or not, even though you might have specified i686 in your make.conf. (also, did your specify p6 with CPUTYPE?=i686)?

      • Nonsense. I don't know where you got this information, but it's flat out wrong. It may have been true in the past, but it is not true now, nor has it been true within the at least the last year. At least on the 5.x branch.
      • Nope, your info is very out-of-date.

        A couple weeks after the release of FreeBSD 5.2.1, the GENERIC kernel was changed to compile an SMP kernel, and SMP support was controlled via a sysctl.

        Just before the release of FreeBSD 5.3, GENERIC was changed to create a UP kernel, as that is the most common install. However, you can compile an SMP kernel and run it just fine on a UP system. You will lose a bit of system performance, but it will run.
        • if it wasn't clear (maybe it wasn't) I was referring to the 4.x branch. which is pretty much (still) all I run. as of 4.11-RCmumble, it would still hang if kernel was built for SMP and booted on a (say) single k7 box.

          might have been the APIC that did it, but the 2 options you need to set to enable SMP in 4.x DO cause a bootup hang on single cpu boxes.

          if 5.x fixed, that, that's cool. its how it SHOULD be. but I won't go to 5.3 until I need to - and 4.x works fine for my needs. I just gotta be careful
          • I had a familiar problem (and not just me) of HTT not being SMP-like enough for 4.x to boot on it (trap on boot), this on a Dell Inspiron 5150 with a P4 3.06 w/ HTT. I ended up turning HTT off and keeping it that way (its advantages are wasted under the UNIXy OSs which have sufficiently responsive schedulers, but a lot more overhead and potentially buggy behavior appears when you turn use it).

            Others did not have this problem (as the mailing lists spoke), so it's probably just a bit of confusion with APICs
          • You didn't mention which version of FreeBSD, just that you tried it recently (which, to me, implies 5.x, as that's the most recent version).

            Yes, in FreeBSD 4.x, you must run a non-SMP kernel on uni-processor systems.

            Yes, this has changed since the release of FreeBSD 5.3.
      • We found FreeBSD doing that as well, except on servers with dual CPU motherboards. So, we install FreeBSD, run "mptable" to determine if it is a MP capable board, and build the kernel accordingly. Easy as premade microwavable pie.

  • Microbenchmarks... (Score:5, Informative)

    by kernelistic ( 160323 ) on Friday January 07, 2005 @04:43AM (#11285951)
    It is important to note that all of the tests that were performed where done on a uni-processor workstation.

    The blanket statement that "NetBSD 2.0 has better threading and process management and network latency than FreeBSD 5.3" does not hold water when the test suite is run on 2 and 4-way SMP systems. FreeBSD 5.x is an amazing engineering effort in which various parts of the kernel have been locked down and decent thread concurancy can occur on multi-proc machines. Part of the latency that you see in these benchmarks are due to the mtx_lock() and mtx_unlock() overhead that is incurred.

    It is also important to note that P4-systems with HyperThreading (As the one used in these tests) have been the "bastard child" on FreeBSD. For the longest of times, compiling anything with CPUTYPE=p4 would produce broken code (In all fairness, this was mostly due to a set of bugs in GCC 3.x). Significant work was put into 5.3 to ensure that the Pentium 4 lived up to the Tier-1 platform robustness standard. In short, it would be fun to see these benchmarks be run on i386/pentium3, i386/Athlon-MP, amd64/opteron and Alpha as well!
    • First paragraph: FreeBSD is waaaay better with SMP than NetBSD.

      Second paragraph: FreeBSD doesn't work well on hyperthreaded P4s, so using one as a benchmark is unfair.

      Hyperthreading is, more or less, SMP (Intel has a fluffy introduction to it here [intel.com]). Sure, there are differences, but you're making exceptions for your exceptions. There's a point where fanboy-ness becomes obvious.

      • I am not making an exception to the exception. In fact, I am not even talking about HyperThreading. What I did say is that using GCC 3.x with P4 as a target produced broken code until very recently and as such, the architecture has not gotten the amount of scrutiny as it could have.

        The default install of FreeBSD doesn't do HTT or SMP without a kernel recompile. My beef with this benchmark is as follows: How about you test the benchmark on other platforms/cpus? In order to truly assertain that NetBSD 2.0 is

      • Re:Microbenchmarks... (Score:5, Interesting)

        by Brandybuck ( 704397 ) on Friday January 07, 2005 @01:27PM (#11289583) Homepage Journal
        Hyperthreading is *NOT* SMP! It is a single CPU that can only do the work of a single CPU. It pretends to be two CPUs but it is not. If you were talking about multiple core CPUs, then you would be right, but p4 HTT processors are single core.

        Treating HTT at SMP is, to quote from the FreeBSD Handbook, "naive". Intel agrees, though they don't use that word. Here's a quote from the article you mentionL "but further performance gains can be realized by specifically tuning software for Hyper-Threading Technology." In other words, you tune software for HTT *differently* than you do for SMP.

  • my reply in the mailinglist (Score:3, Informative)

    by MPHellwig ( 847067 ) * <mhellwig@xs4all.nl> on Friday January 07, 2005 @05:29AM (#11286037) Homepage
    In the freebsd mailing list there was a troll using this test to come down on a couple of highly skilled developers.
    Being lazy by nature I copy/paste my respone in the mailing list for the /. folks to enjoy:

    Benchmark are made to be put into perspective, although everybody has a right to say what (s)he wants to say, this doesn't mean that you have to say it.
    It seems to me that FreeBSD is focusing it performance onto MP 64bit processors. As we can see in the benchmark it has in comparison to other projects a negative impact on UP system.

    But just put it in the perspective of processor developments, AMD (followed by Intel) is heading towards a multi-core 64 bit systems, what probably becomes mainstream at the end of next year.
    With this technology the FreeBSD model could have a winner on there hands.

    Doing the same job but not having the same philosophy on it, is always inefficient, but in the real world it leads to the Darwin effect.
    What means that the best solution gets there chance of survival against the test of time.
    Luckily these are all BSD's, good solution will spread, just take a look at PF.
    OpenBSD has a good user base but not compatible to the sum of user base of the other BSD's. Still PF has spread there wings beyond the user base of OpenBSD.

    FreeBSD is just a name for an OS, if any other OS can give me more "bang for the buck" and provides a full solution, I will use it.
    Be it DragonFly/Free/Open/Net, MacOsX, GNU+Linux, Windows or any of the other hundreds of OS'es out there.
    I like the BSD license so I will tend to stick to "gratis" BSD OS'es.

    All of the disagreements in development is a healthy process to make sure the sort "BSD" an not the specie *BSD will survive.
    Sure I have my disappointments about some decision, but hey so is live, this ain't a fan club for next biggest boy band (he he BSD-Boyz), where using an OS to provide solution for our technologic problems, you favor your solution but don't blind yourself.

    And when you don't blind yourself you re-evaluate your situation and move forward with the best solution for your problem.
    Sure it is a pain to migrate my boxes to another OS (well that is the fun part) and do some massive rewriting of my documentation, but thats my job and I tend to like it. Just standing still and not progress has its attractiveness when you had a very rough ride, but it gets dull very soon and then you find yourself back on the dirty tracks.

    But these are my opinion only, however I like to share them ;-)
  • I just went from FreeBSD 5.3BETA3 to 5.3-RELEASE-p2 on my fileserver, and moved to a new hard disk at the same time. Somewhere between beta3 and -release, the decision to make SCHED_4BSD the default scheduler was made, and SCHED_ULE won't compile because it has an #error macro.

    Anyways, the point of this is that because I was going to a new hard drive, I was doing a lot of dumping and paxing to transfer things over, some before installworld, and some after. I noticed once in 5.3-release that heavy disk acti
    • I use bonnie [textuality.com] to test disk IO. It is a good reliable way to put the filesystem under load, and may help you determine if the scheduler is a problem.
  • Finally, a *BSD related discussion on /. that's constructive and objective, with people giving valid points of view and opinions.. Keep it up!

  • For those bitching that a microbenchmark's not worth beans, I ask that you beat this with FreeBSD (it's a bit old, but hey):

    Aparently the folks from the Swedish University Network (SUNet) at Lulea managed to break their previous Internet 2 Landspeed record for both single and multiple streams, using NetBSD again. Comparison:

    Old record:

    * 838860800000 bytes in 1588 real seconds = 4226 Mbit/sec o
    * Distance: 16,343 km (10,157 miles)
    * 69.073 Petabit-meters/second (12% increase)

    New record:
  • How does a recent OpenBSD (3.5 or higher) on a typical non-SMP PC compare to a recent NetBSD or FreeBSD?

    I hear it is slower but I'm wondering if anyone has any actual experience.

    • Re:As long as there are a bunch of BSD types here (Score:1, Interesting)

      by Anonymous Coward
      As of NetBSD 2.0, it is the fastest of all the BSDs. It is in the number 1 position. Now as far as the number 2 position, if you are talking about FreeBSD 4.9 or earlier, then FreeBSD is number 2. If you are referring to any other FreeBSD including 5.x, then OpenBSD is faster. Therefore, for all of the most current releases, the speed rankings for NetBSD, OpenBSD, and FreeBSD from fastest to slowest is
      1. NetBSD (fastest)
      2. OpenBSD (happy medium)
      3. FreeBSD (slowest)

      Of course, by this time next year the rankings

    • Re:As long as there are a bunch of BSD types here (Score:2, Interesting)

      by Anonymous Coward
      I am not trying to sling shit on OpenBSD here, I've been using it as my primary desktop, net connected servers and firewall OS for very many years and will continue to do so. However I had to do something new recently that caused me to switch to NetBSD 2.0 for this project due to massive performance gains that could not be ignored (this may be due to my scripts being far from optimum at the moment, but)...

      I have been doing some intensive data mining on some large datasets (500MB - 1GB) on a 2GB RAM PC.

      Ope
  • It's a pity that DragonFlyBSD wasn't benchmarked in place of FreeBSD.

Slashdot Top Deals

Someday somebody has got to decide whether the typewriter is the machine, or the person who operates it.

Close