When VMware Performance Fails, Try BSD Jails 361
Siker writes in to tell us about the experience of email transfer service YippieMove, which ditched VMware and switched to FreeBSD jails. "We doubled the amount of memory per server, we quadrupled SQLite's internal buffers, we turned off SQLite auto-vacuuming, we turned off synchronization, we added more database indexes. We were confused. Certainly we had expected a performance difference between running our software in a VM compared to running on the metal, but that it could be as much as 10X was a wake-up call."
Virtualization != Performance (Score:5, Insightful)
Is this a surprise? (Score:4, Insightful)
Amazing! Not running several additional copies of an operating system with all of the needless overhead involved is faster! Who would have guessed?
Sometimes a virtual machine is far more "solution" than you need. If you really want the same OS with lots of separated services and resource management... then run a single copy of the OS and implement some resource management. Jails are just one example - I find Solaris Containers to be much more elegant. Of course, then you have to be running Solaris...
Re:Back to the Future? (Score:1, Insightful)
Virtualization, as far as it is applied to server software, is a kludge. It is a way of making software work on one machine which would otherwise have conflicting OS and security requirements. The virtualization layer provides the abstraction and isolation which should be provided by the OS but isn't. The reason is API complexity: Virtualization deals with a relatively small and low level API. The operating system has a much broader API with much more complex dependencies, so it is much harder to secure and test for incompatibilities.
It should not surprise anyone that removing redundancies (to save money) likely also decreases fault tolerance. On the other hand, it is often beneficial to remove "accidental" redundancy and add redundancy back in with a plan.
Re:Back to the Future? (Score:5, Insightful)
Re:excellent sales story (Score:2, Insightful)
Just something I thought I'd point out.
On another side note what happened to the fine art of trolling, people these days just throw a bunch of racial slurs together and think they're all that. In my day it took a certain finesse to troll properly, you had to be well informed on the issue and then speak truths on the issue and then interpret those truths in a way that will set people off.
oh, well
get off of my lawn
Re:Virtualization doesn't make sense (Score:5, Insightful)
Virtualization DOES make sense, when you're trying to solve the right problem. Do not blame the tool for the incompetence of those using it. It's no good using a screwdriver to shovel dirt and then blaming the screwdriver.
Virtualization is good for many things:
- Low performance apps. Install once, run many copies
- Excellent for multiple test environments where tests are not hardware dependant
- Infrequently used environments, like dev environments, especially where the alternate solution is to provide physical access to multiple machines
- Demos and teaching where multiple operating systems are required
- Running small apps that don't run on your OS of choice infrequently
Virtualization is NOT good for:
- High performance applications
- Performance test envrionemnts
- Removing all dependence on physical hardware
- Moving your entire business to
Your specific concerns:
# Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical
Actually this depends on your virtualization solution
# TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem.
So is hard disk access from multiple virtual operating systems contending for the same disk (unless you're going to have one disk per guest OS...even then are you going through one controller?) Resource contention is a trade-off. If all your systems are going to be running flat out simultaneously virtualization is a bad solution.
# A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest
You can often mount the virtual disks in a HOST OS. No different to needing software to access multiple partitions. As long as the software is available, it's not as big an issue.
# Memory management is an absolute clusterfuck. From the point of view of the host, each guest's memory is an opaque blob, and from the point of view of the guest, it has the machine to itself. This mutual myopia renders the usual page-cache algorithms absolutely useless. Each guest blithely performs memory management and caching on its own resulting in severely suboptimal decisions being made
A lot of operating systems are becoming virtualization aware, and can be scheduled cooperatively to some degree. That doesn't mean your concern isn't valid, but there is hope that the problems will be reduced. However once again if all your virtual environments are running flat out, you're using virtualization for the wrong thing.
Re:excellent sales story (Score:5, Insightful)
Virtualization is an excellent story to sell. It is a process that can be applied to a wide range of problems.
Screw-drivers are an excellent tool. However if you're in a position to buy tools for your company, you should know enough to show me the door if I try to sell you a screw driver to shovel dirt.
Right tool. Right job.
In any industry:
Poor management + slick marketing = Disaster
Re:Sounds about right (Score:3, Insightful)
Re:Virtualization is good enough (Score:3, Insightful)
It's CYA in practise. Here's the usual chain of events:
1. Business makes requirements to vendor: We want X capacity/response time/whatever
2. Vendor to business side: Well, what will you do with it?
3. Business makes requirements to vendor: Maybe A, maybe B with maybe N or N^2 users
4. Vendor to business side: That was a lot of maybes. But with $CONFIG you'll be sure
Particularly if the required hardware upgrades aren't part of the negotiations with the vendor, then it's almost a certainty.
Re:Back to the Future? (Score:5, Insightful)
"What really is the benefit of extended virtualization?
1) The ability to deploy a system image without deploying physical hardware. All those platforms you are meant to have, but don't: a build machine, an acceptance test machine, a pre-production test machine. And if you've done all the development and testing on a VM then changing the machine when it moves from production from a VM to being real hardware doesn't seem worth the risk.
2) IT as a territorial dispute. You are the IT Director for a large enterprise. You want everything in good facilities, what after the last time a cleaner unplugged the server that generates customer quotes, bringing revenue to a screaming halt. The owner of the quotes server will barely come at that. They certainly won't hand over sysadmin control. Their sysadmins like whitebox machines (the sysadmin's brother assembles them), but you'll never have parts on the shelf for that if it breaks. So get them to hand over a VM image, which you run on hardware of your choice, and which you can backup and restore for them.
3) Single hardware image. No more getting a "revised" model server and finding that the driver your OS needs isn't available yet (or better still, won't ever be available for that OS, since the manufacturer really only supports new hardware in their forthcoming releases). And yeah, the server manufacturer has none of the previous model in stock.
And of course there's minor stuff. Like being able to pull up a shiny clean enterprise image to replicate faults.
You'll notice the lack of the word "silver bullet" above. Because virtualisation isn't. But it does have a useful role, so the naysayers aren't right either.
I'm waiting for the realisation that merely combining images onto one physical machine does not do much to lower costs. For a directly-administered Windows OS the sysadmin's time was costing you more than the hardware. Now that the hardware is gone can you really justify maybe $50kpa/5 = $10pa per image for sysadmin overhead? This is particularly a problem for point (2) above, as they are exactly the people likely to resist the rigorous automation needed to get sysamdin per image overhead to an acceptable point (the best practice point is about $100 per image -- the marginal cost of centrally-administered Linux servers. You'll notice that's some hundreds of times less than worst-practice sysadmin overhead).
I'll also be a bit controversial and note that many sysadmins aren't doing themselves any favours here. How often do you read on Slashdot of time-consuming activities just to get a 5% improvement. If that 5% less runtime costs you 5% more sysadmin time then you've already increased costs by a factor of ten.
Re:excellent sales story (Score:5, Insightful)
ESXi is free, and they could have used that. The overhead for most I/O is a fraction that of VMware server's.
If they did this so long ago that ESXi wasn't available for free, then their basis for discussing problems with VMware is way outdated too, a lot changes in 14 months....
VMware Server simply has many issues: layering the VM filesystem on top of a bulky host filesystem. Relying on a general purpose os to schedule VM execution, memory fragmentation, slow memory ops, contention for memory and disk (VS inappropriate host OS caching/swapping), etc.
And it bears repeating: Virtualization is not a magic pill.
You can't deploy the technology and have it just work. You have to understand the technology, make good design decisions starting at the lowest level (your hardware, your network, storage design, etc), configure, and deploy it properly.
It's not incredibly hard to deploy virtualization properly, but it still takes expertise, and it's not going to work correctly if you don't do it right.
Your FreeBSD jail mail server might not work that well either, if you chose a notoriously-inefficient MTA written in Java that only runs on top of XWindows.
Re:XenServer worked for us (Score:5, Insightful)
The company I work for has just about every Midrange VM solution you can imagine: Citrix, ESX (Seperate Windows and Linux clusters), Solaris Containers, and AIX VIO/Lpars. That is more or less the order of stability, btw.
of all the solutions, AIX is the most consistent and stable. Cheap is what they are not, but in our case they are Blue Dollars. It does exactly what it is billed to do, day in, day out.
Solaris 10 Zones a royal bastard to patch, but otherwise perfectly stable. (quite frankly, they are really just jails, just a little more configurable I suppose)
ESX is stable enough, depending on hardware. Certainly easier than anything but perhaps the HMC.
Citrix is the worst of the lot. But with so much invested, they don't want to do anything else.
Re:Virtualization doesn't make sense (Score:4, Insightful)
Inefficiently as fuck, by the way
Uh, why? Even shit applications don't replace or extend the kernel
FreeBSd runs Linux apps just fine last time I checked
Re:This is Ironic, right? (Score:3, Insightful)
Can we ask you then, why are you running it on VMware Server? Use ESXi. Its free. VMware Server's I/O performance is no where near as good as ESXi.
Re:Sounds about right (Score:4, Insightful)
Re:Virtualization doesn't make sense (Score:3, Insightful)
Inefficient as fuck. Whereas if they'd just been processes running under the same OS, the kernel would already know they were sharing the same page.
Re:excellent sales story (Score:5, Insightful)
Wow, how about you make it more obvious that you have no clue what you're talking about.
ESX and ESXi are bare metal hypervisors that run on the hardware directly. They do not require any OS.
Management of the system can be done with the VMWare infrastructure client GUI which runs on Windows.
The management interface is a SOAP service and the API is public, you can admin via a perl script if you want, and indeed VMware has made command line tools (written in perl IIRC) that access the soap interface. These tools are all available as pre-packaged virtual machines if you want, based on Linux VMs and can be downloaded directly from the web server on the ESX or ESXi server.
Now if you want to bitch about the fact that I can't use FBSD as a host for a virtual machine then by all means, but your complaints are just those of ignorance.
I've used FreeBSD since 2.2, and I'm guessing from your post that you're one of those people that still tries to use FreeBSD as a desktop machine. While it obviously can be done, and with enough effort it can be rather usable, FreeBSD really isn't intended for the desktop PC role, you may want to consider using an OS more suited for the task, let FreeBSD remain the bad ass server that it is and let OS X and Windows be the desktop OSes that they are.
Re:excellent sales story (Score:5, Insightful)
I disagree. I consider Xen to be a kernel which other kernels are modified to run inside of, it is just a guest kernel making requests(read system calls) to a hypervisor(a special sort of kernel) that then translates it into requests to the host kernel. But mostly I feel this way because of the way I/O is handled in Xen is very much unlike the way VMware does it (go find my resume, I used to be an ESX developer at VMware).
Because Xen was originally designed to function without special hardware extensions to support virtualization it is a virtual machine in the same sense that Unix is a virtual machine(processes were literally virtual machines from day 1 in Unix). Xen just jams one more layer above processes.
BSD Jails are just a more Unix way of virtualizing a set of processes than Xen is. Xen requires an entire kernel to encapsulate the virtualization, BSD jails do not. In my opinion that is where they differ the most, but that difference is almost unimportant.
Re:-1, Flamebait (Score:1, Insightful)
You really seem to like being right (or at least telling other people they're wrong).
I'd like to point out that ridiculing people and being obnoxious actually lowers the quality of discussion instead of improving it. If you still want to do that, please at least make sure you don't start the ranting in totally inappropriate places like you did now.
Re:excellent sales story (Score:3, Insightful)
I would actually say that the day ESXi became free, it made server completely obsolete for ANYTHING other than initial testing or building.
As you stated, this article really on every level is a ridicuously poorly designed implimentation, I don't get into flame wars as to what's the better OS, etc, etc, so far as I'm concerned whatever is best at doing what I need it to is the solution I aim for, and with ESX I must admit I have been extremely happy with the time and resource savings, as well as the GREATLY reduced management overhead. Throw in the HA, DRS, vMotion, and disaster recovery, and I now sleep a lot better at night, and get far fewer calls!
Re:free beats fee most of the time (Score:4, Insightful)
And nobody's asking you to memorize what LTSP stands for. Just double-click the text in Firefox, right-click and choose search. So much quicker and more effective than asking everyone to spell out abbreviations. It's a win-win!
Re:Virtualization doesn't make sense (Score:5, Insightful)
Thank you for saying that. The purpose of a multi-tasking multi-user OS is to allow running multiple applications with full isolation from one another. If we need some other piece of software - like a VM hypervisor - to do that, then the OS has failed in its duty. But wait, some people say, it's not just about multiplexing hardware, it's about migration and HA and deploying system images easily. These are also facilities the OS should be providing. Again, if we need some other piece of software then the OS has failed.
One could argue that we've evolved to a point where the functions of an OS have been separated into two layers. One layer takes care of multiplexing the hardware; the other takes care of providing an API and an environment for things like filesystems. Better still, you get to mix and match instances of each layer. OK, fine. Given the Linux community's traditional attitude toward layering (and microkernels, which this approach also resembles) it's a bit inconsistent, but fine. That interpretation does raise some interesting questions, though. People are putting a lot of thought into where drivers should live, and since some drivers are part of "multiplexing the hardware" then it would make sense for them to live in the hypervisor with a stub in the guest OS - just as is being done, I know. But what about the virtual-memory system? That's also a form of hardware multiplexing, arguably the most important. If virtualization is your primary means of isolating users and applications from one another, why not put practically all of the virtual-memory functionality into the hypervisor and run a faster, simpler single-address-space OS on top of that?
If we're going to delegate operating-system functionality to hypervisors, let's at least think about the implications and develop a coherent model of how the two interact instead of the disorganized and piecemeal approaches we see now.
DBAs and Virtualization (Score:4, Insightful)
Re:free beats fee most of the time (Score:4, Insightful)
I hate to sound condescending.. but system administration is considered the lower end of the technology community.
You don't sound condescending; you sound ignorant. Routine system maintenance is low end, getting to play with new (to commodity hardware) virtualization techniques and ZFS and SANs and HA systems isn't quite the same as staring blankly at a re-purposed desktop.
Put another way: it's cool that you like writing drivers, but if they suck, I'm the one who gets to blackball your company on purchase orders.
Re:This is Ironic, right? (Score:3, Insightful)
We've deployed a number of ESXi boxes with mixed results. In a nutshell, if you have a underutilized server that's lightly loaded, it's a great candidate to be a VM.
Anything that needs performance should be considered off the list for a VM, unless you can convince the consumers that the speed penalty is worth the ease of management.
* Web server that servers your lunch menu and maybe your HR vacation scheduling system: VM
* Build machine that pounds through 7 GB of source and takes 10hrs of solid compiling to produce output: not a good choice for a VM.
* Domain Controller: VM
I like VM's because they're portable and machine agnostic. The whole virtual infrastructure stuff is polished and works.
However, people tend to think that virtualization is the right path for everthing, which it is not.
It has its place in the IT toolbox with everything else. Who knows, maybe Intel and vmware will hatch an offspring that looks like ESXi but with hardware provisioning... Though I'm sure IBM's lawyers would shit kittens!