Further Insight
Additional thoughts on the performance of a virtualized environment can be read here: http://links.amd.com/HolyGrail
Server virtualization juggernaut - well, at least on x64 iron - VMware is beside itself with glee that a virtualized Linux server running atop ESX Server hypervisor narrowly beat out real Linux boxes on a popular Web serving benchmark test. Yes, I know. This sounds like a perpetual motion machine, getting the hypervisor …
It seems the vmware config was more in line with the 24-core HP config which had 128GB of memory and 100+ disks behind it -
http://www.spec.org/osg/web2005/results/res2009q1/web2005-20090113-00124.html
I'd be interested in seeing the disk I/O associated with each system tested, I'd be curious how effective their disk configurations are and how much wasted I/O capacity is there by segmenting the system out so much, vs striping the data over all of the spindles. The EMC config seemed much more elaborate than the MSA config.
Well, i guess the opening title gives the game away a little bit...
"Fake Server". I have customer who run all of their windows VMs on VMware. A virtual machine is a real to them as physical machine.
However, put this gripe to one side. In the main I can see your problem. It really is impossible to compare the physical machine to a virtual machine - especially with such divergent hardware and no common bench mark process. If we can't trust the data - then we can't make any meaningful conclusions from the data - yet you proceed to do so. I'm afraid you can't have your cake and eat it at the same time. Either the benchmark is meaningful or isn't - and isn't therefore your conclusions become invalid.
By definition physical and virtual are apples and oranges - so it is difficult (impossible to compare them objectively).
As you say "It is hard to say why because the machines are not identical."
In the main people buy vmware not for the performance - but for the server consolidation and management capabilities it offers. Not for raw performance. However, virtualization skeptics have always questioned how on earth you can get good performance if there virtualization layer creates a "virtualization overhead". Quite simply these people don't really understand what virtualization does - using such technologies like TPS and Memory Overcommitment to cancel out those overheads. VMs use "direct execute" to make native calls to the CPU for advanced features - I could go on ad nausesum.
In part I think the reason VMware (especially) and others (less so) love these kind of reports is that they are desperate to counter the widely held assumption that virtualization instanaously means less good performance. If you want to see case-studies of exchange where Plymouth Uni has 25K mailboxes running in exchange on their ESX hosts. Frequently customer report improvements performance that run counter to their expectations.
The reality is that the days of running work-loads on physical is over. The modern servers being produced are being built for hypervisor - not for windows or linux - which can't address that hardware. I remember the Register saying as much a couple of months ago...
And Finally "And what is true about VMware's sophisticated server virtualization software in this regard is equally true of the XenServer stack from Citrix Systems and whatever Microsoft cooks up with partners for Hyper-V."
Er, I'd take issue with that. Not all virtualization platforms are the equal. Anyone involved in the field of virtualization for short time knows this is the case...
As the report points out, there are way too many variables in these platforms to make meaningful comparisons. Storage, memory, application instances, processors and RAM to name a few. It is always the case with vendor-produced benchmarks. They are exercises in marketing, not in objective comparisons. What doesn't help is the paucity of system utilisation information - was the physical server benbchmark I/O bound fo instance? What was the CPU loading in each case, what service times were seen on the storage systems?
There is is another obvious point about the ESX test and that is the configuration has been finely tuned with the number of VMs, use of processor affinity and carefully balancing the loads across VMs to maximise throughput. In a real live environment. Carefully tuning the workload such that the load on each VM fits neatly onto a certain number of cores in order to maximise processor affinity and minimising the pre-emptive rescheduling of VMs is just not practical in a mixed workload where ESX is being used as a consolidation solution, Real workloads are just not so predictable.
ESX is a solution for ease and efficiency of consolidation of small and moderate systems, flexing capacity and not the ultimate platform for performance or cost-effecitveness of very large applications. Horses for courses, and if you really do have very large applications which justify it, bare metal deployment will ultimately be cheaper in hardware and software costs.
General rule - be wary of marketing hype and read between the lines. Of course the real decision will be made by somebody senior in your organisation who hasn't a clue about the technology, but is readily swayed by selective statistics and the promise of a magic wand to solve his/her pressing business problems. This decision will reach the point where it has much of the characteristics of a papal decree and will therefore be inarguable (at least in public).
I think you've precis-ed the article perfectly. All we now are two new icons. We obviously need a "No shit, Sherlock!" icon but I think there is also call for a "Clue by four" icon, which can be waved at the PtaRds responsible for the original claim.
IT?, because this story really has no technical content.
There is a more comparable test of a non-virtualised DL585 server on the SPEC website. Software levels appear to be comparable.
http://www.spec.org/web2005/results/res2008q3/web2005-20080618-00112.html
It's of a 2.31 GHz 16 core DL585 G5 with 64GB of memory and got to 43,854 SPECweb2005 vs the 44,000 rating. So that's 0.4% more on the ESX configuration at the expense of 64GB more memory, processors running 300MHz faster and a considerably more expensive I/O system.
So if the faster clocked server is perhaps 10% more powerful at only very slightly more throughput that might give a reasonable view of the ESX overhead (of considerably more cost concern than a possible 10% CPU overhead is that of memory - 64GB for a DL585 G5 lists at about £5.4K).
Rather interestingly, HP benchmarks a DL385 G5 with 8 x 2.3GHz cores and 64GB at a SPECweb2006 of 30,007, which could well mean that this is one of those benchmarks which doesn't scale so well with single, large instances.
I'd venture that a pair of DL385 G5s with 64GB in a load-balanced arrangement (if your application allows for such a config) would perform a lot better, at much lower cost, than a DL585 G5 with ESX and multiple VMs.
Can the overhead of virtualisation be reduced or mitigated such that a real world configuration runs just as effectively when virtualised as it would non-virtualised, thus, is the virtualised option the same or better value for money ?
For those with foresight, the next question is ... and in terms of future scalability ?
Too much mud in that test to get any particularly good answers.
The cost to virtualize a host using ESX is $0.00-if you use ESXi. True, you don't get HA/DRS/VMotion but you do get encapsulation as well as an abstraction from the hardware. ESXi is the same hypervisor as the Enterprise one. I'm surprised the author didn't bring that out-it certainly skews his numbers regarding the "cost" of virtualization.
Mr Jones stated:
"ESX is a solution for ease and efficiency of consolidation of small and moderate systems."
That was true a few years ago, but with hardware performance increasing along with the technology in VMware's hypervisor, even very heavy workloads such as high transactional database are being run in the virtual space. For those systems, clients will often run a single VM on a host where you get the best of both worlds: hardware abstraction, VM manageability, better D/R but also better performance since the host is not sharing any resources with the exception of the hypervisor overhead-which is generally only 10 to 20%/
How many benchmarks get done for every one that actually is made public?
Deaddownsouth - thanks that is really amusing
- your "very heavy workloads" is some-one elses light workload. And a single VMware to get hardware abstraction with a cost of 20% overhead?
- You did know that VMWare is not the only product giving you "hardware abstraction"?
- Why would you need VM manageability if you have a single instance?
- Better DR - again your DR is somebody elses HA DR is more than just moving a VM-instance around.
- AAAARRGGGHHHH - why am I reacting to this ...
Paris because she also can believes marketing and has no idea of the bigger world out there.
Just one small point - how do your clients manage clock synchronisation? It is quite big issue for Windows running on VMWare versions I'm familiar with, which basically prevents any production use for me (as there would be no way to match log files recorded on different machines, not to mention other problems with timestamps in data they exchange etc.).