back to article Fake server beats real server on Web test

Server virtualization juggernaut - well, at least on x64 iron - VMware is beside itself with glee that a virtualized Linux server running atop ESX Server hypervisor narrowly beat out real Linux boxes on a popular Web serving benchmark test. Yes, I know. This sounds like a perpetual motion machine, getting the hypervisor …

COMMENTS

This topic is closed for new posts.
  1. Jake at AMD

    Further Insight

    Additional thoughts on the performance of a virtualized environment can be read here: http://links.amd.com/HolyGrail

  2. Nate Amsden

    server config

    It seems the vmware config was more in line with the 24-core HP config which had 128GB of memory and 100+ disks behind it -

    http://www.spec.org/osg/web2005/results/res2009q1/web2005-20090113-00124.html

    I'd be interested in seeing the disk I/O associated with each system tested, I'd be curious how effective their disk configurations are and how much wasted I/O capacity is there by segmenting the system out so much, vs striping the data over all of the spindles. The EMC config seemed much more elaborate than the MSA config.

  3. Mike Laverick
    Jobs Horns

    Fake Server...

    Well, i guess the opening title gives the game away a little bit...

    "Fake Server". I have customer who run all of their windows VMs on VMware. A virtual machine is a real to them as physical machine.

    However, put this gripe to one side. In the main I can see your problem. It really is impossible to compare the physical machine to a virtual machine - especially with such divergent hardware and no common bench mark process. If we can't trust the data - then we can't make any meaningful conclusions from the data - yet you proceed to do so. I'm afraid you can't have your cake and eat it at the same time. Either the benchmark is meaningful or isn't - and isn't therefore your conclusions become invalid.

    By definition physical and virtual are apples and oranges - so it is difficult (impossible to compare them objectively).

    As you say "It is hard to say why because the machines are not identical."

    In the main people buy vmware not for the performance - but for the server consolidation and management capabilities it offers. Not for raw performance. However, virtualization skeptics have always questioned how on earth you can get good performance if there virtualization layer creates a "virtualization overhead". Quite simply these people don't really understand what virtualization does - using such technologies like TPS and Memory Overcommitment to cancel out those overheads. VMs use "direct execute" to make native calls to the CPU for advanced features - I could go on ad nausesum.

    In part I think the reason VMware (especially) and others (less so) love these kind of reports is that they are desperate to counter the widely held assumption that virtualization instanaously means less good performance. If you want to see case-studies of exchange where Plymouth Uni has 25K mailboxes running in exchange on their ESX hosts. Frequently customer report improvements performance that run counter to their expectations.

    The reality is that the days of running work-loads on physical is over. The modern servers being produced are being built for hypervisor - not for windows or linux - which can't address that hardware. I remember the Register saying as much a couple of months ago...

    And Finally "And what is true about VMware's sophisticated server virtualization software in this regard is equally true of the XenServer stack from Citrix Systems and whatever Microsoft cooks up with partners for Hyper-V."

    Er, I'd take issue with that. Not all virtualization platforms are the equal. Anyone involved in the field of virtualization for short time knows this is the case...

  4. Anonymous Coward
    Stop

    huh?

    they compared the results of a better system against those of a lesser system and the better system won? no sh*t sherlock. this is like saying a PC equiped with a 2.4GHz Xeon

    with 333 DDR RAM out performs a 2.2GHz Xeon with 266 DDR - virtual or not!

  5. Steven Jones

    Too many variables to compare

    As the report points out, there are way too many variables in these platforms to make meaningful comparisons. Storage, memory, application instances, processors and RAM to name a few. It is always the case with vendor-produced benchmarks. They are exercises in marketing, not in objective comparisons. What doesn't help is the paucity of system utilisation information - was the physical server benbchmark I/O bound fo instance? What was the CPU loading in each case, what service times were seen on the storage systems?

    There is is another obvious point about the ESX test and that is the configuration has been finely tuned with the number of VMs, use of processor affinity and carefully balancing the loads across VMs to maximise throughput. In a real live environment. Carefully tuning the workload such that the load on each VM fits neatly onto a certain number of cores in order to maximise processor affinity and minimising the pre-emptive rescheduling of VMs is just not practical in a mixed workload where ESX is being used as a consolidation solution, Real workloads are just not so predictable.

    ESX is a solution for ease and efficiency of consolidation of small and moderate systems, flexing capacity and not the ultimate platform for performance or cost-effecitveness of very large applications. Horses for courses, and if you really do have very large applications which justify it, bare metal deployment will ultimately be cheaper in hardware and software costs.

    General rule - be wary of marketing hype and read between the lines. Of course the real decision will be made by somebody senior in your organisation who hasn't a clue about the technology, but is readily swayed by selective statistics and the promise of a magic wand to solve his/her pressing business problems. This decision will reach the point where it has much of the characteristics of a papal decree and will therefore be inarguable (at least in public).

  6. Tim

    Overhead

    This provides an ideal means to calculate the overhead to run VMware; 10% better performance achieved by using 100% more ram and 800% more storage. Presumably that is what they are trying to tell us by publishing this.

    Tim#3

  7. Ken Hagan Gold badge
    IT Angle

    Re: huh?

    I think you've precis-ed the article perfectly. All we now are two new icons. We obviously need a "No shit, Sherlock!" icon but I think there is also call for a "Clue by four" icon, which can be waved at the PtaRds responsible for the original claim.

    IT?, because this story really has no technical content.

  8. Steven Jones

    More meaningful comparison

    There is a more comparable test of a non-virtualised DL585 server on the SPEC website. Software levels appear to be comparable.

    http://www.spec.org/web2005/results/res2008q3/web2005-20080618-00112.html

    It's of a 2.31 GHz 16 core DL585 G5 with 64GB of memory and got to 43,854 SPECweb2005 vs the 44,000 rating. So that's 0.4% more on the ESX configuration at the expense of 64GB more memory, processors running 300MHz faster and a considerably more expensive I/O system.

    So if the faster clocked server is perhaps 10% more powerful at only very slightly more throughput that might give a reasonable view of the ESX overhead (of considerably more cost concern than a possible 10% CPU overhead is that of memory - 64GB for a DL585 G5 lists at about £5.4K).

    Rather interestingly, HP benchmarks a DL385 G5 with 8 x 2.3GHz cores and 64GB at a SPECweb2006 of 30,007, which could well mean that this is one of those benchmarks which doesn't scale so well with single, large instances.

    I'd venture that a pair of DL385 G5s with 64GB in a load-balanced arrangement (if your application allows for such a config) would perform a lot better, at much lower cost, than a DL585 G5 with ESX and multiple VMs.

  9. FreeTard

    Why didn't they use Xen on RHEL5

    That would have made a lot more sense to be honest, and like for like while they're at it.

  10. Jason Bloomberg Silver badge

    There's only one question ...

    Can the overhead of virtualisation be reduced or mitigated such that a real world configuration runs just as effectively when virtualised as it would non-virtualised, thus, is the virtualised option the same or better value for money ?

    For those with foresight, the next question is ... and in terms of future scalability ?

    Too much mud in that test to get any particularly good answers.

  11. Anonymous Coward
    Flame

    SMP Anyone?

    Whatever happened to running multiple applications on the same OS? Oh that's right. This stuff has to be rebooted all the time nowadays :) Hide this under the term "manageability" and there you go........................

  12. deandownsouth

    Cost

    The cost to virtualize a host using ESX is $0.00-if you use ESXi. True, you don't get HA/DRS/VMotion but you do get encapsulation as well as an abstraction from the hardware. ESXi is the same hypervisor as the Enterprise one. I'm surprised the author didn't bring that out-it certainly skews his numbers regarding the "cost" of virtualization.

    Mr Jones stated:

    "ESX is a solution for ease and efficiency of consolidation of small and moderate systems."

    That was true a few years ago, but with hardware performance increasing along with the technology in VMware's hypervisor, even very heavy workloads such as high transactional database are being run in the virtual space. For those systems, clients will often run a single VM on a host where you get the best of both worlds: hardware abstraction, VM manageability, better D/R but also better performance since the host is not sharing any resources with the exception of the hypervisor overhead-which is generally only 10 to 20%/

  13. Lou
    Paris Hilton

    Lies, damn lies and benchmarks

    How many benchmarks get done for every one that actually is made public?

    Deaddownsouth - thanks that is really amusing

    - your "very heavy workloads" is some-one elses light workload. And a single VMware to get hardware abstraction with a cost of 20% overhead?

    - You did know that VMWare is not the only product giving you "hardware abstraction"?

    - Why would you need VM manageability if you have a single instance?

    - Better DR - again your DR is somebody elses HA DR is more than just moving a VM-instance around.

    - AAAARRGGGHHHH - why am I reacting to this ...

    Paris because she also can believes marketing and has no idea of the bigger world out there.

  14. Bronek Kozicki

    @Mike Laverick

    Just one small point - how do your clients manage clock synchronisation? It is quite big issue for Windows running on VMWare versions I'm familiar with, which basically prevents any production use for me (as there would be no way to match log files recorded on different machines, not to mention other problems with timestamps in data they exchange etc.).

This topic is closed for new posts.

Other stories you might like