Microsoft has released a beta version of its Dryad platform, a Windows-happy answer to Google's MapReduce distributed number crunching platform. Yes, there's already a burgeoning open source version of MapReduce: the Apache Hadoop project. And at least one Microsoft unit is both using Hadoop and actually contributing code to it …
I really wish somone in M$ has a clue
As someone with a vested interest in M$ doing well (being one of the evil people actually writting production systems on M$ servers) I really do look at what they are doing right now and wonder if they have any game plan.
You've got an open-source product with good take up...so let's make another version of it but with no story about why you would use it rather than the popular one. Then tie it to a server version which costs a sh*t load. Oh, and quietly drop the product in a few years when we're just starting to getting something right on it but a new shiny thing has appeared and we need to make a new version of the shiny thing.
Even at the basic level I can't see what they are trying to do. Why not have this running on the cheapest version of Windows server. I would guess that a lot of people would like a fully distributed framework to run certain types of software which treats the underlying h/w and os as disposable items. Need more power, throw in a new server. Server dies, throw it out and stick in a new one. You don't need to care about availability because the hadoop element takes care of that sitting on a cheap stack of easily replaceable physical bits. That's feasible if the h/w and OS are at the right disposable price but as soon as one of the elements needs to be at the rolls-royce price level then it doesn't work. Given the people already have hadoop I can't see that this will work when they tie it in to Windows HPC which has only a limited visibility/availability to the development community anyway. Another waste of time...
That is called marketing you know
That is called marketing driven product roadmap.
It is a standard issue replacement for innovation in companies whose internal governance prevents them from innovating.
Microsoft is _DEFINITELY_ not alone in this category.
Wow! Now we can code data center applications that require thousands of Windows licenses, one per node. It's still beta, and hasn't really been tested at scale. But think of the POSSIBILITIES. I can see now, this technology will SWEEP THE MARKET!
If you look
You'll see that Dryad has been used internally within M$ for years on hundreds of thousands of nodes.
MS + HPC = SOL
Physics research requires real HPC capabilities and thousands of processing nodes. Hadoop/MapReduce ditto. I can say this from experience and personal knowledge that no one in the real HPC universe would even consider MS operating systems for a couple of good reasons. One, there isn't enough $$ in the universe to pay for them. Two is that MS operating systems are nowhere nearly reliable enough to handle such computing requirements. Today, it is commodity x86_64 processors running Linux, period. Open source is good. Free is better. Open source and free operating systems? Unbeatable.
Absolutely nobody uses Windows HPC, especially not NCSA.
What, they do?!
Well still, NCSA isn't in the 'real HPC universe'...
What's that? What does NCSA stand for?
(let's not even mention the SSC machine - http://www.top500.org/system/details/9787)
Granted it's only 5 of the top 500, Linux certainly dominates, but do some fact checking first eh?
Also, they may only have 5 in the top 500, but the hooks into Excel suggests they are going for existing small to medium shops, rather than new large ones.
Your point about licensing costs is totally valid, but your other comments are ignorant. I know engineers at a major e-commerce company, and their experience is that LInux is surprisingly unreliable under heavy load -- until it has been patched and hacked to run the particular software you are using. One colleague crashed LInux just by running Linpack benchmarks, due to the kernel losing hardware interrupts. That doesn't bode well for scientific computing. The same folks tried using async I/O entry points and again crashed Linux immediately. You seem to think modern MS servers are like running Windows 95, which tells me you do not have any actual experience with server operating systems but are just expressing politcial opinions.
1%, raw assembly used to have a bigger HPC market share than that
After years of sustained marketing and development effort, Windows HPC gets 1% market share. That's really all the evidence you need that the only reason that pig flys is because of the external boost rockets overcoming the pig's drag. If Microsoft ever removes those "have we got a deal for you" boosters, Windows HPC will plummet to earth.
How much were NCSA (and others) paid?
C'mon AC @ 04:04, you don't seriously expect that the HPC sites MS likes to quote have actually paid open market prices for their kit do you? Have you done YOUR fact checking?
In return for an interesting deal on kit, these poodle sites get MS HPC into the top 500, and have to jump when MS PR people say "jump". For a while, anyway, till people realise what went on.
http://www.microsoft.com/presspass/press/2008/jun08/06-18hpcserverpr.mspx and many many others, dating back to at least 2006, but still not really getting anywhere...
You don't seriously expect anyone who is big enough to need any form of supercompute to pay list price for MS' licences do you? That's the first rule of making any volume purchase, beit software, hardware or services - get your purchasing guys to beat up the supplier, until the price is right.
Not quite as dumb as it looks
Windows HPC might cost money but it’s the cheapest Windows OS, and cheaper than licensed RedHat Enterprise on HPC nodes (which very few shops do: CentOS is more popular).. Windows HPC is the simplest and cheapest way to get a Dryad grid.
Were Dryad another MapReduce clone it would certainly be dumb idea, but it is a different beast and more in-tune with the complex mathematical workflows you’d find in a Bank or Hedge fund, and better than the generation-1 grid computing solutions.
Two key questions have not been answered:
1. Why no talk of a Azure version as an alternative to HPC?
2. With F# being opened up, why not make the promise for Dryad?
They don't pay list prices
Microsoft subsidizes a significant portion of the hardware cost of HPC installations that claim to use Windows, so it's a net negative cost to the installation. That means that a US corporation is paying to build supercomputers in China. Yay. For the most part these sites run Windows for the benchmark only, and run Linux the rest of the time.
Microsoft Excel integration with HPC... Oh, God I wish there were words for how ridiculous that is.
@AC First - what they're trying to do is to appear to still be in this game. Fake it 'till you make it. That sort of thing. For HPC the OS doesn't really matter much. All of the operating systems run apps and give them direct access to the hardware, and that's what's required for HPC. Theyd've have done better this pass, but their management app had a signed short int overflow. Who could have predicted that? Nice post though.
Excel integration is for small shops who aren't in a position to get their excel models coded up onto big, distributed linux/unix systems and probably don't have anyone employed who can do that, it's not for big supercompute. It's a bloody good idea, but ridiculous for oil exploration, HEP, general big company stuff.
Actually this makes sense.
Full disclosure: I'm not a Microsoft Fanboi but I am an experienced Hadoop developer.
Ok, I'm never going to use Microsoft's tool or probably ever work on this at any of my clients site(s). But that doesn't mean that this is a bad idea for Microsoft. Actually its a no brainer if you think about it.
There are a lot of companies that have committed to the Microsoft view of the world. .net, C# and visual studios. Add to this Microsoft has SQLServer as their database... So Microsoft centric companies don't have a 'Big Table' option. Sure you can run Hadoop on Windows, but it doesn't really tie in to their environment.
So Microsoft creates their answer and all are happy. Don't be surprised if there aren't any 'crossover' opportunities.
And yes, its the first time I"m actually giving Microsoft a 'thumbs up'. Looks like Hell Michigan just froze over. :-)