Slashdot Log In
ESR and the MindCraft Fiasco
from the stuff-to-read dept.
The Mindcraft fiasco
Microsoft's latest FUD (Fear, Uncertainty and Doubt) tactic may be backfiring.
A 21 April ITWeb story reported results by a benchmarking shop called Mindcraft that supposedly showed NT to be faster than Linux at SMB and Web service. The story also claimed that technical support for tuning the Linux system had been impossible to find.
Previous independent benchmarks (such as "Microsoft Windows NT Server 4.0 versus UNIX") have found Linux and other Unixes to be dramatically faster and more efficient than NT, and independent observers (beginning with a celebrated InfoWorld article in 1998) have lauded the Linux community's responsiveness to support problems. Linux fans smelled a rat somewhere (uttering responses typfied by "Mindcraft Reality Check"), and amidst the ensuing storm of protest some interesting facts came to light.
- The benchmark had been paid for by Microsoft. The Mindcraft press release failed to mention this fact.
- Mindcraft did in fact get a useful answer to its request for help tuning the Linux system. But they did not answer the request for more information, neither did they follow the tuning suggestions given Also, they forged the reply email address to conceal themselves -- the connection was made after the fact by a Usenetter who noticed that the unusual machine configuration described in the request exactly matched that of the test system in the Mindcraft results.
- Red Hat, the Linux distributor Mindcraft says it asked for help, reports that it got one phone call from them on the installation-help line, which isn't supposed to answer post-installation questions about things like advanced server tuning. Evidently Mindcraft's efforts to get help tuning the system were feeble -- at best incompetent, at worst cynical gestures.
- An entertainingly-written article by the head of the development team for Samba (one of the key pieces of Linux software involved in the benchmark) described how Mindcraft could have done a better job of tuning. The article revealed that one of Mindcraft's Samba tweaks had the effect of slowing their Linux down quite drastically.
- Another Usenet article independently pointed out that Mindcraft had deliberately chosen a logging format that imposed a lot of overhead on Apache (the web sever used for the Linux tests).
So far, so sordid -- a fairly standard tale of Microsoft paying to get exactly the FUD it wants from a nominally independent third party. But the story took a strange turn today (22 Mar) when Microsoft spokesperson Ian Hatton effectively admitted [8] that the test had been rigged! "A very highly-tuned NT server" Mr. Hatton said "was pitted against a very poorly tuned Linux server".
He then attempted to spin the whole episode around by complaining that Microsoft and its PR company had received "malicious and obscene" email from Linux fans and slamming this supposed "unprofessionalism". One wonders if Hatton believes it would be "unprofessional" to address strong language to a burglar caught in the act of nipping the family silver.
In any case, Microsoft's underhanded tactics seem (as with its clumsy "astroturf" campaign against the DOJ lawsuit) likely to come back to haunt it. The trade press had largely greeted the Mindcraft results with yawns and skepticism even before Hatton's admission. And it's hard to see how Microsoft will be able to credibly quote anti-Linux benchmarks in the future after this fiasco.
Yes but be careful. (Score:1)
HOW-TO put an end to the fiasco! (Score:1)
--G
Does it matter? (Score:1)
So what is left for NT?
Careful, you can get sued.... (Score:1)
The commercial product I work on would love to publish direct comparison benchmarks against the competing MS product. Our legal department won't let us.
MS is guilty of nothing (Score:1)
Personally, I think MS realized that Mindcraft didn't do a good job of running the tests, so they declined to advertise the study. Mindcraft just put out a release on their own.
You have to understand how these things work. All good software companies run tests of their product under various conditions on various hardware configurations. When they find one that beats some portion of their competition, they try to advertise that fact. To get credibility, they hire an independent lab to reproduce the tests.
In this case, it looks like even MS was suspicious of how poorly Linux fared. My guess is that MS knows that even a well-tuned Linux will lose this particular test, otherwise they wouldn't have hired Mindcraft to run it in the first place.
Interesting Hatton comment (Score:1)
Yes... But... (Score:1)
But with regards to the study, wouldn't it have been more truthful to tune the Linux box appropriately? Then if the results showed that NT was better, a point could be made that Linux needs more work in these areas. I really believe people would accept that as honest criticism (again, IF the results showed that NT was faster, I have my doubts that would happen
Mindcraft obviously didn't make much of an effort in tuning the box (ie hire a decent Linux admin), posted a message to a couple newsgroups (lacking sufficient details, and not responding when someone requested more details), one call to RedHat (which was directed to the wrong group as ESR stated). And yet, they build an entire study around these points, making claims that Linux support is bad and all. For these reasons they were blasted, not for saying NT is better than Linux (as some people I've talked to think is the motive for Linux people's outrage).
Don't forget their motto (Score:4)
"...we work with you to define test goals. Then we put together the necessary tools and do the testing. We report the results back to you in a form that satisfies the test goals."
Since they say Microsoft sponsored the test, we can replace "you" with "Microsoft." So they worked with MS to define the test goals (NT is 2 or more times better than Linux). Then they put together the tools to do that, hacking the registry and all to beef NT up, slowing Linux apache/samba servers. And finally, report the results back in a form that satisfies the test goals, lo and behold NT is 2-3 time faster than Linux. Such a surprise, right?
Apache Benchmarking (Score:1)
Things I could see improve your system:
drop the Apache RPM's and compile it.. Specificaly with the PGCC compiler, I have heard of 30+% speed improvements with it. you could even go for the PGCC based distro stampede for your base install.
look at mylex RAID boards.. they are supposed to work a little better than the AMI cards that Dell uses.
It'd be fun to send that box to mindcraft, and have them test it as same preformance or 90% preformance of the NT box.. but cost less than $5k
not $18,000
Are managers really "used to" NT? (Score:1)
Xenophobia. Managers are scared of Linux because it's not NT, which they're used to.
Managers are used to NT? Where? Managers that propose NT for high-end stuff definitely never seen NT for long enough to be "used to" it -- but they are "used to" being bombarded by M$ advertising/propaganda of NT, and this, not mythical "they are already used to NT" should be counteracted.
Hard to find... (Score:1)
Maybe everybody is at Comdex? (Score:1)
Given all of that, it's a little early for sponsored benchmarking by members of the Linux community. This stuff takes time, and if it's a choice between doing Comdex right and repeating a discredited benchmark, doing Comdex right comes up at the top of the list every time.
Isn't that what I said? (Score:2)
There are valid limits to Linux scalability, problems that need fixing, and honest benchmarking can help us find those limits. Unfortunately, Mindcraft's benchmarking was so flawed by misconduct and poor judgment that it is not useful for that purpose.
BTW, I do agree with the Microsoft spokesman who said that he was certain that NT would have come out on top even with an honest test. I suspect the SAMBA results would have been quite competitive, within 3% (more or less) of the NT numbers, but the Apache server has never been known for its static file serving speed (though mod_mmap_static may change that!). On the other hand, there is a big difference between the 5%-10% advantage that I bet Mindcraft would have found, and the ridiculous numbers that they actually reported. They actually shot themselves in the foot here, because if they'd reported the real numbers, the Slashdot Crowd would have howled, but Jeremy Allison and other technical heavyweights would have stayed on the sidelines working on fixing the problems found, and the media would have ignored the Slashdot Crowd.
Just count it as another example of Microsoft Arrogance (tm) outweighing their good sense. It's amazing how such bright people can do such stupid things.
-- Eric
Linux-Tuner Applets? (Score:1)
Put an end to the fiasco! (Score:1)
Everyone is complaining how Microsoft rigged this test, but no one is surprised or doing anything about it. Why doesn't someone just sponsor an HONEST test, and let the best system win.
How about a range of machines? (Score:1)
Wasn't the test supposed to be about how nt can scale up but Linux can't? We all know the test was rigged, and I'd like to see it run again, but over a range of machines.
What does nt vs Linux look like on a 486 66 w. 8 meg of ram? Hmmm?
Proud of the Linux community and I learned a lot! (Score:1)
I work for a fairly large company thats coming out with a kick ass machine pretty soon. Its enterprise class for sure. Lots of processors. I would love to see linux tuned for it.
I've wondered about this too... (Score:1)
Bear in mind that Intel's integer core is more primative than either AMD or Cyrix's current design. That means that for the vast majority of non-3D/CAD software you'll see a marked improvement over a PII 300 when you use even Cyrix's relatively weak MII chip (whatever P rating actually runs at a 300MHz clock speed).
I sat through an Intel marketing presentation back when I worked retail, and was told flat out that floating point power was more important for word processing, and that every processor that runs at a given clock speed generates the same heat, because it's the little clock crystal that generates all the heat (not the resistance as electricity flows through the chip).
Just goes to show you can't trust marketing to give you straight facts.
There's a difference (Score:1)
Yes... But... (Score:2)
That is a natural consequence of the open source
development. Many of the features of Linux are there because some users needed them (scratch an itch, as ESR says...)
So, bearing in mind that there aren't many Linux
users with quad xeons with 4 Gb Ram, it's only natural that issues relating with that kind of machine have a lower priority to the Linux user community
Mainframes (Score:1)
Mainframes will never die. The legacy system of tomorrow will be mainframe transaction-processing systems fronted by SP/2 analysis clusters, with something like Linux or NT gating the whole mess to the web. I can almost guarantee it.
Of course, Linux is being ported to the S/390...
About Netcraft, semi-off-topic question (Score:1)
I'm curious about this too. I wonder if people who pay Netcraft money get that kind of breakdown.
One thing to remember is that Netcraft counts domains, not IP addresses. ISPs that host sites for clients probably dominate the survey and most of them run Apache. Companies that run their own web site off a T1 or ISDN line probably favour NT, but they would be in the minority.
I like to take each months numbers and pretend that nobody is switching (clearly not true, but might be a good approximation). This month, for example, Apache gained 423063 sites and MS gained 132943. So new Apache sites outnumbered new MS sites by MORE THAN 3 TO 1! That's impressive.
Huh? (Score:1)
Not an easy problem. (Score:1)
I would still like to see some more conclusive stuff on Linux's highend SMP abilities (4 or more), bother on i386 and on Alpha and UltraSPARC. Alan Cox claimed there were some speedups to SMP late in 2.1.x (i think) that should have significantly improved hi end perf. Perhaps VAR would sponsr a test?
If Linux doesn't beat all on up to 16 or so processors, we should fix it.
It's not that simple. Currently the biggest problem with the Linux SMP implementation is the IO subsystem. (SCSI and IDE, etc.) The problem is that this subsystem isn't SMP safe. So whenever the kernel enters this portion, it grabs what is known as the kernel lock. Thus disk activity can only happen on 1 processor at a time (bad). In Linux 2.3 they're going to strip this away, so that Linux 2.4 (or 3.0, or whatever) will probably scale far better than Linux 2.2 (of course they'll be making other nice optimizations along the way). But this isn't something you can just fix on a whim.
Not to side with Mindcraft/MS, but... (Score:1)
One could interpret it that way, but considering that they willfully avoided being helped to tune Linux (ask for samba help anywhere but on the samba lists/newsgroups, call the wrong helpline, don't ask for the right one, when they accidentally do find a helpful person, ignore him, etc...) it seems that the goal must have been to find that NT outperforms Linux.
It reminds me of an old Mad Magazine comic, A father orders his steriotypical hippy son living at home to go get a job. The son, dressed in his most casual attire (torn cutoff shorts, open shirt, many beads etc), goes to a clothing store and says "You ain't got no jobs do you?". Later to his father, "Well, I TRIED!".
The only difference is, Mindcraft actually did get an offer, so they had to ignore it.
And...? (Score:1)
I feel the need to ask why this piece was worthy of "airtime", and can reach only one conclusion: That a refutation can somehow become more valid because it comes from the pen of Eric Raymond.
I'm now happy that Katz periodically contributes his dozen screenfuls of drivel, and I don't have a problem with Eric either, but would I have got a whole item on /., had I summarised the obvious flaws of that ridiculous benchmark? I doubt it.
Matthew
- what happened to our great meritocracy?
Microsoft's credibility (Score:1)
what have you been smoking?
with the rise of n-tier architecture and thin client, mainframe-style computing is getting stronger than ever. centralised data and logic, just the display on the desktop.
So, is linux faster than NT on a 4-way w/ 2GB mem? (Score:3)
I would like to know if Linux does scale as well or better than NT with 4 and 8 processors -- both systems properly tuned and using the same webserver. When that question is answered, I'd like to know what to expect in the future. Is Linux going to leave NT in the dust, or will this be the key niche ground for NT servers that Microsoft will defend to the end, and Linux will never conclusively defeat?
No - you're wrong. (slightly) (Score:2)
But both Samba and Apache have been heavily tested on very high end servers. The Samba crew have even been heavily involved in making Samba fast on high end servers.
HostNameLookups (Score:3)
I strongly believe however that their httpd was running under inetd, and that would cause the effect they saw.
Apache Benchmarking (Score:5)
Server:
- Hand built by our best hardware guy
- PIII 500 (single CPU)
- Adaptec 2940U2W SCSI Adapter
- 10,000 rpm LRW drive. 1 drive only.
- 100Mb/s network card
- 256Mb PC100 RAM.
- Linux 2.2.6, upgraded from stock Linux-Mandrake box
- Apache 1.3.6, configured for best performance.
No changes to the
Want to know the results so far?
Well, we can get about 2200 requests per second out of that box. The Quad Xeon NT box that mindcraft tested got 3700 requests per second at its maximum rate. We are at very early stages so far, and I think I can squeeze more out of the box by dumping Apache and using thttpd or something else that uses a threaded model. But since this is to be a pure mod_perl box I don't think that's important.
Things to remember:
The mindcraft server had 1Gb of RAM.
The mindcraft server had RAID (RAID/0 I believe).
The mindcraft server had 4 10/100 network cards.
We're so far pretty pleased with our little Linux box... It was a fair bit cheaper than Mindcraft's server....
Possibly only using one processor? (Score:2)
Mindcraft also used the v0.92 MegaRAID driver. An SMP race condition was fixed in v0.93 which was almost certainly available from the AMI web site long before the Mar 10-13 test. So SMP NT "beat" a non-SMP Linux on a quad-Xeon server. Big hairy deal.
Original poster is "Doc" Savage. Original post [linuxhq.com] 14 Apr 99.
Eric made a factual mistake (Score:3)
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying "the study was paid for by Microsoft" and that "a very highly-tuned NT server was pitted against a very poorly tuned Linux server".
That is, the claim attributed by Eric to Ian Hatton was really made by reacting Linux supporters.
What Hatton did admit, was:
"Microsoft did sponsor the benchmark testing and the NT server was better tuned than the Linux one."
This isn't much, but it is sufficient. Hatton admits that "the NT server was better tuned than the Linux was" and even without adjectives that invalidates the report.
Scaling is what counts (Score:2)
Remember that a certain number of sites really need big-iron servers (hey, slashdot isn't exactly gentle on its hardware, although in that case I suspect database performance may be more of an issue), and even when they don't it's the results from high-end server tests which impress the management the most.
Having seen Linux/SMP in action and made some subjective judgements I'm quite confident that, properly configured, it ought to scale fairly well onto hardware of the class Mindcraft were `testing'. But it would still be nice to have some number...
benchmarks. (Score:3)
Seems to me that what we really need is a bench marking rebuttal; is there another
benchmark going on? I saw that in Jeremy Allison's article he was working with PC Week,
does anyone else know any other active bench marking going on?
I think that the only way to prove against FUD is education, bench marking can go a long
way.
I have about 7 Linux servers with no down time, great performance on lesser hardware then
my commercial servers in my company, that should be proof enough; but my pointy haired
boss still asks "Why not NT?". I do not need any more fuel for that fire.
We need Benchmarks on larger servers, with more memory, RAID, and a high-end server
guide.
Proud of the Linux community and I learned a lot! (Score:2)
Also, I must say I really learned a lot by following the debates. Next time I need to install Apache and Samba you can bet I'll be referencing the responses to Mindcraft to see the proper way to optimize this stuff.
Kudos and thanks to the Linux Community!
Spread the word (Score:2)
Admit, "M$ guilty of consumer fraud" is a better headline than "NT beats Linux on all fronts".
Spread the word (Score:2)
This is an alramingly common mistake - poor netcraft.
Microsoft's credibility (Score:3)
The plain fact is, Microsoft did this to appeal to middle/upper-management, not us. They need to keep feeding them reasons to keep their NT investment without looking stupid. Remember the mainframe days? Shortly after the PC came out, a torrent of similar "debate" emerged from the mainframe community. First they laughed, then they fought, then the PC community won. Suprise. History repeats itself.
--
The Mindcrap Affair: second-order effects (Score:3)
--JT
Content is beside the point. (Score:2)
The 'Rush Limbaugh' principle is a very valid point, especially in this context. Don't forget the target market for this study is Microsoft partners and WinNT-based shops.
Aside from all the meaninless numbers (who cares if your web server can saturate a 100BT line with static pages!), the study drives home an important point to NT Administrators - If you've invested in a high end IIS system, and you've got it tuned, there's probably no good reason to switch that box over to Linux. If the Linux box was tuned correctly, I doubt the difference would be that great performance-wise.
Of course, the study didn't address stability, which is the number one problem with IIS.
--
Samba article (Score:3)
A few interesting points -
* In the often referred-to ZD Samba versus NT benchmarks (where Linux+Samba wins), the Samba/Linux configuration was tuned by a Samba team member. Objectively, this makes the ZD benchmark actually less valid as the Mindcraft study, because as far as we know, a Microsoft-employeed SMB developer wasn't actually there tuning the server.
* Tuning Linux properly involves cryptic commands such as:
echo "80 500 64 64 80 6000 6000 1884 2" >/proc/sys/vm/bdflush
echo "60 80 80" >/proc/sys/vm/buffermem
While I'm sure these commands are documented somewhere, this sort of tuning makes the NT Registry Editor look like a model user interface. Low level tuning like this really needs a nicer front end, or preferably, a daemon which monitors system activity and dynamically tunes these settings.
It sounds like the Mindcraft study has been a kick in the pants for the Linux community to get some high performance documentation together. I'd like to see a nice How-To which lays out some of the more obscurantist tricks such as echoing strings to the
--
Content is beside the point. (Score:5)
The truth or falsehood of the Mindcraft study is irrelevant to its intended audience. The point is to give NT "believers" something to quote in arguments, that's all. It's the Rush Limbaugh Principle. In a disagreement, it's helpful to have official-sounding statistics to back up your point. It doesn't matter where they came from, and it doesn't matter whether they're even remotely accurate. What counts is that somebody "important" (read "well-known") said it in public, which "validates" it. This "validation" isn't about truth. What it means is that the proper forms have been followed, and so it's acceptable to introduce the "evidence" in an argument. What's being offered is not evidence in the conventional sense, but the appearance of evidence, or the outward form of evidence. In poker, what does the four of diamonds mean? It means the four of diamonds. It's pure, disembodied symbol.
Disagreement and debate in our culture (especially on the net) isn't a whole lot less stylized (nor a whole lot less predictable) than Noh drama. You have to play by the rules and observe the forms. The content of the Mindcraft study is arbitrary. The study is a signifier, or token. A yacc parser says, "hey, this token is a function, hey, that one's an operator." The actual content of the token is not significant; what matters is what kind of token it is.
Everybody should learn at least a bonehead popularized minimum of semiotics (which is all I know, obviously
While we're at it, let's be honest with ourselves: How many of us are going to check Eric Raymond's facts for ourselves -- even to the minimal extent of clicking on the links he provides? And how many of us who don't check the facts are going to run around repeating them? Quite a few, probably. Dammit, I think Raymond's right on the money with this, and I'm confident that he's done his homework -- but I don't have the time to go about proving it. As far as many of us are concerned, Eric has given us a counter-signifier. Some "good spin" to match against the "bad spin". (That makes it sound dishonest, but IMHO if the "good spin" is factual and accurate, then "good" is a perfectly reasonable thing to call it.)
Think about it.
(Experienced sysadmins are a bit of a special case here. They can judge for themselves. The Limbaugh Principle applies mainly to people who are arguing in an area outside of their field of expertise -- I don't recall who it was who said that "every man is gullible outside his specialty", but it's true even of the best of us.)
"Once a solution is found, a compatibility problem becomes indescribably boring because it has only... practical importance"
No, cjr made a referential mistake. (Score:3)
"Linux supporters have reacted violently to the Microsoft SA release (Independent research shows NT 4.0 outperforms Linux) published on ITWeb yesterday, saying 'the study was paid for by Microsoft' and that 'a very highly-tuned NT server was pitted against a very poorly tuned Linux server'. In response, Ian Hatton, Windows platform manager at Microsoft SA, says these comments are valid."
No more benchmark... a contest (Score:3)
In a benchmark their are great odds that the benchmark will be sponsored by one of the party (M$ in this case).
If you do a contest, like the best ratio performance/price : you benchmark the performance of all the competing teams and then divide by the price the team involved in the hardware (not the software because due to Linux openness many people would say Linux price biased the contest).
If someone do so you can have a M$ team which will try to tune NT to is best, a Linux/Samba/Apache team which will try to tune Linux to his best, a Novell team, a Sun team...
You could choose your hardware so small team can try to compete. Even companies unrelated to NT/Linux/Novell/Another OS could compete so that can do a lot of publicity to these companies if they are well placed in the results.
It would be a good thing so every people supporting an operating system and so knowing how to tune it would be able to compete and their would be a greater range of results than in a single benchmark.
Of course we now need to find somebody to finance the contest
So, is linux faster than NT on a 4-way w/ 2GB mem? (Score:2)
The interesting question to me isn't "How much power can you get out of hardware X with OS Y", but rather "How much hardware do you need to throw OS Y on to do job Z".
From what I've been reading, NT does better SMP than Linux does. Frankly, Linux doesn't need SMP nearly as badly as NT. If uniprocessor Linux can do the same job as SMP NT, who cares how good or bad SMP Linux is?
Stooping to their level (Score:2)
Keep the high ground, folks. It's really in your best interests.
Interesting Hatton comment (Score:2)
Trust? Obviously you don't have too much confidence in NT, Ian.
Fight FUD with FUD (Score:2)
Instead, why not fight FUD with FUD? Mindcraft claims the study's still valid even though the systems weren't tweaked equally. If that's the truly the case we're home free! Do a study designed to show how *badly* an NT server can be tweaked, and publish the results. As long as you promote the results as "just as valid as the Mindcraft benchmarks", you are being perfectly honest.
So next time MS throws out the invalid Mindcraft survey (NT 2.5 times better), don't attack the survey. Just throw out the new Linux survey (Linux 153 times better) done using the "Mindcraft method".
Why can't they run it again? (Score:2)
My question is: Why cant they do it again? Just do the tests again....
I realize it will be expensive. But someone paid for it originally and came up with flawed results. Most companies would be looking at how to do it right the second time instead of saying
"Well ya know if we were to do it again we would not screw it up (But since microsoft isn't interested in a real test that isn't going to happen)"
Most companies do their best to cusion bad publicity. But microsoft seems to be proving time and again that ANY publicity is good.
Even if its bad.