See this is just like the IQ stuff and also the EP stuff: the trolls are so dishonest there's basically no incentive for competent people to get involved at the level of blogs.
The attack is basically gibberish. Reading the discussion and trying to interact with D/avid K/ane just showed how incoherent the whole line of reasoning was.
The statistics in the Lancet paper are basically banally ordinary. The problem for K/ane is that they are frequentist calculations (which is what most papers use), but K/ane is a Bayesian. There's nothing wrong with being a Bayesian (other than the fact that all Bayesians are crazy people), but it means that he doesn't really understand the point he's trying to make.
Yay, Shalizi's (semi-) lurking!
So, I read dsquared's first post, but I haven't bothered to read the original David Kane piece. I take it that this is some kind of Confidence Intervals Gone Horribly Amok thing, right? One Too Many Assumptions of Normality?
"Stay tuned for more science, readers, because until this case of Red Bull runs out, I am going to be a blogoscientific force of nature!"
This sentence alone made the entire thing worth the price of admission.
Agreed. I read the whole thing out loud to Mr. B., between giggles.
Arthegall: The Lancet paper reports the average mortality before and after the war, with associated confidence intervals. For these intervals, they assume asymptotic normality. For their main result, estimating the ratio of the pre and post war mortalities, they use a more sophisticated method, which produces more accurate confidence intervals. K/ane claims that the same data set could not possibly produce both the first pair of confidence intervals for the absolute mortalities, and for the ratio. But his argument is hopelessly confused.
4: Yeah, that's it. At my primitive level of understanding (that is, I know what a standard deviation is), the argument is that if you include the Fallujah cluster, and then calculate the confidence interval under the assumption that the data is normally distributed, the confidence interval gets so broad that it includes zero excess deaths as a possibility. And therefore excluding the Fallujah cluster was cherry-picking the data to make it look more reliable than it was. The response, again at my same primitive level of understanding, is that if you include the data from the Fallujah cluster, it's insane to assume that the data is normally distributed, and so calculating a confidence interval under that assumption would have been likewise insane, which is why the original authors didn't do it.
and 1: This sounds enormously promising.
7: Right, I understand that. Because I'm too lazy to read the original Lancet paper, what's the "more sophisticated method" that produces the "more accurate" confidence intervals?
What people have been talking about as the relevant method is bootstrapping, about which I know nothing other than what's in the linked article.
This is an argument about bootstrapping? Really??
They use a generalized linear model, a la McCullough-Nelder, and estimate the confidence intervals using bootstrapping. (You do have to be suspicious of any academic discipline that has both a "general linear model" and a "generalized linear model", though.)
Possibly -- at least it gets mentioned.
This was a funny one to read -- while I don't know much about statistics, I tend to follow along with arguments like this when they come up because I find them interesting, and usually feel as though I understand them. This one lost me instantly -- I don't know if it's incomprehensible because people are spouting bullshit, or whether the points of disagreement are just over my head.
12 - As far as I can make out, K/ane thinks that because they reported an ordinary mean with a confidence interval computed using asymptotic normality, that they are commited to using that method forevermore. So somehow using bootstrap for their main method is cheating. Fuck if I understand the logic.
13: Sweet! Just what I was looking for, thanks :-).
Seeing the posts on CT reminded me that I had saved, and had been meaning to read carefully, the Lancet paper... in my Copious Free Time, of course.
13 - K/ane is really realy confused, which makes everything he says hard to understand.
16 - The paper actually is very short. The statistical details make up maybe 2 pages. Papers in medical journals are always very short for some reason.
I found that argument very interesting despite being entirely, 100%, incomprehensible, even for someone who's taken two statistics courses.
Now that I think about it, this David Kane may be the first person I've ever seen arguing so whole-heartedly and seemingly honestly against the "left-wing" position at Crooked Timber. That is, the first person who isn't obviously a liar, a lunatic, a libertarian, or a person brainwashed by Fox News. It's nice to see an actual difference of opinion.
He used to be a contributor to this blog (I don't know if he still is?) -- and they're a pretty respectable bunch. I remember when this whole attacking-the-Lancet thing came out, some of it started as posts over there... people were criticizing the blog's student editor for even allowing his posts to remain up. And even though they were sorta crazy sounding, and even though they were eventually taken down, I remember that it seemed pretty unseemly.
To me. At the time, at least. I dunno.
I know less statistics than LB. After a lot of reading, my guess is that the Fallujah numbers were so exceptionally high that if they were included in the averages they would throw them all off. One way to deal with this would be to put it in a framework (Bell curve) assuming the hypothetically possibility of an equally large deviation in the opposite direction (exceptionally low mortality locally, rather than exceptionally high). But as D^2 says, it's not possible for a cluster to be as much lower as the Fallujah cluster was higher (i.e., mortality less than zero).
The fiendish thing about this is that apparently a statistical approach is being recommended according to which a very large increase in mortality in a single location is evidence for the possibility of equally low mortality at a different location. That seems like an artifact of methodology and rigid formalization.
Kane seems to be asserting an approach as obligatory even though it produces some impossible conclusions in the case in question.
But as I said, I know no statistics. I'm improvising.
21 - That's basically right. But since that's so absurd, K/ane can't just come out and argue that. Instead, he argues that the Lancet people are committed to doing just that for some mysterious principle of consistency that doesn't exist. (Since the paper does report a few results that implicitly make the bell curve assumption, apparently you are committed to using the bell curve forevermore. It's like joining the mob.)
Anyway, I think K/ane's real point is that he thinks he's shown that the Lancet people faked their data. He doesn't want to come out and say it, since it's hard to prove, so he wants to you to think that their result is "impossible", and leave the obvious inference to you.
8 basically summarizes both my level of expertise and my position, with the exception that, as I interpret it, if you include the data from the Fallujah cluster, it's insane to assume that the data is normally distributed, and so calculating a confidence interval under that assumption would also assume the possibility of resurrection.
There are multiple ways to derive a confidence interval from a bootstrap procedure. It appears to me (playing around with the data linked here) that the authors of the original study chose the worst one (based on mean and sd of the bootstrap distribution, not its percentiles). This was a mistake, but it didn't matter, because for the data without Fallujah the CIs are pretty well behaved and all look pretty much the same whichever method you use. But if you add Fallujah back in, you need to switch to something else because the first way is dumb. If you don't, you get dumb results. The K/ane criticism is just emphasizing the dumb results that it's possible to get from doing dumb statistics and claiming they mean something.
NB: If you do things sensibly, you get sensible results.
So I had forgotten that the incident I mentioned in 20 was already re-linked to in dsquared's first post on this new thing. Blah.
Also, picking through the deltoid comment thread, this Kane quote seems particularly telling.
Take the exact same computer code that produces the 98,000 (CI 8,000 - 194,000) estimate. Now, don't exclude Falluja. What does the code produce?Now, without the exact code, it is tough to know the answer to this. I argue that the authors do a lot to try to imply that the mean estimate would rise to 300,000 or so and that the confidence interval would get wider but still safely exclude zero, something like 150,000 to 600,000.
But this is, I am fairly certain, false. Using the exact same code, I bet that the lower bound would be well below -100,000. See the paper for details. Moreover, it is almost certain that the authors knew this and that they purposely organized and wrote the paper in such a way as to hide this fact, to purposely mislead readers (including smart readers like Tim and dsquared) into thinking that including Falluja would move the lower bound of the confidence interval for excess deaths up. After all, excluding Falluja is "conservative!"
Lame.
24 - If you are computing an average, and you have a large enough sample size, then the distribution of the data doesn't matter. You can treat it as if the data is normally distributed (the proverbial "bell curve"). That's why papers will report a confidence interval as if the underlying data is normal. If you have enough data then it doesn't matter. The Lancet paper is one of the cases where it does matter, so you use another technique (such as bootstrap).
The statistics in the Lancet paper are basically banally ordinary. The problem for K/ane is that they are frequentist calculations (which is what most papers use), but K/ane is a Bayesian
by the way, he keeps saying this but I've seen no evidence, nish clish, that he actually is one, as opposed to someone who is wretchedly confused about confidence intervals but believes that saying "Bayesian!" every couple of sentences will conceal the fact.
Bayesian!
Bayesian!
Say, that does work!
27: woohoo central limit theorem! My minimal statistics knowledge pays off!
I've been trying to work the Hairy Ball Theorem into everything recently. I'm sure that it proves something about Iraq mortality.
Let's not abuse the Hairy Ball Theorem, John.
25: I really need to come up with a pseudonym
Ooh, can we help?
32 -- "Spacial Bear" is still up for grabs. Also "El Hombre Muy Magnifico".
Also "Straightforward, Unambiguous Ned".
A source for the discerning, classy pseudonym consumer.
"El Hombre Muy Magnifico" is a thing of beauty.
I would really like to apologize for all my griping about pseuds. Really, I only have three categories of objection -- pseuds that are some variant of 'lurker'; pseuds consisting of two or three initials; and pseuds of the form 'common first name with trailing initial' (as in DaveB and DaveL, who I know well enough to differentiate now, but who intially made my head hurt. A peculiar first name would be fine; just not DaveMattMikeDougJeanDebraRachel and similar.)
But no one should be dissuaded from commenting by my griping. I'm not setting blog rules, I'm just complaining about names I can't keep straight. And there's absolutely no expectation that names be witty -- given that nothing's witty the fortieth time you see it, there's no need to try.
I really need to come up with a pseudonym
Done, and done.
37: And stay off her lawn, goddammit.
as in DaveB and DaveL, who...initially made my head hurt.
Ba-dum ching!
You could always one-up Armsmasher and choose CarcassSplitter. Or the inexplicably still-unclaimed Bungdropper.
5:
Entirely worth the price of admission was this comment by Emerson:
Having won this battle, perhaps Shannon and David should shift their energies towards showing that electric production in Baghdad may be higher than before the war, rather than much lower as the lying media claim. I fear that they're focussing their energies too narrowly on this particular question, while letting dozens of other media lies pass by unrefuted.They might also take a look at the bogus claims that more than two million Iraqis have fled the country, and all the silly talk about dead bodies. It seems pretty likely that the dead bodies are just leftovers from the Saddam regime which are being trucked around to be discovered again and again and again.
Posted by John Emerson ยท July 28th, 2007 at 4:06 pm
37: I think you are entirely correct, LB. You now have the support of the coveted M/tch M/lls demographic.
DaveMattMikeDougJeanDebraRachel strikes me as a peculiar first name. But LB's a New Yorker; maybe it's common there.
Holy smokes. I just looked at the Pew Research Center News Quiz and this is one of the questions:
In 2006, were more Iraqi civilians or more U.S. soldiers killed as a result of the fighting in Iraq?
Think about this. While we're having esoteric arguments over statistics in the Lancet study -- with no one seriously disputing that hundreds of thousands of Iraqis have died -- professional researchers have apparently decided that asking Americans whether more than a couple thousand* Iraqis have been killed is a legitimate way to assess their news savvy.
* A few questions earlier, the survey asks:
Since the start of military action in Iraq, about how many U.S. soldiers have been killed? To the best of your knowledge, have there been around 1,500, around 2,500, around 3,500, or around 4,500 military deaths in Iraq?
Way back in 20 I remember when this whole attacking-the-Lancet thing came out, some of it started as posts over there... people were criticizing the blog's student editor for even allowing his posts to remain up. And even though they were sorta crazy sounding, and even though they were eventually taken down, I remember that it seemed pretty unseemly.
I think it's more that he directly accused Roberts of falsifying his evidence based on vague suspicions that their response rate was too high. This is obviously a hugely serious allegation, and I think the editors' decision to take it down lest they be seen as endorsing his accusations was the right one.
46 -- as I recall, he was accusing someone of falsifying at least some of the data. If not Roberts, then one or more of the on-the-ground Iraqis who did the polling.
But honestly, I don't care if he was accusing Roberts of falsifying his mom -- I still think that it was completely unseemly and aggressive, for people like Healy (who I totally respect and who is, along with dsquared, almost certainly right on the facts in this case) to go after Per/fors in a case like this. Go back and re-read the last paragraph in the Floating the Fraud Balloon post from last year. The SSSB was a group blog. Per/fors, as a grad student, was a volunteer editor for the enterprise, but this isn't like a newspaper editor -- she wasn't required to "sign off" on every post that was written up there by every contributor. All of the posts were written with the author's name on them.
What do you think it looks like, when K.H. comes in with (essentially) "I wonder what the director would think of all this?"
They basically browbeat a grad student into unpublishing someone else's writing. This is somewhat lame, even if the ones doing the browbeating are ultimately on the Side of the Angels. They should have refrained from trying to influence some other blog's posting policy, and then just used the power of their own bully pulpit to heap scorn on Kane for having written such a dumb analysis and offensive suggestion in a public place under his own name.
Suggestions like Healy's,
The blogosphere being what it is, I expect posts with titles like "Harvard statistics blog says Iraq survey results may be fraudulent" to start popping up pretty soon.don't do much for me. Who the hell cares, if other people are too dumb, or too disingenous, to distinguish between clearly-marked authors on a group blog? Other people being dumb doesn't excuse you for acting a little like an ogre.