Re: Bored Statistics Mavens, Take Heart!

1

I am so going to regret this...


Posted by: Cosma | Link to this comment | 07-29-07 7:36 PM
horizontal rule
2

See this is just like the IQ stuff and also the EP stuff: the trolls are so dishonest there's basically no incentive for competent people to get involved at the level of blogs.


Posted by: Gonerill | Link to this comment | 07-29-07 7:58 PM
horizontal rule
3

The attack is basically gibberish. Reading the discussion and trying to interact with D/avid K/ane just showed how incoherent the whole line of reasoning was.

The statistics in the Lancet paper are basically banally ordinary. The problem for K/ane is that they are frequentist calculations (which is what most papers use), but K/ane is a Bayesian. There's nothing wrong with being a Bayesian (other than the fact that all Bayesians are crazy people), but it means that he doesn't really understand the point he's trying to make.


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:07 PM
horizontal rule
4

Yay, Shalizi's (semi-) lurking!

So, I read dsquared's first post, but I haven't bothered to read the original David Kane piece. I take it that this is some kind of Confidence Intervals Gone Horribly Amok thing, right? One Too Many Assumptions of Normality?


Posted by: arthegall | Link to this comment | 07-29-07 8:08 PM
horizontal rule
5

"Stay tuned for more science, readers, because until this case of Red Bull runs out, I am going to be a blogoscientific force of nature!"

This sentence alone made the entire thing worth the price of admission.


Posted by: apostropher | Link to this comment | 07-29-07 8:09 PM
horizontal rule
6

Agreed. I read the whole thing out loud to Mr. B., between giggles.


Posted by: bitchphd | Link to this comment | 07-29-07 8:12 PM
horizontal rule
7

Arthegall: The Lancet paper reports the average mortality before and after the war, with associated confidence intervals. For these intervals, they assume asymptotic normality. For their main result, estimating the ratio of the pre and post war mortalities, they use a more sophisticated method, which produces more accurate confidence intervals. K/ane claims that the same data set could not possibly produce both the first pair of confidence intervals for the absolute mortalities, and for the ratio. But his argument is hopelessly confused.


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:18 PM
horizontal rule
8

4: Yeah, that's it. At my primitive level of understanding (that is, I know what a standard deviation is), the argument is that if you include the Fallujah cluster, and then calculate the confidence interval under the assumption that the data is normally distributed, the confidence interval gets so broad that it includes zero excess deaths as a possibility. And therefore excluding the Fallujah cluster was cherry-picking the data to make it look more reliable than it was. The response, again at my same primitive level of understanding, is that if you include the data from the Fallujah cluster, it's insane to assume that the data is normally distributed, and so calculating a confidence interval under that assumption would have been likewise insane, which is why the original authors didn't do it.


Posted by: LizardBreath | Link to this comment | 07-29-07 8:27 PM
horizontal rule
9

and 1: This sounds enormously promising.


Posted by: LizardBreath | Link to this comment | 07-29-07 8:28 PM
horizontal rule
10

7: Right, I understand that. Because I'm too lazy to read the original Lancet paper, what's the "more sophisticated method" that produces the "more accurate" confidence intervals?


Posted by: arthegall | Link to this comment | 07-29-07 8:33 PM
horizontal rule
11

What people have been talking about as the relevant method is bootstrapping, about which I know nothing other than what's in the linked article.


Posted by: LizardBreath | Link to this comment | 07-29-07 8:36 PM
horizontal rule
12

This is an argument about bootstrapping? Really??


Posted by: arthegall | Link to this comment | 07-29-07 8:40 PM
horizontal rule
13

They use a generalized linear model, a la McCullough-Nelder, and estimate the confidence intervals using bootstrapping. (You do have to be suspicious of any academic discipline that has both a "general linear model" and a "generalized linear model", though.)


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:43 PM
horizontal rule
14

Possibly -- at least it gets mentioned.

This was a funny one to read -- while I don't know much about statistics, I tend to follow along with arguments like this when they come up because I find them interesting, and usually feel as though I understand them. This one lost me instantly -- I don't know if it's incomprehensible because people are spouting bullshit, or whether the points of disagreement are just over my head.


Posted by: LizardBreath | Link to this comment | 07-29-07 8:44 PM
horizontal rule
15

12 - As far as I can make out, K/ane thinks that because they reported an ordinary mean with a confidence interval computed using asymptotic normality, that they are commited to using that method forevermore. So somehow using bootstrap for their main method is cheating. Fuck if I understand the logic.


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:44 PM
horizontal rule
16

13: Sweet! Just what I was looking for, thanks :-).

Seeing the posts on CT reminded me that I had saved, and had been meaning to read carefully, the Lancet paper... in my Copious Free Time, of course.


Posted by: arthegall | Link to this comment | 07-29-07 8:45 PM
horizontal rule
17

13 - K/ane is really realy confused, which makes everything he says hard to understand.


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:46 PM
horizontal rule
18

16 - The paper actually is very short. The statistical details make up maybe 2 pages. Papers in medical journals are always very short for some reason.


Posted by: Walt Someguy | Link to this comment | 07-29-07 8:48 PM
horizontal rule
19

I found that argument very interesting despite being entirely, 100%, incomprehensible, even for someone who's taken two statistics courses.

Now that I think about it, this David Kane may be the first person I've ever seen arguing so whole-heartedly and seemingly honestly against the "left-wing" position at Crooked Timber. That is, the first person who isn't obviously a liar, a lunatic, a libertarian, or a person brainwashed by Fox News. It's nice to see an actual difference of opinion.


Posted by: Cryptic Ned | Link to this comment | 07-29-07 8:50 PM
horizontal rule
20

He used to be a contributor to this blog (I don't know if he still is?) -- and they're a pretty respectable bunch. I remember when this whole attacking-the-Lancet thing came out, some of it started as posts over there... people were criticizing the blog's student editor for even allowing his posts to remain up. And even though they were sorta crazy sounding, and even though they were eventually taken down, I remember that it seemed pretty unseemly.

To me. At the time, at least. I dunno.


Posted by: arthegall | Link to this comment | 07-29-07 8:53 PM
horizontal rule
21

I know less statistics than LB. After a lot of reading, my guess is that the Fallujah numbers were so exceptionally high that if they were included in the averages they would throw them all off. One way to deal with this would be to put it in a framework (Bell curve) assuming the hypothetically possibility of an equally large deviation in the opposite direction (exceptionally low mortality locally, rather than exceptionally high). But as D^2 says, it's not possible for a cluster to be as much lower as the Fallujah cluster was higher (i.e., mortality less than zero).

The fiendish thing about this is that apparently a statistical approach is being recommended according to which a very large increase in mortality in a single location is evidence for the possibility of equally low mortality at a different location. That seems like an artifact of methodology and rigid formalization.

Kane seems to be asserting an approach as obligatory even though it produces some impossible conclusions in the case in question.


Posted by: John Emerson | Link to this comment | 07-29-07 9:05 PM
horizontal rule
22

But as I said, I know no statistics. I'm improvising.


Posted by: John Emerson | Link to this comment | 07-29-07 9:06 PM
horizontal rule
23

21 - That's basically right. But since that's so absurd, K/ane can't just come out and argue that. Instead, he argues that the Lancet people are committed to doing just that for some mysterious principle of consistency that doesn't exist. (Since the paper does report a few results that implicitly make the bell curve assumption, apparently you are committed to using the bell curve forevermore. It's like joining the mob.)

Anyway, I think K/ane's real point is that he thinks he's shown that the Lancet people faked their data. He doesn't want to come out and say it, since it's hard to prove, so he wants to you to think that their result is "impossible", and leave the obvious inference to you.


Posted by: Walt Someguy | Link to this comment | 07-29-07 9:16 PM
horizontal rule
24

8 basically summarizes both my level of expertise and my position, with the exception that, as I interpret it, if you include the data from the Fallujah cluster, it's insane to assume that the data is normally distributed, and so calculating a confidence interval under that assumption would also assume the possibility of resurrection.


Posted by: foolishmortal | Link to this comment | 07-29-07 9:24 PM
horizontal rule
25

There are multiple ways to derive a confidence interval from a bootstrap procedure. It appears to me (playing around with the data linked here) that the authors of the original study chose the worst one (based on mean and sd of the bootstrap distribution, not its percentiles). This was a mistake, but it didn't matter, because for the data without Fallujah the CIs are pretty well behaved and all look pretty much the same whichever method you use. But if you add Fallujah back in, you need to switch to something else because the first way is dumb. If you don't, you get dumb results. The K/ane criticism is just emphasizing the dumb results that it's possible to get from doing dumb statistics and claiming they mean something.

NB: If you do things sensibly, you get sensible results.


Posted by: I really need to come up with a pseudonym | Link to this comment | 07-29-07 10:21 PM
horizontal rule
26

So I had forgotten that the incident I mentioned in 20 was already re-linked to in dsquared's first post on this new thing. Blah.

Also, picking through the deltoid comment thread, this Kane quote seems particularly telling.

Take the exact same computer code that produces the 98,000 (CI 8,000 - 194,000) estimate. Now, don't exclude Falluja. What does the code produce?

Now, without the exact code, it is tough to know the answer to this. I argue that the authors do a lot to try to imply that the mean estimate would rise to 300,000 or so and that the confidence interval would get wider but still safely exclude zero, something like 150,000 to 600,000.

But this is, I am fairly certain, false. Using the exact same code, I bet that the lower bound would be well below -100,000. See the paper for details. Moreover, it is almost certain that the authors knew this and that they purposely organized and wrote the paper in such a way as to hide this fact, to purposely mislead readers (including smart readers like Tim and dsquared) into thinking that including Falluja would move the lower bound of the confidence interval for excess deaths up. After all, excluding Falluja is "conservative!"

Lame.


Posted by: arthegall | Link to this comment | 07-29-07 10:24 PM
horizontal rule
27

24 - If you are computing an average, and you have a large enough sample size, then the distribution of the data doesn't matter. You can treat it as if the data is normally distributed (the proverbial "bell curve"). That's why papers will report a confidence interval as if the underlying data is normal. If you have enough data then it doesn't matter. The Lancet paper is one of the cases where it does matter, so you use another technique (such as bootstrap).


Posted by: Walt Someguy | Link to this comment | 07-30-07 12:15 AM
horizontal rule
28

The statistics in the Lancet paper are basically banally ordinary. The problem for K/ane is that they are frequentist calculations (which is what most papers use), but K/ane is a Bayesian

by the way, he keeps saying this but I've seen no evidence, nish clish, that he actually is one, as opposed to someone who is wretchedly confused about confidence intervals but believes that saying "Bayesian!" every couple of sentences will conceal the fact.


Posted by: dsquared | Link to this comment | 07-30-07 1:38 AM
horizontal rule
29

Bayesian!

Bayesian!

Say, that does work!

27: woohoo central limit theorem! My minimal statistics knowledge pays off!


Posted by: Sifu Tweety | Link to this comment | 07-30-07 7:36 AM
horizontal rule
30

I've been trying to work the Hairy Ball Theorem into everything recently. I'm sure that it proves something about Iraq mortality.


Posted by: John Emerson | Link to this comment | 07-30-07 7:57 AM
horizontal rule
31

Let's not abuse the Hairy Ball Theorem, John.


Posted by: heebie-geebie | Link to this comment | 07-30-07 7:59 AM
horizontal rule
32

25: I really need to come up with a pseudonym

Ooh, can we help?


Posted by: heebie-geebie | Link to this comment | 07-30-07 8:01 AM
horizontal rule
33

32 -- "Spacial Bear" is still up for grabs. Also "El Hombre Muy Magnifico".


Posted by: Cryptic Ned | Link to this comment | 07-30-07 8:02 AM
horizontal rule
34

Also "Straightforward, Unambiguous Ned".


Posted by: heebie-geebie | Link to this comment | 07-30-07 8:03 AM
horizontal rule
35

A source for the discerning, classy pseudonym consumer.


Posted by: DS | Link to this comment | 07-30-07 8:10 AM
horizontal rule
36

"El Hombre Muy Magnifico" is a thing of beauty.


Posted by: Walt Someguy | Link to this comment | 07-30-07 8:10 AM
horizontal rule
37

I would really like to apologize for all my griping about pseuds. Really, I only have three categories of objection -- pseuds that are some variant of 'lurker'; pseuds consisting of two or three initials; and pseuds of the form 'common first name with trailing initial' (as in DaveB and DaveL, who I know well enough to differentiate now, but who intially made my head hurt. A peculiar first name would be fine; just not DaveMattMikeDougJeanDebraRachel and similar.)

But no one should be dissuaded from commenting by my griping. I'm not setting blog rules, I'm just complaining about names I can't keep straight. And there's absolutely no expectation that names be witty -- given that nothing's witty the fortieth time you see it, there's no need to try.


Posted by: LizardBreath | Link to this comment | 07-30-07 8:11 AM
horizontal rule
38

I really need to come up with a pseudonym

Done, and done.


Posted by: Jesus McQueen | Link to this comment | 07-30-07 8:12 AM
horizontal rule
39

37: And stay off her lawn, goddammit.


Posted by: apostropher | Link to this comment | 07-30-07 8:12 AM
horizontal rule
40

as in DaveB and DaveL, who...initially made my head hurt.

Ba-dum ching!


Posted by: heebie-geebie | Link to this comment | 07-30-07 8:14 AM
horizontal rule
41

You could always one-up Armsmasher and choose CarcassSplitter. Or the inexplicably still-unclaimed Bungdropper.


Posted by: apostropher | Link to this comment | 07-30-07 8:15 AM
horizontal rule
42

5:
Entirely worth the price of admission was this comment by Emerson:

Having won this battle, perhaps Shannon and David should shift their energies towards showing that electric production in Baghdad may be higher than before the war, rather than much lower as the lying media claim. I fear that they're focussing their energies too narrowly on this particular question, while letting dozens of other media lies pass by unrefuted.

They might also take a look at the bogus claims that more than two million Iraqis have fled the country, and all the silly talk about dead bodies. It seems pretty likely that the dead bodies are just leftovers from the Saddam regime which are being trucked around to be discovered again and again and again.

Posted by John Emerson ยท July 28th, 2007 at 4:06 pm



Posted by: Brock Landers | Link to this comment | 07-30-07 8:32 AM
horizontal rule
43

37: I think you are entirely correct, LB. You now have the support of the coveted M/tch M/lls demographic.


Posted by: M/tch M/lls | Link to this comment | 07-30-07 9:17 AM
horizontal rule
44

DaveMattMikeDougJeanDebraRachel strikes me as a peculiar first name. But LB's a New Yorker; maybe it's common there.


Posted by: Brock Landers | Link to this comment | 07-30-07 9:21 AM
horizontal rule
45

Holy smokes. I just looked at the Pew Research Center News Quiz and this is one of the questions:

In 2006, were more Iraqi civilians or more U.S. soldiers killed as a result of the fighting in Iraq?

Think about this. While we're having esoteric arguments over statistics in the Lancet study -- with no one seriously disputing that hundreds of thousands of Iraqis have died -- professional researchers have apparently decided that asking Americans whether more than a couple thousand* Iraqis have been killed is a legitimate way to assess their news savvy.

* A few questions earlier, the survey asks:

Since the start of military action in Iraq, about how many U.S. soldiers have been killed? To the best of your knowledge, have there been around 1,500, around 2,500, around 3,500, or around 4,500 military deaths in Iraq?


Posted by: Witt | Link to this comment | 07-30-07 9:21 AM
horizontal rule
46

Way back in 20 I remember when this whole attacking-the-Lancet thing came out, some of it started as posts over there... people were criticizing the blog's student editor for even allowing his posts to remain up. And even though they were sorta crazy sounding, and even though they were eventually taken down, I remember that it seemed pretty unseemly.

I think it's more that he directly accused Roberts of falsifying his evidence based on vague suspicions that their response rate was too high. This is obviously a hugely serious allegation, and I think the editors' decision to take it down lest they be seen as endorsing his accusations was the right one.


Posted by: snarkout | Link to this comment | 07-30-07 9:21 AM
horizontal rule
47

46 -- as I recall, he was accusing someone of falsifying at least some of the data. If not Roberts, then one or more of the on-the-ground Iraqis who did the polling.

But honestly, I don't care if he was accusing Roberts of falsifying his mom -- I still think that it was completely unseemly and aggressive, for people like Healy (who I totally respect and who is, along with dsquared, almost certainly right on the facts in this case) to go after Per/fors in a case like this. Go back and re-read the last paragraph in the Floating the Fraud Balloon post from last year. The SSSB was a group blog. Per/fors, as a grad student, was a volunteer editor for the enterprise, but this isn't like a newspaper editor -- she wasn't required to "sign off" on every post that was written up there by every contributor. All of the posts were written with the author's name on them.

What do you think it looks like, when K.H. comes in with (essentially) "I wonder what the director would think of all this?"

They basically browbeat a grad student into unpublishing someone else's writing. This is somewhat lame, even if the ones doing the browbeating are ultimately on the Side of the Angels. They should have refrained from trying to influence some other blog's posting policy, and then just used the power of their own bully pulpit to heap scorn on Kane for having written such a dumb analysis and offensive suggestion in a public place under his own name.

Suggestions like Healy's,

The blogosphere being what it is, I expect posts with titles like "Harvard statistics blog says Iraq survey results may be fraudulent" to start popping up pretty soon.
don't do much for me. Who the hell cares, if other people are too dumb, or too disingenous, to distinguish between clearly-marked authors on a group blog? Other people being dumb doesn't excuse you for acting a little like an ogre.


Posted by: arthegall | Link to this comment | 07-30-07 10:54 AM
horizontal rule