That guy appears to take delight in building a his ontology of bungling.
The results of some of the lab assignments in my first semester college class in physics cast some doubt on the existence of gravity. We handwaved it away by blaming the instruments.
Often 80% or more of my keystrokes are correct.
You should ask Randall Munroe what he thinks.
I only read the abstract. Was it a joke that doesn't get explained until the end or something?
I don't think the reading of that article is an experiment worth replicating.
That's an impressive display of some skill or another. Probably should have been a lawyer.
Is Harvard having trouble? Maybe some of you have cut down your donations and they can't hire good people now.
There are two kernels of sense here: (i) learning how to do an experiment properly is hard, and you can never give a complete account of every step needed of every procedure. (Dr. Polanyi, Dr. Michael Polanyi, to the white courtesy phone.) (ii) A failed attempt at replicating provides more negative evidence when it also explains why the original experimenters were deceived.
But yeah, he should invest in cold fusion, and submit this piece as an editorial to the Journal of Evidence-Based Haruspicy.
Hmm, will ST weigh in? Comment but under deep, deep cover? Or stay the hell away?
Four legs Original researchers good!
Failed replicators bad!
12: Up to the point where I stopped reading, there's a good case for why you have to do a really good job of running a replication experiment. But this apparently leads to the conclusion that the very idea of replication is a personal offense against the original experimenters who would never have gotten a meaningful result if their experiment hadn't been perfect in every way and you don't want to undermine scientists, do you? Do you?!
15:
"But they were from his own lab! And you saved his experiment!"
"I didn't let them fail to replicate it, and that's not the same thing."
11: Given that we're talking about the university of Marc Hauser, Harvey Mansfield, ...
16: Right, sloppy attempts at replication are of little use, except to say that the effect isn't very robust. But then the "how dare you impugn my honor" stuff comes in, which is just weird. (I seriously wonder how much of the craziness in the essay might be prompted by the Marc Hauser affair.)
I tried to see the canals on Mars and all I got was another meaningless failed replication. Bungled again!
What 20 said.
And, also, if the author were a real scientist he'd know that the main reason you try to replicate someone else's work is as the first step in building on it. If there's a paper that says, I don't know, "Bleaching in this species of coral is slower when the water's alkaline", then you might read it and wonder "hmm, I wonder if that's true in other species of coral?" And if you want to try it in your own lab, then step one is to see if you can reproduce the original result with the same species, before you start trying it with others.
Also also, this is flawed: "Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way" - logically, an equally likely explanation will be that the original experimenter bungled something along the way.
Also also also, the "black swan" analogy is flawed. Most scientific practice doesn't depend on finding a single data point which will irrevocably disprove a hypothesis. It's about getting statistical significance. The existence of one black swan does not, in fact, irrevocably disprove the hypothesis that all swans are white. If 999 subsequent expeditions to Australia had failed to find any black swans, I might start to suspect that the first black swan had been an experimental error. Maybe it wasn't actually a swan but a misidentified goose. Maybe it had just had some india ink dropped on it. Maybe it was a fake.
I _am_ a philosopher of science.* I have already dispatched my ninja-monkey-death-squad to deal with this guy.
Also, he's a neuroscientist. That figures. No offence to Sifu, but neuroscientists are right up there with engineers and physicists in the 'glibly solving** other disciplines problems.' stakes.
* retired variety.
** i.e. not solving.
22.2 huh. I assumed he was a social psychologist, because that's what he seems to be talking about most.
I _am_ a philosopher of science.* I have already dispatched my ninja-monkey-death-squad to deal with this guy.
Thus I refute you! *roundhouse kick to the head*
re: 23
Working on social psychology neuro stuff. Yeah.
Also, he's a neuroscientist. That figures.
I assumed he was a social psychologist
He's a social neuroscientist, which is really the key. This is a field that has been hit by controversy over false positives before. Also, and more specifically causal in this case, there has been an ongoing effort by, ah, let's say more statistically sophisticated psychologists and neuroscientists to replicate some of the major claims in the social psychology literature (see e.g. here and here). These efforts have not always been taken with good humor by the authors of the original studies (the authors of this study, for instance, have quite strongly pushed back against the critics).
One (very) prominent member of the department to which the author is attached has sort of taken on the role of pushing back at the replicators (as bullies and "second tier scientists") mostly by mocking people on his twitter. As best I can tell this essay is a (really, really poorly executed) attempt to add intellectual heft to this pushback, and position the department as sort of ground zero for the anti-replicators.
Needless to say, this move is not regarded fondly in some other corners of the department -- the scientist who runs the psychfiledrawer site is an alum of a different lab at the institution -- in part because the Marc Hauser affair is so fresh in memory, and one one would hope that department members would have a little bit of humility, in part because it's just really horrendously poorly argued, and in part because many people's first reaction is to say "well, Harvard and/or neuroscientists, what do you expect" when an easy majority of neuroscience done by department members is careful, designed to be replicable, built on large effects, and highly concerned about methodological robustness. Built, in other words, to be replicable (and to hopefully be replicated). And/or which begins with replications of the work of others as a baseline. You know, as science ought to be done (not to step on ttaM's turf).
Anyhow, there has been critcism of the linked piece on the neuroscience internet, as you might imagine, although most of it has been generously polite and mostly focused on the really, really obvious statistical fallacy that ajay points out in 21.
Tal Yarkoni raises an interesting empirical/sociological question here, of how many null results people need to see before they stop believing a result.
Is Harvard having trouble?
I'm told rather frequently that the quality of faculty they hire these days just isn't what it used to be.
Someone else here linked to a tweet from Dan Gilbert saying that replication was useless. That was really weird, because I like his work, and I sometimes show his TED talk to students. Now I feel like I have to distrust his work.
What's up with this trend?
28 "Is Harvard having trouble?"
According to Greg Mankiw, their endowment isn't getting a rate of return commensurate with their obvious awesomeness.
When I interviewed for a post-doc position in neurolinguistics at Cornfed Football and Boy Molesting University several years back, they asked me what projects I'd like to do. I said since my PhD work produced a novel result, with non-trivial theoretical implications, it'd be nice to replicate it in a different lab, with some different parameters. There was open shock and horror on their faces.
I have mixed feelings about that now. Part of me feels I did exactly the right thing, but the part of me that has since been socialised into the garbage practices rife in my field is embarassed at how green I was. Replicate your PhD results, lol.
How many papers have to say replication is useless before people will believe it?
re: 30
Allegedly their annual return on their endowment is more than our* endowment.
* place I work.
32: Either one or infinity, depending on whether you phrase that as a positive or negative claim.
I'd guess that 28 has been true for almost as long as there's been a university there. And when you try to hire high-profile scholars, you're going to hire more than your share of charlatans.
Building on ajay's 20, one of my favorite things about (some kinds of biology) is the use of positive and negative controls. It's harder to do in other fields, but I like seeing in-experiment checks of experimenter/equipment error.
I don't understand how the author can so rapidly dismiss the value of learning critical experimental parameters and the self-corrective nature of science. He acknowledges that fraud happens, but lots of fraud is discovered when a result remains stubbornly irreproducible. (Anybody following the STAP thing at RIKEN?) If independent reproducibility isn't important, why do statistical analysis at all? Why not case studies on individuals? Nobody would expect n=1 to reproduce cleanly, so problem solved!
I noticed that the arguments are carefully limited to social sciences. I'm trying to be charitable, but I don't see why social sciences would be different than, say, clinical drug trials. It's a complicated field, but if professionals who are publishing in the field can be compared to seventh grade science lab, I'm not sure the training programs are doing their job. Are there no SOPs?
Also, 28 is really outrageously rude.
31: By the time you finish, you're supposed to be so bored with your dissertation topic you can barely stand it.
37: Sure, but that's where Grit comes in.
...mostly by mocking people on his twitter.
It's Dr. Electoshock Because Boredom again.
I was going to reference the STAP issue. One nice thing about biology/chemistry is that a lot of useful things are about making reagents for other useful experiments as in the STAP case (stem cells- useful!) Aside from understanding the mechanism by which something like STAP was proposed to work, if no one can reproduce your procedure, no one will give a shit about your work after a couple years, regardless of whether they found you actually photoshopped the cell images. The fraud revelation just made it faster to jump ahead to the "ignore this result" stage.
At this point I'm starting to think that a useful heuristic is "If it appears in Nature and it concerns stem cells, it's presumptively BS until proven otherwise". The editors there seem really determined not to learn any lessons on this particular subject.
@28, 36 last: I certainly hear more inane statements from Harvard faculty then I used to, but I suspect that's just because the web/social media is so great at empowing people to embarrass themselves.
29: First, none of Gilbert's work has failed to replicate, as far as I know, so it's not bald self-interest in that sense.
Without any firsthand knowledge, I think there might be a few things going on: first, Gilbert is likely friend to or coauthor with some of the authors whose work has failed to replicate; he probably feels like he is looking out for them, and for his students. Second, social psychology has had -- and really, is continuing to have -- something of a renaissance; departments are doing a lot of hiring, public interest is very high, excitement is high within the field. I think he is likely concerned that focusing on replications will take the wind out of those sails. Third, it really is the case that some foundational papers to modern social psychology have been falling down -- the walking speed primed with age words study is quite famous, and is, I believe, discussed somewhat extensively in Gilbert's textbook -- and Gilbert might (quite rightly) feel like the whole field is under attack, and somebody should defend it. Finally, I think he might just have a visceral disinterest in replication; he believes that social psychology should be about telling interesting and englightening stories about human nature, with powerful narratives that are accessible to non-scientists. Replication, and arguments over the validity of null-hypothesis significance testing, are at best orthogonal to that.
Mitchell is a more complicated case; he is relatively young (he just got tenure) and possibly is trying to make a name for himself in a department that has some incredibly heavyweight names (Gilbert, Pinker and Banaji, among others) with reputations that extend well outside of their subfields into (certainly in the case of Pinker, but also Gilbert) the popular consciousness. It seems like maybe he is a bit competitive with them (particularly with Gilbert and Pinker, who both also teach intro psych and are sort of legends among the undergrad population) and trying to stake out his own territory. Also, he is historically a pretty combative scientist, having first gained notice by fighting with Rebecca Saxe at MIT over the nature of Theory of Mind representations in the brain.
The bad thing about biology/chemistry is that if you ever have anything to do with either of them, you get spam email for knockout mice and antibodies for the rest of your life.
It's interesting to note that the Facebook study everybody was up in arms about is squarely in the realm of social psychology/social priming and is also (because of its tiny effect size and enormous sample) inherently unreplicable.
And targeted ads on every site I browse! I never knew political blogs were so concerned with kinase inhibitors.
@44: I find the barrage of "Come to our conference in Dubai" and "Submit a paper to this journal that you've never heard of" emails even more annoying than the ads for antibodies.
You get invited to phony conferences in Dubai? My phony conference opportunities are all in provincial Chinese cities.
If people keep starting new journals, people will have to public more replications just to fill the space.
Or maybe I should apologize for replication.
I get ads for kinase inhibitors based on google searches for "plasma diagnostics."
Some of the people who were visiting Beijing the same time I was took side trips to conferences in provincial cities. Mostly because all the speakers were given fat envelopes full of cash as some kind of per diem or honorarium or something, I think. That seems to be a thing in China. "Ah, you're our seminar speaker today! Welcome. Here's a big wad of cash."
36.2: Not coincidentally, many of the scientists who are pushing replication come from subfields where effect sizes are large enough that you actually can do perfectly good experiments with an N of 2. And case studies (of individuals with brain lesions) are actually relatively common in neuroscience, and although they are themselves inherently sort of unreproducible have generally produced results that have been corroborated much more easily than the social priming stuff has.
36.3: Truthfully the methodology of the papers in question is not (should not be) that complicated. You show somebody a list of words, and then they walk to the elevator, and you time how long it takes: should that really be more sensitive to the vagaries of experimental conditions than (say) measuring the cosmic background radiation?
If you change the font, you aren't doing a proper replication.
Finally, I think he might just have a visceral disinterest in replication; he believes that social psychology should be about telling interesting and englightening stories about human nature, with powerful narratives that are accessible to non-scientists. Replication, and arguments over the validity of null-hypothesis significance testing, are at best orthogonal to that.
Too bad Harvard scrapped the Core program, with its emphasis on "ways of knowing," for the less ambitious General Education program, which is basically just distribution requirements.
58: You are, I think, kidding, but the actual debate has been at about that level (see the link in 43). I'm not sure if font choice has been mentioned but it wouldn't surprise me.
You show somebody a list of words, and then they walk to the elevator, and you time how long it takes: should that really be more sensitive to the vagaries of experimental conditions than (say) measuring the cosmic background radiation?
Maybe the effect disappears if they're walking on a carpet, or if the hallway is too narrow, or if they get distracted by too many things on the wall...
The Mona Lisa is science. It's just got an extremely exacting protocol.
43.2: I actually think that most of these aren't bad arguments, but is it impossible to build in some sort of small scale replication when doing follow-up work? Maybe that's a really naive question, but it seems like the sort of thing where a control group could serve to replicate an old result within a larger study. I don't disagree that doing everything two or three times independently is a good way to advance the field, but it seems like some sort of partial replication is common elsewhere. For example, releasing raw data in supporting sections so it can be independently analyzed, having detailed decision trees for study enrollment and inclusion of subjects into final analysis, or designing a smaller study-within-a-study to confirm previous conclusions aren't replication per se, but don't seem like onerous requirements (although maybe the last is simply impossible if it requires a thousand subjects or something). I'd hate to have based a follow-up study on something ephemeral.
57.3: But that's what the linked essay is arguing. I'm happy to grant that studies with actual people are complicated compared to petri dishes. He suggests it's impossible to replicate details such as how they coax subjects to stay still in the MRI. He also suggests that it's likely that failure to replicate can be attributed to experimenter error, which I don't doubt, but if most failure is categorized as experimenter error, there's a big problem with standard procedures. Also, he dismisses the importance of learning which parameters are key to getting reproducible results. If you measure cosmic rays, and someone else does it differently and gets different results, you can argue over whose measurements are more accurate, but if two labs do it in the same way and get different results, there's something subtle that suggests no data should be compared until the method is at least sort of standardized for whatever variable is causing the different values.
He suggests it's impossible to replicate details such as how they coax subjects to stay still in the MRI.
Chloroform.
Anyway, this arguing against replication is pointless. What science really needs is an argument against correcting for multiple comparisons. Or somebody to correct for multiple comparisons so I don't have to do it.
64.1: Yes, of course, and that happens a lot. Actually, one of the points of the psychfiledrawer site I linked above is to allow people to report results of incidental replications that they did in the course of pursuing other studies (since they usually won't rise to the level of a publishable project on their own). Gilbert, though, seems to think that these efforts are per se a distraction, and that the current situation (where replications are not particularly publishable) is perfectly correct, because replications (or non-replications) are not interesting.
64.2: Right. I think those arguments (that these experiments are so surpassingly sensitive that it is impossible to correctly report how to replicate them, and likewise impossible to replicate them) are ludicrous, and either wrong (the experiments are not that sensitive and the results are not true) or indicative of massive methodological problems in the field of social psychology. (Really, I tend to think both of the preceeding are valid to some degree.)
Honestly I stopped reading after line 1. "...unsuccessful experiments have no meaningful scientific value." When you define "unsuccessful" as "didn't get the result I wanted/that happened before" you're not doing science.
Granted, we sometimes say that as a shorthand, but it's always with an implied reason- that experiment didn't work (because the cells were contaminated.) If you think you did everything right and throw out the result anyway because it didn't "work", just quit your job.
If you think you did everything right and throw out the result anyway because it didn't "work", just quit your job.
Throwing out the result would be a massive improvement on what I constantly see in my field - massaging the result and bullshitting around it until, hey look, it does support the hypothesis, after all!
It all just ends up being, 'Here's a bunch of stuff we did, and a thing that happened.' That's not a scientific claim. It's an anecdote.
His argument would also support stopping publishing the method section of a paper. Why would you want to know? Replicating the experiment would be useless, and there's so much tacit knowledge there that the method section is incomplete. Really you should just publish the conclusions.
Limitations of this study include that I kept the five dollars I found. Therefore, the work is not replicable.
Even if we take the claim at face value--that there are real, statistically significant effects that can only be measured under very precise and carefully controlled conditions and so are unlikely to show up when other researchers do the same experiment in a slightly different way--it seems like such effects would, consequently, be almost completely unimportant in the real world.
re: 74
Yeah, it points to a failure to ask the right questions at the appropriate level of generality.
To bend over backwards in fairness to social priming researchers, they tend to talk about these effects as manifestations of a subtle but pervasive tendency for social behavior to be influenced by implicit cues. So they would say that the fact that any one experiment is fragile is irrelevant, because while the effect size for any given result is small, the effect is omnipresent (if hard to test at the appropriate level of generality) and thus has a large aggregate effect on individual behavior; they're doing indirect and momentary tests of a fundamental and constant but mostly hidden phenomenon, which aren't ideal, but which are the only way to get at it.
At some point, of course, the subtlety and inaccessibility-to-experiment of a phenomenon becomes indistinguishable from its nonexistence.
'Here's a bunch of stuff we did, and a thing that happened.'
This sort of thing can be really useful if you're thinking of doing something similar. It's not a scientific claim, but it is still worth sharing with peers and colleagues. Unfortunately there is no journal of scientific anecdotes that I'm aware of, so these things tend to get shared informally.
79: Like the now well-researched 'rubber hand' phenemenon, which I'm told got started from clowning around with a rubber hand someone brought to a lab xmas party...
People are more likely to bend over backward in fairness to social priming researchers if the researchers are wearing white lab coats over corsets.
79, If only we could go back to the thirties when journals were full of articles with names like "Studies on Pneumococcus Replication, Part X: Replication On Meat Agars".
squarely in the realm of social psychology/social priming and is also (because of its tiny effect size and enormous sample) inherently unreplicable
Not to mention fuckin' sentiment analysis.
78 sounds disturbingly similar to justifications for the irreproducibility of homeopathy.
84: Returning to an upright position, I agree. Especially when you're studying effects that you are in theory claiming are innate or at least fairly universal features of the human mind.
Executive Summary: Silly twit haz access to internet.
85.1: there's no actual need to fall to the ground every time I post a comment. Rise, sir, from this semi-recumbent position. It is most indecorous.
Silly twit haz access to internet.
And, in the case of Gilbert, tenure at Harvard and a huge popular following.
Mitchell also has tenure.
91 is the nicest thing anyone's said about me all month.
And most of the month so far has been spent entirely surrounded by people speaking either Malay or Chinese. I got a lot of reading done.
Do you know enough Malay and Chinese to be sure somebody hasn't said something nicer than 91?
Well, no. I mean, no one was pointing at me and saying "ang moh", but beyond that they could have been saying anything.
No one really understands us.
"We have secret sauce that explains why we can get an effect and you can't" is exactly -- exactly -- the argument pro-psi researchers use to defend the fact that people who believe in psi can make the experiments work, and people who don't can't. I hope most of these people would recognize it as bullshit in that case. Who knows? Maybe it's true that a pro-sci aura is a precondition for the psi effect to reveal itself, but -- I'm about to make a claim that an actual philosopher of science will tell me is wrong, I'm sure -- it seems to me that the work of science is fundamentally intersubjective persuasion, so you have to be able to find methods that can accomplish that goal. If you believe your shit so hard, then answer replication failures with the more robust methods Jason Mitchell is going on about. The world will live through a period of uncertainty about whether priming effects, etc., are real. I love social psychology and yet I also know it's not as urgent was whether some drug has deadly side effects.
I'm okay with non-replication as long as the results are published in the Journal of Just So Stories.
The Journal of Irreproducible Results is funny for a reason.
100: Would definitely be an improvement over PNAS.
As a name, for the journal, I mean. Because I think PNAS is full of bad studies. Fuck, it's too hot, I'm going home.
I agree with the essay. Replication is totally overrated.
Just so everyone else has the same earworm I do, the title of this thread has turned into a very badly scanning version of "Lola".
That scanned so poorly I didn't catch the earworm.
105. Glad it isn't just me. But I find an awful lot of phrases do that.
You all have some fertile soil for earworms.
It works a little better because I misremembered the actual words to Lola -- I had "Now, I'm no fool" where it should have been "Now I'm not dumb". Three words in a row are enough to lead me down any garden path you like.
I was trying for "Now, I'm not the world's most passionate guy."
Mitchell also has tenure.
And he was promoted internally, not poached from elsewhere, and it looks like they gave him tenure a year early. So a lot of people in that department must think he's a superstar.
110 is me too. I couldn't make it work either.
Or they want to use him the way a hoplite uses a shield.
Now I've got Weird Al Yankovic's "Yoda" running through my head. Thanks a lot.
111: he did his PhD and one of his two postdocs in the department, per his CV, so yes. His research is also quite prominent -- he is unquestionably one of the biggest names in social neuroscience.
he did his PhD and one of his two postdocs in the department, per his CV, so yes.
Nepotism, hooray!
And I now have the idea of a generic Mitchell and Webb sketch running in my head.
(My advisor's advisor, my advisor's advisor's advisor, and my advisor's advisor's other advisee are the three other faculty in my department working in my subfield.)
Since the Piketty thread petered out in a welter of links to Jacobin, let's see what that august publication has to say about Harvard faculty...
We just hired the wife of a current faculty member who himself got his doctorate in our department. She rules, though. Totally the right hire.
Yay! It's good when departments can solve two-body problems.
Yep. She's one of four (!) hires this year, at least three of which (including her) I'm pretty excited about.
Oh look, I have a paper coming out in PNAS next week.
I thought the deal with PNAS was that papers submitted through the normal channel were generally good, and papers submitted through the "I know a NAS member" channel were generally terrible?
Oh look, I have a paper coming out in PNAS next week submitted through the "I know a NAS member" channel.
Well, I'm sure your parents are very proud.
But I'm only like 17th author so I don't take that personally.
I mean, I don't think there's, like, a rule that those have to be terrible. I just definitely know about some that are terrible.
Do all 17 authors have a sign a thing averring a substantial contribution and final approval?
What do you mean all 17? There are plenty more after me.
PNAS is one of the journals that now requires a "Smith planned the work, Jones did experiments x, so-and-so did experiments y, Smith Jones and so-and-so wrote the paper" section. That must be big when 17+ authors are involved.
On earworms: My personal remedy for them is to imagine the song as sung/performed by William Shatner. Now, that means you have the song as-by-Shatner in your head briefly, but I find that this version tends to overwrite the previous earworm and then go away.
@134: Are you implying that Shatner's version of Mister Tambourine Man doesn't rock?
Nah, it's like four lines long. A bunch of people designed it, others ran it, others analyzed it.
138: What happens when brains "light up" too brightly and for too long.
In the interest of completeness, here is a not-entirely-full-throated defense of Mitchell's piece, by somebody who doesn't agree with him but who would like to find common ground.
I do not personally find it terribly convincing even on its own terms, but it is mostly thoughtful.
I found it still basically moronic. I agree the replicating an experiment is often very difficult, but so is staging the same intervention across a whole series of participants for any one project. The only actual way to know that your experimental protocol can actually deliver a clean intervention is to do it again.
Or to split your sample and test different sections, but that seems pretty much the same thing. Anyway, I continue to object to arguments that basically amount to, "You aren't smart enough to check my work."
142: What blows my mind here is the lack of self-awareness. If my phenomenon is that finicky and that hard to induce, and I know I (or my minions) often make mistakes, what reason do I have to think the phenomenon is real? Maybe if we're talking about something like making the chemicals in the reaction vessel change from blue to orange, or chunks of dull metal go boom, OK, once is enough. But as near as I can tell everything in social psychology is either a correlation, which is a pretty abstract statistical pattern, or a difference between correlations, which is even more abstract.
The part that I thought was reasonable was that the replicators are in fact coming into things with some biases about what they think isn't true , which could give failed replications a sociological weight over and above the statistical weight they should carry.
I have a secret for you. I think having a bias against experiments done by people who argue that replication is a bad thing is a perfectly reasonable bias.
But as near as I can tell everything in social psychology is either a correlation, which is a pretty abstract statistical pattern, or a difference between correlations, which is even more abstract.
It gets quite a bit dodgier than that.
146: Actually, it pretty much boils down to faith.
There's some social psychology that isn't particularly dodgy. I knew a bunch of them back in the day. I never knew the neurosocial social psychology people.
But, yes, the lack of self-awareness is very much astounding.
I found Mitchell's attitude fairly strong evidence for the "most published findings are false" hypothesis. If a superstar with tenure at Harvard basically thinks of himself as a publication machine with a great knack for coming up with positive-but-hard-to-replicate findings, that doesn't exactly bode well for the overall ethos of his field.
Apparently, CrossFit (incorporated) isn't exactly happy with the scientific method either.
If my phenomenon is that finicky and that hard to induce, and I know I (or my minions) often make mistakes, what reason do I have to think the phenomenon is real?
I thought the implied answer was pretty clear: the paper got published in a prestigious journal and the researcher got tenure at a prestigious university. Why would these things happen if it was all just bullshit?
BTW, I'm sure this is just a selection effect due to the fact that I usually come across social science research only when it's been hyped enough to make it to Slate and Facebook, but every social science article I've seen in PNAS has been pretty much crap.
142, 143:
we from the same hood, we come from the same shit
You got gonorrhea too? We fucked with the same bitch
Getting money is necessary; see me I'm a visionary
And I'm saying that house should be a crack house
Now see it how I see it or I'll bring the straps out
The tech and the mac out, the Sig and the Taurus
There are certainly good psychology papers in PNAS, although I can't claim to have read many from the social psychology end of things. I'm happy to link to a good one if it would make you feel better about the journal.
I thought the implied answer was pretty clear: the paper got published in a prestigious journal and the researcher got tenure at a prestigious university. Why would these things happen if it was all just bullshit?
Non-imposter syndrome. Probably far more common than the reverse.
146, 148: I'm not saying all social psychology is dodgy! I also think very abstract statistical patterns can be pretty damn real. I even know some excellent, careful scholars at Harvard. But when someone writes like there's no such thing as a false positive...
152: the paper got published in a prestigious journal and the researcher got tenure at a prestigious university. Why would these things happen if it was all just bullshit?
I'd agree that's a reasonable heuristic for outsiders to the field, but it's useless for those within it. (The problem with a lot of social epistemologies.) Replicators are peers, like the original reviewers.
146, 148: I'm not saying all social psychology is dodgy!
Oh, I didn't think that you were. I was saying that I find a lot of social psychology quite terrifyingly dodgy. But I'm the ghost of a famously careful behaviorist, so I suppose that's no surprise.
161: of course! I just think this attitude is implicit in Mitchell's (and Daniel Gilbert's) commentary on this topic.
In particular:
Ted Williams once said of baseball that it's "the only field of endeavor where a man can succeed three times out of ten and be considered a good performer." He apparently never submitted an NIH grant or applied for an academic job. Science is a tough place to make a living. Our experiments fail much of the time, and even the best scientists meet with a steady drum of rejections from journals, grant panels, and search committees. On the occasions that our work does succeed, we expect others to criticize it mercilessly, in public and often in our presence. For most us, our reward is simply the work itself, in adding our incremental bit to the sum of human knowledge and hoping that our ideas might manage, even if just, to influence future scholars of the mind. It takes courage and grit and enormous fortitude to volunteer for a life of this kind.
So we should take note when the targets of replication efforts complain about how they are being treated. These are people who have thrived in a profession that alternates between quiet rejection and blistering criticism, and who have held up admirably under the weight of earlier scientific challenges. They are not crybabies. What they are is justifiably upset at having their integrity questioned.
In short, life is hard enough for the scientist, so why bring epistemology into the mix? (And to spell it out, life is hard because getting grants is hard, getting a job is hard, getting tenure is hard...)
The post linked to in 156 is funny, and the post linked within that post is both funny and insightful. I want to hear more from The Journal of Evidence-Based Haruspicy
Part of what confuses me about this is that my field has the opposite pathology: there is very strong pressure on experiments to not publish any effect that conflicts with the null hypothesis at more than 95% confidence, unless it's like 99.999999% confidence. I know one large experimental collaboration where this is basically a standardized policy: all effects are either less than 2 sigma or more than five. Nothing in between can be publicly discussed.
If somebody asks a question at a conference, what do they say?
If someone asks to see a result that they haven't shown? They'll either say it isn't ready or that they don't have permission to show it.
151 reminds me when I am jogging and hating it I sometimes say to myself "at least I'm not doing crossfit."
I wasn't thinking of somebody asking for a specific result but the more general, "have you considered the relationship with X?" sort of thing.
Anyway, we're usually in a hurry to publish or publicize whatever we know.
170 is more the model I'm familiar with. I'm impressed by the discipline required for 167 to be the standard.
168 -- jogging every day is much more likely to produce injury than (competently done) Crossfit.
I'm still unclear on exactly what "social psychology" is, but it seems to mean "the kind of psychology that's not ev psych that gets reported on in the newspapers," and that always seemed to have a slight air of quackery to it, or at least way too broad exptrapolations taken from very limited findings.
Is there an a priori standard for doing crossfit properly? Or jogging, for that matter?
I'm no philosopher, but basically, yes -- good form in both prevents injuries, but if you run every day an injury is basically inevitable. Less frequent running with good form is just fine.
172: No chance in hell I'd ever run every day. Narrow scape!
Oh you all suck. I guess I'll go erase my whole post that I was going to post and had written already to post.
Or just put up the post and move Halford's self-serving nonsense there.
"the kind of psychology that's not ev psych that gets reported on in the newspapers,"
That's not a bad summary at all, really.
Look, I don't really care, but haven't you personally (and a bunch of other runners here) been injured running, killin ot as a fitness program, just as long as we're on the topic of self-serving nonsense?
182: I managed to hurt my left big toe three times in a year running, and it will never be the same again. It does sound ridiculous, but I really would rather have avoided it. Whether some other sort of exercise would've been even more injury prone, I couldn't say. (I have no idea what crossfit even is.)
164: I will be sure to pass on any communications I receive from the journal.
184: Somehow, this reminded me of Katisha from the Mikado: "But I have a left shoulder-blade that is a miracle of loveliness. People come miles to see it. My right elbow has a fascination that few can resist."
I seriously fucked up my hand running, because I tripped and hyperextended my wrist trying to stop my fall. Its been a couple months and it still hurts when I do things like open jars.
I wasn't running for exercise, though. I was trying to teach my son to ride a bike.
Wow hand running is a sport? I mean just walking in your hands is pretty impressive.
Okay, read that stuff again (OP): he's possibly the stupidest professor to ever live. This is seriously amazing stuff!
154: sure, I'll take you up on that! I spend too much time looking at things that suck.
I'm a fan of this paper, which lies pretty squarely in the world of Psychology.