I feel better that I stopped doing peer reviews. Unless it's me, I'm the problem, it's me for doing that.
Oh, and there was this follow-up post - apparently he got some unsettling anonymous hate mail for the first one.
I was in such a rush that I forgot to finish the OP title and now I'm out of the house! I'll fix it in an hour.
Peer review is absolutely broken beyond repair. It was near the breaking point before covid, and it's been in complete meltdown since then.
The situation is especially bad in math where papers are long and detailed and traditional refereeing standards involve a very close reading. I think we just need to get rid of refereeing for most papers. Keep anonymous refereeing for new authors, anonymous quick reports for all journals, and for the top 5ish journals with the really important papers have anonymous quick opinions but make the checking part something you get your name printed in the journal for. The traditional role of referees verifying the results is increasingly unnecessary in a world where most papers have coauthors.
The other big problem is just that there's too many damn papers. We need to change the incentives so that we don't have this huge glut of uninteresting papers that no one reads but which still have to get slotted into journals.
From the second link:
"That's also why I'm not worried about an onslaught of terrible papers--we've already got an onslaught of terrible papers. You learn very quickly to ignore them, just like you learn to ignore junk mail, stupid Netflix shows, and spam calls. Again, science is a strong-link problem. I'm not worried about how many bad papers there are; I'm worried about how many good papers there are."
I like parts of this guy's argument. I particularly like reading slightly stylish or fun papers and resent reviewers trying saying "but your prose MUST be boring." So I'm on board with his more casual writing style in his sample self-published paper. But a big part of his argument seems to be "not enough good things are getting published"* and I disagree with that part. In my field I'm flooded with good to great studies, most of which I ignore because it's just a firehose. But I don't know that a quality filter is a bad thing.
*not enough of MY things are getting published.
5: Maybe that should go back to being the role of journal editors? They pick informally what they think is good, and maybe half of them actually will be, and there's your filter. Plus with open access you can more easily figure out which journals have a better filter.
I'd go a bit further, the fundamental failure here isn't just peer review, it's making science about *papers*. The goal of science isn't to produce stacks of paper that no one reads! Thousands of separate papers isn't a good way to organize knowledge! Everyone knows this, but yet we all have to do it because it's how we've decided hiring and grants work (unless you're enough of a superstar that you can opt out of the stupid system).
Publication strategizing (e.g. one paper in a good journal or split and have two papers in lesser journals) is kind of fun to listen in on when it doesn't impact you much.
Basically 90-95% of papers should get less attention than traditional peer review (or not be written as papers in the first place!), and the remaining 5-10% should get more attention than traditional peer review (e.g. the referee should replicate the experiment, and publish a companion piece along with the original article).
Ah, so some system to reward merely adding to a dataset without it having to create a paper?
10: Yes! Or even just credit for organizing the data better. The Stacks Project is much more valuable than papers, the OEIS is much more valuable than papers, the Lean mathlib is much more valuable than papers, etc.
A comment I particularly liked is:
"15000 years per year is an extraordinary number, but phrasing it like that really undersells it. Just to put it in perspective, that means that at any moment, 15 thousand scientists are working on peer review. A normal full time job has you working about a quarter of the time (5 out of 21 eight hour blocks per week). This means that the peer review project uses the equivalent of sixty thousand full time science positions.
Sixty thousand! That's a lot. It's five CERNs. It's bigger than any scientific organization I could find in 2 minutes on Google. If I were employing sixty thousand scientists, I would be expected to show a lot of results."
12: I think some of that is people who only know their own field assuming it's the same in all fields. A lot of papers and peer review are done in medicine and related fields where most faculty work like 75% to 90% on the science and where senior people have staff working 100% research.
Or wikipedia for science but with a different bar for notability. Math wikipedia is already pretty amazing, there's just a lot of stuff that isn't on it and probably would be hard to get on because of notability. (There's the nlab, which is also a great accomplishment, but something like that but without the idiosyncratic viewpoint. Or just more than one such place but with different idiosyncrasies.)
My brother has been fortunate enough via his research institute to be able to focus on building software tools for others to use in his field of study, another underrewarded endeavor.
The other great thing about incorporating papers into a larger project is that it does a better job of verifying than peer review. You're going to need to synthesize papers in a way that requires you to understand what's in them, and that will cause you to catch errors in a way that reading typically won't. What causes errors to be caught is when the results get genuinely used by someone else.
Pele has passed. Absolute legend.
I can't tell from the outside whether peer review has done or will do any good addressing the problem of preprints coming out and making news while being unreliable and not something that should be reported as having been established as fact.
17: I didn't know he could still play.
First: if scientists cared a lot about peer review, when their papers got reviewed and rejected, they would listen to the feedback, do more experiments, rewrite the paper, etc. Instead, they usually just submit the same paper to another journal.
I think this is unfair. Pretty much every time I've been on a paper that was rejected, it was submitted to another journal only after thinking about the feedback and responding to with revisions or new analyses. Second, plenty of rejections have no feedback beyond "Nice paper, just not important enough for this journal. Please submit to Journal of Lesser Sciencing."
Agree, if his post had gone through the peer review process we could have made him remove the point in 20.
20 cuts both ways. You'll put some work into improving the presentation to improve your chances. But it would take some very harsh reviews or a sequence of failures before you'd say, "They're right, this work is no good and I should throw it away." It's pretty close to just buying another lottery ticket.
I think I largely agree with the indictment of peer review but I wonder how many of the identified pathologies are more demand-side, driven by increased public investment and a larger quantity of research from a larger population of researchers, chasing grants and appointments that can't all be mediated by a tiny number of elite gatekeepers? I.e., if you've got 10,000 researchers in search of tenure churning out 20,000 papers a year, what recourse do you have but to hornswoggle 60,000 "peers" into certifying their work meets some kind of professional standards? Would 1,000 unaccountable doyens be preferable?
"They're right, this work is no good and I should throw it away."
I mean, not after the first time, but after a few bad reviews, it gets pushed to the back of everyone's workload and dies. Or at least it used to. Now it gets sent to arxiv.
22.2 Yeah, you run into the problem that young people without permanent jobs write tons of papers, and there aren't enough people with permanent jobs to do the service needed.
On the boring prose thing, it kind of blew my mind when blogs came out, and I started to see written by smart people on interesting, well-researched or developed ideas, and with an engaging informal style. Like the pairing of informality with high quality content was a revelation.
The problem with enwordiation in science is that you usually need to explain a complex bit of research, tie it into a literature, and explain why anyone should give a fuck. You've got 2,000 to 4,000 words. It's going to suck as a piece of literature because you've got to use whatever way takes the least words.
23: The logic here is something like: a good paper has a 60% chance of getting accepted; a bad paper has a 15% chance; it takes about three tries, with some spit and polish, to convince yourself you are in the true-negative/bad-paper situation and not in the false-negative/good-paper one. I'm not saying good peer feedback doesn't get incorporated but there's a substantial discount applied due to the large quantity of bad/lazy/axe-grindy feedback.
To me, that's the biggest advantage of moving away from journals. It's got nothing to do with peer review though. I can bloviate to my heart's content. The problem being, so can other people if there isn't a structure to force them to get to the point.
27: That's just life. Good feedback at the right time is important, but hard to get.
A good science-thing needs to do three things:
1: Produce a reasonable list of things to read that will let me say "I tried hard enough."
2: Produce a fixed thing I can site to people that will say "they said that then and I can show you where it was written."
3: Produce a goal and I point to and say "I did this good thing where people can see it."
Peer review helps with #1, though not very well and I'm not sure something else couldn't do it better. Peer review isn't required for #2, but just putting up a pdf absolutely does not work. It needs to be something like arxiv, but that would be fine. #3 is already disconnected from peer review for me since so much of what I do is not peer reviewed anyway.
Obviously, since no system will work perfectly for everyone, the easiest thing to do is make it work as effectively as possible for me.
Too much time passed and I started to feel like I was no longer allowed to fix the post title, by the rules of the internet of whatever.
Just give the post a DOI and that'll make it legit.
I like Liam Kofi Bright's ideas on crowdsourced peer review instead if pre-publication peer review.
He's a philosopher of science and irreverent high volume Twitter poster, which seems about the right set of qualifications for fixing peer review.
I'm so glad that I'm focused on teaching rather than research now, because academic publishing irritates me on so many different levels that I just don't want to deal with it. I review maybe 1-2 journal articles a year these days, and only for the quite rare cases in which it's hard to think of anyone else in my original field who is familiar with the specific literature and methodology used.
The two main problems with crowdsourced refereeing is:
1) what to do with the papers no one is interested in reading
2) how to deal with sexism, racism, and harassment.
If you think 1 isn't an issue because those papers don't need refereeing, think about cases like grad students or faculty at schools with no but nonzero research requirements writing a paper with undergrads.
But 2 is the big unsolvable one. An online space that doesn't devolve into harassment requires intense moderation. Just limiting it to experts doesn't even begin to solve the problems.
Well, it needs to be a coordinated system with paid employees who work on those issues. It can't coast on goodwill. But it shouldn't be for-profit nor pretend there's a scarcity of pixels, either.
Math does have something similar to 40, Mathematical Reviews (aka mathscinet). It is a significant revenue source for the AMS (though the AMS is non-profit...). But math reviews are just one per paper and they're a neutral summary not an opinion, it certainly doesn't require reading the paper at any level of detail to write a MR (in fact, sometimes they're just copied from the abstract). To do that they have a full-time staff of 15 mathematicians (plus presumably additional non-PhD staff).
Since this is the academic thread, I'm going to point out that Pitt is, to my moderate surprise, holding a lead against UCLA. This would be nice revenge for all the Californian people buying nicer houses here than me.
The peer review post aside, that guy's blog (substack?) is full of great things.
At one point in my life, I regularly read and thought about things that Yglesias wrote. I didn't even disagree with him more than I disagreed with most, but I'm afraid of returning to that time in my life. So I'm staying away from substack as a whole.
What did Barbara Walters know about 2023 that she avoided it so suddenly?
39: Sure, but professional networks exist, and moderation systems exist. So these two things are more a cost to pay for much better peer review, versus all the costs and downsides of the current system, imo.
But have you ever seen Barbara Walters and Pope Benedict in the same room?
Moderation systems for the most part (here partially excepted) don't "exist." Ever major internet space has serious problems with harassment. Almost every woman I've ever talked to has immediately said some variant on "over my dead body" to posting their papers somewhere that allows comments.
Yawnoc: Pretty much a daily deposit of comment spam. The front pagers usually clean it up quickly.
Huh, never refreshed at the wrong time before
I can't believe comment spam is still a thing. Crazy how incentives created by Google fucked up the open part of the internet so bad.
Watching Simon Le Bon performing is a good way to really have it hit you that 1982 was forty years ago.
We watched Glass Onion. It was great! Daniel Craig's southern accent was so bad that I thought it was going to be some kind of plot twist that he wasn't actually southern. But aside from that, great!
He had the same accent in the first one.
I thought it was a Foghorn Leghorn thing but the internet says no.
You gotta see the first one!
really have it hit you that 1982 was forty years ago
I'm still internalizing that the 21st century, which just [expletive] began, is nearly a quarter of the way done.
In 3.5 years the US will be 250 years old. I'm really curious what that will look like. I don't remember the bicentennial, but my impression is that it was an occasion for a hokey patriotism which feels further removed from contemporary politics.
I was in elementary school during the bicentennial and we had a little parade where the kids dressed like founding-era figures. I don't remember who I was supposed to be but I was marching beside Betsy Ross.
Thankfully my kids will be surly teens by 2026 so I probably won't have to endure too much hokeyness.
Where will Leonardo DiCaprio live when the US turns 250? Montenegro?
57: it is truly weird that the Falklands War was closer to WW2 than to the present day.
59: I was really annoyed I missed Glass Onion in cinemas but we watched it online the other day and it was not good. Very, very slow, and a distinct drop in quality from Knives Out. And you have to care about characters for a film to work! I'm really not sure I minded that one obnoxious incredibly rich tech person had a murderous grudge towards another obnoxious incredibly rich tech person.
The heroine in Knives Out was literally a nice person. No one in Glass Onion was a nice person and the supposed heroine wasn't even the least unpleasant of them. She was the worst person there apart from the actual murderer!
73 confuses me, but I feel bad risking spoilers for others if I were to respond. Janelle Monae's character isn't nice? maybe we can be circumspect.
74: no, definitely not nice. She does a terrible thing at the end of the film, purely out of spite. The other characters are all just rather selfish and unprincipled, but she is truly awful.