Why We Sleep: A Tale of Institutional Failure

Table of contents

Chapter 1. The latest chapter in Why We Sleep, a Saga of Research Misconduct

Chapter 2. Why do we keep writing about this?

Chapter 1: The latest chapter in Why We Sleep, a Saga of Research Misconduct

In our previous installment of this podcast, we learned from independent researcher Alexey Guzey that famed sleep researcher Matthew Walker seems to have committed research misconduct in his bestselling book, Why We Sleep. Walker is a University of California professor, Google researcher, and Ted talk star, and his book received rave reviews in the press and was endorsed by Bill Gates, among others.

Sometime after that, software developer Yngve Hoiseth contacted the University of California to report Walker’s violation of their research misconduct policy. The university’s response was to dodge the complaint.

Here’s the key bit of the exchange between Hoiseth and the university. Hoiseth looked up the Berkeley research misconduct policy and wrote:

I.A.b [the relevant section of the policy] states:

Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record (emphasis mine [Hoiseth’s]).

As documented by Guzey, Walker clearly omitted data by removing a bar from a chart when reproducing it. The omission makes it seem like the research clearly supports his hypothesis when, in fact, it doesn’t. How did you conclude that this does not fit your definition of research misconduct at least well enough to warrant further investigation?

The university replied:

Thank you for your interest in this matter, which we have pursued in accordance with our policy. In conversation with Walker and with the professor who conducted the inquiry, the conclusion was that the bar omitted from the graph on the book did not alter the findings in an appreciable way, and more importantly, that the bar was not omitted in order to alter the research results. . . . It seems that there is a difference of opinion as to the significance of the errors and the omission of the bar from the graph, and difference of opinion is explicitly addressed in our policy here.

Hoiseth responded:

It’s clear that Walker omitted “data or results such that the research is not accurately represented in the research record.” I don’t see how the sentence about difference of opinion is relevant in this case.

Anyway, I feel like we’re getting a bit lost in the weeds here. The reason I’m unsatisfied with your response is that I think research integrity in general is very important for society, and in this specific case I personally spent quite a lot of time and resources on Walker’s book. I bought a pile of them which I gave away to friends and family, and I got gear to track and improve my sleep.

Guzey uncovered many errors by thoroughly reading a single chapter of a book Walker even cites in his own papers. And Gelman says that we’ve entered research misconduct territory. Now, I don’t know what to believe. I worry that science dies by a thousand such cuts.

The university replied:

We have completed the inquiry into Mr. Guzey’s allegations to our satisfaction. I don’t wish to go into the weeds either in order to contest any of your statements, especially given the current crisis situation we are all experiencing, so I will need to conclude our conversation here.

Give them full credit for being polite and non-aggressive, but I agree with Hoiseth that removing the bar from the above graph is an unambiguous case of “omitting data or results such that the research is not accurately represented in the research record.” This has nothing to do with “significance of the errors” or “difference of opinion.”

Also, and now this is me speaking: the bit about the error not being significant, that really annoys me. We see this all the time. We saw it when Susan Fiske had to change her t statistics from 5.03 and 11.14 to 1.8 and 3.3, and then she declared that the fix “does not change the conclusion of the paper.” We saw it when Brian Wansink had something like a thousand papers retracted but he claimed that none of this mattered because his research claims were all still true. We saw it when Marc Hauser declared that all his findings were just fine, even though he wouldn’t let anyone see the tapes of his monkeys. We saw it when that gremlins dude had to make correction after correction after correction in his paper, but his conclusions never changed. And lots more examples of the years. This all bugs me because . . . these people always make a big deal about their data, science science data data experiment experiment statistical significance bla bla—until the moment the data are revealed to be wrong, and then it turns out the data didn’t matter.

If the data didn’t matter, then why did you include them in your damn book in the first place? If the removal of the bar from the graph didn’t matter, then why did you remove the damn bar?

I’m just sick of all this. I propose the following solution:

Instead of these researchers piling up a bunch of questionable evidence into a book which they can hype to high heaven, and then when the errors eventually get caught, they say that the data didn’t matter anyway, why not cut out the goddamn middleman and just start off your book with a statement such as, “None of the data in this book matters.” Whichever of our claims turn out to be fake, let us assure you that none of our claims will be affected by these errors. Just get it out in the open. Give these people official titles such as Tenured Professor of Scientific Claims That Don’t Have to Be Supported by Data.

OK. It was good to get this off my chest.

Hoiseth describes this as “a tale of institutional failure,” and I agree. I don’t like that the university official talked about going “into the weeds either in order to contest any of your statements.” This bothers me for two reasons. First, there are no “weeds” here, it’s a simple pair of graphs. Second, why should the university official assume that their job is to “contest” a statement from an outsider? Why not evaluate the evidence? I get the impression that they don’t want to find anything negative so they go to some trouble to not look so hard.

I also found this quote from Bill Gates:

Now that I’ve read Matthew Walker’s Why We Sleep, I realize that my all-nighters, combined with almost never getting eight hours of sleep, took a big toll. . . . Walker, the director of UC Berkeley’s Center for Human Sleep Science, explains how neglecting sleep undercuts your creativity, problem solving, decision-making, learning, memory, heart health, brain health, mental health, emotional well-being, immune system, and even your life span. . . .

I [Gates] don’t necessarily buy into all of Walker’s reporting, such as the strong link he claims between not getting enough sleep and developing Alzheimer’s. In an effort to wake us all up to the harm of sleeping too little, he sometimes reports as fact what science has not yet clearly demonstrated. But even if you apply a mild discount factor, Why We Sleep is an important and fascinating book. . . .

Does everyone really need seven or eight hours of sleep a night? The answer is that you almost certainly do, even if you’ve convinced yourself otherwise. In the words of Dr. Thomas Roth, of the Henry Ford Hospital in Detroit, ‘The number of people who can survive on five hours of sleep or less without impairment, and rounded to a whole number, is zero.’ . . .

Some interesting things here:

1. The UC Berkeley affiliation may really make a difference here. It’s the usual exchange: Walker gets to share in the university’s reputation; the university gets some of his fame and some of his grant money. Also it’s hard to fire a tenured professor. And if you’re stuck with the guy, I guess there’s not much motivation to issue a finding that he committed research misconduct.

2. Gates recognizes that Walker exaggerates! But he doesn’t seem to recognize that Walker flat-out misrepresents data. As Guzey writes, it’s almost as if the expectation of mild dishonesty inoculates the book author from thoughts that he might actually be lying.

3. No evidence is given that you “almost certainly” need seven or eight hours of sleep. The closest thing to evidence is the statement of some doctor somewhere (there’s this weird thing in the U.S. where anyone who’s a medical doctor is considered an authority), but even that quote is just talking about the problem with “five hours of sleep or less.” There’s a big range between 5 and 7 hours! What about the people who can survive or even thrive on 5.5 hours, or 6 hours, or 6.5 hours? Maybe such people are very rare, but no evidence is offered in support of this claim.

Personally, I’m a 9-hours-a-night person, so I’m inclined to believe some of Walker’s claims. But I’m annoyed that he presents claims without evidence, and then distorts whatever weak evidence is available to make his points.

Chapter 2. Why do we keep writing about this?

There’s coronavirus, for chrissake. The world is burning around us, so why am I posting—not even with a 6-month delay—on this bit of trivia? Or, what’s my beef with that sleep researcher, anyway? Am I some sort of resentful Stasi terrorist?

First, no, there’s nothing wrong with Guzey and Hoiseth and me geeking out over this. Recall the Javert paradox. It’s completely reasonable to write about scientific misconduct, and yes sometimes we have to scream a bit to get heard over all the chatter of the scientist-as-hero press.

Now on to why this is important.

First, expertise matters in this world, and we’re learning this now when fighting the virus. But we need real expertise, not fake expertise. I’ve said it before and I’ll say it again: Gresham, baby, Gresham. If we don’t contest the fake expertise, I’m seriously worried it will be crowding out the real stuff. As late as 28 Feb—less than a month ago!—we had a Harvard-credentialed expert telling us not to panic about coronavirus. OK, he wasn’t experts in epidemiology, he was an expert in . . . ummm, I dunno, expertise? Just an all-around public intellectual, a kind of Edmund Wilson of social science? I’m not sure. Anyway, I think we need to fight this kind of thing when we see it. I don’t mean that we all need to fight it, or that I need to spend most of my time fighting it, but I do mean that some of us have to spend some of our time fighting it, not letting empty claims of expertise slip by.

In some way, I feel that the Why We Sleep author and the University of California are worse than media entities such as David Brooks who fling around fake stats and don’t correct clear errors. Brooks is an opinion columnist. If you believe anything he writes, that’s on you. But the Why We Sleep author, he’s like Dr. Oz—he borrows authority from serious institutions of scholarship. This is bad.

So, yes, when the University of California hears about research misconduct by their faculty and sweeps it under the rug, this can keep their reputation afloat (after all, how many people read this blog? And even the people who read this can just say it’s me being cranky) and they can keep raking in that sweet, sweet indirect cost recovery from government grants. But they’re letting the side down. They’re not doing their part as a component of the scholarly enterprise.

And, to return to coronavirus . . . we need real expertise here, more than ever.

from Hacker News https://ift.tt/2UgXusA