The examples in the textbooks are not representative of real world problems

Bill Harris points to this cartoon from Randall Munroe. I think he (Munroe) is really on to something here. Not about Bayes’s theorem but about the sorts of examples that are in textbooks.

Consider the following sorts of examples that I’d say are overrepresented in textbooks (including my own; sorry!):

– Comparisons and estimates where the result is “statistically significant” and the 95% interval for the effect of interest excludes zero.

– Simpson’s paradox situation where adjusting for a factor changes the sign of an effect.

– Regression discontinuity analyses where there’s a pleasantly smooth and monotonic relationship between the predicted outcome and the running variable.

– Instrumental variables analyses where the instrument is really strong.

– Probability examples that are counterintuitive.

In short: in a textbook, most of the examples work. And when they don’t work, the failure illustrates some important point.

What’s going on here? Textbook examples are there to illustrate a method. The recommended approach is to first give two examples where the method works, then one where it fails. But we, the textbook writers, are busy, so we’ll often just give the one that works and stop. As I wrote once, “the book has a chapter on regression discontinuity designs. That’s fine. But all we see in the chapter are successes.”

Then again, on this blog, sometimes all you see are the failures. That’s not representative either

And then there are some examples such as aggregation and conditional probability where it’s just standard to demonstrate the principle with a surprising example.

As you can see, I have no overarching theory here; it just seems like a topic worth thinking about. Maybe in my next textbook I’ll try to address the issue in some way, maybe just to bring it up to remind the readers about this inevitable selection problem. I’m also reminded of the famous book, “What They Don’t Teach You at Harvard Business School.” Selection creates opportunities!

from Hacker News https://ift.tt/0WorvKX