Karl Smith, Ryan Murphy, and Empirical Economics

Karl Smith had a good response to Russ Roberts's repeated "scientism" accusations yesterday. This part was especially good:

"After all no one has – in my jargon – taken a ruler to the sun. No one has actually trekked from the earth to the sun with a tape measure to get its distance.

Indeed no one has even been to the sun or even out of earth’s orbit. People confidently mock those who say the sun is not at the center of the solar system but has anyone been outside the solar system to look down and check? Certainly not.

All of this is based on measurement and inference. And people trust the measurements and inferences of physical scientists even when they make wild conclusions based on highly technical derivations, complex models and slight differences in obscure measurements.

Two guys screw together a few half-silvered mirrors and all of a sudden the passage of time is just a fancy illusion. Is that more convoluted than Donohue and Levitt?"

The simple argument is that making inferences is what all scientists do, and it's strange that Russ finds that so controversial. But clearly economics is different from physics (as it is different from all sciences and all sciences are different from each other). Ryan Murphy seems particularly concerned with that. He responds to Smith:

"The ratio of content to vitriol in Roberts’ original post is a bit low (though I’d be a hypocrite for being too harsh on someone for their vitriol), but it is rather obvious that Smith has no idea why anyone would believe that the problems in econometrics fundamentally differ from the problems of measurement in physics [I don't find it obvious that Smith has no idea about this - so perhaps Murphy could clarify]. Actually, let me rephrase- when straightforward distributions aren’t applicable in physics, they actually use the correct distributions instead of assuming asymptotics and doing other stupid things. Dr. Smith, if you look at the data in economics, you see fat tails. When you have fat tails, even according to mainstream figures like Peter Kennedy, you can’t do meaningful hypothesis testing. Which means coefficient estimates are bullshit. Actually, I’ll just cite him directly for simplicity’s sake.

The consequences of non-normality of the fat-tailed kind, implying infinite
variance, are quite serious, since hypothesis testing and interval estimation
cannot be undertaken meaningfully.

Found in a footnote on page 63 of Kennedy, Peter. 1998. A Guide to Econometrics. Fourth Edition. Cambridge, MA: The MIT Press."

I have Kennedy's fifth edition (this appears on page 70 there) and maybe the book changed since the fourth edition, but if it hasn't then Murphy has failed to quote the next line where Kennedy references an entire chapter on robust specifications and non-parametric approaches to use in these situations. In other words - precisely what physicists do that Murphy claims economists don't do.

The whole physics vs. economics game always seems strange to me. Economics is most obviously like biology, not physics - and for obvious reasons. We are studying the social behavior of a primate. When we study the social behavior of any other species of primate besides homo sapiens we usually call those scientists "biologists", but in the case of this one species we've chosen to call them "economists". Fair enough - there's good reason for the different nomenclature (we're not just primates, after all - we're pretty special). But for all intents and purposes we are biologists. The physics that non-physicists usually have the opportunity to interact with (admittedly only a subset of physics) is fairly stable, mechanical, deterministic, etc. It's not like the complex system of the economy or other subjects of biological research. I think Smith's point is exactly right - scientists do inference. It's legitimate to give a physics example to make that point. But otherwise physics is a poor standard to measure economics against because it's too different. Russ Roberts has this weird imbalance in the way he usually approaches these questions where he also agrees with me that economics is most like biology, but for some reason he assesses the empirical bona fides of economics by comparing it to physics. Why? I don't know (and I've asked him and while he's responded to other questions and comments of mine he's never answered that question).

Murphy also highlights the real empirical hurdle of economics: endogeneity. Exactly right. Because of endogeneity we have to tackle the problem of measurement and inference differently from physics. Endogeneity is what makes empirical economics so interesting (to me at least). Endogeneity is a reason to be skeptical of results - the disposition of a scientist is always skeptical. But Murphy shouldn't confuse the obligation of having a skeptical eye with an obligation to consider something illegitimate (which is often what Roberts mistakenly does).

Smith ends with more great thoughts, this time from Robin Hanson:

"They key difference, I think, is that more interested parties see themselves as losing if the public listens to economists, and these parties therefore dispute economists in public. Such interested parties also influence individual economists, and so weaken within-economics consensus. In contrast, few care enough about what physicists say to dispute them in public."

I do think this is the primary difference, and this bothers me about the way Russ Roberts approaches these questions. Let's take two examples - the big macroeconometric models and the impact analyses that instrument for endogeneity problems.

I personally accept the sort of parameterized macroeconometric modeling that is done by IHS Global Insights, Moody's Analytics, etc. as having limits, but completely legitimate. It's just modeling the impact of policy given what we know from similar past policies. Russ Roberts repeats Arnold Kling's argument about the illegitimacy of these approaches whenever they come out with a justification of fiscal stimulus using these models. Fair enough - not everyone is going to agree with me on the methodology. But when Heritage produces a criticism of the stimulus using the exact same methodology, and when that analysis makes a huge splash in the media and the blogosphere does Russ Roberts make a peep? Does he raise the same objections he did when the same model produced pro-stimulus results? No. Russ should be raising hell about that, but he doesn't. It's the exact same methodology. Not only is it the exact same methodology - but it's the same damn model (IHS Global Insights, in that case) with a different set of assumptions. The only explanation for the difference in Russ's response that I can see is that one supports stimulus and one doesn't.

The same goes with studies that instrument for stimulus. I like these because like most people I see endogeneity as the primary problem for multiplier estimates. They are hard, but that's part of life - and that's also what makes them interesting to work at. Russ, of course, doesn't like these studies. He's called them a "great place for faith based econometrics", and has downplayed their value in discussions with Ed Leamer on EconTalk. And, predictably, Russ has criticized Romer's estimates using this sort of IV approach which finds a reasonably sized fiscal multiplier. You would think, then, that when Robert Barro uses the same sort of approach and finds no evidence for a multiplier (and publishes this in the WSJ - his was not a low-profile study) Russ would be equally critical of Barro? If you think that, you'd think wrong. To my knowledge he never mentioned it, and when his co-blogger Don Boudreaux favorably cited Barro's estimate Russ didn't take the opportunity to make the same critique of Barro that he did of Romer.

I would like to come to a different conclusion, but I simply can't. When macroeconometric modeling comes out with a pro-stimulus result Russ criticizes it but when the exact same approach comes out with an anti-stimulus result, he doesn't. When intruments come out with a pro-stimulus result Russ criticizes it, but when the exact same approach comes out with an anti-stimulus result he doesn't. What else am I supposed to conclude here? It's exactly what Robin Hanson says: "more interested parties see themselves as losing if the public listens to economists, and these parties therefore dispute economists in public". Most of what Russ says about econometrics I now consider to be a politically motivated statement rather than a scientifically motivated statement, because I simply can't find much scientific consistency in the way he talks about.

Notice that the two studies I always praise here are Barro's and Romer's. Notice when I raised questions about the Heritage numbers, it was never because I objected to their use of the IHS Global Insights model - I've always considered that to be a good decision for Heritage. Notice that the studies I don't like are the cross-state studies, and I criticize these studies on here when they come up with pro-stimulus results or anti-stimulus results. This isn't a political game, much as some people like to treat it like one. The empirical work is hard, and we need to work to keep doing it better. But if you stake out a methodological position you need to apply it consistently. Too much criticism of empirical economics seems to me to be grounded in politics rather than science.