More on the Conley-Dupor ARRA Paper

More people have weighed in on the ARRA evaluation I mentioned this weekend. I think all of these miss what I still think is the major point - which I reiterate at the end of this post.

- Noahpinion's opinion has been getting a lot of coverage. His major point seems to be that the results are not especially statistically significant. OK, but the point estimates are pretty consistently negative... I'm not sure I entirely buy this as the major point against the paper.

- Noahpinion and Brad DeLong all pile on Karl Smith and Greg Mankiw for posting the paper without comment. I think this is a little much. I post stuff I don't necessarily agree with but find interesting all the time. Often I provide thoughts, but sometimes I don't. We ought to be more critical readers than this.

- Paul Krugman's critique is odd to me too. He argues that Conley and Dupor make no effort to control for the differential effects of boom and bust in different states. But this is the whole point of the Conley-Dupor instrumental variable approach! Krugman seems suspicious of the instruments (not a bad posture to take - people should always be skeptical of instrumental variables), but it's not clear to me exactly what the problem is. The highway funding formula sounds like a fairly good instrument for the spending. I could see some problems with the sales tax intensity instrument. There might be some public finance tradeoffs between sales taxes and property taxes, and low property taxes might be associated with housing booms - I don't know. But Krugman fails to get very specific here.

- Arnold Kling says that Conley-Dupor isn't reliable without elaborating on why he thinks that. Instead, he embraces Taylor and Cogan who make absolutely no effort at all to identify a counterfactual! Ugh. Taylor's work on the stimulus has - in my mind- been essentially worthless.
- Dean Baker has some OK sensitivity suggestions here but he also falls into this "I bet they're doing something dishonest but I really don't know" line. Look, I think we should assume honesty of scientific peers until we have reason not to. I think we should assume there's no blatant data mining going on here. Dean Baker doesn't like the instrument I find convincing (the spending instrument). I wish I knew why he doesn't like it but of course he doesn't say.

*****

The real problem with this paper: The real problem with this paper, which was a problem with an earlier state-level analysis that found a positive effect of stimulus too, is that it is state-level. Fiscal stimulus works through two major mechanisms: (1.) increasing demand for loanable funds, and (2.) the multiplier effect. Both mechanisms will bias state-level estimates in an open economy (like ours) downward. Let's think of the case of Maryland and Virginia. The impact estimator in this paper is produced by asking "what effect does a change in (VA stimulus money - MD stimulus money) have on (VA job growth - MD job growth)?" In other words, if we marginally increase stimulus money, what is the associated marginal increase in job growth? There's one big issue with estimating this that the authors do make an effort to solve. We would think that lower expected job growth would cause higher stimulus payments, right? The instrumental variables are intended to address this "endogeneity bias". Let's assume for the moment that they addressed that sufficiently.

A far bigger problem is that in an open economy we know that financial markets are national markets. So any impact that the stimulus has on these markets that might encourage job growth is going to increase both VA job growth and MD job growth simultaneously. If they both increase simultaneously, you're not going to observe the change with a state-level regression. In the same way, any multiplier effect in an open economy is going to be biased downward. We can decompose the impact estimate I discussed above (with VAst=VA stimulus, VAjc=VA job creation, etc.) into:

(dVAjc/dVAst) - (dVAjc/dMDst) - (dMDjc/dVAst) + (dMDjc/dMDst)

Now - notice which marginal effects have a negative sign in front of them when you decompose this. It's the two cross-state derivatives! Makes sense, right? If higher MD stimulus causes higher job creation in VA, then a regression that uses the difference in VA and MD stimulus to estimate the impact on VA jobs is going to be lowered in the regression even though there's a positive effect in the real world! So the question is, how big are (dVAjc/dMDst) and (dMDjc/dVAst)? A good start would be to figure how substantial interstate commerce. I tried the BEA's state GDP page, but they don't seem to calculate it (and why would they within a monetary and political union like the United States?). Does anyone know of any estimates of interstate commerce as a percent of GDP in the U.S.? That would be a good place to start to estimate how much of an underestimate this is.

So how would we go about checking on this? My first thought is to regress the residuals on state population or state GDP, but I'm not entirely sure that makes sense. I would think that interstate commerce makes up a smaller share of output for larger states, but that could be wrong - after all, a lot of big states are also major economic hubs that you would think smaller states would be more likely to do business with.

The point is, I think everyone is missing the biggest problem with this paper: state level analyses can't capture (1.) interstate effects, and (2.) impacts on national markets.

That's an enormous omission.