Wednesday, May 18, 2011

More on the Conley-Dupor ARRA Paper

More people have weighed in on the ARRA evaluation I mentioned this weekend. I think all of these miss what I still think is the major point - which I reiterate at the end of this post.

- Noahpinion's opinion has been getting a lot of coverage. His major point seems to be that the results are not especially statistically significant. OK, but the point estimates are pretty consistently negative... I'm not sure I entirely buy this as the major point against the paper.

- Noahpinion and Brad DeLong all pile on Karl Smith and Greg Mankiw for posting the paper without comment. I think this is a little much. I post stuff I don't necessarily agree with but find interesting all the time. Often I provide thoughts, but sometimes I don't. We ought to be more critical readers than this.

- Paul Krugman's critique is odd to me too. He argues that Conley and Dupor make no effort to control for the differential effects of boom and bust in different states. But this is the whole point of the Conley-Dupor instrumental variable approach! Krugman seems suspicious of the instruments (not a bad posture to take - people should always be skeptical of instrumental variables), but it's not clear to me exactly what the problem is. The highway funding formula sounds like a fairly good instrument for the spending. I could see some problems with the sales tax intensity instrument. There might be some public finance tradeoffs between sales taxes and property taxes, and low property taxes might be associated with housing booms - I don't know. But Krugman fails to get very specific here.

- Arnold Kling says that Conley-Dupor isn't reliable without elaborating on why he thinks that. Instead, he embraces Taylor and Cogan who make absolutely no effort at all to identify a counterfactual! Ugh. Taylor's work on the stimulus has - in my mind- been essentially worthless.
- Dean Baker has some OK sensitivity suggestions here but he also falls into this "I bet they're doing something dishonest but I really don't know" line. Look, I think we should assume honesty of scientific peers until we have reason not to. I think we should assume there's no blatant data mining going on here. Dean Baker doesn't like the instrument I find convincing (the spending instrument). I wish I knew why he doesn't like it but of course he doesn't say.


The real problem with this paper: The real problem with this paper, which was a problem with an earlier state-level analysis that found a positive effect of stimulus too, is that it is state-level. Fiscal stimulus works through two major mechanisms: (1.) increasing demand for loanable funds, and (2.) the multiplier effect. Both mechanisms will bias state-level estimates in an open economy (like ours) downward. Let's think of the case of Maryland and Virginia. The impact estimator in this paper is produced by asking "what effect does a change in (VA stimulus money - MD stimulus money) have on (VA job growth - MD job growth)?" In other words, if we marginally increase stimulus money, what is the associated marginal increase in job growth? There's one big issue with estimating this that the authors do make an effort to solve. We would think that lower expected job growth would cause higher stimulus payments, right? The instrumental variables are intended to address this "endogeneity bias". Let's assume for the moment that they addressed that sufficiently.

A far bigger problem is that in an open economy we know that financial markets are national markets. So any impact that the stimulus has on these markets that might encourage job growth is going to increase both VA job growth and MD job growth simultaneously. If they both increase simultaneously, you're not going to observe the change with a state-level regression. In the same way, any multiplier effect in an open economy is going to be biased downward. We can decompose the impact estimate I discussed above (with VAst=VA stimulus, VAjc=VA job creation, etc.) into:

(dVAjc/dVAst) - (dVAjc/dMDst) - (dMDjc/dVAst) + (dMDjc/dMDst)

Now - notice which marginal effects have a negative sign in front of them when you decompose this. It's the two cross-state derivatives! Makes sense, right? If higher MD stimulus causes higher job creation in VA, then a regression that uses the difference in VA and MD stimulus to estimate the impact on VA jobs is going to be lowered in the regression even though there's a positive effect in the real world! So the question is, how big are (dVAjc/dMDst) and (dMDjc/dVAst)? A good start would be to figure how substantial interstate commerce. I tried the BEA's state GDP page, but they don't seem to calculate it (and why would they within a monetary and political union like the United States?). Does anyone know of any estimates of interstate commerce as a percent of GDP in the U.S.? That would be a good place to start to estimate how much of an underestimate this is.

So how would we go about checking on this? My first thought is to regress the residuals on state population or state GDP, but I'm not entirely sure that makes sense. I would think that interstate commerce makes up a smaller share of output for larger states, but that could be wrong - after all, a lot of big states are also major economic hubs that you would think smaller states would be more likely to do business with.

The point is, I think everyone is missing the biggest problem with this paper: state level analyses can't capture (1.) interstate effects, and (2.) impacts on national markets.

That's an enormous omission.


  1. Daniel,

    This post is a great example of why this blog has become my favorite in recent months. Thanks for the intellectual honesty, Daniel!


  2. Thanks - that's very kind.

    This sort of post is actually where I really have the most experience: thinking through microeconometric issues. I'm largely self-taught in my macro, I have little interest in micro outside of externality questions, and I haven't done anything with macroeconometric issues in years and a lot of it is a mystery to me. But microeconometrics is essentially what I get paid to do all day, so I think I have a good sense of the issues involved.

  3. I'm pretty sure they mention this limitation in their conclusion (the second paragraph). They also suggest approaches in the future that can address these concerns. My prior is that interstate commerce is not going to dominate within state effects and seeing that their estimates suggest no effect on private sector jobs, getting that number up to the 3 million or so provided by the President's advisors is going to be hard.

    To be honest, most of the discussion of this paper in the blogosphere (not yours) seems to be focused on the introduction or even abstract. I don't expect bloggers to analyze the paper at the level of a referee report, but the level of discussion in the blogs so far wouldn't even pass as seminar comments.

  4. More deeply, the problem with this--and all--cross-state studies of stimulus is that they implicitly hold monetary policy constant, and what monetary policy is doing really really matters.

    Nakamura and Steinsson ( implicitly make the point that state-by-state estimates are totally uninformative as to what happens if there is monetary policy (since their attempts to get their estimates to narrow down plausible models fails).

  5. I certainly agree that it would bias it downwards. I'm having difficulty imagining how it would bias it downwards into the negative-- surely dMDjc/dMDst > dVAjc/dMDst under all plausible assumptions?

    What's the case where it's not? Do you have to assume that MD stimulus spending is such horrible malinvestment that it causes MD jobs to be poorly allocated, but the stimulated increased aggregate demand is allocated in a more efficient way in the state without directed stimulus funds?

  6. John Thacker -
    I'm really not sure what the magnitudes are like.

    Virtually all the retail value added and a fair amount of the wholesale value added of what I buy here in Virginia is Virginian. But the rest? I'm looking at my food, the books on my shelf by my computer (my computer!), everything around me and there's not much that I would simply assume was made here. And we buy so much online now that makes it even harder. I don't really know how the value added stacks up, but I would guess the American economy is a lot more open and a lot more goes on interstate than people think. It would not surprise me at all to learn that half the value I consume is produced outside of Virginia, but perhaps that's totally wrong.


All anonymous comments will be deleted. Consistent pseudonyms are fine.