Friday, January 17, 2014

Thinking about the specifications of the Dube, Lester, and Reich minimum wage study: Part 1

Bob Murphy recently probed some important questions about the Dube, Lester, and Reich paper on the minimum wage (hereafter DLR). Ultimately I think Bob's concerns are misplaced, but these are exactly the issues we should be thinking about and arguing about with the minimum wage literature - not petty assertions that the other side doesn't understand the law of demand. So let's jump right in - Bob writes:
"I admit that I have not studied them in depth, but if you look at the discussion in the survey article I mentioned above, here’s what it sounds like: You can take all of the adjacent counties in the country for which you have continuous data over a long period (such as 16 years), where the applicable minimum wage is occasionally different in each county (because they fall in different states). If you “naively” run a regression on this dataset, then the classical consensus emerges: It does indeed seem that a higher minimum wage is associated with a slowdown in the growth of employment."
I think this is an entirely fair read of DLR. Their paper has two samples - a full county sample and a contiguous county sample that is a collection of county-pairs (with some counties repeated if they are paired with multiple border counties). They find this classical result in both samples if they just run a regression akin to Neumark and Wascher. This is a good thing - we want confirmation of results across datasets (Neumark and Wascher use states). Bob continues:
"However, this could be a spurious result, because states with high population growth might just so happen to also match the federal minimum wage, rather than setting a higher state level. To correct for this, the newer studies introduced a regional dummy variable into the regression analysis, at which point the negative effect of the minimum wage almost disappears.

If indeed what I just described is what’s going on, then that seems ludicrous. The point of matching contiguous counties is to isolate all other relevant variables, except for the applicable minimum wage. You can’t use the weather (one of the explanations given in the survey article to explain the flaw in the original studies, which did not correct for geography) to explain why people would flock to one county versus the adjacent one
This is where I think Bob starts to get things wrong. In the first place I don't think he's understanding what DLR are doing. The regional dummies and the matching are not being done together. See DLR's description here (specification 1 is the naïve regression described above):

Remember that even when you use the second sample of contiguous counties they aren't automatically matched yet. It's just a whole bunch of counties thrown in together that happened to be on state borders. When you start introducing geographic controls into the fully county sample (specifications 1 through 4), you very quickly lose the negative correlation. In other words, spatial heterogeneity matters crucially for these results. The finding that even regional time specific dummies in the full sample will lose you the negative effect is the whole reason why they are justified in going further and "matching" (as Bob phrases it) the counties. But there is no matching until specification 6. Specification 5 runs the naïve regression on the raw contiguous counties file, and specification 6 finally introduces county-pair dummies that "match" the counties (it would be more accurate to say that they eliminate the between-pair variation and only rely on the within-pair variation to identify the minimum wage effect). You might have to go to table 2 in the paper to see it clearly enough, but only the regressions where the red outlined contiguous county dummies are included are "matched":

Now Bob also raises concerns about why they're tossing other stuff in when they've already got a county match. I hope it's clear that they are not "matching" counties until the sixth specification, but this point is still worth addressing because it is relevant for other issues that Bob raised with the paper almost a year ago. Finding a good comparison group is the principal task of the microeconometrician. It's identical to the issue of "identifying your model". But when you find a good comparison group it doesn't mean there's no room left for improvement. So you'll often include other controls or matching procedures after finding a good comparison group (in this case, contiguous counties).

When I was at the Urban Institute, one project I was on was to evaluate a job training program for high-growth industries (often advanced manufacturing and ironically now, construction). We had individual level data, but we paired cases that showed up in contiguous localities (in this case a WIA designated area, not necessarily a county). One county's workforce investment board would direct people to the program we were looking at, the other would just have the training options that were generally available. That was a big step forward compared to what we're calling here the "naive" approach of comparing our treatment cases to anyone in the country that ever sought out job training. By focusing on contiguous localities we got a much better comparison sample. But there was still a lot to be improved on. So after that, we used propensity score matching to further match the characteristics of the group, and then on top of that we included other control variables in the ultimate model specification.

The point being that you never have a perfect comparison group. Using contiguous counties in DLR is a big improvement on Neumark and Wascher but that hardly means that everything is controlled for.

This gets us into the controls they add - specifically, time trends. I'll get that up in Part 2, hopefully soon. that deals with the concerns raised by Bob about a year ago. I wanted to get it in here, but I realized there's a lot to say to clarify the basic specification issues raised in his more recent post.

To sum up - even very simple controls for spatial heterogeneity like regional dummies strongly imply that you need to control for geography, motivating DLR to introduce their contiguous counties strategy.


  1. "When you start introducing geographic controls into the fully county sample (specifications 1 through 4), you very quickly lose the negative correlation."

    I don't have time to read the entire study. Can you provide some kind of description of how the researchers were able to control for the loss of degrees of freedom. Generally speaking, adding any sort of variable to a statistical model increases R and decreases the significance of even highly important variables. What specifically about this model would lead us to believe that what this model tells us is more than a pure mathematical artifact?

    1. My recollection is they did a cross-specification test and these always account for differing numbers of constraints (that's often the point of doing such tests). I can go back and search for that.

      I don't think this is it. First, notice that for the employment effects the point estimates change, and not just the standard errors. But for the earnings and effects and the coefficients on population, you don't lose significance. This suggests that it's not some general over fitting problem, but that they are addressing spatial heterogeneity that is biasing the employment effect. It would have to be a weird sort of overfitting to affect one but not the other.

      Finally, the sample size just seems too big to have degrees of freedom problems. Remember this is a panel. The total county sample has something like 90,000 observations. The contiguous pairs panel has something like 70,000. If anything you'd risk finding statistically significant results that have no real economic significance.

  2. Daniel - I think that was a good explanation of the border-discontinuity method we used. Moreover, the degrees of freedom issue is not a first order one given N >> parameters. Moreover, see our followup paper where we find sharp reductions in employment flows but not stocks. That and other findings are highlighted in our most recent working paper "Credible Research Designs for Minimum Wage Studies" here:

    1. Great - thanks for the IZA link and for visiting the blog, Arin! I'm glad to hear I reported faithfully :)

      I will take a close look at that. I'm realizing I need to pay attention to these spatial heterogeneity issues more for my first dissertation chapter. I'm doing a county-level analysis of job creation tax credit in Georgia. I'm identifying based on the tiered roll-out of the benefits (so an RDD), but I am not doing anything currently to account for these spatial heterogeneity issues. The benefit tiers are quite regionally concentrated within the state, so that is likely to be important. I just haven't done much with that so I need to spend some time thinking about how to implement it.


All anonymous comments will be deleted. Consistent pseudonyms are fine.