Saturday, February 16, 2013

Don Boudreaux on empirical social science

I don't want to give the impression that this post is one that I largely disagree with. Actually a lot of it is strong, particularly his discussion on the role of theory (I also liked the prior post where there seemed to be some recognition that reductio ad absurdum is often a terrible way to evaluate an economic claim). As Cafe Hayek posts go, this one is quite good. But I did want to highlight this passage, not to trash Don but just to raise concerns about an attitude that bothers me a lot:
"...there is no shortage of empirical studies that document the minimum-wage’s detrimental impact on low-wage workers. There are also, of course, other empirical studies that find the opposite effect, and yet others that find no significant effect.

That’s the way it is with social-science (including economic) data: they almost never speak for themselves clearly and without significant exceptions. Someone can always challenge even the most consensus-supported studies as having left out important variables, gotten causality backwards, employed inappropriate aggregation, examined inappropriate time periods, and on and on."
I don't like this attitude that empirical analysis is all over the map and you can  criticize all studies. Of course as in any science we're faced with a distribution of point estimates, which we usually take to be around a "true" point estimate. We are usually most confident about this if the studies at the center of mass seem to be the best designed and come out as the most precise. If you find some studies with a positive minimum wage effect and some with a negative, and some with no effect there's a good chance that there's actually no effect of the minimum wage.

Here is an excellent graphic from a new CEPR publication on the employment effects of the minimum wage. The estimated elasticities are on the x-axis, but what I really like is that the reciprocal of the standard error is on the y-axis. That means lower variability (not necessarily more accurate - we'll get to that) estimates have a higher value on the y-axis.

This is really not the picture Don is painting of the empirical literature for the economic impact of the minimum wage. This reaffirms my view from early meta-analytical work that the minimum wage essentially has no employment impact. Now there's a substantial cluster of slightly negative results in this graph, which is the origin of the Neumark and Wascher point that we do have a consensus for a slightly negative impact of the minimum wage. I'm actually willing to embrace that "negative but small" conclusion for essentially the same reason that Don points out - it sure has a theoretical basis. But embracing that conclusion as a practical matter doesn't change the fact that the zero-effect findings look very strong , and anyway the empirical evidence is not wishy-washy at all as Don seemed to suggest. You either have a "no effect" option or you have a "maybe a teeny effect" if you're being objective.

Claims like Jeff Tucker's assertion that one's position on the minimum wage is something of an intelligence test (in his case implying that you are not intelligent if you don't oppose it) fares very poorly.

Now I also alluded to the difference between precision and accuracy above - or what econometricians usually refer to as efficiency and bias. Bias - that your point estimate may not be centered on the "true" estimate - is the primary concern.

One of the biggest sources of variability in any set of studies is the inherent bias of a particular identification strategy. Economists almost never have a true experiment, so we have to use a variety of tricks to identify the relations we're interested in in a system that is simultaneously determined. Often these tricks will influence the point estimate itself. In immigration studies, for example, you have a lot of natural experiment studies and a lot of what I guess you'd call "shift-share" studies. The former tends to find positive effects and the latter tends to find negative effects (none substantial unless you're looking at very close labor market competitors).

Presumably there's something systematically different about these approaches. The natural experiment approach usually involves looking at a large influx of immigrants at a single point in time. The problem with that is that there is still a non-random aspect to (1.) which immigrants go, and (2.) where they go. Immigrants are going to be attracted to local economies where they have a shot at succeeding and where they might have a comparative advantage. You're going to naturally pick up lower negative effects or even positive effects as a result. The strategic behavior on the part of immigrants is good to know and may even reassure us, but it does not present an unbiased estimate of the impact of a truly exogenous migration shock. The shift-share analyses associated with Borjas apparently have some negative biases just like the natural experiment studies have some positive biases, although I'm personally less well versed in those criticisms.

The point is, when you consider these points the meaningful variability between empirical estimates starts to decline a lot. When you really understand why one set of studies is higher and one set of studies is lower - and if the methods of both studies have implicit biases, then you start to see them as upper and lower bounds - not as widely varying estimates of the same underlying parameter.

This is true of fiscal multiplier studies too. War spending studies give low estimators (I have a research note on this that will hopefully see the light of day at some point), studies where there is spill-over produce low multipliers, studies that look at state or county level spending by the federal government produce high multipliers because all jurisdictions pay federal taxes (so the counterfactual gets a negative hit).

A good empirical economists should be able to do what you might call mental meta-analyses like this. To really know an empirical literature you need to have a sense of the different strategies that people use for doing empirical work and the strengths and weaknesses of each strategy because a lot of the variability of the results are baked in by the differences in the strategies. When you start to think of the literature in these terms you realize there really isn't as wide of a range of results as you first thought. Often you have one group of studies using a strategy that introduces a negative bias with lower results and another group of studies using a strategy that introduces a positive bias with higher results. Your reaction to that should not be "wow there's a wide range of results" - it should be "we can be pretty sure that the real answer lies somewhere in between".

Empirical economics is not a free-for-all, but it does require effort. Most questions are actually clearer than they appear at first glance, but you have to really understand what the studies are doing that makes them different.


  1. Excellent post.

    An example that I like to use on this question of inherent bias in identification strategies, is the use of twin studies to decompose the relative importance of nature vs nurture.

    It might seem like an ideal situation -- from an experimenter's perspective -- to analyse how identical twins (that may have been separated at birth) fare in later life. However, can you really disentangle natural genetic ability (nature), from the impact/reaction that society has to that person's ability (nurture). As a simple example, consider someone who is born with an inherent athletic prowess. It seems clear to me that, all else equal, they will receive more training and encouragement of their skills from a younger age than kids who didn't exhibit this potential. Their nurture will in some sense then be dependent on their nature.

    1. Ya, a lot of people don't realize that it's a game of whack-a-mole with biases. They focus so much on solving one they either make another come up or they let others slip by.

      The example in this case that concerns me a lot right now is labor policy evaluations that use counterfactual cases who are participating in the same labor market. If you have a policy that improves the chances of your treatment group at the expense of your counterfactual group (because they are competing against each other), you're biasing you're results upward.

      This was an issue I tried to tackle in my econometrics paper last fall, and I did find some of evidence that it's a real problem for job creation tax credit evaluations... I'm going to try to get the paper submitted somewhere this spring.

  2. I wonder if the reason Boudreaux doesn't consider these kinds of biases when deciding not to trust empirical research has something to do with his general mistrust of scientific reasoning in economics. Perhaps he's just so committed to his anti-empiricist viewpoint that he's now in a feedback cycle; empiricism doesn't work, so I don't trust it, and now I refuse to find any way that empiricism is valid.

  3. I have always wondered if their real objection is unemployment means wages are too high, therefore by eliminating the minimum wage we eliminate unemployment; there are jobs to be had but the unemployed are simply unwilling to take them and all unemployment becomes voluntary.

  4. Please forgive any ignorance in this question as I'm not an economist but why couldn't (shouldn't?) this statement:

    "This reaffirms my view from early meta-analytical work that the minimum wage essentially has no employment impact."

    Be rewritten as:

    "This reaffirms my view from early meta-analystical work that the minimum wage essentially has no employment impact."

    This is why I've a difficult letting go of the reductio approach to this. I think we'd all agree that a federally established minimum wage of $100 per hour would cause all sorts of problems, right? But what about $12.00? My first thought is, "maybe?" Further, thanks to variation in the cost of living, it wouldn't affect all places the same and aggregate numbers might not reflect those differences. For example, a $12 minimum wage in Manhattan may not be a big deal but it could be devastating to rural areas.

    The reductio doesn't 'prove' anything but I use it to point out that since there is a point at which we'd all agree there would be problems, things are happening as we continue to increase it, and we can't predict with any accuracy when those things will occur, what they will be, and the magnitude of them.

  5. I'm not sure what happened between "preview" and "publish" but the second sentence was supposed to read:

    "This reaffirms my view from early meta-analystical work that the minimum wage *as enacted* essentially has no employment impact."

  6. Daniel,

    I'm not sure how to read that chart, but is it fair to say that all of the studies combined, suggest there is an elasticity of negative 0.8?

    OK, so the president has proposed to increase the minimum wage by almost 25 percent.

    In a country the size of the US, how many people will that throw out of work?

  7. BTW I have looked at those papers and I realize they are saying the elasticity is much closer to 0 than negative 0.8. All I'm saying is, just eyeballing that graph above, it sure looks to me like the naive average of those points gives you a decent (negative) elasticity, such that a proposal to raise the minimum wage by 24 % (Obama's proposal) would throw a lot of people out of work.

    So I'm asking, how do you read the above graph, and what does it mean in light of a 24 percent increase?

  8. Matthew Martin has some data on this. It would affect 6% of workers, but 5% already earn less than the current minimum wage (yes, the minimum wage has many loopholes). That is a fairly small effect.

  9. Lord--be careful with that data! I just pulled that dataset up because it was what I happened to have on hand. I think it is representative, but it is definitely not the right data for empirical analysis.

  10. Excellent and fascinating post. I'd really love to hear an Econtalk podcast with a discussion about empirical economics and the role of econometrics between Daniel and Russ.


All anonymous comments will be deleted. Consistent pseudonyms are fine.