1. They reject endogeneity and simultaneity problems - they don't think they are important at all in economics, or
2. They think endogeneity and simultaneity are important problems to think of in empirical economics and they just don't think we do a good job addressing the problems.
The first one seems highly unlikely to me. Russ likes to talk a lot about Bastiat (and presumably Bob shares his admiration), and Bastiat was deeply concerned with these issues. That's the whole point of the seen and the unseen, after all. In that sense Bastiat very much has the sensibilities of a modern, mainstream macroeconomist. It also seems unlikely because despite my differences with both Russ and Bob, I don't think either of them are dummies - and you honestly don't need that much economic sense to agree that these are critical problems for economics.
So I assume it's the second issue - they don't like how economists deal with endogeneity and simultaneity.
The trouble is, if this is the problem then they really need to give their readers more before they start throwing around terms like "intellectually bankrupt" (particularly given how they tend to pout when others make accusations like that). What fault do Russ and Bob find in the way these estimates are generated? I have no idea.
This bothers me a lot because although I'm not really an empirical macroeconomist, I am principally an empirical economist. Everything I've ever written on economic history, history of thought, Keynes, Hayek, etc. I've written for free. What I do for a living and a career is empirical economics and I spend a lot of time worrying about precisely these problems that Bastiat raised - how to parse out the seen and the unseen in a non-experimental setting.
We've talked about multiplier estimates here (although again - it's not my expertise). I've come out strongly against estimates that use cross-state and cross-county variation, for example. Some "windfall revenue" studies are nice, but they still have demand leakage issues. I've also come out against war spending estimates because we're not dealing with depressed situations in many cases, and because war spending is not exogenous.
We've talked on this blog about the pros and cons of different minimum wage studies too.
We've talked on this blog about why I am skeptical of most instrumental variable papers.
We've talked on this blog about where you need to be careful using propensity score matching.
We've talked on this blog about why I love natural experiments and regression discontinuity designs.
We've even talked a little on this blog about the pitfalls of random assignment if there are general equilibrium effects.
So if I express skepticism about some kind of empirical finding I hope at least I've expressed why or that my history of blogging could give you a clue about why I'm skeptical.
But with Russ and Bob, I come up empty. They like Bastiat so they have to agree with me that counterfactuals and endogeneity are important. They obviously don't like the methods to deal with it. But what I don't understand is why they don't like it - what the issues are - why Russ is happy to cite Ramey or Barro and Redlick or Selgin, Lastrapes, and White favorably but calls a Krugman claim "intellectually bankrupt" when the estimates he cites are struggling with the same goddamn empirical head-scratchers that Ramey/Barro/Redlick are.
I think I have a right to expect a little more detail on what the concern is with these estimates for a couple reasons:
1. Russ went to Chicago and Bob went to NYU (ranked second and tenth respectively). I am currently attending American University (ranked 205th). Despite some peoples' complaints about these rankings, they fall out how they do for a reason. If I can provide detailed justifications for my thumbs ups and thumbs downs on empirical economics, they certainly should be up to the task.
2. Ramey, Barro, Redlick, Selgin, Lastrapes, and White all come down with conclusions that Russ likes. Feel free to name others that Russ has linked to favorably on Café Hayek and whom he has not called "intellectually bankrupt" for using modern empirical methods too. I can't recall a single case - in the years following the blog - that Russ has praised an empirical analysis that goes against his political views. This is troubling. It does not mean he is biased, but it is an AWFULLY good reason to expect more details on what he does and doesn't like about the analyses.
3. The other reasons they give are really bad. Stop complaining about how mainstream economists are copying physicists. Physicists don't use these methods. Economists INVENTED several of these methods after all. And if they did use them, who cares? In what universe is the unique usage of a method by one group of scientists anything like a standard for judging the validity of the method?
On behalf of all of us who actually do and care about empirical economics, I just think we deserve a little more than this.
Look Daniel, part of why I read your blog is that you are so thoughtful on these issues. But look at the laundry list you just gave of all the pitfalls in estimated multipliers etc.
ReplyDeleteDo you agree with me that that Macroeconomic Adviser estimate was WAY THE HECK off, for what (at the time of the projection) was only one quarter in the future?
Is it even possible in principle for these empirical techniques to be so useless in macroeconomics, that we should stop using them? What would my case against them have to look like, besides listing all the problems that are wrong with them and showing how the best economists in the business make a baseline forecast that's off by 58% (if their assessment of the sequester is right) for the next quarter's GDP growth?
They seem way off. I haven't looked at the analysis yet, though.
DeleteThe trouble is, I really don't know forecasting methods. I did one forecasting problem set in time series almost a decade ago. I do know impact analysis and quasi-experimental methods. So I feel free to comment on them. I don't feel prepared to weigh in on forecasting methods, much less sling terms like "intellectually bankrupt" about it just because of error in the estimate.
What I do know about forecasting is that in complex systems, you are dealing with a depressingly short time horizons.
And as with Romer/Bernstein - make sure you don't tie impact estimates up with forecasts.
DeleteOK let me reformulate it: What is the level of confidence you have in the multiplier estimates? You agree that it's *possible" this is a huge waste of time, right? So why do you think that the "empirical approach" is worth doing in economics?
DeleteClearly economists haven't done for governments what, say, the physicists have done.
You don't think economic science has dramatically improved the quality of public policy? I guess I'd just disagree.
DeletePhysicists - because of the nature of what they're studying - can clearly control things in a way that we can't control society. I guess I just don't see the fact that we can plan out how atoms are organized and that we can't plan out how societies are organized as a reason to fault social science. That's just the way the world works.
Is there reduced precision in, say, meteorolgy, seismology, much of biology, and social science? Sure. I'm not clear on why you're titling posts with things like "intellectually bankrupt" for that.
This is awesome. We are getting into good stuff here. Unfortunately I have to do work and you have a kid on the way.
DeleteI'll try to write something next week.
Daniel, what's your take on experimental economics (e.g. Vernon Smith)? Should there be more emphasis on controlled, repeatable experiments in econ? (as I think is the case in pyschology--I may be wrong)
DeleteVirtual worlds seem potentially useful laboratories for macro experiments. Some work has been done in MMORPGs, but I'd be interested to see a big scale project like SimCity, but with real players. A virtual Biosphere 2, but for econ research.
"Learning about real economics in virtual worlds"
http://economics.rutgers.edu/dmdocuments/DavidAbrams1.pdf
"As real as real? Macroeconomic behavior in a large-scale virtual world"
http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=pdf&doi=10.1.1.193.6326
"Physicists - because of the nature of what they're studying - can clearly control things in a way that we can't control society. I guess I just don't see the fact that we can plan out how atoms are organized and that we can't plan out how societies are organized as a reason to fault social science. That's just the way the world works."
DeleteThis is precisely why Austrians say the scientific method isn't appropriate for economics. Austrians aren't faulting social science, they are faulting economists who approach economics as if it is a hard science like physics.
Dan -
DeleteBut that has nothing to do with whether it is a science or not. We can't plan the weather either, but that's still a science.
Now you're bringing something else in with the last sentence, and I agree. If you approach economics like physics that's obviously dumb. One should approach economics like economics, which is as far as I can tell what most economists do. Look at Bob's evidence in his own post for what acting like a physicist consists of - they estimate counterfactual behavior with data! Come on - give me a break.
Very important John S. And the "virtual worlds" thing jumps over another option - agent based modeling. Virtual worlds essentially combine experimental economics and agent based modeling.
DeleteThe trouble is in the real world institutions and social relations are critical to the structure of economic relations and behavior. Experimental econ and virtual worlds sanitizes all that out. So I don't expect it to replace statistical analysis - there's a lot that experimental econ won't be able to accomplish. But it definitely has an important place.
Correction: Since their forecast was made (I think) in the beginning of January, then maybe it's more correct to say their horrible baseline projection was for 2 quarters in the future.
ReplyDeleteDaniel, I think the problem is that the statistical methods economists use (you're right that it weren't physicists who invented them) aren't very scientific. First of all, statistical inference is in general, strictly speaking, not scientific. No one knows what probability is at the bottom level and it is not an intrinsic, fundamental thing like energy (except, perhaps, for the collapse of the quantum wave function).
ReplyDeleteAt the practical level, probability is now defined as a share of a certain outcome of the experiment in all the observed multiple outcomes of such experiment. Even if we ignore the previous point and treat this approach as scientific there is a big problem for applying the method to economic phenomena. The problem is that choices aren't repeated, separate experiments, they are unique events.
I know you'll say that statisticians have found a way to go around this problem by showing that certain metrics of distributions of non-experimental data-generating processes are very similar to those of the distributions of outcomes of what they believe are repeated random experiments (like throwing dice). But this go-around is a non-sequitur. It requires a question begging assumption that probability is applicable where certain metrics of the distribution are similar to those of the distributions that are believed to be random.
All events are unique events. The quality of the analysis is reduced to the extent that the factors that make the events unique can't be accounted for or controlled for.
DeleteIf this sort of thing trips you up then nobody, anywhere should be using statistics. (you seem to say as much in the first paragraph). You seem to be approaching this logically - that the use of statistics requires certain foundational premises and if they don't hold the whole enterprise is a bust. It ought to be approached as a tool - given imperfect data (the data and our knowledge of them are ALWAYS imperfect - ALWAYS), does statistics allow us to characterize a phenomenon with a good degree of clarity and accuracy.
Can't I answer "OK, but completely irrelevant" to your logical question and "yes" to my tool question?
btw, this gets back to a point I was making the other day with a guy about the problem with a lot of philosophy of science.
DeleteMuch philosophy of science is just applied epistemology, and then they've chosen to attach the word "science" to that and use those standards to judge the quality of science. Most scientists could care less about epistemology, we care about getting a decent description of the world that helps us understand the world better.
That's why the most useful philosophers of science are the ones that take science to be what scientists do, that try to understand research agenda formation, the course of progress in science, etc. - in other words, the best philosophers of science are the ones that know that science isn't applied epistemology.
=Can't I answer "OK, but completely irrelevant" to your logical question and "yes" to my tool question?=
Delete=Much philosophy of science is just applied epistemology, and then they've chosen to attach the word "science" to that and use those standards to judge the quality of science. Most scientists could care less about epistemology, we care about getting a decent description of the world that helps us understand the world better.
That's why the most useful philosophers of science are the ones that take science to be what scientists do, that try to understand research agenda formation, the course of progress in science, etc. - in other words, the best philosophers of science are the ones that know that science isn't applied epistemology.=
You beautifully summed up the fundamental point of disagreement from which most of our particular disagreements stem. You of course can dismiss my basic criticism and continue doing what you're doing. But my response will be that you shouldn't then be surprised about people like Bob proclaiming the intellectual bankruptcy of mainstream econ. Because what he calls intellectual bankruptcy stems exactly from this essentially postmodernist approach to science.
= If this sort of thing trips you up then nobody, anywhere should be using statistics. (you seem to say as much in the first paragraph). You seem to be approaching this logically - that the use of statistics requires certain foundational premises and if they don't hold the whole enterprise is a bust. It ought to be approached as a tool - given imperfect data (the data and our knowledge of them are ALWAYS imperfect - ALWAYS), does statistics allow us to characterize a phenomenon with a good degree of clarity and accuracy.=
Not all events are necessarily unique. All quanta are for example the same. Provided the experiments are done correctly, statistics probably has a use in quantum physics. The same probably concerns phenomena which are at their basis probably quantum, like mutation.
=does statistics allow us to characterize a phenomenon with a good degree of clarity and accuracy=
It's herein that the problem lies. If we are able to understand (and I agree that we are) when a phenomenon is characterized correctly why the hell used patently unrealistic methods? IOW, you're undermining your own case. To be able to understand that something is a good metaphor of a phenomenon you have to understand it. This allows you to do without the metaphor.
As an aside, I don't believe that science is applied epistemology. It's the other way around. Epistemology is part of science broadly understood which includes knowledge that we can know things. How we do it at the basic level we'll probably never know.
DeleteCan you at least accept that you seem to be the one redefining words here? Your position on empiricism seems to only admit a small sliver of physics at best. Is EVERYONE else "intellectually bankrupt"? And now epistemology is science?
DeleteIf you want to use words this way at least recognize that you're completely bucking their meaning for most people that talk about these things.
And this in a lot of ways poses more problems for Bob - who does empirical stuff all the time - and Russ - who happily posts a series of empirical analyses that are friendly to his position. You, as far as I can tell, stick to a few underlying philosophical questions and don't touch the rest of it. So when I demand a little more specificity from Russ and Bob on what they have a problem with and what they don't have a problem with, that's part of the context - they are NOT making many of these strong claims that you are.
If we're at the point where quantum physicists are about all that can use this stuff then you and I are from different universes. I can say "Daniil's a nice guy, and interesting", but we really can't even trade arguments.
DeleteNow - how do YOU KNOW that quanta are amenable to this? Since a buttoned down epistemology is important to you this seems like a question you need to provide an answer to. We've done empirical work in quantum physics of course, but surely you can't use that to justify the claims that allow for empirical work in the first place.
When you get down to it, you're every bit as much a postmodern as I am, with one exception - you don't like the idea of talking like one.
"It's herein that the problem lies. If we are able to understand (and I agree that we are) when a phenomenon is characterized correctly why the hell used patently unrealistic methods? IOW, you're undermining your own case. To be able to understand that something is a good metaphor of a phenomenon you have to understand it. This allows you to do without the metaphor."
DeleteThis is bizarre. Think about it in the context of gravity--Newton observes an apple falling, but in order for him to say that gravity is what's pulling it down, you require him to know what gravity is, to know that such 'forces' exist in the universe, and that apples are subject to them. Whether or not you're right in an epistemologic sense, it's a definition that doesn't include any reasonable form of human 'science', so who cares?
Daniel, I frankly don't understand why you're interpreting my position the way you do. I think it's perfectly compatible with all the natural sciences. Could you explain why it isn't on at least one example?
DeletePseudonymous, by understanding I didn't mean being able to define say gravity in some extensive terms. Sorry if my formulation was confusing.
DeleteI meant that you need to see from a certain experience that there is a force and to have intuitive grasp of what force is to do physics. Of course you don't need to be able to grasp everything about say gravity sitting in the armchair. This understanding guides the scientist later.
Now the analogy in economics is, for example, the phenomenon of human choice. You need to understand what choice is to do economics but of course you don't need to be able to deduce everything about all choices sitting in the armchair. The mainstream approach to economics is faulty not because it leaves the armchair but because it is based on assumptions that contradict basic economic understanding.
=If we're at the point where quantum physicists are about all that can use this stuff then you and I are from different universes. I can say "Daniil's a nice guy, and interesting", but we really can't even trade arguments.=
DeleteDaniel, did you purposefully ignore my caveat that if individual quanta of particular kinds are indeed the same statistics can be used in relation to phenomena which are fundamentally based on quantum events? There are probably quite a few such phenomena and I suspect that it's in relation to them that statistics is legitimately used in natural sciences.
=Now - how do YOU KNOW that quanta are amenable to this? Since a buttoned down epistemology is important to you this seems like a question you need to provide an answer to. We've done empirical work in quantum physics of course, but surely you can't use that to justify the claims that allow for empirical work in the first place.=
I am not even close to being an authority on quantum physics. That said I suspect people started using statistical methods in quantum physics only when they had reasons to do that from experiments.
On the other hand, I disagree with Bob Murphy to the extent that he maintains that praxeology isn't empirical. It isn't empirical in the narrow, positivist sense of the word. But it is empirical in the Aristotelian realist sense of the word. All our knowledge is ultimately based on experience, including the knowledge of what action, means, money, etc. are..
ReplyDeleteDaniel,
ReplyDeleteI would argue that the main problem with economics is that a lot of it cannot be observed. For example, in physics when a rock bumps into another rock, the former is the cause that the latter moved. In economics, why people do what they do is because their actions aim at definite ends and these ends cannot be observed. The ultimate causes are pretty much invisible.
Now what we can do is to try and interpret the behavior around us by making assumptions about the ends people aimed at and whether or not they were successful and whether this was for the right or the wrong reasons. Some interpretations will be more plausible than others, but whether or not you will consider one interpretation more plausible than another largely depends on your core assumptions. It will depend on the assumptions you are least likely to abandon in the face of evidence. It's your Lakostian hard core or close to it anyway.
Macroeconomics in this sense is more like literature, than like physics, where observations take the role of a text. All interpretations are true, provided the assumptions hold, but the assumptions cannot be tested because the assumed ends and beliefs are invisible. Reasonable people, reasonable as they are, have reasonable assumptions and can therefore disagree about how to interpret a particular situation.
Daniel, I'm always surprised by your surprise when you write these kinds of posts. What do you expect? Read the comments sections (I know you have!). There's a choir to preach to. There's no incentive for them to be anything more than vague and critical.
ReplyDeleteCan we reset this conversation to actually be about econometrics instead of a lot of pontification about how econometrics is bad by people who have never run a regression?
ReplyDeleteDan has had a lot of of interesting things to say over the years about estimations of multipliers and it totally misses the point of this post to make these broad statements that are ultimately meaningless assertions that we can't know anything.
Can we reset this conversation to actually be about econometrics instead of a lot of pontification about how econometrics is bad by people who have never run a regression?
ReplyDeleteDan has had a lot of of interesting things to say over the years about estimations of multipliers and it totally misses the point of this post to make these broad statements that are ultimately meaningless assertions that we can't know anything.
"Intellectually bankrupt" is just shorthand for not able to understand this and can't be bothered to try. To ask for details is to ask for the impossible.
ReplyDeleteMy follow-up post should settle all further disputes on this topic.
ReplyDelete