During lunch, I stumbled across an interesting discussion between Richard Dawkins and Lawrence Krauss that wanders in a couple different directions, but starts with an interesting way of thinking about how to compare the genius of a man like Darwin with the genius of a man like Einstein.
Dawkins says that we can think of a scientific theory as a ratio between what can be explained and what must be assumed. Dawkins argues that Darwin wins out over Einstein on this ratio, but also notes that this has implications for the underlying genius of Einstein. He jokes "any fool could be a Darwin". Krauss disagrees somewhat and gives Darwin more credit, and the discussion goes on from there.
But I think this ratio idea is a good way to think about science. It's appealing because in a lot of ways it is a productivity measure - informational output per unit of informational input. It really explodes the hard/soft science distinction as well. Yes, social sciences have complex, imperfect, imprecise informational inputs - but the phenomenon it is trying to explain is equally complex, imperfect, and imprecise. The "hardness" of a science doesn't make sense under this schema (or at least it doesn't seem important... all "hardness" really means is "complexity/precision") because we are normalizing outputs with the input.
This also presents a nice way of thinking about scientific advances. Copernicus revolutionized cosmology because he was able to produce the same output (the observed course of astronomical bodies) with substantially fewer inputs. One way of thinking about this is that Copernicus was "right" where others were "wrong"... but as science has advanced we've realized that that sort of hubris may be uncalled for. A better thing to say than that Copernicus was "right" was that he was "more efficient". His theory had higher scientific productivity.
Without having read any Kuhn, this sounds Kuhnian to me - but I think the expression of this ratio as the scientific productivity of a theory has other advantages. After all, with a productivity measure you can have a marginal productivity measure (or really an expected marginal productivity measure because science is a discovery process not really a production process). Once you have a marginal productivity measure you can make all kinds of claims about the behavior of scientists.
I see a few wrinkles that may present obstacles in making use of this sort of marginal productivity measure, but I have to chew on it for a little while. Thoughts or claims to the effect of "ya - so and so essentially already said that" would be appreciated - I'm not as well read as I'd like to be on philosophy/history of science.
Wednesday, December 1, 2010
Subscribe to:
Post Comments (Atom)
Posts
I think of scientific 'hardness' in very different terms. Something like, the difference between physics and economics is that physics makes very accurate predictions and economics doesn't.
ReplyDeleteThere's a large body of economic reasoning that is held in high esteem by economists, but if you use that reasoning to make predictions you don't get very good results. In physics, theory that doesn't give accurate predictions is generally held in low esteem by physicists.
Economics could easily be a hard science if economists simply refused to esteem theory that doesn't give accurate predictions.
Economics would also be greatly improved if economists read more Popper and used falsifiability to distinguish between claims that can properly be considered by science and claims that cannot. Actually, that's something that annoys me about Dawkins - he asserts that science has disproven claims that are in fact non-falsifiable and outside of the purview of science.
What do you think of fields such as epidemeology, meteorology, etc.?
ReplyDeleteWhat of geology/seismology too? Old Faithful is a very dependable case, but the very fact that it is named for that dependancy indicates how unusual it is. I cannot "predict" what recessions will occur over the course of the next century. Can a geologist "predict" what eruptions will occur? Can a meteorologist "predict" what hurricanes will occur? Can an epidemeologist "predict" what outbreaks will occur? What about an evolutionary biologist? Can they "predict" what species will evolve over the next million years?
I think we need to be very circumspect about the standards that we use to measure "science", and "prediction" is an easy one to abuse. Economic forecasts are produced with an indicated degree of error (simply by virtue of the probability distributions we know about), and should also be accompanied by a recognition that we are making claims about an extremely complex system that we can never hope to project in any deterministic way.
But economics isn't alone in that. We can't project climate, genetic heredity, or geological activity in any deterministic way either!
The physics comparison can be useful but it can also be very, very misleading. Quantam mechanics helps somewhat with illustrating this, because the very sense of what "prediction" means under quantam mechanics is shot to hell.
In reality, I think economics fairs better than you suggest. Evolutionary biologists have put a lot of work into studying past evolutionary behavior, and they can account for a great deal of that behavior. And then of course some of it remains a mystery - no evolutionary biologist can trace all the steps for why a particular branch of a species evolved in the way it did or failed in the way it did. But they have a good framework for understanding it and do a decent job. They can also take that information and apply it in very useful ways to the world around us - in medicine and a variety of other fields. But what can they really "predict"? Well they can't "predict" much of anything in the sense that those who accuse economists of being "soft" expect economists to predict things. The system under study - the genome, to say nothing of the organism housing it or the ecosystem housing that organism - is hopelessly complex. "Prediction" in the way that people expect economists to "predict" things simply cannot be. And yet there is no doubt at all the evolutionary biology is solid science.
ReplyDeleteI don't know... I still don't see it. I feel like people take physics and chemistry (and really, what is chemistry but more atomic-level physics? That's really all it is.) and elevate it to this princely status among sciences. The problem is (1.) the system under study is fundamentally different... all the inhabitants of this system only obey a few laws and have a few properties, and (2.) no other thing we call a "science" comes close to what physics/chemistry does. And all these other sciences that don't even come close remain separated from physics/chemistry because of their complexity. So it really all boils down to the complexity point.
As for falsifiability... I think that would be nice and is useful in a lot of circumstances (particularly where experiments can be brought to bear), but I think that's largely futile. See my discussion of falsifiability here: http://factsandotherstubbornthings.blogspot.com/2010/11/practice-of-macroeconomics-few-rules.html
Do Popperians generally accept the way physics does falsifiability? Has there ever been an audit of falsification in various scientific fields? I don't know - more and more I feel like Popperians have a lot to say about epistemology, truth, and philosophy, and less to say than I originally thought they did about science and the practice of science. The mindset it useful - "reject the null" and all that. But that's not even real Popperianism anyway. Still, practically speaking it's a useful and much healthier mindset to have than the way that people usually approach things.
And like I always say - if Dawkins gave a talk on the social behavior of monkeys and how and why that social behavior is the way it is on the basis of certain scientifically determined patterns and theories, everybody would call it science.
ReplyDeleteLet that monkey evolve for a few million years into a human and give the exact same talk, and now you've gone "soft".
I personally don't get it.
Today I used a dataset with dozens of detailed variables on the behavior of 22,000 highly evolved monkeys. Yesterday I used a dataset with a couple hundred variables that tracked the behavior of 3,000 highly evolved monkeys over the course of ten years.
That's just with the science I've been doing this week - I've used bigger and more detailed data on other highly evolved monkeys. How much you want to bet my data is a lot better than the data available to most primatologists?
Sorry for the extensive commenting... think of this as an addendum to my original post and not entirely a response to you, Robert :) As always - I enjoy reading your thoughts.
ReplyDeleteI don't think it's too hard to answer most of your objections. Take evolution for example. Sure the science of evolution doesn't do much to predict which new body forms are going to evolve, but it does predict which body forms SHOULD be present in the fossil record. And those predictions can be made with amazing accuracy.
ReplyDeleteMeteorology is a good example of a science that has hard and soft parts to it. Meteorologists can't predict the weather with great accuracy very many days into the future, and there's an awful lot of softness in many attempts at climate modeling. But meteorologists can predict the weather tomorrow pretty well, and they clearly know something (maybe not as much as we'd like) about phenomena like El Nino.
I like to say that technology demonstrates knowledge. There is SO MUCH technology that has come out of the knowledge that physicists have generated. Those technologies pervade the modern world. What similar technologies have come out of economics?
To be more clear, my problem isn't with economics. My problem is with economists who tolerate and even endorse weak, poor quality science. E.g. insisting that a particular body of theory is right when there really just isn't enough evidence to determine that. Accurate prediction that can be done over and over again is strong evidence that there is real understanding held by the one doing the predicting. Physicists insist on this kind of evidence before accepting a body of theory as truth. Economists are more lax.
Now, I'm willing to cut economists some slack and say that the work they have undertaken is extremely difficult (due to complexity, lack of experimental evidence, messy data, etc.). The problems in economics today are MUCH harder than were the problems in physics at the beginning of the 20th century, for example. But that just means that economists need to accept that progress is going to be very slow, and that they will probably have little to show for a lifetime of hard work. Those who don't like that should work on problems that are easier to solve. They shouldn't publish weak science and pretend they KNOW things that they don't.
I'm going to read your post on falsifiability. Maybe I'll have more to say once I have.
Sure the science of evolution doesn't do much to predict which new body forms are going to evolve, but it does predict which body forms SHOULD be present in the fossil record. And those predictions can be made with amazing accuracy.
ReplyDeleteWell, it doesn't really predict this - it explains it after the fact, after looking at the fossil record in question. A lot of this is speculation about why certain things happened the way they did... I may need a little more detail on what exactly you mean by "amazing accuracy". Anyway - I'm quite sure economists do this too, don't they?
On climatology - exactly right! We can predict near things better than far things, and what is "near" and what is "far" depends on the subject of study. What science is this not true of?
As for technologies... this is tough of course because we're talking about social technologies. Corporations use scientific studies of markets all the time to formulate business plans. A host of government policies are social technologies. There are a lot out there, but it's hard to point to them physically in the same way you point to technology that emerges from physics. Social technology is harder to think about, but it's no less real. This seems like an odd standard to me anyway. We have "technology" from geology certainly insofar as we know about how to use and work with minerals, but what "technology" really emerges out of seismology, except simply tools for doing more seismology? Some science is simply useful because it provides informtaion to know and navigate our world better. Similarly, a lot of high level theoretical physics offers us almost no technology but it's still science. I think this is a good thing to remark on but I would not use at as a standard for determining the scientific qualities of something.
OK, I have something to say briefly about falsifiability and scientific method. Science tries to build models of reality. The scientific method is to build a model of some part of reality and then to try to see whether the model is a good or poor match for reality. A good model behaves very much like the thing being modeled. A poor model doesn't.
ReplyDeleteFalsifiability just means that there is some meaningful way to compare the model you've constructed against the part of reality that it is supposed to mimic. If you don't have falsifiability, you can't know whether your model is a good or bad match for reality.
In my opinion, Popper had it right. You just CAN'T do science on non-falsifiable claims, because there's no way to know whether the claims accurately reflect reality, or whether they are a bunch of nonsense.
This is what's wrong in the soft sciences, and what's wrong in in the soft parts of hard sciences - no one can tell which models are any good and which are garbage, because there's no practical way to test the models for similarity with reality.
That's why I get all hung up on prediction - because it's one really good way to test a model to see whether it matches reality.
Moving in your last three paragraphs I see you're moving back to physics again - and I restate that I think this is very problematic.
ReplyDeleteInstead of physics, let's take biology. The insights of Adam Smith - which were based on some empiricism, but which have been tested and corroborated with evidence much more rigorously over the years - I would argue are every bit as "proven" as the insights of Charles Darwin. Would you disagree? I'm really surprised by your claim. Do you think there is something suspect about an economist proclaiming the validity of the claim to market efficiency? We can't model market efficiency in the same way that Newton can model things - but we can do it at least as well as biologists can. And often biologists do their modeling of the basic concepts of Darwin much less formally.
Now - we clearly argue over details. When we try to model and understand very specific processes we do fight more. But doesn't this occur in biology as well? When they get into tracing and explaining more detailed evolutionary branches or the impact of more detailed evolutionary pressures they argue too. And that's very good!
Evolutionary biologists still don't have a very solid grasp on the causes of the Cambrian Explosion - but they've got a lot of good, plausible ideas about its causes. We don't know exactly the causes of the Industrial Revolution, but we've got a lot of good, plausible ideas about its causes.
I don't know... I think edging back into physics is getting you in a rut.
A few laws, a few forces, a few properties. If the economy had that... if the biosphere had that... it would be a different story entirely. But I think the physics comparison is deleterious to sound discussion of science.
If "soft" just means "you don't have a handful of well defined laws, forces, and properties" then certainly economics is "soft" along with everything outside of physics and chemistry. That doesn't really seem to have anything to do with science which is a practice. It seems to have more to do with the nature of the subject of study.
Paleontologists can tell you an awful lot about species that haven't actually been observed in the fossil record. Because they understand how evolution acts on the bodies of living things, they can predict what body forms have existed at particular times, even though fossils of those forms have not been discovered. And (working with geologists) they can figure out where they should dig if they want to maximize their chances of finding one of these fossils. That's how Lucy was found.
ReplyDeleteI'd say seismology is actually a very hard science, and that it is regularly used to make predictions. Which parts of San Francisco do you not want to live in? What kind of construction do you want for your dwelling? Where are earthquakes likely to happen? How is the interior of the earth composed? A seismologist can tell you.
There are some kinds of knowledge that don't have much practical use, and so technology doesn't come out of them. There's probably a lot of astronomy that fits that description. But economics is about things we care about because they have practical import. Real economic knowledge should produce valuable, observable technology.
So I don't know Popper in the sort of detail that lot of commenters on here do. I generally would agree with you that falsification is more challenging in economics, and really any complex science (and I include biology in this as well... really there is little difference in my mind between economics and biology - I am a primatologist, essentially). I make this point in the link to an earlier post of mine I provide above.
ReplyDeleteI think where I diverge from Popperians is the extent to which I consider this meaningful. If you think of science as the search for absolutely truth this is a deal-breaker of course. I don't think that is what science really is. Science provides workable models for reality, as you say. And it does need to approach falsification to make a solid case that those models are workable and useful. Nobody is arguing with this. But absent strict falsifiability (which... I ought to add... itself doesn't provide a solid "foundation for belief" anyway - brian in a vat and all that), corroboration and weaker falsifiability is still useful. And on this spectrum of corroboration and weak falsifiability, I (1.) don't see that much difference between economics and non-physics, non-chemistry sciences, (2.) I think economics has proven to be "useful" for understanding the economy and it has provided usefulness using the scientific method, and (3.) any shortcomings that I can readily observe come from two sources: (a.) the complexity of the subject at hand (which I don't see as relevant to the question of scientific identity), and (b.) ideological and political factionalism that conflates positive economic science with warring ideologies.
The latter, of course, I am fully in support of purging from economics proper.
*absolute truth
ReplyDeleteI'd say seismology is actually a very hard science, and that it is regularly used to make predictions. Which parts of San Francisco do you not want to live in? What kind of construction do you want for your dwelling? Where are earthquakes likely to happen? How is the interior of the earth composed? A seismologist can tell you.
ReplyDeleteProbability distributions - all of them. You really think economists can't make similar predictions?
The only difference is that the subject of study changes when people use the insights of economists in a way that seismology does not change when people take the insights of seismology.
So we have feedback loops. OK. This distinguishes "social science" from "natural science" and I have never balked at that distinction. But I still don't see how it changes the fundamental practice of science.
Real economic knowledge should produce valuable, observable technology.
Right - and I described it, did I not? When we design markets, when we launch products, etc. this is all a product of a scientific study of the market. This study of the market does not adhere to the methods of textual exegesis, the methods of logic, or some artisanal method - it adheres to the scientific method and produces output according to that method.
But it is a social technology, not physical technology so when you say "observable" you have to take that into account. There is also, of course, the entire expanse of monetary and macroeconomic policy. This is all technology that has emerged from economic science. Like early rocket tests, sometimes this technology is a little bumpy and doesn't work so well. But nobody questioned whether physics was really a science when those seeking to harness it got off to a rocky start.
What can a seismologist tell you about your house that a financial economist can't tell you about your investments? Good, sound advice within a certain probability distribution with the admitted possibility of "unknown unknowns" lurking around. It's what a construction company distills from the work of seismologists, and it's what a financial planner distills from the work of financial economists.
ReplyDeletehttp://blogs.discovermagazine.com/80beats/2010/12/01/the-estimated-number-of-stars-in-the-universe-just-tripled/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DiscoverBlogs+%28Discover+Blogs%29
ReplyDeleteThe BEA regularly revises its GDP estimates... but could you imagine if the BEA announced that it had to triple its estimate of nominal production?
Notice - I react to this news understanding that this is life - measurement is imperfect and we constantly learn more. It's no black mark on physicists.
Good discussion - I very much appreciate it, and am always, always, always open to readers noting that Popper demonstrates I don't know what the hell I'm talking about.
ReplyDeleteI still feel pretty strongly about this - I'm essentially a very specialized primatologist. But nevertheless, a fruitful discussion.
"Probability distributions - all of them."
ReplyDeleteNo, not really. It's not a simple extrapolation from historical data. The mechanisms are understood - that's why the technology of elastomeric bearings (those earthquake-resistant mounts that they use under buildings and bridges) has been successfully developed. And my position is certainly not that economists can't make predictions.
But that's not the point anyway. I have no problem with probability distributions. They aren't unscientific. They are a perfectly useful tool for building and testing models. Statistical knowledge is real, usable knowledge.
"But nobody questioned whether physics was really a science when those seeking to harness it got off to a rocky start."
And it's not my position that economics should be questioned as a science because of the limits of what it can successfully model today. But models that don't give accurate predictions shouldn't be held aloft and defended with undue certainty and venom. When you don't know, you should just own that. But I'll bet I can find plenty of economists in public forums insisting that they KNOW which policies will give which results. But their failed predictions belie them.
Right... the mechanisms are understood... I'm still not grasping the difference you're suggesting is there.
ReplyDeleteThey aren't unscientific. They are a perfectly useful tool for building and testing models. Statistical knowledge is real, usable knowledge.
Right, I wasn't suggesting you had a problem with them - I was just pointing out that that's what economists can provide too.
But models that don't give accurate predictions shouldn't be held aloft and defended with undue certainty and venom.
What do you really mean by this Robert? I think you and I are working off entirely different assumptions about what economic models do and do not do. And I'm certainly not clear on what the venom is supposed to be about.
When you don't know, you should just own that.
Without question!
But I'll bet I can find plenty of economists in public forums insisting that they KNOW which policies will give which results. But their failed predictions belie them.
Really???? I think you'll find economists who have reasonable faith in their models and do not have insistence that they can predict outcomes with perfect clarity, but who - given the extent of what they do know - would emphatically insist on certain policies. We don't exactly have a grasp on what makes for successful macroeconomic policy. But not doing something is as much a policy choice as doing something, so given our imperfect knowledge I would emphatically insist on certain policies. But that's very different from me claiming I am 100% sure of what's going on.
A meteorologist will emphatically tell you to pack an umbrella, but that doesn't mean he's under the impression that his forecasts are picture perfect.
"But not doing something is as much a policy choice as doing something, so given our imperfect knowledge I would emphatically insist on certain policies."
ReplyDeleteBut that's my point. How can you emphatically insist that your model is correct (and therefore the policies you propose will be successful) if your model is not well and thoroughly tested - e.g. by repeatedly making successful predictions?
"...do not have insistence that they can predict outcomes with perfect clarity..."
I am almost inclined to call that disingenuous, but it all rests on what you mean by perfect clarity, I guess. But there are endless supplies of examples such as this: http://gregmankiw.blogspot.com/2010/01/unemployment-update.html
But that's my point. How can you emphatically insist that your model is correct (and therefore the policies you propose will be successful) if your model is not well and thoroughly tested - e.g. by repeatedly making successful predictions?
ReplyDeleteWell sure - if its not well and thoroughly tested then nobody should be claiming anything emphatically. I would agree with you on that. My point is that we have well and thoroughly tested models that seem to be providing a good framework, but still involve a lot of uncertainty in forecasting (like any complex system). So I would be pretty emphatic about the thrust of policy without having a complete handle on predictions or even some of the details. And a big part of the reason is that even though I'm not sure exactly how much good to expect from, say, fiscal and monetary stimulus - I know we know enough to say emphatically that it will be much, much safer than doing nothing.
I don't think that's a problematic case, is it?
I am almost inclined to call that disingenuous, but it all rests on what you mean by perfect clarity, I guess. But there are endless supplies of examples such as this: http://gregmankiw.blogspot.com/2010/01/unemployment-update.html
ReplyDeleteYou would do well not to think of that as disingenuous because I meant every word of it. The kinds of forecasts economists are expected to do are quite substantial. And we do it because a so-so forecast is better than flying blind (it's the same reason why meteorologists do forecasting). Forecasting of complex systems is never good. Forecasting of complex systems undergoing an unusual event is even worse.
You show me how well the CDC projects epidemics, Robert. You show me how well an evolutionary biologist predicts the next Cambrian explosion. These are complex adaptive systems and they are cascade events.
When scientists talk about predictive power, they mean "if we know x, y, and z is true then we predict a, b, and c will happen". The problem is when you are studying a complex system you don't know x, y, and z.
What do you expect? Did Romer plaster that projection all over the place smuggly as you suggest? No. It was a quick forecast a long ways out (MUCH longer than these things are usually forecase) that was made before all the information was in. She wasn't smug about it - it was a very rough cut. You know who's been posting it all over the place? Republicans and stimulus skeptics. If you want to know who is misrepresenting the conditional nature of forecasts of complex systems you can look in that direction - not to economists.
I have been arguing that there are softer and harder forms of science, and that the distinction between the two is in the STANDARD for supporting evidence that must be obtained before a body of theory can be highly esteemed. If you want to argue that epidemiology and evolutionary biology have softness in them, I'm right there with you. I'll even agree that parts of physics can get a little mushy -- that weakly evidenced theories can gain popular support, especially when no one in the field has anything better to offer.
ReplyDeleteHowever! I think that softness is bad for science. It wastes resources and confuses the public. It reccommends policies that are unproven and perhaps detrimental. It makes room for charlatanism. It ought not to be tolerated -- and yet it often is!
"The problem is when you are studying a complex system you don't know x, y, and z."
I understand this problem extremely well. But the answer is not to go ahead anyway, making your best guesses about what x, y, and z might be, and then to publish to a credulous audience. That's not science! And if you can't make accurate predictions, then for crying out loud, don't make predictions! Don't be a pretender to knowledge that you don't have.
Your last line is a bit funny since the link I posted is to the blog of a very prominent economist. Look -- the fact that there is such deep disagreement among many of the top experts in the field suggests that the economic policies that are being promoted are not the product of strongly evidenced science. As long as that's the case, let's just own up to that. If we can't recognize bubbles when we're in them, and can't stave off recessions before they happen, and can't give accurate predictions of the effects that our reccommended policies will produce, then let's just admit that we might be wrong and the other guys might be right, or maybe everybody's wrong and maybe hands off is a better policy than compulsively trying to fix a complex system that we don't understand and don't know x,y, and z about. Or even that we don't know what we're doing but we're pretty sure that doing nothing is going to leave us screwed so let's just try and hope for the best.
Let's just not pretend that it adds up to science when we do that stuff.
First - I'm continually confused by statements that you make like "if you can't make accurate predictions" or "if your model is not well and thoroughly tested". I think you're conflating precision of a science with accuracy or thoroughness. To use the language of econometricians, you're conflating efficiency with bias. Yes, some sciences are going to have more efficient testing than others. But if you want to talk about "accuracy" that is a question of bias in the estimate and there's no good reason to believe that this varies as much as you suggest.
ReplyDeleteSo what is "softness" - the relative complexity of a subject? I've said several times now that that seems to be all it boils down to, but its a relatively vague term so I'm not sure.
You then go on to say "let's admit we might be wrong", "let's admit our policy recommendations might not actually be the best", etc. OK, sure. I'm all for this. But what does this have to do with anything? Who is arguing we shouldn't act this way, Robert?
Ultimately, if "soft" just means "complex" and it includes essentially everyone but physics and chemistry I guess I could use it as a synonym for "complex". But that's not how its been used - it's been used to mean "not entirely scientific".
And let me bring this back to the original subject of the post - the whole reason why I liked thinking about this in terms of a ratio was that it offered a way to think about normalizing the complexity of the results by the complexity of the assumptions. It's not like it's a tractable measure but its a way of thinking about how to compare these sciences with varying degrees of complexity without falling into the trap of thinking about complexity as impurity.
ReplyDeleteAnd also - I want to reiterate that you're noting disagreements on very, very detailed sub-questions to the field (which of course will occur in a science that has such a big impact on society and that is so complex).
ReplyDeleteBut the foundations of economics - Adam Smith and what we know about market exchange - is every bit as solid as Darwin.
"So what is "softness" - the relative complexity of a subject?"
ReplyDeleteI'm sure I've been clear about this. "...there are softer and harder forms of science, and that the distinction between the two is in the STANDARD for supporting evidence that must be obtained before a body of theory can be highly esteemed."
You'll notice that I don't mention complexity in that quote.
"Who is arguing we shouldn't act this way, Robert?"
Essentially, I think that YOU are. I think that when you say,
"We don't exactly have a grasp on what makes for successful macroeconomic policy. But not doing something is as much a policy choice as doing something, so given our imperfect knowledge I would emphatically insist on certain policies."
I think that you are saying that it is alright for a scientist to argue FOR one policy position and AGAINST another, when there is insufficient evidence to substantiate either position. I think that scientists should have the courage to refrain from offering policy recommendations until theory has been well tested and demonstrated to generate accurate predictions about the behavior of the system.
As I keep reiterating, for me the question is about living up to a very high standard for evidence before embracing a theory.
"You'll notice that I don't mention complexity in that quote"
ReplyDeleteRight, but I don't see any difference in treatment of evidentiary standards (and I don't see anywhere where you've raised any differences in evidentiary standards) except the expectation of relative levels of precision in the evidence emerging from the complexity of the subject of study. So I can't conclude anything other than "soft means complex"... which is OK if that's what is meant, but usually it is intended to mean "less faithful to the scientific method".
think that you are saying that it is alright for a scientist to argue FOR one policy position and AGAINST another, when there is insufficient evidence to substantiate either position.
I'm not sure where you're getting this. What I'm saying is that all evidence is imprecise, and evidence about the mechanisms behind a complex system are necessarily more imprecise. I would not say that someone should make a claim when there is insufficient evidence. What I'm saying is precisely that they should hold off for sufficient evidence, but should also act rationally in response to sufficient, imprecise evidence. But sufficiency of evidence is absolutely crucial and I have no idea how you've construed me as saying otherwise.
I think that scientists should have the courage to refrain from offering policy recommendations until theory has been well tested and demonstrated to generate accurate predictions about the behavior of the system.
I could not agree more and I'm not sure why you interpret me to be disagreeing.
See - you say "sufficient evidence", but then whenever I press for what that means, you turn to high-fidelity forecasting. I don't see how they could possibly be the same thing.
ReplyDeleteI do think economists should stay quiet unless they have sufficient evidence. I do not think they need to be astrologers that see the future.
"...you say "sufficient evidence", but then whenever I press for what that means, you turn to high-fidelity forecasting. I don't see how they could possibly be the same thing."
ReplyDeleteReally? So if I have a model for some real world situation or thing, my model is still a good model even if it gives incorrect predictions of how that real world thing will behave? How does that work?
Here's a practical example: Let's say I'm a mechanical engineer and I have an equation that tells me how much force it will take to bend a steel bar. Except, the number I get out of the equation doesn't match the actual force at which the bar bends. How good is my equation (my model) then? Not very good.
Here's another example: Let's say I'm an economist and I have an equation that tells me how the unemployment rate will change if I change government spending. Except that the numbers I get out of that equation don't match reality very well. How good is my model? Well, maybe it's not very good, or maybe I guessed wrong on some of the x's, y's and z's that I didn't know. Either of those explanations for the failure of my predictions is possible.
Ultimately, it doesn't matter much, because I KNOW that my model doesn't give reliable predictions. The only fair thing to do is to say: "I don't have any science that's good enough to confidently predict what outcomes will be produced by different policies. Sorry."
So, what is sufficient evidence? The ability to predict accurately, repeatedly. Models that do that are faithful to the thing they are modeling.
Really? So if I have a model for some real world situation or thing, my model is still a good model even if it gives incorrect predictions of how that real world thing will behave? How does that work?
ReplyDeleteDo you have sufficient information to produce a forecast from the model or not? In some cases you do, and in some cases you don't. In some cases the informational inputs are probabilistic or chaotic in which case you certainly don't have the right informational inputs to make a forecast. But none of that seems to speak to the scientific quality of the model at all. It only speaks to the extent to which the model can be practically applied for a very specific purpose. I am not prepared to call climatology less scientific than physics because of its forecasting potential because I don't see what that has to do with science. I will be prepared to call it "complex" (and when I use this word "complex" I alway mean it in the formal mathematical sense of the word).
You seem to be identifying forecasting with evidence and I think that's wrong - and that seems to be why all I can pull out of your claims that makes sense to me at all is "economics is a science of a complex subject, and should be regarded by economists and the public as such".
I suppose what's hard for me to swallow about your position is that you are wrapping up (1.) science, (2.) engineering, and (3.) forecasting into the same thing. If the subject of science, engineering, and forecasting is an astronomical system then we end up getting pretty good scientists, engineers, and forecasters of that system. If the subject of science, engineering, and forecasting is a progressively more complex subject we don't have as good forecasters (although they're usually still good enough to be useful), and we don't have as good engineers. But we do still have pretty good scientists.
You seem to be taking the limitations of an economic forecaster or an economic engineer and blaming it on economic science. That doesn't make sense to me.
As far as I can tell, science is about building models of reality. How do you evaluate those models? By how accurately they predict real outcomes. Period.
ReplyDeleteThe fact that engineers also use models (to help them figure out how to engineer outcomes) shows that science and engineering are closely related. That's not conflation.
Also, you keep missing my point. Having poor quality models doesn't make your work unscientific. That just means the science isn't very far along yet. But it IS unscientific to pretend your models are better than they are. E.g. to make policy recommendations when you're working from a low quality, low fidelity model.
"Also, you keep missing my point. Having poor quality models doesn't make your work unscientific. That just means the science isn't very far along yet."
ReplyDeleteThe first sentence encourages me and when you alluded to this earlier it equally encouraged me. The second sentence ends up confusing me. "Very far along" to what? To some ideal state of knowledge? Well sure, I'd agree that economics is less far along than other sciences. Is this what you mean by "soft science"? I'm just having a lot of trouble pinning down exactly what you mean by "soft science", particularly because a lot of other people use "soft science" to mean "less scientific". At first the difference I could see was complexity. If all you mean is that economics is more complex, I'd agree. If what you mean now is that there's a lot left to figure out I'd also agree. I'd strongly agree with both of those points (whether you're making them or not) and I don't think I've indicated that I haven't.
"But it IS unscientific to pretend your models are better than they are."
Well... that's probably just more "unprofessional" or "inaccurate" than "unscientific", but yes I agree.
E.g. to make policy recommendations when you're working from a low quality, low fidelity model.
Hmmm... still not clear on why making policy recommendations with imprecise model projections is tantamount to "pretending models are better than they are". Do you think the weatherman is being unscientific when he advises you to bring an umbrella? Do you think knowledgeable folk living in a flood plain are being "unscientific" when they advice you to get flood insurance? I'm really not seeing why this policy recommendation thing bothers you so much.
I'm actually pretty baffled about how I've been unclear about what I mean by 'soft'. I've said it repeatedly.
ReplyDeleteSoft = less scientific = pushing models that have weak evidence to support them.
And supporting evidence for models is inextricably bound up with whether those models give accurate predictions.