I just found a coding error in some stuff I did last night, which is not surprising because I was burning the midnight oil.
It was pretty obvious - some people had GPA's of over 20.
That's not right.
Turns out in one line of code instead of weighting the average grade in a given field in the first semester by the number of credits in that semester, I weighted the credits by the credits - so a few people that took a lot of classes in one field in their first semester of college (that's not a lot of people, but some) got big GPAs.
I found the coding error quickly with some descriptive checks (always important), but most importantly because I knew what results made sense and what didn't. I was looking to make sure my GPAs ranged between zero and four (they're standardized to that range). Now they do.
Coding errors are easy to find if you can apply the smell test to your results. Multipliers of 5 do not make sense, even to the most ardent Keynesian. Negative multipliers ought to give the most ardent libertarian pause. This is why we do lots of studies on things like supply and demand elasticities - because they give us a feel for what makes sense or not.
And if it doesn't something's wrong. Maybe it's a statistical artifact, but maybe you did something wrong.
What does this mean for Reinhart and Rogoff? I don't know. Their result isn't wildly implausible particularly since in their own paper they always acknowledged the feedback between economic performance and debt. Maybe the big tipping point result was a little less plausible. At least the tipping point should be conditional on some other factor associated with creditworthiness or the monetary system (England during the Napoleonic Wars, the U.S. after WWII, Japan today, Belgium, etc. ought to give us that intuition).
The smell test probably isn't quite as cut and dry as it was for me this morning when I knew something was wrong, but the point still remains: good theory and a good sense of how the world works fosters good empirics, which in turn contribute to a better understanding of how the world works. And perhaps most importantly, this iterative relationship between the quality of theory and the quality of empirics opens the door to multiple equilibria, including good and bad scientific equilibria.