Wednesday, August 15, 2012

Always dig around in your data

I am much more excited about this restricted Dept. of Education data than I was when we first applied for it. I had no idea it had monthly longitudinal employment records for these people. I figured I'd be stuck with a couple employment variables every couple of years (for each wave of the survey).

Nope - monthly history.

The thing is this is the Dept. of Education, so they don't care about that. They used all that great data to create a couple summary variables and then stuck it all in a source file and buried it way down in a sub-folder. But I sniffed it out! If this were a Bureau of Labor Statistics dataset that would have been front and center (in fact it is front and center in things like the NLSY).

So the lesson is: always dig around in your data and get to know it. When you get started with some data treat it like a long-term relationship: a woman you're trying to woo, rather than a one night stand to satisfy your more immediate needs. I got that sort of relationship with the NLSY and have since repeatedly turned back to it because I knew the data like the back of my hand. Trying to develop that sort of relationship right now with the SIPP and the B&B.

Microdata is complicated and when people make investments in big surveys like this there's usually a lot of moving parts involved. There is always more to it than there first appears to be. Get to know it!


  1. I suppose it's one of the joys of research, eh Daniel Kuehn? Will this data be used for an article on economic history, or is it all contemporary empirical work?

    1. This is contemporary - covers the period 2007-early 2010 with new waves coming out in the future. Actually I'm a little worried it's too contemporary because the crisis hits right in the middle of it. Most of these students graduate in spring of 2008, so I'm hoping they get an ample chance of finding a job before it really gets bad (we're just looking at STEM students). I may have to dredge up the 2003 panel for comparison, though.


All anonymous comments will be deleted. Consistent pseudonyms are fine.