Are immigration restrictions "extremely unjust" and why?

I generally favor loosening immigration restrictions and not making distinctions on the basis of different classes of workers, but I think that because it's smart and mutually beneficial. There are specific immigration enforcement measures I would probably call "unjust" but I've never really thought of immigration restrictions themselves as unjust much less "extremely unjust", the way that Bryan Caplan does. Bryan says that we should consider the justice of these laws before we talk about whether Obama went against the "rule of law" in making his executive decision (I think most people don't talk about the justice of the law because most people think that question is settled, not because they disagree with Bryan on the order in which these arguments should proceed... Bryan brings up slavery and the Holocaust but he's very lonely in thinking immigration restrictions bear similar "extremely unjst" characteristics)

This isnt' the first time Bryan has made an ethical point he thinks is obvious that I think is ridiculous and poorly argued (for example, his pacificism), but he provided a link to his Cato paper so I thought I'd check out why. He begins:

"Consider the following thought experiment: Moved by the plight of desperate earthquake victims, you volunteer to work as a relief worker in Haiti. After two weeks, you’re ready to go home. Unfortunately, when you arrive at the airport, customs officials tell you that you’re forbidden to enter the United States. You go to the American consulate to demand an explanation. But the official response is simply, “The United States does not have to explain itself to you.”

You don’t have to be a libertarian to admit that this seems like a monstrous injustice. The entire ideological menagerie—liberals, conservatives, moderates, socialists, and libertarians—would defend your right to move from Haiti to the United States. What’s so bad about restricting your migration? Most obviously, because life in Haiti is terrible. If the American government denies you permission to return, you’ll live in dire poverty, die sooner, live under a brutal, corrupt regime, and be cut off from most of the people you want to associate with. Hunger, danger, oppression, isolation: condemning you to even one seems wrong. Which raises a serious question: if you had been born in Haiti, would denying you permission to enter the United States be any less wrong?"

 

 

Huh?
Am I crazy, or is he completely missing what's actually wrong with this? What's wrong is that my home, all my stuff, and my wife is in Falls Church, Virginia not Port au Prince, Haiti! What's wrong is that I've been a contributing citizen and a productive employee in the United States, not Haiti! What's wrong is that I was born in Texas, not Haiti, and that my parents were born in the United States, not Haiti, and that my grandparents were born in the United States, not Haiti, and that I've got people born in the United States back to the early seventeenth century!

It has nothing to do with how hard life is in Haiti.

A couple summers ago when we visited Paris we stayed in what seemed to be a very nice part of town. Our room was a little small, but we were a stone's throw from the Louvre and everything seemed to be very ritzy in the vicinity. I ate much better there than I do at home. It was a pretty cushy week for me. And contrary to what I'm often told, the French were very friendly and for the most part they tolerated when I wandered around armed with the half dozen French words I knew (Kate knows French, but I would often wander out again when she napped). Paris is a wonderful corner of this planet in virtually every respect.

My point is that if custom's officials denied my entry on my way back from Paris I would be just as outraged as I would be if they denied me entry on my way back from Haiti. It has nothing whatsoever to do with condemning me to an impoverished existence. It has to do with the fact that I have claims to entry that ought to be honored that aren't honored.

The question at hand is: who ought to have claims to entry? I do think we should be fairly liberal about this point, but I think Bryan is making some very bad arguments about why.