First and foremost, definitions are what we decide they are - so the worst thing about the heterodox push-back isn't that any of it is wrong per se, it's that they're wrong for browbeating Noah for using by far and away the most widely used definition and asking, for clarity's sake, that people use that definition when it's appropriate. I couldn't agree more.
In my view "neoclassical" economics is economics that cites optimizing behavior as the principle explanation of human choice under conditions of scarcity. "Self-interested", of course, is not always "optimizing" (although it can be) so we usually date neoclassicism to the marginal revolution rather than what we usually call the "classics". Of course there are hints of marginalism (and therefore proper optimization) earlier and that's fine - these things can be fuzzy. Neoclassicism does not mean hyper-rational actors. Neoclassicism can include things like game theory (that just introduces a strategic framework to the optimization problem) and even behavioral economics (I'm thinking of things like hyperbolic discounting - but probably a lot of behavioral economics can be imported into an optimization problem, though it doesn't have to be).
Neoclassicism shouldn't mean "mainstream" because there was a time when "mainstream" wasn't neoclassical and it might very well not be neoclassical again. "Mainstream" has sociological, not analytic meaning.
This, I think, is basically along the lines of what Noah said.
This isn't the only definition one could use to describe these ideas. Most notably Keynes (borrowing from Marx) called the classics and the marginalists "Classical". Again there's nothing inherently wrong with that, most people just decided that there were more useful ways to divide and sub-divide, and so no one uses Keynes's terms any more.
Matias Vernengo has some good stuff and some bad stuff to say about neoclassicism, but what most struck me was how he bristles at the way Noah used the term "heterodox":
"So Noah Smith thinks that I, Lars Syll and Steve Keen, and other heterodox bloggers (in which he adds Austrians; you see why they should teach History of Thought?* For a discussion of the meaning of heterodox economics, including why Austrians are not so, go here) use the term neoclassical economics as a pejorative term."The only people who use "heterodox" the way Vernengo uses "heterodox" is fellow Post-Keynesians and Marxists. It's not how the JEL uses the term (they include Austrians) and it's not how most historians of economic thought use the term. Noah (and my) definition of heterodox has a lot going for it for that reason alone, but even if you were to come at this issue with a blank slate I think it makes more sense. Austrians, like Post-Keynesians, have several elements to work that few mainstream economists use, think are the best way of doing the science, or even know about in a lot of cases. That is what heterodoxy is. "Heterodox", like "mainstream" is a term with sociological rather than analytic meaning. Some (not all) Austrians can be described as neoclassical. Same with Post-Keynesians (Joan Robinson did a lot of great neoclassical work for example - Kalecki did similar things but with non-neoclassical [one could probably say "classical"] exposition - they got along because of a common heterodoxy, but their analysis was in two distinct traditions). That doesn't really matter - they're still heterodox (just like a lot of non-neoclassical behavioral economists are mainstream). There's no litmus test either - lots of people work in both traditions and there's a lot to respect in both.
If Matias wants to use a different definition that's fine. I work with a lot of Post-Keynesians in my department and you all know I think they have some important things to contribute. Among them, when I hear "heterodox" I know how the term is being used and it all works out fine. But one thing they don't do (and something I don't think Matias should do) is lecture everybody else about how they need a class in the history of economic thought when they use a different definition.
Matias seemed to agree with a lot of what Noah had to say about neoclassicism, but he preferred to distinguish neoclassicism for its conclusions about "endogenous distribution". This doesn't make sense to me at all. Obviously what Matias calls "endogenous distribution" emerges in neoclassical economics, but it seems to also emerge in most Post-Keynesian economics (it just doesn't emerge in the same way - often, though not always, factor prices are given and it is the wage share that emerges endogenously - but that's still the distributional outcome). He also mentions "minimal government that protects property rights" as going hand in hand with neoclassicism, which I guess might be true if it is understood very broadly and very weakly, but I don't think these sorts of normative claims have much of anything to do with what we're talking about.
I do agree with Matias's point about the non-neoclassical mainstream work that Noah brings up being within the "protective belt" of the neoclassical paradigm. But this is instructive. Most of this work is what we would call "institutionalist" which is most decidedly not neoclassical. It's published next to neoclassical papers because it provides excellent economic analysis within a different analytic framework. Meanwhile there are other "institutionalists" that are heterodox. There are also "institutionalists" like Doug North who is celebrated by the mainstream despite his best efforts to push the mainstream away and insist he's working on a completely different project! All this is quite instructive! What matters for being in the "mainstream" is mostly about who you are willing to work with, what ideas you are willing to accept as decent ideas, and where you publish. It's a sociological term. To be out of the mainstream you need two things: (1.) you need to have elements of your analysis that is different from most of the mainstream, and (2.) you really need to be exiled from the mainstream or (more commonly) you need to exile yourself. Some institutionalists thrive in mainstream economics. Some institutionalists are very adamant about not being in the mainstream and criticizing the mainstream. The latter are heterodox, but really as a matter of choice and affiliation.
Similar problems come up with the responses of people like Lars Syll (who thinks neoclassicism implies deductivism) and Deirdre McCloskey (who wants neoclassicism to mean Paul Samuelson). You can click through the links (I won't cover them here) but the problems with their thoughts are pretty much the same problems with Matias's: they take entirely reasonable definitions and conditions that they like a lot and are flummoxed that most people seem to think differently.
Here's the point: definitions are useful if (1.) they make meaningful distinctions, and (2.) a lot of people use the same definition to mean the same thing. You need both elements. Noah's use of terms like "neoclassical" and "heterodox" meet both elements. Matias's only meets element (1.) (probably - although maybe not) in general, although it also meets element (2.) if he's talking to other Post-Keynesians.
And Noah uses it how most history of economic thought people use the terms and how most economists use the terms.
So give the guy a break.