Mathematical modeling really isn't the problem, this is:

"But the people who saw the collapse coming weren’t people who built better models; they were people who questioned the assumptions in the existing models and figured out how dependent they were on those unquestioned assumptions." - Dan McLaughlin (HT - Jeff Friedman)

I can't speak for the criticism of Nate Silver. Unlike a lot of it out there, this does seem like an informed criticism. But I like his take on the use of modeling and what makes a good model. People who act like formal modeling is the problem miss the point, I think. Prose is a model too - just a whole lot fuzzier and harder to pin down. The problem (and strengths) of any model always lie in the applicability of the assumptions it makes.

In that sense I sort of disagree with how McLauhglin phrased it. If your model is simpler and less sophisticated but jettisons a bad assumption that misleads a more complicated, more sophisticated model, then you have built a better model. But I still agree with McLaughlin's general sentiment.