Its assumptions are much too strong for the real world: pure rationality, perfect mutual knowledge of the rules of the game and the payoffs for each player. Game theory does certainly provide insight into many realistic phenomena - the prisoner's dilemma is the classic example, variants of which can be seen all over the economic world. But I have never been especially inspired by the exercise of working out Nash equilibria, because they usually illuminate only the stylised game that has been written down, and not the world that it is meant to model.
However, two relatively new variations on game theory are making the field more interesting.
The first is the idea of learning. For example, Fudenberg and Levine's book, The Theory of Learning in Games, sets out ideas of how real people playing games can come close to rational behaviour by trial and error, paying attention to each other and learning from their mistakes.
The second is a rich set of heuristics used by players and analysed in behavioural game theory. Their specialism in this area of work was the only thing that tempted me to apply for a PhD at London School of Economics (though in the end there wasn't quite enough appealing about it). Camerer's book, Behavioral Game Theory: Experiments in Strategic Interaction, is the bible of this field and explores some key issues which can actually make the analysis of gameplay realistic:
- theories of moral obligation (why does the ultimatum game work?)
- limits on foresight (I think that he thinks that I think that he thinks...how deep can we go?)
- experience and learning
These make all the difference in the applicability of game theory to real life - or, for that matter, to real game design!
If you're designing a set of economic incentives, or making a game for real people (that is, not economists or mathematicians) to play, these heuristics will provide much more useful tools than the study of payoff matrices and Nash equilibria.
Indeed, unlike standard behavioural economics - which is only a supplement, or a complement, to mainstream economics - the insights of behavioural game theory could make the study of rational game theory more or less pointless.