Why does capitalist market theory have limited applicability to nonstate societies?

That is because the economic sphere is intertwined in other spheres of life in non-state societies

In state societies, the economy is a separate entity from the life of the people but the people do participate in it. It is regulated by the state and applies to all equally. In non-state societies it's much more close to people because it's a part of everything they do.