What were some the hardships that did not allow Greece to have a large population?

Little farmland, they were only able to grow grapes and olives and no grains. Little fresh irrigation water that could not sustain the entire population.

During 800 BC, Greeks began facing overpopulation, their land was not able to support a large population by normal agriculture due to the sole conditions and water irrigation system so they were only able to grow grapes and olives. They began to establish colonies sending their people throughout the Mediterranean world to create economical ties selling olives and grapes in return to grains with those lands but they didn't established political control.

One of the main problems that Ancient Greece faced was that, contrary to popular perception, the land quality in Greece was not optimal. Terrain tended to be rocky and grasslands were not common.

Animal husbandry, therefore, was restricted to a small percentage of the population and was not well developed in Greece. Moreover, crops that were grown were resilient, but labor intensive (such as olives) or low in nutrients (for example, barley, as opposed to wheat). During the hot summers, irrigation was necessary, and due to the small number of farm animals, Greeks did not use animal manure or animal power, both of which greatly reduced their agricultural productivity.

All these factor meant that Greece could not achieve the large population that can be seen in large Ancient empires.