During much of human history, birth rate was about the same as death. Population did not increased and if it does increase you might have a problem. Unless there is a new technology or something that allows to grow more food on the same space you will end up unable to feed everyone. Less food meens that you might only have have a lower life quality that will reduce birth rate or increase death rate. Or you could have up to 10% of the people die of starvation in 3 years only, like in the 14th centuy.

Another reason why population was not increasing is because of frequent desease and famine. These two where pretty frequent and killed millions of people. Even in the 20th century and today in some poor countries.

War is also a major cause of death. The Taiping rebelion in China is probably the dedliest conflict in human history. It was a civil war and there where other conflicts in the region at the same time !!! if I remember correctly. I guess that's what happends when you loose the mandate of heaven... Not only people die on the battlefield but lands/ villages get burned, many civilians are massacred, and these is still famine and desease because of crops got burned or have rotten in the soil since everybody is busy elsewhere. That is what is going to happend in Game of Thrones...