I told you how large are the buildings. Don't you think that a two-story building, having a total inhabitable space of ~300mq (~3200 square feet) would be enough for four families living in it (in the relatively cramped conditions of a technologically undeveloped society)? I think it would. Four families means ~20-25 people in this context, so 371 housing buildings you counted in the beginning could house ~7200-9200 people (if they all were two stories high).
I guess the basic point of contention is that it does not make sense to count by buildings, without considering how large those buildings are. 556 or 371 structures could house tens or even hundred of thousand people, if they were all residential skyscrapers. Or they could house merely a few hundreds, if they were all small shacks where 1 or 2 people lived. That's why you have to consider the area first, not the number. Are we on the same page on this??
The farms as showed in the pic are not too much for the population, rather too little, maybe way too little. If it is true that you need a square mile of farmland to support 180 people, then to support 5000 you would need 27.7 square miles (equals to ~71.7 km^2), which is almost exactly the entire area depicted in the whole map (not the cropped version showed in this last step, but the huge 10k*7.2k pixels I work with). This would be quite too much, but in any case it is clear that what you see now is too little. I will put some more, but I won't cover the entire map in farmlands, even if that would be more realistic (assuming 1sqmile*180 people is right, it seems a bit high to me, but I don't know anything about such things).