I can't really point you to specific books as it's actually a more complicated question than it might first seem. You're probably going to have to do a bit of digging to find answers to all your questions and you'll find it it's not the same for every culture, place, and time.

But basically it comes down to things like transport and most transport is walking if we're going with a traditional medieval fantasy setting. Cities often saw big changes in size when new transport options became available, which is why they tended to be small up until recently. You have to walk between the market/bakery and your place of residence/work to get food. You don't wanna be walking 5 miles there and 5 miles back if possible. If you're traveling from town to town along a well used road, you'll find villages at least within a days walk of each other.

In regards to castles, they have a err... "zone of control" or sphere of influence I guess you can call it. So you can only send out soldiers so far if you expect them to fight the same day. That's why the high density of castles in Wales was effective in bringing it under english control.

But the circumstances can be different from place to place! So I'd start by thinking "how long will it take me to walk there".