You could even put in a separate walk, around the barracks if you really wanted to keep people segregated. That would certainly give it a feel of a harsh military town.

How easy is it for people to come and go between the sections?

Can anyone go through the barracks and into the keep, or are we looking at strict inspections, possibly paperwork at each gate/