Well the idea, which I guess I didn't get across very well, is that the 'castle/keep' in the north is very old, the very first building constructed, hence why it is so small, and there is little room to expand it on the island. I'll probably need to add some interior 'buildings' within the walls, to fill it out a bit.

When I get a chance I'll try removing the 'outline' and see how it looks.