I assume you mean the clutter west of the river. Thats supposed to be a large slum with ppl propping up shacks and hovels everywhere there's room.
The town grew out of the old town at the mounth of the river and the rich people got the east side by the castle and that became the nice part of town, with merchants temples and official buildnings. The west side grew from a small slum to a huge slum area, merging with the large farms on the southern hill and starting to spill out beyond the relatively new walls.

The star shaped fort on the hill is for the local Paladins of Heironeous chapter with 'modern' sloping walls and heavy bastions.
The older castle overlooking the sea is converted from a medieval castle

Oh, i really need to find a reason for the campaign to stop of at a sizable fortress, or find some other excuse to make a map of one.