Thinking further on why the market bugs me, I think it stems from the dock location. The keep is the main reason the town formed, right? If you were building a keep, wouldn't you want the docks as close as possible? I think that's what drives my vague discomfort with the overall layout. Maybe another dock just south of the bridge on the west bank connects up with the open square area and a second, more upscale trade area?
It depends on the docks, really. If it's docks that can house bigger ships than you might want to keep it away from your keep. Any water is a great way for invaders to get in, especially if it's difficult or impossible to close it off from the outside (like a sea). The closer people can dock to your keep, the quicker they could get to it and perhaps even overpower it before whoever lives in the keep can do something to stop them.
Okay so I decided to flip the map and rethink the layout a little. The way I see it the eastern bank was the original settlement as most of the trade would come from the settlements to the east, so here you would have the wealthy merchants establishing their estates/houses relatively close to their businesses and the center of trade I would imagine. This would of course mean that they would be shoulder to shoulder with the industry/craft.
Over the water, in a more defensive location, stands the Barons keep, along with with what was initially the only temple that was to one of the Major gods. All this is away from the hubub, trade and noise of the town.
As the town begins to grow and expand (though it's still not a massive town and doesn't get that many passing travelers) it expands around the center of trade at the docks, as more people filter in the wealthy merchants no doubt slowly move away from the noise of the docks and relocate to the more peaceful western bank and thus the "Lord's District" is founded (though they're not necessarily lords). The buildings they originally owned in the Old Town may still be occupied by their families, or sold to newly arriving merchants, or converted into guild buildings etc. The Old town expands and then when it hits the walls begins spilling over into the eastern bank, but to the north and so still away from the Lord's District and a secondary bridge was built.
Also, added a personal dock to the keep
Scale of the buildings is still all over the place, and I've just shifted around images from my original map.
the map is great, the intention if ink sketch make it real