Thinking further on why the market bugs me, I think it stems from the dock location. The keep is the main reason the town formed, right? If you were building a keep, wouldn't you want the docks as close as possible? I think that's what drives my vague discomfort with the overall layout. Maybe another dock just south of the bridge on the west bank connects up with the open square area and a second, more upscale trade area?