Alright, I made some changes.
I changed the colors, added a river, modified the street layout.
Ok, the story of the city: The road was a trade route. The river is the only fresh water for miles, so this point was a necessary resting point. The traders would as well sell stuff here while they were resting. After getting some fame/money they built the port to connect the water trade routes with the land ones. But with fame comes evil-doers. They were really vulnerable to attack, being in open field. So they built a Garrison up on the hill. The city got pretty big, and they joined a kingdom. After a few decades, through clever politics (read assassination and lying) one of the citizens became the king. Originally the capital was not here. The city was almost burnt down in a fire, but they are very proud people, and decided to build the city in the same place with rock from the closest mountain (apx 4 days to the N-W). The king saw this as an opportunity to move his court to a place safer for him, so he moved here.
Don't worry, lostatsea, I don't feel discouraged. I'm really thankful for all the help.