Well, medieval cities and even walled cities did have gardens. The herbs were used by households for medicinal and cooking purposes. Some city gardens even had apple and pear trees. Only in the most populous cities was garden space almost completely crowded out, but public parks sprang into existance in later ages.
I am now quoting from "Life in a Medieval City" by Joseph and Frances Gies, two amateur historians. A great book I recently bought.
Also, I know lots of cities in the Netherlands had herb gardens, some even existing today. I did a search in books.google.com and got more examples, including a book about herb gardens. A quote:
"The enclosed privy garden, close by the town walls or flanked by other buildings in the crowded, fortified medieval towns, can be adapted as a pattern for modern herb garden designs."
And a final piece of evidence. Most of the old city maps I have seen feature garden space. Some examples:
These maps were mostly made in the 16th century, but I can't imagine those cities didn't have gardens a few centuries earlier.
Anyway, I am not saying the medieval city had a lot of space for gardens, but they certainly weren't as rare as I once believed myself.
(they were probably more rare in the early Middle Ages, but looking at those maps they might even have been common in the late Middle Ages).