No no, don't give up, I love what you have going here! You could easily add clarity by writing "1. Bider House", "2. Mengin House" etc. And then just simply add the numbers beside the appropriate building like on any top-down city map. Easy and effective, not much more work

Cheers,
Tainotim