I think if you look at real buildings like Westminster Abbey, you'll see that quite large buildings can be constructed. Keeping in mind the Westminster has very tall walls with very little inside for strength, which is why it has all the flying butresses. If you were to add floors some additional supports and outbuildings connected by raised walkways that structure could easily rival most fantasy buildings.

The thing that always holds me back is.. why would you need a building that large, what do you do with all the rooms, what purpose is it being built for?
Then even less important for mapping but really important for gaming, is how would this really large building work in a game situation?