There was a show on just recently that had a guy that is a professional tree house builder. He built a tree house that spanned two trees and had a floating floor to allow the trees to move independently. Very cool. During the show they had some incredible tree houses. One of them was a three story house that was about 2500 square feet if I remember correctly and it was fully furnished to boot!

Now in a fantasy setting with magic at hand, I don't think there would be any problems with building a huge tree house castle especially if you could simply ask the tree to help.