Who is going to pull it down? It is 50-60 miles to the nearest village, the road might be about 100 to the nearest decent sized town. Even if a few locals haul away stone for their own things, there was likely a village closer by that would make easier pickings for cut and dressed stone. If I was going to knock bits off a castle to salvage stone for my own use, I would likely nip away at parts of the upper walls where gravity is going to help me, and I'm not risking bringing half the castle down on me . Look at how many large castles stood abandoned in Europe. The biggest thing he'll have to work out is just WHY someone built that big of a castle there in the first place, and what caused the reason for it to exist to disappear.