Trees are a funny thing. While any fortification worth a darn would cut back the forest outside the wall at least to projectile range (to give your archers a good shot at approaching bad guys), there might be trees either left inside the wall or planted asap after it was built. Animals waiting near a stable will be in better shape if they can take shelter from the sun in the shade of a tree. The blacksmith needs to set his forge in a stone building with a roof to protect his tools from the weather, but he will do his hammering under an open shed or beneath the branches of a tree.

Some trees are grown and protected as part of the defense industry - wood for arrow shafts, wood for bows, wood for axe handles - all needing certain characteristics and potential candidates for inside a castle wall.

Or of course, your castle might have no trees.
I just wanted to point out that 'trees' or 'no trees' is completely YOUR choice - history and logic support either choice.