I almost never hand over the source files for anything, the source files are my property, the resulting jpg/pdf/flattened image etc is what I'm selling. In theory sourcefiles also hold information about how you do your magic. In addition to that the work also consists of using different brushes, textures etc - some of those could be brushes you bought from others and if so are not legal to share with others.
A compromise could be delivering a unmarked map as well, as it probably will be names and stuff that might change. If there suddenly appears a rift through the continent I'm guessing the client would want an edited map for that anyway.