Well with that in mind ^.^ I agree that mountains like the one in Pasis' tutorial would look out of place. to clear up the coastlines, maybe go back to the flat black & white map created from the clouds and play with the sharpness? You might actually have to do some serious hand-editing. For the mountains, if you're going for hand-drawn antique style you should try the photoshop brushes created by PickeBu (on deviantart). At least one is called the "Old Map Brushset"; there are 3 sets, all of which contain quite a few very nice brushes. There's a forest set, a mountain set, and a sort of miscellanesou set. Everything looks like hand-drawn ink. I haven't used them yet, but I've played around with them and they're really nice looking. Hope those work for you, and looking forward to whatever you come up with, using these brushes or not! Good luck!

- Alizarine