I've been using this for my latest map (and a few others). I was asked while posting a WIP on my most recent map "how did you do that" so I promised to put up a step by step of the process. I know a lot of us use the techniques in here, but we don't seem to have a tutorial that's right here on the site for it.. (before you say yes we do, let me explain)
I want to say very clearly that this is based on the technique here:
Wood block stamp style water (S. John Ross' PS tutorial)(+in GIMP)
I checked the thread before posting this - the map linked in the first post (holycarp, freaking awesome) is there when you follow the link - but the link to the tutorial gets redirected and I can't find the tutorial anymore
So, instead of referencing that tutorial and mentioning my own tweaks as an addition, I'll include my entire process. Note that there's a lot of opportunity for fiddling to come up with your own look.
Note: This tutorial assumes the use of Photoshop - I originally worked with this technique in photoshop 7, and used it in CS3 and CS5. To make the wavy lines, this particular process makes use of the Eyecandy Jiggle filter - I have NOT been able to use it with CS5 as the filter is not compatible with 64bit (I only installed the 64 bit version, so it should likely work with cs5 32bit). There are likely other ways to get the wavy effect using various filters, though since I have Eyecandy still installed for my CS3, I did not put the work into developing alternatives.
Step 1: Draw out your land shapes. I highly recommend putting the outlines on their own layer (I name it Land Outlines). Once I've drawn the outlines, I duplicate the layer and fill in the shapes so each island/continent is a solid colour. I name it Base Land Shapes. Select your land shapes from Base Land Shapes. I do this by Ctrl-Clicking the thumbnail on the layer palette
Step 2: Expand the selection 100 px (you can use less or more depending on the look you're going for - but remember this number for the next step)
Step 3: Feather the selection by 100px or whatever number you used in step 2. If you want to get really wacky, you can start messing around with different values for each.
Step 4: Create a new layer, I usually just call it Grey. Fill your selection on this layer with the paint bucket using a medium grey colour. Note that the darker you go, the heavier the lines will be later on, lighter grey will give you more sparse lines. You could even experiment with gradient fills.
Step 5: Right click on the layer in the layer palette, and duplicate the layer - in the requester box that pops up you want to tell it to go into a New file - to do that, where it has the filename, drop down the arrow to say New (there will be a lit of currently open documents).
Continued in next post due to image count per post limitations.