Ok, so this is only a mini-tut by any standards. I just did a search to see if there was already a tutorial covering this and I found that RobA's tutorial was almost the same but I hadn't checked it out because it was titled for GIMP.
Ah well, so this is for anyone else like me, who is too stupid to check what is already out there.
Anyway... this is for when you have a rough idea of what shape you want your landmass to be but have trouble drawing the random jagginess required. When I started mapping I followed Jezelf's tutorials so my base is always black land on white sea, so that's what I'll do here. You can use that as a springboard to find the outline etc. if that is what you prefer.
1) So... start with a layer and fill it with white. This is just so you can adjust other layers opacity later.
2) New layer. Get your shape on there by whatever means necessary. You don't have to be too precise - if you draw too many detail in at this stage they will probably get lost in the later processes. You can use a brush if you like but I usually use the lassoo tool and roughly draw what I want and fill it in with black. When you have your shape or shapes set this layer blend mode to MULTIPLY.
3) New Layer. You should still have your landshape selected, if not just ctrl+click on the layer you did it on. Select->inverse (Ctrl+shift+i) your selection and fill it with white. Now, depending on how jaggy you want your landmass you can either just deselect it (ctrl+d) and give it a gaussian blur of 20-50 pixels. Or, if you want it extra craggy, you can modify->contract your selection by about 10-50 pixels then feather the selection by an amount somewhere in the same range, invert the selection again and cut the pixels away (ctrl+x). Or of course you could use a layer mask instead of cutting away but I find they actually slow the process down somewhat.
4) Go back to your land layer and give it a gaussian blur of about 20 pixels.
5) Start a new layer below the land layer but above the white background. Filter->Render->Clouds. This probably won't be jaggy enough so use Filter->Render->Difference Clouds a few times (CTRL+F to repeat the last filter). You might like to wait until the last step before doing though so you can see each adjustment to your coast.
6) Add a Layer->New Adjustment Layer->Threshold and make sure it is at the top of the stack.
7) Now just adjust settings until you get something you like. You have four variables you can play with - the opacity on your 3 working layers and the Threshold level on your adjustment layer. You can also keep running Filter->Render->Difference Clouds on your cloud layer until you get something you like.
8 ) Once you have something you like, flatten the image and run a small gaussian blur on it to take away the hard edges. Something like 0.3-0.5 should be fine.
Job done.
Here's a before and after...
Coastline1.jpgCoastline2.jpg