I developed my own method of creating realistic coastlines. I'm very pleased with how it turned out:
Old Guy Gaming - Creating Realistic Coastlines
I'm ready to begin work on the actual map. My sketch shows the basic shape and dimensions that I want but the outline is too smooth. I wanted to rough it up to make it look more realistic. I looked through various tutorials but everything I tried distorted the original outline more than I wanted. Probably the result of my being new at this and not following the tutorials properly.
I kept at it though until I hit upon a process that I'm pretty happy with. This was done in Adobe Photoshop.
I started with my island in white and the surrounding area in black as shown in the first image.
* Filter | Noise | Add Noise (gaussian, monochromatic, set to 100).
* Filter | Blur |Gaussian Blur (set to 2).
* Image | Adjustments | Threshold (adjust to taste). *
* Select Magic Wand Tool, click in black surrounding area. (contiguous checked)
* Shift-Ctrl-I to invert the selection
* Fill with White. (contiguous unchecked)
That's all there is to it. Now I have the beginnings of my very first Photoshop map.
* -- Move the slider to the right. You want to eat away at your landmass. Don't worry about how the land looks at this step. We will fill it back in. We just want to roughen the edges. By creating "holes" everywhere, we are trying to let the outside creep in a little bit. If there are too many holes, the outside will never hit anything to stop it. You want just enough holes to let it creep in a little but then run into a boundary. You may have to undo and try different amounts a few times to get the effect you want. Once you've done it once, it will be much easier to do than it is to explain. : )
EDIT: There is an updated version of this tutorial at Old Guy Gaming.