...Continuing on:
Step 3: Building up and tearing down
As I mentioned before, you can either start by stroking your selection with black or with white. Black will cut away from the land and white will add to it.

Either way, the key to the method is to alternate between black and white a little, using progessively smaller brushes.

So, next, we're going to build up on what we started last time. Before we proceed, unselect everything.

Then, let's set up our brush. You'll want the foreground this time to be the opposite of what you used the first time (this time I'll use white). You'll want to select a smaller brush size, but not too much smaller. To achieve the right balance, you may need to play with the scale setting of the brush. Again, you'll also want to increase the spacing setting. However, as you use smaller and smaller brushes, you'll want to increase the spacing less and less.

Because our last step already redefined our coast a little, we'll need to reselect the new coastline. Use the same procedure of wand-selecting the black region and inverting your selection. (If you don't invert when you wand-select the black ocean regions, when you stroke the selection, you'll also get little white bits, assuming you're using a white brush, on the outside edge of the map...)

Here's I'm setting up the brush:

Click image for larger version. 

Name:	Coastline Tut Step05.jpg 
Views:	290 
Size:	238.6 KB 
ID:	7278

With your brush set up and your new region selected, you're ready to stroke the selection again.

Step 4: Wash, Rinse, Repeat

This step's fairly easy... just repeat Step 3, decreasing the brush size and changing the brush color, reselecting and re-stroking. After two or three iterations, you will probably reach something that is looking mostly good, if a bit rough around the edges.

Click image for larger version. 

Name:	Coastline Tut Step06.jpg 
Views:	217 
Size:	238.5 KB 
ID:	7279

Click image for larger version. 

Name:	Coastline Tut Step07.jpg 
Views:	303 
Size:	215.2 KB 
ID:	7280

Once you've done this, it's time to move on to:

Step 5: Cleaning Up a Bit

You may notice that there are a few relatively strange things going on. Our first attempt at using the jitter left a few circular holes close to our new coastlines. (Howbeit, your results may vary... which of course is the whole point.) Assuming those look as odd to you as they do to me, or if something else is bother you, it's time to clean that stuff up. In addition, I found the results a little too fractal-looking.

To clean up, I wand-selected the seas again, inverted to get the land, and then used a large brush set to white to fill in the erroneous-looking circles. This also cleaned up some of the over-fractalized edges.

Click image for larger version. 

Name:	Coastline Tut Step10.jpg 
Views:	272 
Size:	232.5 KB 
ID:	7281

Once you've got this where you like it, you're ready to move on with the rest of RobA's regional map tutorial, or use this however else you'd like to use it.

If you're going to use this as the "land mask", again select the black oceans, invert, and go to Select > Save to Channel. Or, you can use the "edge detect" at Filters > Edge-Detect > Edge... (I prefer the "Gradient" setting if you do this) to get a line-drawing of the coastline. (You'll get a white line on black field... just go to Colors > Invert to turn it into a black line on a white field.)

Anyway, here I move on with the next few steps of RobA's tutorial, using the results I've created in this tutorial as the "land mask" channel.

Click image for larger version. 

Name:	Coastline Tut Step11.jpg 
Views:	366 
Size:	282.3 KB 
ID:	7282

Final take-aways: this takes a little longer than the method RobA detailed in his tutorial, but gives you more control over the final result. It will take an artistic eye to find where the fractalizing goes too far and clean it up a bit, but it's easier than trying to draw all those squiggles by yourself.

I hope this method proves useful to you!