[Award Winner] Using GIMP to Create an Artistic Regional RPG Map

[Seretur]

Ah, that could do it. You skipped a step:

Now invert the layer (Color->Invert) and blur the image (Filters->Bur->Gaussian Blur) The amount of blur will determine how random the final coast will be. The larger the blur, the more random the coastline will be (in the blurry area). If you have a well defined coast already (like here) use a smaller blur (50px). If you have a poorly defined coast (as is the case when using a random coast technique) make it a large blur.

If you want to preserve the mini-inlet northeast of the big bay thing ( I'm incredibly geographically literate, aren't I? ), but randomize the rest a bit, try a 50px overall blur, select the inlet area with the lasso, and Select->Invert. Blur that (50-100px) and the inlet should be preserved, while the rest of the coast is a little more random.

If you think your coast is just fine exactly how it is, you don't need to worry about this at all. This step is just to randomize the coast. If your coast is well-defined, and you like it, feel free to skip the Land Noise and the Land Clip, and go straight to the Land Mask.

(( In the future, adding a new post with the attachment would be most convenient for everyone. Now it looks like I'm double posting ))

EDIT: While I was typing, I was looking at your outline JPG. A lot of the definition got lost when you traced it, and I wouldn't recommend leaving your land layer how it is unblurred. If you want it to look closer to what you originally drew, try this:

Click on your Background layer, hide everything else.
Using the magic wand tool, select the sea.
In your Land layer (or a new one if you don't want to destroy what you've got), fill the selection with black.
Invert the selection (Select->Invert), and fill the new selection with white.

This way, it will keep your original coastline exactly (or almost). Then you can leave it unblurred, if you like.