How weird, I've just today done a map for someone who asked for a map based on the same picture. Go to this thread to see what I arrived at. To get the frilled effect I drew a few vertical lines with the same proportional length and spacing as the frill lines on a piece of paper with a black pen, scanned them in and saved them as a brush (you could probably just draw a few vertical lines in Gimp, come to think of it) and use those for your brush. I used a vector program which has raster effects and the brushes work differently, and I don't use gimp, but the settings you need for your brush will be that the brush pattern stays 90 degrees to the angle of the line and you might want some size jitter in there as well. Then it's just a matter of stroking the outline of the continent with the brush.

If you're a new with Gimp, you may need to wait for one of the gimp users to come along and explain it better!