I spent many hours experimenting with this method (the GIMP version) because since I started to be interested in the mapping activity (not long ago), I have been looking for a way how to transform a handdrawn coast in a realistic one.
Unfortunately nothing I tried did that.
The reason is that a realistic looking coast is fractal. And fractality means self similarity. That is when you zoom on the coast in different resolutions, you see a similar irregular shape.
The problem with all pixel based Tools is that they only transform a line on a scale of a few pixels - say 2 or 3. So if you had a too straight coastline and apply these methods, once you zoom out you still have a too straight line with a few pixel sized bumps and nooks.
Increasing the roughness isn't a solution either because you loose then continuity - the coast dissolves in a multitude of pixel sized islands and "rivers".

So I thought that to make a realistic coast line, one would need at least 2 scales of transformations. Of course in the nature there are many more but we have computer constraints.
For instance a 2-3 pixel scale is taken care of with methods like this one.
But we'd need above it another scale (say 8-15 pixels) which doesn't work with pixel transformations but with line transformations and makes sure that continuity is conserved.

This transformation would be just a rotation and a length. Say you start at a point P which is outside of the handrawn line. The angle of the next segment is random but not Gaussian - the probability is higher in the direction of the handdrawn coast (works like a spring attracting towards the original coast) so that one doesn't loose the general shape. The length may be gaussian with the mean being the distance of P to the handrawn coast and standard deviation a parameter governing the irregularity.
This defines the new point P', the next segment PP' is drawn and rince and repeat.
The result is a fractal line at a scale of about 10 pixels which broadly follows the handdrawn shape but creates irregularity at this scale.
If one makes the scale a parameter, one could create a very realistic looking coastline by starting at 50, then 10 and last 1 with a pixel transformation method like this one.

I am pretty sure that I don't invent anything because the maths are basic but I didn't find something like that in the Tools I have (GIMP, FT3, Wilbur, World Machine).
However this may be because I am still a newbie in using those Tools.
So if some expert among you knows of a way how to make this, I would appreciate a hint.