Just wanted to point out this tutorial for gimp (which should be transferable to photoshop) on creating not so random coastlines. It is a bit easier than the cut/paste/rotate/fit process you follow.

You could also probably apply a difference clouds several times at step 4 to make it a bit rougher, but you would want to make the initial blur smaller.

-Rob A>