There is a great tutorial by Oldguy in the tutorial sections on creating coastlines. You roughly draw the coastline you want and using his method adds the 'crinkly bits'