One thing I found really helpful was to look through the Coastal Landforms portal on Wikipedia (it's one of those nifty groups of topics with an index to the portal at the end of each page). In addition to being fascinating reading for a map nerd like myself, it gave me a good idea of which shapes are possible and why they tend to happen. The second piece is to guard against uniform variance (this is the part I struggle with). It consists of actively tryibg to make your squiggliness random, and takes some practice.
Of course, this is all if you're hand drawing the coast from nothing; there are all kinds of tuts for random (and semi-random) coastline generation in the tutorial section.
Sorry for lack of links ... tapatalk makes it impossible to hop out of a post in progress to grab a link and then come back and not lose what you already typed!
Sent from my iPhone using Tapatalk