I've had pretty good results in photoshop by drawing the outline of the coast, selecting the area outside it with the magic wand, inverting the selection (this allows for fewer steps if you enjoy having lots of islands), and then playing around with the smoothing, expanding, and contracting before finally painting the land color on a separate layer. It leaves a fairly smooth looking shape for the land with a minimum of fiddling around.