For feathering out that woodcut there are a lot of different techniques. The easiest is to select the landmass, increase the selection, feather the selection, invert the selection, then delete. If you want your lines to be pointy on the end then you'd have to make a dotted line around the coast then use some wind or motion blur. If you want squiggly lines then you also have to add a ripple or a wave. Myself, I use a small black and clear checkerboard pattern, then wind, then ripple, then feathered selection delete to soften the points.

What you have so far looks pretty nice