If that's on two layers, here's what I would do. First use Rob A's Tutorial on Randomized Coastlines to break up that straight edge. The human mind is really great at seeing that sort of thing. Now that you have a nicely jittered hard selection, this is the time to blur the transition. Don't blur the image itself, but the alpha channel for the upper layer.

Hopefully this is somewhat clearer than mud...