Hi Smithy I'm not sure what it is you're trying to do here. If what you are after is lessening the opacity of the sea around the coast in a feathered style, then the quickest way I can think of (there are probably a host of other ways) to do that would be to select the land mass, expand the selection by the number of pixels you want the feather to operate, (Select/modify/expand), then feather the selection (Select/modify/feather) and hit delete.

I'm not sure if that's the effect you are after?