I am making a map using the Ascension Atlas style tutorial video I found here and I've run into a bit of an issue: when zoomed in, placing rivers I realized that I hadn't feathered the coast early on an have random pixelated noise. I tried created an inverted selection from the Base layer, modifying, and deleting, but this seems to have no effect. Anyone got an idea? Otherwise I'm starting from scratch.

I'm using PS 5.1, and am a photoshop novice.