Well I figured out a way to do it. The only noise effect that seems to do anything with Lock Alpha layer on is Hurl. I applied that noise, and then just desaturated the color out of it. That broke up the coastline outlines. Though I'm still confused why the other noise effects don't seem to be doing anything. Then I went through RoBA simple mountains in gimp tutorial quickly and made a mountain range that I'm pretty happy with. So later today I'll probably make some more mountain ranges on the map.
Love all the tutorials here! So many to go through and learn from!
I also just noticed that my selection strokes of the coastline also stroked the border of my image! heh. I'll have to erase that.
I'm not sure that the old paper background is really appropriate for this map, but I'm just going to keep going and see what it ends up like.