If you're using photoshop, you won't see the feather since it's sampling the swamps and the layer below it. Hit delete after 5 (if your swamps disappear, invert the selection and then hit delete).

A more controlled way to feather is to select your swamp shapes and then create a new mask (swamps white, everthing else black). Alt click the mask layer and then use some gaussian blur on it. Using this method you can control where you want the feather and where you want to keep hard edges by using a brush with black / white or gray.

best

Ravs