One thing you can try to eliminate or reduce the lace-like appearance of the coastline is Filter>>Other>>Maximum followed by Filter>>Other Minimum with the same settings (assuming sea is black and peaks are white). The maximum will spread the terrain and fill in 2 pixels areas; minimum will open the terrain edges back up but leave the small holes filled. This is the equivalent of a trivial morphological closing operation, if I recall.