Hmm...It seems that your coasltines were very pixellated before you applied some blur, so you probably have to blur them a lot but doing this, you lost outline quality. I'd suggest you get first a landmass that is less pixellated.