Less pixellated coastlines?

[Asdrubael]

Hi guys! I've been lurking for quite a long time while I make a map for my headworld. I'm trying to figure out how to get less pixelated coastlines since I used the cloudfilter/threshold layer to generate my landmasses, but now my coastlines are pretty jagged and weird looking when you look at it in full res. Which kind of goes against how I wanted it to look in the first place. Basically, I'm looking to see if I can fix them without needing to pen in the coasts myself. If not, then I guess that's okay, probably would go better with the style that I'm working with anyway.

Here's my map so you can look at it.



Thanks in advance!