Quote Originally Posted by Ascension View Post
Once you get your landmass how you want it and then merge things so that the land is white and the sea is black on one layer then use Filter - Noise - Median with a setting of 1 or 2 and that will take out a bunch of tiny pixels. OR you could use the magic wand tool to click on the landmasses, then Filter - Select - Modify - Expand = 1, new layer fill with white and merge down. This also takes out stray pixels. OR you could use Filter - Pixellate - Crystallize = 12 (then go back with a 9 a 6 and a 3) then Image - Adjustments - Brightness/Contrast and max the contrast. OR you could make up a new technique. The bottom line is that there are many ways.
I think a lot of that pixelization comes from the type of brush you use to 'create' the landmasses before merging the layers to create the base layer. It looks like Serene used a soft brush instead of a hard-edged circle brush, which creates that coastal effect. If you want smoother coastlines that still have interesting shapes, just use a hard-edged brush with a very low flow and change the size as needed.