View Full Version : Smooth coasts & Style?

02-15-2012, 06:44 PM
Hello forum,

I have been trying to make my coasts smoother or at least a lot less pixelized. I have followed the tutorial, which is the "Saderan" tutorial, straight down to the last word, but I can't get my to get smooth, they are always pixelized.

If its possible, does anyone know how to get coasts like the style in this map (http://www.cartographersguild.com/showthread.php?6795-Westeros-by-Tear), or at all? Usually I like the more realistic look, but I think the smoother coasts would be better for this one map. :blush:

I have PS CS5, GIMP, and Corel Painter (Student Edition), if it helps. :P


02-16-2012, 12:38 AM
I've had pretty good results in photoshop by drawing the outline of the coast, selecting the area outside it with the magic wand, inverting the selection (this allows for fewer steps if you enjoy having lots of islands), and then playing around with the smoothing, expanding, and contracting before finally painting the land color on a separate layer. It leaves a fairly smooth looking shape for the land with a minimum of fiddling around.

02-16-2012, 09:28 AM
Post an image. Typically, I (using GIMP) use RobA's coastline tutorial process. Of course, it also depends on if your doing print or web and the size. If you are printing out an image at 100(or less) DPI, then yea, things will look really crappy.

02-16-2012, 05:41 PM

Oh, so if I understand right, you use the expanding etc etc tool and then with the selection you fill in the land? I'll try it out next.

I too have used that tutorial, but the end results leave the image even more pixelized or clouded, if that makes sense?

Sure, here is an image of some of the coast line (my map is pretty big ). I tried the threshold option and even messed around with the settings, and I tried blurring it even. But nothing seems to be working for me...

02-16-2012, 09:52 PM
Alex: Yes, that's more or less what I do. Attached is an image of the results of such a method (sorry for the lack of an image before)


Edit: Alex, I hope you don't mind, I've taken the liberty to fiddling with your image using my method and I've attached the results. I did have to up the image size to a width of 1000 pixels in order to keep things smooth, but other than that, all I did was use the magic wand trick (with a smoothing of 1 px).

02-16-2012, 11:51 PM
I do not mind at all.

That is SO much better than my version! I like the pixel look most of the time, but for this map, that's what I wanted to do!

Do you perhaps have a post explaining what you do in fuller detail then?

02-17-2012, 12:17 AM
I don't have any in-depth explanations or descriptions of the process above as of yet. I can try and draft something up if that would be helpful though. If so, any input you have on what you would find useful would be very helpful as I've never really posted much in the way of such things.

Mostly, I just make liberal use of layers and the magic wand tool and fiddle about until I get results I like.

02-17-2012, 09:40 PM
Oh yes, that would be very helpful. :)

Hmm...I am pretty much use to CS5, so explaining the tools wouldn't be necessary or anything. So anything or anyway you can explain how you got the above would be nice. I'm am not sure if there is anything else. Maybe including if this can be done on any size picture, or is there a default size...?

02-17-2012, 10:33 PM
I just select the crappy pixel-landmass than I make a new layer and smooth the selection bye 2 pixels and fill in the new layer or Alphachanel with the now smooth lines. By the way I just found out myself to do so...

02-18-2012, 01:20 AM
Alex, I hope the attached pair of images will be useful. All images were made in Photoshop 7 (I've used CS5 as well but I don't personally own a copy, but the general results should be similar.)

The image itself is 1400x1400 pixels. The outline was drawn with a 3px round brush using just a mouse on a separate layer. I then selected the the the area outside the outline (the "water" if you will) with the magic wand tool. I then inverted the selection, smoothed the selection under Select > Modify > Smooth by two px. On a new layer I painted the landmass green with a large brush.

As you can see, some of the coastline details from the outline are lost in the process. If you spend a lot of time selecting tiny areas carefully, and/or selecting inside the outlines to see if that returns a better result, you can generally get a fairly close approximation of the what your outline originally looked like.

I hope this is useful.

02-18-2012, 01:16 PM
The "pixellizing" effect you're seeing is called aliasing. It is an artifact of using a digital medium—you cannot get a truly smooth angled or curved line when the image is built of discrete pixels that are either "on" or "off." The solution is to use some intermediate shades in places along the boundaries to create the illusion of smoothness. This is called anti-aliasing. For reference, I posted some example images a while ago here: http://www.cartographersguild.com/showthread.php?1641-Image-Export-Formats&p=14957&viewfull=1#post14957

Most of the time, you want to start with anti-aliased lines because there's not really any easy way to reliably and accurately anti-alias them later. To that end, avoid using the Pencil tool, and be careful not to threshold things too harshly. Also, work at a resolution high enough that you are not relying on pixel or subpixel accuracy for your detail.

Now, if you really need to fake antialiasing, there are a number of approaches you can use. Personally, I like to use some resizing tricks. In the Preferences > General menu, set Image Interpolation to Bicubic Smoother. Enlarge your image to 200% in both directions. Give it a 1 pixel Gaussian blur. Go back to the Preferences and set Image Interpolation to Bicubic Smoother. Reduce your image to 50% in each direction (returning it to the original size). The lines should be somewhat smoother. Try it with and without the blur step, try different combinations of the interpolation algorithms (except nearest neighbor—that one will actually enhance the aliasing instead of reducing it), and see if going to 400% or 800% and back down does anything more for you. Different sizes of blur will give you different results, too.

This is not truly antialising, though, as it softens the lines rather than blending them properly. You could also try some tricks with layer effects. A 1-pixel stroke or even a very small outer glow might do the trick for some shapes and styles.

Here's a sample of what I did while I was experimenting. In the upper left is my original, aliased shape. In the top center, I simply increased the size by 200% then reduced it by 50% with Bicubic interpolation both directions. Top right, I did the same thing with a 1-pixel Gaussian blur. Bottom left, I used the resizing process I described above. Bottom center has an outer glow set to Blend Mode: Multiply, 65% opacity, Color: Black, Technique: Softer, 0% spread, 1 px size, and a linear contour.


02-21-2012, 02:08 AM

I apologize Schwarzkreuz, but I am having a difficulty understanding what you mean. :(( You selected it with the wand, right? How did you smooth out the layer if it was blank? Or was there something you added on the new layer?


Thank you very much for the images and explanation, Sular! I was just about to say that the wand forcefully grabs parts I don't want to, but I'll keep trying to get it right. I like the image's coast/lines in the left picture you uploaded as well. :D I'll try the option out now. :)


I see! Thank you very much for the link and post, Midgardsormr. I did not know about "anti-aliased lines". Also, nice post with the descriptions as well, very useful (especially for my spriting!)! *rep* I will try this option as well! :)

Once I finish each version, I'll try and upload them so I can show them off (in case I'm being stupid xD)! Oh, and I'm going to rep you all too! Promise. (:

*Schwarzkreuz, it says I cannot rep you yet. I will when I spread. :>*

03-03-2012, 05:57 PM
Sorry for the double post, just reporting back.

I tried each of the methods, but none of them made it smoother. I tried increasing and decreasing from 800 and 900 as well with the blurs and even tried increasing the blurs, but the terrible pixelized look was still there. Blurring it erased some of the coasts and all of the islands. The magic wand too method's coast lines ended up being the same as well.

So I tried Paint SAI Tool to recreate my landmasses. I was wondering if the lines looked smooth, or if it didn't work right:

Besides the above, I have tried everything I could think of, even redoing the tutorial with a new map. Same old pixelized coasts.

03-03-2012, 06:12 PM
Hrm, that is somewhat strange. I did try playing around with my tutorial images to see if I could create a jagged coast. I did manage it by selecting the interior of my linework rather than the outside and then reversing the selection. Even after smoothing the interior-selection version was more jagged than the other. So, clearly, the way one selects things matters.

One thing that occurs to me is that the image resolution might well be a factor. I have mine set at 300 pixels per inch.

03-03-2012, 06:20 PM
Oh man, that would be a shame if it was the problem. I would have to find a way to recreate the map with a higher reso. How can I check the current resolution of my image? (assuming that is what you mean?)

03-03-2012, 09:29 PM
In PS it is listed under "Image Size" as resolution. You can adjust the resolution in the text box to whatever you want it to be.

03-03-2012, 10:43 PM
If 300 is recommended as the minimum, then I definitely believe its the problem. The image's size is 1990x4212, with a reso of 72! :(

If I changed the reso in the image, would it work, or would I have to restart? And if so, is there away way to preserve what I have now...? I would hate to have to lose what I have now. :((

03-03-2012, 11:58 PM
If you alter the resolution it will likely resize your image again, which might help. That said, 1990x4212 is not a small image at all. Would it be possible for me to see what you've got and to play around with it? I may be forgetting a step or something and having a sample to work with may help me figure this out.

03-04-2012, 12:16 AM
I changed the reso to 300, and it increased the image. It helps a bit, but much of it was sorta changed and still a lot pixelized.

Sure thing, I do not mind. The PSD file right, right?

03-04-2012, 12:21 AM
If you can, sure. If that is too large, then just the coastal outlines should be helpful.

03-04-2012, 12:33 AM
Alright then, tell me if this is good. If not, I'll try and upload the PSD file instead.


03-04-2012, 01:24 AM
Alex, I took your image, set it to the pixel size you gave of 1990x4212 at a resolution of 300. The image I downloaded was smaller and I wanted to use the dimensions you had. I then selected all of the black areas of the map with the wand tool, inverted the selection, smoothed it by 3 pixels, and painted a the selection green on yet another layer. I used a large round brush (just the basic photoshop round brush) set to 771 pixels to paint this region of the map. This size probably does not matter all that much, I just wanted a large brush.

I've attached the full-sized file. As you can see, at 100% magnification the coasts are smoothish but as you zoom in it gets a bit more jagged as one would expect.

03-04-2012, 02:16 AM
That is strange...tinypic must have shrunk the image. :o

And that is much, much better! Would this technique work in any color? I would need the land white and the "ocean" black to continue off the tutorial where I left off. I retried it and it looks very blurry in white and black... :(

03-04-2012, 02:31 AM
Alex, I colored the image black and white in the manner of the source file. If this will work for you feel free to use it.

03-04-2012, 03:01 AM
Hey Alex - just a little note on resolutions. The DPI (72, 150, 300, what have you) doesn't have anything to do with the image size - it's just a measure of dot density, basically that goes along with the image saying how many dots fit on an inch (i.e. how many pixels are packed into a linear inch), which then specifies to what size the image should be output.

I.e. - a 6000 x 6000 pixel image is a 6000 x 6000 pixel image, regardless of DPI. However, the DPI tells you what size the final output will be. On a computer display it will measure approximately 140 x 140 inches at 100% zoom (where 1 pixel in the image corresponds to 1 pixel on the screen). That same image will measure 20 x 20 inches when printed at 300 DPI, or 40 x 40 inches at 150 DPI. All without any resizing whatsoever.

Oh yeah, for large scale printing, 150 DPI is usually enough.

03-04-2012, 05:49 PM

Sorry this is late, I was on my PC, but the storms knocked my internet down. And yes, black and white! That works perfectly, thank you so much! Way better looking than the pixelized look I had.

The technique you explained works for me now, I just have to repeat the steps and fill in about four times with the bucket to get it rounder.

Oh, thanks for explaining Lukc. :)

So the DPI is set when the image's size is set?
*I thought DPI was something you could edit along side the image's size.

03-04-2012, 07:45 PM
I'm glad it's working better for you, and good luck with the map itself.

05-22-2012, 08:33 PM
I may be late to the party but what works for me is to use the smooth feature on selections as mentioned above + I import the outline into Adobe Illustrator. Then I use the Live trace feature and choose black and white and not only does it turn it into a vector (you can resize infinitely without loss of quality) but it gets rid off all the pixelness and attains an almost smooth, flowly pen like outlining.