I recently joined this forum because I needed to make a map for a story. This map is for my own self edification and to use as a reference for my writing. I also wanted it handy in my computer and have the ability to change it up. This wasn't for a contest or anything, but the methods are good.
From what I've seen, is no good mapmaking software for the Mac. I could use CC3, but it looked clunky and I needed control. I also got to a point of diminishing returns for my search and instead of spending any more time finding a program, I was just going to make it in PS. I'm a fairly savvy PS user, but mostly for color correction. I read many posts here on how to make a coastline, and they're pretty good, but again, didn't fit my needs. I have a very specific map that needs specific boarders. All I wanted was some jitter and randomness to it. After failing to accomplish some of the tutorials here (I think they missed a step or two in their writing or they don't translate from Gimp) and not satisfied with the results of ones that I did try, I came up with my own method. You could also use this tecnique if you don't have your own map; you would just have to draw one out in PS. One thing about PS is there is way more than one way to skin a cat.
One thing I didn't like about other tutorials is the lack of feedback and control. I'm a big fan of changing things up to how you like it without having to go back many steps to redo something.
What I came up with will give you great control over your coastline.
I feel compelled to share this because it is fairly easy, and I came up with it after fooling around for a couple hours. But, bear with me, because I'm not a teacher and I don't have much patience for posting all this. I hope it doesn't take me a long time.
It is going to help greatly if you have a tablet, but not necessary. It is also going to assume that you have a working understanding of photoshop. I'm using CS4. There are a few features in this that came out at some point (I have no idea when), so an older version of PS may not have everything listed here. Some of my settings might differ greatly from yours if you are not working at a similar size and resolution.
Here we go.
1) I have a drawn map that I want to make. Get it into your computer somehow. Mine was 16 x 20, so I took a picture. Clean it up whatever way you want (I used levels) so that it is clear and legible.
2) Next, create a new document that you want your created map to be. I started with a 20 x 20 at 300 dpi. That may sound big, but it's a big map and I always work bigger than I might need. Also, I may want to print it out at some point for reference. You can always size down at the end, but you can't size up. (Yeah, you can with certain programs, but depending on the original, it may not look good) Copy and paste your map. Resize it so it fits the window. Crop as needed. My doc ended up being 16 x 20. Great.
3) Trace your coastline.
I use paths for this. You could draw out your coastline with a pencil tool as well and fill lin the results. I still find paths finicky and a pain, but it offers much more control and editablility. I am going to assume you know how to use paths with regards to what channel they're on, adding, joining, subtracting, etc. If you are going to use paths, here is some useful tips. When tracing, use the freeform pen tool with a curve fit between 2 and 5. 5 for the broader coastal areas, and 2 for the tighter peninsulas, and islands, etc where you need more detail. I used 5 on almost everything. These numbers will be different depending on the size and resolution you are working in. You can also subtract for lakes on the interior. This doesn't have to be super precise, that'll come in later. A CU of mine looks like this:
4) Fill your path. (There's more than one way to do this)
Create a new layer. Fill it with black (Edit>Fill>Black). Go to paths tab. Select your path (should be called work path unless you renamed it). Right click>Fill Path>White.
5) Create clouds
Here is one of the big differences between the other tutorials I've seen and my approach. OK so. Create a new document. For my purpose, I made mine 4 x 4 at 300 dpi. Fill with black. Filter>>Render>Clouds. It is important that you get a pattern with a variety of black and white areas. That's also why it's so big.
6) Create seamless texture
This is a really old trick. If you already know how to do this, skip ahead. Filter>Other>Offset. Choose horizontal and for this size I chose 200 pixels. You want it offset enough to give you room to work and have enough area to clone. Choose the clone stamp tool. Make your brush big (200) with a soft edge. There's a bit of an art to cloning, but the point is that you want to erase the hard edge. Keep in mind that when picking your source, try to keep overall lights and darks. Vary up your sources, and make sure you go off the page to get all the hard edge but your source doesn't.
Next do the other edge. Filter>Other>Offset Vertical and 200 pixles. If you messed up before, you'll see an edge here. That's ok, just clone it out. You can check your work by doing more offsets and see if there is an edge. Congratulations! You just made a seamless texture. This techinque can be used in a huge variety of ways. You'll also never need to render clouds again unless you just want a different pattern. Next, Edit>Define Pattern. I named mine Clouds 1 Tile Large. I would suggest at this point, even though you may not use it in this tutorial, you resize the image to 3x3 and 2x2 and define patterns with names such as Clouds 1 Tile Medium & Small. They will come in handy at some point.
Back to your map document.
7) Create a pattern fill layer. (I use the half black/white circle icon on the bottom of my layers palate.) From the drop down menu, choose the large cloud pattern. Shrink down the scale to something small. Mine was at 21.
Add a Threshold Adjustment Layer. I have mine set to 101. You can change this up later to your liking.
9) At this point, so you can see what's going on with all the settings below, you may want to duplicate your map outline layer just above it and give it a gaussian blur of about 7 (This will vary depending on your resolution, etc.). You can keep this, modify it, or throw it away and edit your original, hard edged map outline below it. I don't like the way the blur messes up some of my finer coastal details, but you can make that more precise later.
Now you should have 5 layers. From the top, Map sketch, Threshold, Pattern Fill, Map Outline Blurred, and Map Outline.
9) Some magic. Double click on the middle of the Pattern Fill layer in the layer palate to bring up the blending options. Set blending mode to multiply. Next, Adjust the sliders on the bottom to something like this: Blend if <Gray>, This layer 52 / 169, Underlying Layer 0 / 244.
Play around with these sliders. You should be able to get direct feedback no to see exactly what they do. The important thing is that the underlying layer white end needs to be clipped just enough so there are no clouds in the body of land. The sliders on "This Layer" play around with until you get about the desired turbulence you want. There shouldn't bee too many islands or inlets, unless you really want that. This turbulence is also going to be highly dependent on the "Pattern Fill" scale amount, so play around with that, too, and the threshold adjustment layer setting (Which shouldn't be adjusted too much. A few ticks in one direction can drastically change things).
Your result should look something like this without your map sketch visible:
>Note: Totally not necessary for this tutorial. If you want to get really creative here and change things up, duplicate your pattern fill layer, move it below the one you created. Change the scale to something bigger. Go back to the original pattern fill layer and create a reveal all mask. Then in the mask, use a wide soft brush add black to reveal the other pattern fill underneath. You're now blending two turbulence patterns.<
10) Play god. Now to refine and perfect your coast. This is where a tablet is really handy. Make sure your foreground background is black and white. Pick a medium size (around 70), round brush. 0% harndess, 1% Spacing. Shape Dynamics: Size Jitter 0%, Pen Pressure, Minimum Diameter 0%. Other dynamics: Opacity jitter 0%, Pen Pressure, Flow jitter 0%, Pen pressure. I also set my airbrush flow to 33%.
Go into your Map Outline Blurred layer and paint along your coastline. Depending on whether you brush is black or white, you will either add or subtract. Tip: you can easily switch your foreground / background colors by pressing x and change your brush size with the bracket keys. You will also notice that the turbulence is different because we are getting rid of the even gaussian blur. Now, make your map sketch visible and set the opacity to about 50% or whatever is good for seeing your drawn coast and the underlaying effect. Paint away! The goal here is to make little grayscale adjustments. Little islands and inlets are just a matter of changing your brush size and the pressure. Your original Map Outline didn't get something, add it now. Made a mistake? No problem. Switch your colors and approach it from the other side. Fun!
11) When your done, select the Tolerance layer, select the magic wand tool, tolerance 0, click in the land. Select > Save Selection > New and give it a name. I further refined my selection with a great tool called refine edges. If you don't have that, you may want to do a small feather and smooth (resolution dependent) then save the selection again with a new name to preserve the original.
Your done! You now have a coastline to do with what you want. Here's my work in progress. The coastline has a small stroke to it.
Phew. That took a while. Let me know what you think and if there are any problems with the post, like pictures not loading. Thanks!
PS. In writing this, I accidentally overwrote my map file beyond recovery. I have to start over from scratch, but in the meantime, I've noticed that a few of the settings have been slightly different just because I like the results better. This just goes to show how there are many results you can get from this depending on your taste.