So.... I have had a few people who really like the ISO mountains I have made in the past and since there is not another tutorial which addresses how to do this style. So, I will take a quick break from my mapping and attempt to help anyone who wishes to try this style a helping hand.
First, I begin with a new map, in this case, I choose 2000 wide x 1000 high and 150 ppi. Feel free to choose whatever size you wish, but the higher pixels per inch you choose, the better the overall look. However, this also increases the size of the file, both on disk and in RAM and when exported to jpg or png format, so you have determine the trade off that is best for you.
I begin by creating a color layer as the background(in this case I chose a green), followed by new transparent layer. I choose a 3px fuzzy brush at 1 scale. Then begin drawing a few updside-down V's, but don'ttry to get them too perfect. It's both perfectly ok and even very desirable to have some crookedness to your lines in order to give your mountains some character. Also, for the mountains in the "back" rows, have some of the lines going "down" meet it's neighbor. For hills, have those closest to the mountains be more round, while the ones lower/closest to the foreground to be a lot more flattened. In many of the screenshots below, notice that I am zoomed in to 400% or even 800%.
Next you want to block in some color for your mountains and shadow. Begin by creating a new transparent layer and place it below your outline layer. Next, determine which side you want your shadows to be on, I have determined I want my shadows on the West side. Then choose your colors. In this case, I choose pure black and a medium brown (hex number: 806b50), but feel free to choose whatever colors for primary and shadow you wish. You may like to do two shades of the same color, though I tend to prefer solid black for my shadows. I generally tend to segment the mountain shape roughly in half based on the outline for the particular mountain I am working on, choose which side will be shadow(ie, black) and block in the two colors. Also, I swap brushes after putting the main colors and add a bit or my primary color to the shadows side and a bit of black to my primary color side.
Once you have your color blocked in, its time to switch to the smudge brush. Again, I want to point out that I am VERY(again 400-800%) zoomed in to do this work. I pick a 3-5 px brush with the scale at 1. The most important part is the Rate setting, which controls just how much the smudge pulls color from one area to another. I generally set this to around 40-50%. Any higher than than and you pull to much color at a time. If you have a tablet, this is a perfect place to set the Rate based on brush pressure and practice with a medium to light hand stroke.
Begin using the smudge brush to pull the colors down, typically dragging at around a 45 degree angle toward the South-West. Note that this is a general rule you will follow around 50% of the time with the other 50% smudging in whatever direction you think will get the desired effects. Also, make sure that you stoke you stroke BEFORE you finish crossing over your outline and into the "light" side of your neighnoring mountain body(or non mountain region as the case may be.)
What you are trying to get here is to mix around 1/3 solid shadows, 1/3 sold primary color, and 1/3 of a mix between the two. Since you are zoomed in so close of course it will be very pixellated, but thats exactly what you want since you want to be able to see a solid color, a few pixels of progressivly transitioning color followed by the other solid color, ie, a blur of several pixels between the two different solid colors.
Note, and this scale (again, 400-800%, it may not look like much, bbut when you zoom out, it should look very nice(feel free to zoom out... I will show a 100% screenshot in a minute)
Next, we will block in another mountain with color, and some different "shapes" to the interior:
Wash, rinse, repeat. Now, go ahead and block in all the color for the entire set of mountains/hills.
The rest will come shortly as I am out of image space for this post.