So, I have this map project for the world of Myst. I'm sure some of you have played these games when you were younger

I am creating digital Myst books, you can view Myst and the beginning of Riven here:

And in them, I have drawn maps of the various islands of Myst. Some are available for download:

While there are a lot of things I'm not happy with in these maps, the most obvious thing is the mountains or stoney surfaces. On the last, Myst Island, it's the hilltop of the observatory and the edges of most of the island. The eroded rock that slopes down into the water.

It doesn't look horrible, but not very authentic either. I want to make really authentic-looking hillsides/mountains/erosion effects. I'm also looking to do these automatically, not hand-drawn. I am currently drawing my maps all by hand using a Wacom Cintiq. But ideally I would like some form of software where I could draw the land and the erosion would automatically be generated. And I can selectively raise/lower the terrain until I am satisfied. I am using a Mac, but I do have a PC at hand if needed.

If you've used Bryce, you know about the greyscale elevation map editor. Something like that, but a bit nicer, is what I'm looking for. It had controls for adding erosion and "ruggedness" to the terrain.

Failing that, I would like to have some really good pointers for creating good looking stone surfaces on a map.

Any help is appreciated.