Just as an exercise, I played with this one a little bit. Photoshop lacked some of the tools I wanted to use, so I did this in the feature film compositing program Nuke. Unfortunately, that means that the source file is worthless to most anyone else, but somebody else might at least find this image useful.
For the sake of explaining what I did, in case anyone wants to know: I dilated all of the channels by 4.5 pixels, then reversed the dilation. That gave me soemthing pretty close to the original shape of the landforms but without the high-frequency detail from the repeating textures. Then I ran a color key on the blue channel to isolate the water. I inverted the resulting key (making the water dark) and played with the levels to get the water black, the reefs just a little bit brighter, and everything else a little darker than middle grey.
I ran another color key on the red channel to isolate the mountains. I then ran my dilate filter at -3 and again at 9.5 to remove most of the remaining texture in the mountains and pop the big peaks out a little higher. That was mostly undone, though, when I did the blurring a little later. But first I used a levels control to set the mountains to the brightness I wanted. Then a large blur of 19 pixels and a huge Gaussian erode at 120 pixels.
I then shuffled the alpha channels for each branch into the color channels (turning the transparency into grayscale layers), and I merged them with a maximum blend mode. I suppose the Photoshop equivalent would be "lighter color."
I think if I were going to continue working on this image, I'd find some real-world heightmaps at an appropriate scale to overlay on the image.
The end result isn't super-accurate, but it didn't take long, and some painting would probably get it the rest of the way there.