1 ) I'm assuming you've already figured out your coastline.
2 ) I set things up to do the elevation one continent at a time, since it was easier that way. Make a layer from one continent: the land fully opaque and the sea fully transparent. Make it white if black is your max elevation color, or black if white is your max elevation color. You may want to do this in a separate document, if your map high res enough to be slow. Most of the layer's you'll make will be flattened when you are done.
3) On another layer sketch in where you want your mountains, continental divides, or whatever you have as a guide. The brown in my example is where i want my mountains.
4 ) Open some elevation maps like these. You just want greyscale without any shadows. For something more exotic use elevation maps of the moon, mars or other celestial bodies.
5 ) Look for parts on the real elevation map that more or less match what you want on your map. If your continent is large or complex you will need to look for one piece at a time. The Rocky mountains here gradually sloping down across the great plains is basically what i want for my continents, since this one is pretty simple (as examples should be). Select what you want with the lasso, feather your selection and then copy and paste it in as a new layer. I'm using white as down, so i need to invert this bit of the rockies.
6 ) Move your cut-out piece of elevation over your shoreline layer from step 2. Then "Create a Clipping Mask" with the cut out bit of elevation. In Photoshop you can do that by OPTION clicking between the two layers, or from the contextual menu. If you do it right, it will look the following. The clipping mask will only show the part of your elevation layer that is over an opaque part of the coastline layer, thus anything over the ocean isn't shown.
7 ) Resize, Flip, Rotate etc. the elevation layer until it goes where you want it to. It won't be a perfect fit, but we'll refine that later. Turn on your guide layer when you need to remember your original idea, turn it off, when it gets in your way.
8 ) Chances are the bit of elevation you copied doesn't have everything you want. That's fine. Go grab another piece of elevation as per step 5. Paste it into your document and position it above the previous bit of elevation-- don't worry if it overlaps. You'll want to "create a clipping mask again" with this one and any additional layers to your base coastline. All should be clipped to your base coastline layer.
9 ) OK, now the tricky part is getting these disparate bits of elevation to line up, right? Don't worry, it's not tricky my way. If down is white, you'll set the blending mode of all your elevation layers to "Multiply". If down is black use the "Screen" blending mode. What this does is add both elevations together, assuming that some of the layers of copied elevation overlap-- which they should. The only hard edges you see should be at your coastline. If you are more familiar with your app, you can probably think of variations on this method to achieve different effects, but i won't go in to that.
10 ) I find it hard to gauge what going on just staring at greyscale elevation. So i made a gradient map with a series of colors like you'll find on a topographical map. This gradient map layer sits above all the other layers. While you are painting with black or white on one of the elevation layers, the gradient map instantly converts the appearance to nice topographical color.
11 ) Now it's time to refine the elevation. You can paint with fuzzy, semi-transparent white and black brushes on any of the elevation layers to emphasize or de-emphasize any feature. For wide-spread, gentle changes, use a radial gradient tool, set to low opacity.
12 ) When you are happy with the appearance, merge all your layers together (but not the gradient map layer, or your guide layer) and your are done.