It may sound overly simplistic, but it's basically just draw in colors of gray. Draw the ocean in black (or a very dark gray), the solid-filled coast in a medium gray, and mountains in white (or very light gray). Then blend as appropriate (I have seen the smudge tool used to good effect for pushing things along river canyons, for example). The more gray levels you get in there, the better the final result is likely to be, assuming that you've done a plausible job of placing things in the first place.

http://www.cartographersguild.com/ho...made-maps.html has a basic discussion for using Wilbur to create simple black and white masks for coastline, mountains, and rivers.

http://www.cartographersguild.com/tu...ght-place.html offers good suggestions for getting rivers and their good buddies, mountains, where they would plausibly be for a physics-based scenario. The most important rules? water flows downhill and water erodes rocks, moving the resulting sediment downhill with it.