Actually for a test, this didn't come out too bad. The beaches were.. interesting to do. The brush I was using was doing some unexpected things, but I think that was because of the layer style combined with the soft brush combined with the mask. Perhaps it's normal for it to act strange, i don't know - this is my first attempt at working with layer masks to any real extent :)
So, a combination of paying attention to the Masks and Such with Craig John (i forget the username of who made and/or posted it, argh), a few layer styles (bevel options) inspired by pasis, whoever it was that wrote that one tutorial that included the Jiggle filter (wow, it's my new friend) and me just monkeying around, this is what ends up.
I know I could probably get the layer masks to work more efficiently in a few areas, but it was really starting to frustrate. I found a way to keep almost everything working the way I wanted to, but there was a spot where I was trying to get an outer glow and an inner glow to show up at the same time and the masks wouldn't let me. I also gave up on using the textures as pattern overlays, because they were messing up th layer styles I was trying to work with. So until I can sort out a few issues, this will do.
I took a screenshot of my layer palettes just to show exactly why i'm semi-proud of this map :) On the left are the layers as is, the right shows the expanded smart object (land layers). Yes, each layer is 100% texture filled, all terrain work is done through the masks alone. The mask on the smart layer is where the rivers came in (yay, i figured that one out, lol). This is also my first attempt at working with smart objects. Nifty keen! Though I wish there was a way to "un-smart" the layers and put them all back as separate objects/layers - there likely is and I haven't found it yet.
Ok! that was fun!