Painted Ocean Waves in PS
Creating a painted ocean
Note that this tutorial is done using Photoshop and a pressure sensitive stylus tablet. Everything can be done with a mouse, but you will generally make your brush sizes smaller and paint with half the opacity that I list here. The illustrations for this tutorial show waves that are vastly out of scale for the landmass, I did it that way to better illustrate the technique, even so I think the ideal waves for a painterly looking map will be many times bigger than real waves may be.
1. Create your landmass, use any method you care too...My focus is just going to be on the ocean, sea area. Using a large soft edged round brush with pressure sensitivity on -paint in the ocean on a layer below your landmass, keep the pressure light around the land and heavy as you get further out until you have something close to the first picture. The focus here is to keep some irregularity and inconsistency. You can change colors if you like as you radiate outwards and inwards building up the sea. I stuck with just one color and relied on the white of the underlying background layer to show through the transparency to create my shade variations.
Addendum: I have added a closeup view which I did at a larger scale, basically doubling my brush sizes was all I did differently to create this close up. I am also adding more details and options to each step to help explore individual looks which may appeal to you or the methods of your map better.