A convention on many maps today--especially older styled--are the wave lines up and down the coasts that look stamped by a carved wood block.

Here's a great example by a very talented, professional fantasy cartographer, S. John Ross; here he uses the process for the entire ocean: http://www.deviantart.com/deviation/...e+-in%3Ascraps

Thankfully, Mr. Ross is a very generous supporter of the RPG community, so he has created a great tutorial for how to recreate the process using Photoshop: http://www.io.com/~sjohn/uresia-maps.htm

I can vouch for his process, for I've used it myself--albeit not as well!