One way to get a (very) rough approximation to a continental shelf is to blur your land layer and then trace around a roughly constant intensity in the blurred area. As Azelor points out, this won't be geologically accurate, but it will smooth over deeply-indented areas like your two gulfs while not extending too far out into the ocean areas.

http://lance-modis.eosdis.nasa.gov/i...roject=aeronet is a useful resource for looking at continental shelf areas. Two good examples are the west coast and east coast of North America (and South America to some extent). The west coast has little to no continental shelf area, while the east coast has a nice sloping coastal plain accompanied by relatively broad offshore areas.

Once you have the shelf area, use it to modify the lightness channel of the deep-sea color areas (increase lightness a bunch and possibly add a slight hue shift toward cyan from blue, especially for tropical areas). Then you can go back and put in some swirly bits of light green and muddy brown along coastal areas down-current from major river outlets.

One thing that I noted is that your coastline is almost uniformly fractal (it looks a lot like OldGuy's coast-wrinkling tutorial). Real coastlines are multifractal; that is, they are smoother in some places than others. The uniform coastline looks decidedly artificial.