The first that hit me was actually the forest, not the shelf. The forest edge needs to be broken up so here's a few steps to fix that:

1. Make a selection of the forest layer (ctrl-click on that layer in the layer stack).
2. Create a new layer and fill this selection with white.
3. Deselect.
4. Create a new layer and fill it with total black.
5. Move this black layer below the white forest shape layer.
6. Merge these two layers.
7. Filter-artistic-spatter. Mess around with the sliders until you get something with a good broken edge.
8. Select-color range=black.
9. Delete the blacks.
10. Deselect.
11. Hide this layer by poking out the eye in the layer stack.
12. Ctrl-click on this hidden layer then click on the forest layer.
13. Select-inverse.
14. Delete then deselect and you should have a nice broken edge on your forest.

For the shelf, expand the size of the bevel (default is 5 so try around 30-50) and then lower the height of the bevel (default is 100% so try 25%) to reduce the effect of the white highlight and black shadow. You might also want to mess around with the contour of the bevel (this is basically a profile view of what it actually looks like). The gaussian curve is usually nice but for this I usually use a cove shape (like cove molding, it curves inward or down instead of outward or up). What this does is to make a smooth transition from sea floor to gently rounding up.