Ok, so you want to break up that edge that looks too much like the coast and give it some randomness...right? There are three ways to go about this that are very easy:
1. Use a pencil with a scattered brush tip and paint in some randomness in certain places. The scattered brush tip will give you lots of lil bumps along the edge but keep going over that same area and the blank spaces will fill in.
2. Put your shelf layer (solid white) on top of a black layer, merge them, hit it with a spatter filter (under brush strokes), select-color range = blacks, hit delete then deselect, apply layer styles.
3. Use the eraser (in pencil mode, not brush) and select a scattered brush tip and erase parts out to make it more random.