What I do if I want to zoom in on an area for a larger map is I make the main landmass white and put it onto a black layer. Then filter > pixelate somewhere between 3 and 12 (this puts a bunch of white, gray, and black jaggie shapes in where the white and black meet). Then Image > adjustments > brightness/contrast and turn the contrast up all the way to remove the gray. Now you have a black sea and white newly jagged landmass.